Guest post by
René Aquarius, PhD
Department of Neurosurgery
Radboud University Medical Center, Nijmegen, The Netherlands
After a recent zoom-call where Dorothy and I discussed several research-related topics, she invited me to write a guest blogpost about the experience I had as a peer-reviewer for MDPI. As I think transparency in research is important, I was happy to accept this invitation.
Mid November 2023 I received a request to peer-review a manuscript for a special issue on subarachnoid hemorrhage for the Journal of Clinical Medicine, published by MDPI. This blog post summarizes that process. I hope it will give some insight on the nitty-gritty of the peer-review process for MDPI.
I
decided
to review the manuscript two days after receiving the invitation and
what I found was a
study like many others in the field: a single-center, retrospective
analysis of
a clinical case series. I ended up recommending rejection of the paper
two days after accepting to review.
My biggest gripes were that the authors claimed that data were collected
prospectively, but their protocol was registered at the very end of the
period
in which they included patients. In addition, I discovered some
important discrepancies
between protocol and the final study. Target sample size according to
the
protocol was 50% bigger than what was used in their study. The minimum
age for patients also differed between the protocol and the manuscript.
I also had problems with their statistical analysis as they used more
than 20 t-tests to
test variables, which creates a high probability of Type I errors. The
biggest
problem was the lack of a control group, which made it impossible to
establish whether
changes in a physiological parameter could really predict intolerance
for a
certain drug in a small subset of patients.
When filling out the reviewer form for MDPI, certain aspects struck me as peculiar. There are four options for Overall Recommendation:
- Accept in present form
- Accept after minor revision (correction to minor methodological errors and text editing)
- Reconsider after major revision (control missing in some experiments)
- Reject (article has serious flaws, additional experiments needed, research not conducted correctly)
Regardless of which of the last two options you select, the response is: "If we ask the authors to revise the manuscript, the revised manuscript will be sent to you for further evaluation".
Although reviewer number 2 is often jokingly referred to as "the difficult one" it couldn’t be further from the truth in this case.
The reviewer liked the paper and recommended accept after minor
revision. So with a total of two reviews, the
paper got the editorial decision of rejected, with a possibility of
resubmission after extensive revisions only one day after I handed in my peer review report.
Revisions were quite extensive, as you will discover below, and arrived only two days after the initial rejection. I agreed to review the revised manuscript. But before I could start my review of the revision, just four days after receiving the invitation, I received a response from the editorial office that my review was no longer needed because they already had enough peer-reviewers for the manuscript. I politely ignored this request, because I wanted to know if the manuscript had improved. What happened next was quite a bit of a surprise, but not in a good way.
The manuscript had indeed undergone extensive revisions. The biggest change, however, was also the biggest red flag. Without any explanation the study had lost almost 20% of its participants. An additional problem was that all the issues I had raised in my previous review report remained unaddressed. I sent my newly written feedback report the same day, exactly one week after my initial rejection.
When I handed in my second review report, I understood why I initially got an email that my review was not needed anymore. One peer reviewer had also rejected the manuscript and had concerns similar to mine. Two other reviewers, however, accepted the manuscript. One with minor revisions (English needed some improvement) and one in present form, so without any suggested revisions. This means that if I had followed the advice of the editorial office of MDPI, the paper would probably have been accepted in its current form. But because my vote was now also cast and the paper received two rejections, the editor couldn’t do much more than to reject the manuscript, which happened three days after I handed in my review report.
Fifteen days after receiving my first invitation to review, the manuscript had already seen two full rounds of peer-review by at least four different peer-reviewers.
This is not where the story ends.
In December, about a month later, I received an invitation to review a manuscript for the MDPI journal Geriatrics. You’ve guessed it by now: it was the same manuscript. It's reasonable to assume this was shifted internally through MDPI's transfer service, summarised in this figure. I can only speculate as to why I was still attached to the manuscript as a peer-reviewer, but I guess somebody forgot to remove my name from it.
![]() |
from: https://www.mdpi.com/authors/transfer-service |
The manuscript had, again, transformed. It was now very similar to the very first version I reviewed. Almost word-for-word similar. That also meant that the number of included patients was restored to the initial number. However, the registered protocol that was previously mentioned in the methods section (which had led to some of the most difficult to refute critiques) was now completely left out. The icing on the cake was that, for a reason that was not explained, another author was added to the manuscript. There was no mention in this invitation of the previous reviews and rejections of the same manuscript. Although one might wonder whether MDPI editors were aware of this, it would be strange if they were not, since they pride themselves on their Susy manuscript submission system where "editors can easily track concurrent and previous submissions from the same authors".
Because the same issues were still present in the manuscript, I rejected it for a third time on the same day I agreed to review it. In an accompanying message to the editor, I clearly articulated my problems with the manuscript and the review process.
The week after, I received a message that the editor had decided to withdraw the manuscript in consultation with the authors.
Late January 2024, the manuscript was published in the MDPI journal Medicina. I was not attached to the manuscript any more as a reviewer. There was no indication on the website of the name of the acting editor who accepted it.
Note from Dorothy Bishop
Comments on this blog are moderated so there may be some delay before they appear, but legitimate, on-topic contributions are welcomed. We would be particularly interested to hear from anyone else who has experiences, good or bad, as a reviewer for MDPI journals.
Postscript by Dorothy Bishop: 19 Aug 2024
Here's an example of a paper that was published with the reviews visible. Two were damning and one was agreeable. https://www.mdpi.com/2079-6382/9/12/868. Thanks to @LymeScience for drawing our attention to this, and noting the important clinical consequences when those promoting an alternative, non-evidenced treatment have a "peer-reviewed" study to refer to.
At last, someone who reviews with integrity and honesty. Thank you
ReplyDeleteif "Target sample size according to the protocol was 50% bigger than what was used in their study." is an issue, then 1000s of studies in top journals need to retract.
ReplyDeleteI think that would be a fair conclusion. You can, at the very least, specify why you did not reach (or need to reach) the intended number of participants.
DeleteI have been asked to review manuscripts for various MDPI journals in areas in which I have no expertise numerous times. Furthermore, I have often been confronted with reviews from other referees that do not even mention issues that I consider to be of paramount importance in my field of research. Based on this experience, I feel that the selection of reviewers by MDPI editors is often inadequate. Furthermore, I have found that recommendations for rejection are often not taken seriously by the handling editors, even in cases where there are clear methodological flaws that affect the outcome and invalidate the authors' conclusions.
ReplyDeleteI have also had similar experiences with other manuscripts from MDPI. I very clearly indicated some statistical flaws and issues with a given experimental design of a given manuscript. The editor rejected it in one journal and 1 week after I was asked again to review in a different MDPI journal. None of the changes I had suggested were incorporated so I proceeded to write to the editor indicating my concerns. 2 days later the manuscript was accepted pretty much “as is.”I made sure to write to the new editor, indicating my concerns and the flaws with the paper. In fact, the response from one author was literally “I have never been asked to do that, so I won’t do that now.” For anyone in the field it would have been fairly obvious that manuscript was unpublishable. After that, I chose not to ever review for them again.
ReplyDeleteThe problem is that now also well reputed publishers start to use the MDPI system to publish every article submitted. See e. g. Heliyon from Cell Press. My collegue delivered a review rejecting a paper full of errors and without proper use of methods for the discussed problem. He strongly suggested rejection. A second reviewer was obviously absent. Then he was asked whether he would review the revised paper.
ReplyDeleteMy MDPI story is that I reviewed a paper for Entropy. It was not quite in the crackpot category, but pretty bad. I recommended reject, and it did get rejected. About a year later, I saw the article published -- in the same journal. The authors had changed the title and resubmitted it with essentially the same content (and no addressing blatant mistakes I had noted in my review). Worse, it was in a "Special Issue" of Entropy, one of only two papers in the issue, and the editor of the Special Issue was the senior author of the bad paper. So he had volunteered to organize a special issue so he could get his own paper published.
ReplyDeleteI contacted the journal editor and an ethics address for MDPI to inform them about this misconduct (the resubmission of a rejected paper under a different title; I had not noticed the Special Issue aspect at that time). After a couple of weeks, I got a reply that they did not find any wrongdoing. I guess "no wrongdoing" is MDPI's code for "their check didn't bounce."