My experience as a reviewer for MDPI

 

Guest post by 

René Aquarius, PhD

Department of Neurosurgery

Radboud University Medical Center, Nijmegen, The Netherlands

 

After a recent zoom-call where Dorothy and I discussed several research-related topics, she invited me to write a guest blogpost about the experience I had as a peer-reviewer for MDPI. As I think transparency in research is important, I was happy to accept this invitation.  

 

Mid November 2023 I received a request to peer-review a manuscript for a special issue on subarachnoid hemorrhage for the Journal of Clinical Medicine, published by MDPI. This blog post summarizes that process. I hope it will give some insight on the nitty-gritty of the peer-review process for MDPI.

 

I decided to review the manuscript two days after receiving the invitation and what I found was a study like many others in the field: a single-center, retrospective analysis of a clinical case series. I ended up recommending rejection of the paper two days after accepting to review. My biggest gripes were that the authors claimed that data were collected prospectively, but their protocol was registered at the very end of the period in which they included patients. In addition, I discovered some important discrepancies between protocol and the final study. Target sample size according to the protocol was 50% bigger than what was used in their study. The minimum age for patients also differed between the protocol and the manuscript. I also had problems with their statistical analysis as they used more than 20 t-tests to test variables, which creates a high probability of Type I errors. The biggest problem was the lack of a control group, which made it impossible to establish whether changes in a physiological parameter could really predict intolerance for a certain drug in a small subset of patients.

 

When filling out the reviewer form for MDPI, certain aspects struck me as peculiar. There are four options for Overall Recommendation:

• Accept in present form
• Accept after minor revision (correction to minor methodological errors and text editing)
• Reconsider after major revision (control missing in some experiments)
• Reject (article has serious flaws, additional experiments needed, research not conducted correctly)

 

Regardless of which of the last two options you select, the response is: "If we ask the authors to revise the manuscript, the revised manuscript will be sent to you for further evaluation". 

 

Although reviewer number 2 is often jokingly referred to as "the difficult one" it couldn’t be further from the truth in this case. The reviewer liked the paper and recommended accept after minor revision. So with a total of two reviews, the paper got the editorial decision of rejected, with a possibility of resubmission after extensive revisions only one day after I handed in my peer review report.

 

Revisions were quite extensive, as you will discover below, and arrived only two days after the initial rejection. I agreed to review the revised manuscript. But before I could start my review of the revision, just four days after receiving the invitation, I received a response from the editorial office that my review was no longer needed because they already had enough peer-reviewers for the manuscript. I politely ignored this request, because I wanted to know if the manuscript had improved. What happened next was quite a bit of a surprise, but not in a good way. 

 

The manuscript had indeed undergone extensive revisions. The biggest change, however, was also the biggest red flag. Without any explanation the study had lost almost 20% of its participants. An additional problem was that all the issues I had raised in my previous review report remained unaddressed. I sent my newly written feedback report the same day, exactly one week after my initial rejection.

 

When I handed in my second review report, I understood why I initially got an email that my review was not needed anymore. One peer reviewer had also rejected the manuscript and had concerns similar to mine. Two other reviewers, however, accepted the manuscript. One with minor revisions (English needed some improvement) and one in present form, so without any suggested revisions. This means that if I had followed the advice of the editorial office of MDPI, the paper would probably have been accepted in its current form. But because my vote was now also cast and the paper received two rejections, the editor couldn’t do much more than to reject the manuscript, which happened three days after I handed in my review report.  

 

Fifteen days after receiving my first invitation to review, the manuscript had already seen two full rounds of peer-review by at least four different peer-reviewers.

 

This is not where the story ends.  

 

In December, about a month later, I received an invitation to review a manuscript for the MDPI journal Geriatrics. You’ve guessed it by now: it was the same manuscript. It's reasonable to assume this was shifted internally through MDPI's transfer service, summarised in this figure.  I can only speculate as to why I was still attached to the manuscript as a peer-reviewer, but I guess somebody forgot to remove my name from it.

from: https://www.mdpi.com/authors/transfer-service

The manuscript had, again, transformed. It was now very similar to the very first version I reviewed. Almost word-for-word similar. That also meant that the number of included patients was restored to the initial number. However, the registered protocol that was previously mentioned in the methods section (which had led to some of the most difficult to refute critiques) was now completely left out. The icing on the cake was that, for a reason that was not explained, another author was added to the manuscript. There was no mention in this invitation of the previous reviews and rejections of the same manuscript.   Although one might wonder whether MDPI editors were aware of this, it would be strange if they were not, since they pride themselves on their Susy manuscript submission system where "editors can easily track concurrent and previous submissions from the same authors".

 

Because the same issues were still present in the manuscript, I rejected it for a third time on the same day I agreed to review it. In an accompanying message to the editor, I clearly articulated my problems with the manuscript and the review process.

 

The week after, I received a message that the editor had decided to withdraw the manuscript in consultation with the authors.

 

Late January 2024, the manuscript was published in the MDPI journal Medicina. I was not attached to the manuscript any more as a reviewer. There was no indication on the website of the name of the acting editor who accepted it. 

Note from Dorothy Bishop

Comments on this blog are moderated so there may be some delay before they appear, but legitimate, on-topic contributions are welcomed. We would be particularly interested to hear from anyone else who has experiences, good or bad, as a reviewer for MDPI journals.

 

Postscript by Dorothy Bishop: 19 Aug 2024 

Here's an example of a paper that was published with the reviews visible. Two were damning and one was agreeable.  https://www.mdpi.com/2079-6382/9/12/868.  Thanks to @LymeScience for drawing our attention to this, and noting the important clinical consequences when those promoting an alternative, non-evidenced treatment have a "peer-reviewed" study to refer to.