Acceleration of the arms race between fraudsters and honest researchers

I was recently asked to review a paper that piqued my curiosity. From the abstract it sounded as if it was a secondary analysis of a dataset that I had deposited in a repository. But the described data didn't quite match what was in the repository. 

As I usually do, I said I'd only review it if data and code were available. The authors complied: I was pointed at the repository where my data were stored, and when the editor requested it on my behalf, a script in Python code arrived. 

As I had suspected, the whole thing was fabricated. The analysis would have required linking data from two separate repositories - but the linkage information was not publicly available. No mention was made of the repository where the bulk of the supposedly analysed data were held. And the python script didn't clarify how the data were processed, and didn't even run with dummy code. 

So far, so depressingly predictable. But, despite all these issues, I was shocked at how plausible the paper appeared to be. 

In the past, I've been able to detect fraudulent stuff because it is either incomprehensible, formulaic, full of tortured phrases, had results that were far too good to be true, or described methods that were so implausible as to defy the laws of probability. Of course, I have no idea how often I've missed stuff that has slipped through the net; particularly in recent months, I've reviewed papers that have aroused deep suspicions but without sufficient evidence to justify calling it out as fraud. 

This latest case, though, seemed in a different league. The paper was not formulaic - it was in a rather specialised area, and purported to ask an original and interesting question. One of the things that made it seem believable was that the results were rather messy, raising more questions than answers. I wonder if I'd have identified the fraud if it hadn't concerned my own datasets. 

I assume this is one of the fruits of all those data centres that are popping up everywhere - they're facilitating the generation of more plausible fakes. So we're going to need yet more ways of defending the literature against them. I have strong doubts as to whether the world of academic publishing is ready for this, and I'm not confident that any kind of AI-screening will cope with the more sophisticated kinds of AI-fraud. I cling to the faint hope that human experts still might be able to trap some of it, but I'm feeling more and more like a member of the Light Brigade, galloping into the Valley of Death-of-Research-Credibility. 

For what it's worth, I'll offer my thoughts on things reviewers can look out for, but to be honest, the frauds are getting so good that these methods now seem inadequate. 

1. The first rule is to assume that what you are reading might be fraudulent. Alas, the age is long gone when it was safe to assume that a new manuscript was trustworthy. In 2021, Richard Smith, former editor of the BMJ, suggested we should assume health research is fraudulent until proved otherwise, and there's no reason to suppose that other fields are not similarly affected. 

2. Are data and code available? As noted above, I won't review unless they are, because it's just too easy to make stuff up. In the longer term, I worry that AI can generate entire papers with a replication package, but I hope that requiring these may at least slow down the fraudsters. I strongly recommend that all journals reporting empirical research require a replication package, as has become standard in the field of economics (see e.g. here and here).  Of course, actually checking data and code can take up a lot of time, which is the one thing most reviewers don't have. 

3. Do the cited references exist and are they cited appropriately? Hallucinated references are widely recognised as a hallmark of AI-generated material; the paper I refer to above didn't have any of those. But it did have a lot of "misaligned" citations - the kind of thing described in this post by Jill Walker Rettberg. She noted that as well as citing sources that were at best tangentially related to the referring text, there would be an absence of citations to the key work in the area. Checking this stuff is yet another tedious addition to the reviewer's workload. 

4. Does the article make sense? Failure here could involve lack of coherent structure, use of nonsensical "tortured phrases", and/or headings that follow the Introduction/Methods/Research and Discussion (IMRaD) structure seemingly inserted at random points in the text (see Matusz et al 2025). This kind of thing is highly diagnostic if you find it, but I suspect it is becoming less common as the fraudsters up their game. 

5. Consider whether it is plausible that the researchers did what they said they did within the stated time frame and with the stated resources. Some examples where this was a red flag are described here. This may be hard to judge if no time frame is given but for certain kinds of research, such as clinical trials, this should be stated.   This is also a key check in InspectSR - see points 2.4 and 2.5 here.  Even so, implausible research can be hard to detect unless the reviewer has conducted similar studies themselves.

6. Are the researchers and their affiliations legitimate? We need to be particularly careful here, because by considering provenance of research, we risk moving back to a time when research was trusted only if it came from people with some kind of track record, and/or who were personally known to the reviewer or editor. Research publishing should not be a walled garden that keeps out newcomers. But equally, if we take the walls down, we need to be vigilant to ensure the garden is not overwhelmed by weeds. As with point 5, plausibility is also relevant here: if two researchers with no track record from different countries claim to have collaborated on a technically complex analysis without any funding, I'd want to look at whether their CVs explained how they had acquired the skills to do this. 

Perhaps the most depressing thing of all is that just when we need to be mustering competent researchers who can undertake the careful and time-consuming work of reviewing new submissions, academic publishers are pursuing rapid growth, which will provide ever more opportunities for fraudsters to find an outlet for their work. They don't seem to appreciate (or some would say they don't care) that there's a shortage of competent and motivated people to serve as editors and reviewers in the existing system, let alone in a vastly expanded one. I don't think this will end well.

Trouble at t' (review) mill: How MDPI lets down authors

 I guess many readers will have had peer reviews where the reviewer doesn't appear to have understood, or even properly read, the paper they are criticising. But I doubt that you'll have experienced anything like Pérez et al (2023). Their growing exasperation with increasingly irrelevant comments can be sensed in their response to reviewer 2, who asked them to revise statements that were unproblematic, to add information already in the article, to correct information that was not in the article, and to add two references that bore no relationship whatever to the topic of the article.

So, how can this happen? Well, all becomes plain when you read this article by Maria Ángeles Oviedo-García (2024), who has documented what she terms "review mills": groups of reviewers who generate boilerplate peer reviews that do not evaluate the content of the article, and so can be applied to almost any field. Their main function appears to be to act as citation vehicles for members of the review mill, who encourage authors to "improve" their article with "latest literature", with references to papers by themselves or confederates.

When you see numerous reviews by the same set of people side by side, the boilerplate nature of the reviews becomes obvious, with the same phrases repeated across completely different contexts, e.g.: 

1. The title of the manuscript is not impressive so rewrite the title. 
2. In the abstract, the author should add more scientific findings. 
3. Keywords: There are so many keywords in this manuscript and reduce it. So, modify the keywords. 
4. In the introduction part, the introduction part is not well organized and cited references should cite recently published articles such as 10.3390/toxics10110657, 10.1016/j.rinp.2022.105817

Oviedo-García meticulously documented the evidence for review mill activity on PubPeer, and on 13 February 2024, MDPI's communications department wrote a report outlining how they planned to deal with the situation, noting "MDPI is committed to transparency and integrity in scholarly publishing. We will provide updates as the investigation progresses and appreciate your understanding and cooperation in addressing this matter." 

Neither Oviedo-García nor I have heard of any update to the report, and so in an idle moment I thought it would be interesting to revisit the list of 84 papers to see if anything had been done in the 17 months since that statement. A manual check on 18-20th July 2025, revealed that 46 articles had no action, 13 had a “Journal notice”, stating that they were being looked at, two had the “Review Reports” tab deleted on the website, and the remaining 23 had a Correction. I have added information about Notices and Corrections to PubPeer.

The cases of Correction reveal a lack of consistency in how matters have been handled. The most popular option has been to just delete the peer review from the review mill from the website. The Correction is at explicit that this has been done, but it can be confusing because it often leads to some renumbering of reviews, making it hard to track earlier PubPeer comments against what is online. The other thing that is usually done with the Correction is to remove the coerced references from the article.

Serious problems arise, however, if there were only two original reviews, because MDPI's procedures specify that two reviews are needed; hence, if one has been deleted as improper, they have to seek another review. No doubt the editor fervently hopes that the replacement review will not raise new issues, and sometimes that was the case, but there are instances where the new reviewer wanted substantial changes to the article.

This then creates a most unfortunate circumstance for all concerned. Authors, who thought their article was accepted two or three years previously, are now told that the review process was corrupted, and are expected to revise the article in light of a new review. As far as I can tell from what is publicly on the record, a few authors agreed to do that, but others did not. If it happened to me, I'd be incandescent.

I've only looked at 84 articles from one review mill, but Oviedo-García has documented many more examples, nearly all from MDPI journals, and she suggests urgent actions that publishers should take: be vigilant for fraudulent and plagiarised peer review, blacklist review millers, and develop guidelines for publishers to follow in this situation. Until this is done, authors should be aware that their published article could be slapped with a "Journal notice" expressing concern about the review process, and they could be asked for revisions, even when they are entirely innocent of any wrongdoing.

Reference

Oviedo-García, M. Á. (2024). The review mills, not just (self-)plagiarism in review reports, but a step further. Scientometrics. https://doi.org/10.1007/s11192-024-05125-w 

Pérez, I.; Heitkamp, T.; Börsch, M. (2023) Mechanism of ADP-inhibited ATP hydrolysis in

single proton-pumping FoF1-ATP synthase trapped in solution. Int. J. Mol. Sci. 24, 8442. 

https://doi.org/10.3390/ijms24098442

 

Tomatoes roaming the fields and canaries in the coalmine: another embarrassing paper for MDPI

 

Many publishers are getting nervous about infiltration by paper mills, who can torpedo a journal's reputation when they succeed in publishing papers that are obvious nonsense. In a recent Open Letter, a group of sleuths drew attention to an example in Scientific Reports, published by Springer Nature.

After the Open Letter was published, the paper that instigated our concern was promptly retracted by the journal, but as far as I can tell, not much else has changed. The point about a paper like this is that it is so blatantly bad that it cannot have been through any kind of serious editorial scrutiny or peer review. It acts as a canary in the coalmine: if gobbledegook is published in your journal, it's an indicator that you need to look very carefully at your editorial processes, and act immediately to remove editors who let this stuff in. Sadly, I haven't yet seen much evidence of that happening at Scientific Reports.

This post, however, concerns another publisher, MDPI, who have regularly featured on my blog, and not in a good way. Last month, I commented on the strange state of affairs whereby Finland had downgraded its classification of 187 MDPI journals because of evidence of "minimum time spend for editorial work and quality assessment", at the same time that German universities had secured a national publishing agreement with MDPI. The story I have to tell here may confirm Finland's judgement, and give Germany pause for thought. It concerns this article: Abbas, R., Amran, G. A., Hussain, I., & Ma, S. (2022). A Soft Computing View for the Scientific Categorization of Vegetable Supply Chain Issues. Logistics, 6(3), https://doi.org/10.3390/logistics6030039 

As with the Springer Nature example, the first indication of problems came via the Problematic Paper Screener, the excellent system that checks articles for various red flags, including "tortured phrases". These provide an indicator that a paper has probably been plagiarised but then passed through a process that substitutes synonyms for main words, with the aim of evading plagiarism detection software. So, as noted on PubPeer, in this case we have "fluffy logic" for fuzzy logic, and "unaided ML" for unsupervised machine learning. 

However, example sentences in which tortured phrases were embedded indicated a deeper problem. Most of the text is incomprehensible, and things start to get seriously weird when the authors get on to tomatoes. We are told: 

.... the third creation framework considered for the creation phase is tomatoes. This creation framework is devoted to developing homegrown creatures brought up in rural settings to create vegetables. This can bring domesticated tomatoes likewise to broad or serious frameworks. Broad frameworks include creatures wandering meadows (ordinarily under the oversight of a herder). Differently, serious tomatoes are situated in shut foundations and are outfitted with ICT innovation, which empowers creatures to be observed continuously. Inside these creation frameworks, the most run-of-the-mill issues we run over are meadow observing [75], creature government assistance [76], creature conduct following [77], and tomato creation forecast and enhancement [78,79], as displayed in Figure 3. 

According to a VSC point of view, the formal meanings of these issues are recorded beneath. 

• Field checking: This issue is connected with the exact recognizable proof of meadow inventories to separate between the most reasonable sorts for tomatoes purposes. 

• Tomato government assistance: This is centered around the example arrangement of the dehydration way of behaving in brushing creatures for investigations of creature nourishment, development, and well-being. 

• Tomato growth checking: This depends on the utilization of conduct investigations to recognize early indications of medical problems and advance early negotiation. 

A clue to the origin of this material comes from the cited references, which are about pigs and cattle. Anonymous PubPeer commenter Nerita vitiensis found that a substantial part of the text was adapted from a previous work by different authors, but with the topics of "livestock and fish" changed to "tomatoes and cruciferous vegetables". This explains the description of tomatoes as "creatures" under the oversight of a herder. 

The authors of this piece seem seriously out of their depth, as evidenced by the bland comments apparently written by Chat GPT that they provided on PubPeer. 

Now, one very good thing about MDPI is that it generally identifies the academic editor who handled a paper, and it sometimes also makes public the reviewer reports. This should mean that when a major foul-up like this occurs, it should be possible to identify and purge those responsible for accepting the work. 

The academic editors who accepted this article are Xue-Ming Yuan, who is currently soliciting papers for a special issue in the MDPI journal Mathematics, and Anrong Xue.

The MDPI website shows reports from three named reviewers. 

The first reviewer, Edyta Kardas, was concerned about the use of first-person language, and punctuation, but not apparently about statements about animated tomatoes. She reviewed 8 papers for MDPI journals in 2024.

The second reviewer, Alejandro Vega-Muñoz focused solely on the structure of the article, but apparently did not look at the content. He has edited two special issues for other MDPI journals. 

 The third reviewer, Francesco Barreca, attempted a synopsis of the article (which I could not understand) and then had just two suggestions:

"The work is well done but I have some remarks:

• The figures should be review, the dimension are variable

 • Moderate English changes are required"

The "moderate English changes" were unspecified. Barreca has a track record of editing a special issue of another MDPI journal.

 
Last week, I contacted Publication Ethics at MDPI to draw their attention to this article, noting the dereliction of duty by reviewers and editors, and suggesting that as well as retracting the paper, they should remove the editors and peer reviewers from their database. They replied to say: 

"We confirm that the Editorial Office is investigating the concerns related to this paper following the guidelines of the Committee on Publication Ethics https://publicationethics.org/ of which we are a member and our policy https://www.mdpi.com/ethics#_bookmark29.

We would like to inform you that this case is a priority for us, and we are actively working to resolve it. We will update you on the outcome of this investigation as soon as possible."

I await developments with interest. It is widely recognised that COPE guidelines are not well-suited for dealing with this kind of situation: they make the default assumption that authors should be consulted to give their perspective when criticisms are raised - a reasonable assumption in many cases, but not when there is such blatant evidence of fakery.

The most serious case of infestation of a publisher by nonsense occurred in 2022-3, when the publisher Hindawi (owned by Wiley) was targeted by paper mills who, among other things, generated numerous papers that I labelled as AI gobbledegook sandwiches. Eventually, the publisher withdrew literally thousands of papers and closed the Hindawi brand, after complaints by shareholders started impacting profits.

Like many of the sleuths who track down paper mills, I have become cynical about the commitment to research integrity that is claimed by many publishers, including MDPI. But I do believe they will act when it is in their interests to do so. As the amount of nonsense and disinformation in the scientific literature increases, I think we'll enter a new phase where trustworthiness of journal contents will start to have much higher value. If you want to be taken seriously as a peer-reviewed journal, you just cannot continue to pump out articles accompanied by superficial verbiage from "peer reviewers" that makes no real contact with the subject matter. Publishers will need to act now to clean up their editorial boards if they want to stay in business.   

Update 7th February 2025

 

I am pleased to report that I have now heard from MDPI as follows:

Following a thorough investigation of this paper and according to the recommendations of the journal's Editorial Board, we have decided to retract this publication, in line with MDPI’s retraction policy: https://www.mdpi.com/ethics#_bookmark30

Further details regarding this retraction can be found at: https://www.mdpi.com/2305-6290/9/1/20

However, as far as I can tell, no action has been taken against the editors and reviewers who were responsible for accepting the article.  The only thing I notice is that the peer reviewer comments are no longer linked to the main page for the article on the journal website (though they are still available here).  I have written to MDPI Publication Ethics to ask for clarification as to whether any action will be taken to replace the editors, and to remove the peer reviewers from their register.  Unless a robust approach is taken to removing those who admit such nonsense into the journal, readers and potential authors can have no confidence in the integrity of MDPI's editorial processes.

Update 12th February 2025

I have now received a response from MDPI Publication Ethics to my query about editors and peer reviewers.  The plan is to educate them rather than remove them.  

We are happy to provide additional details regarding our approach to this situation. 

In line with the recommendations from COPE for ethical and transparent scholarly publishing (https://publicationethics.org/guidance/guideline/principles-transparency-and-best-practice-scholarly-publishing), we are adopting an educational approach. 
This implies sending notifications to the reviewers and the editor to inform them of this retraction, clarify their responsibilities in the review process, and outline the potential consequences of a superficial review, according to our guidelines for reviewers (https://www.mdpi.com/reviewers#_bookmark11) and information for editors (https://www.mdpi.com/editors). 

Given the significant responsibility that comes with the role of peer reviewers, which directly impacts the credibility of both the journal and the broader scientific literature, we take proactive steps to offer additional guidance and resources beyond our standard guidelines. An example of this initiative can be found on our MDPI Blog: https://blog.mdpi.com/2025/01/22/reviewer-responsibilities/

Moving forward, we will closely monitor the activities of both the editor and reviewers to ensure adherence to the highest standards of academic publishing.

We remain at your disposal for any further information.

Papermillers must be feeling very cheerful right now.

Note: Comments on this blog are moderated, so there is a delay before they appear.  Anonymous or off-topic comments are not accepted. 

Retrospective look at blog highlights of 2024: What happened next?

It's always interesting for a blogger to look back to see which posts have garnered most attention. In 2024 there were three standout items, my resignation from the Royal Society, an Open Letter about editorial failings at the journal Scientific Reports, and a guest post by René Aquarius about his experiences as a reviewer for MDPI. For each of these blogposts, it's interesting to consider not just the number of hits, but also the broader impact. Blogging is a great way to let off steam, but does it actually achieve anything?

Resignation from the Royal Society 

As I said at the time of writing the post, I don't expect my resignation to change much at the Royal Society, but I was surprised at the amount of interest in the matter. The story was picked up by national media (e.g. here and here), and even got covered in France and the Netherlands. I had anticipated I might be attacked by a swarm of trolls, but there was no more than a handful of rude emails, saying the Royal Society was better off without me. What was more remarkable was the outpouring of positive reactions. My inbox was swamped with people thanking me for taking this step - not just academics, but people from all walks of life. It's customary when elected to the Royal Society for colleagues to send congratulatory emails: the number of congratulations I received for resigning was about five times as many as I had for being elected. I felt I had struck a chord with many people who were fed up with unsuitable individuals being honoured. British people in particular were outraged that the Royal Society was honouring someone who was spreading misinformation with the apparent aim of undermining our democracy.

I've seen only one serious attempt to challenge the arguments in my blogpost - by Toby Young in the Spectator. I can imagine that the comms team at the Royal Society may not have been overjoyed at his main line of argument, which was to say that there had been many Fellows of the Royal Society with rebarbative opinions who hadn't been thrown out, so why should they start now? He described me as a left-wing Torquemada, which I rather enjoyed, except that his implication was that my objections to Musk were political. Had he read my blogpost, he'd see that politics didn't come into it - it is the lack of scientific integrity that is the issue.

As a delightful twist to the story, Toby Young, who is notorious for his foul-mouthed comments on women and disabled people, has just been appointed to the House of Lords, showing that the Royal Society is not alone among the British establishment in honouring those who represent the antithesis of their core values. I can only assume that Kemi Badenoch, who was responsible for his elevation, did it to own the libs.

Open letter regarding Scientific Reports

This letter, cosigned by 22 scientific sleuths, noted that laughably awful papers were getting published in the Springer Nature journal Scientific Reports, which is generally regarded as a serious journal. It looked to us as if the journal was being infiltrated by paper mills who were using it as an outlet for fraudulent or low quality work. For this to happen, there need to be editors who are either turning a blind eye to such articles, or were actively working with a paper mill to accommodate them. A quick analysis of the editors listed on the journal website turned up 28 who had records on PubPeer indicative of involvement in research misconduct.

So what has happened since? We received a reply from Chris Graf, the Research Integrity Director at Springer Nature, thanking us for our letter and emphasising the extent to which the publisher was putting resources into tackling research fraud and paper mills. The article that had sparked off the Open Letter was retracted fairly promptly.

But when I rechecked the Editorial Board list today, 26 the 28 editors we'd listed were still in position - even though we'd given links to PubPeer entries that specified problematic behaviours of all of them. The two who are no longer listed are Ilyas Khan, and Achyut Shankar. I have no idea whether their disappearance from editorial roles at Scientific Reports has anything to do with the Open Letter. (Rule 1 of sleuthing: don't expect anyone in power to tell you if they have acted on information you provided).

And a final wrinkle to the story: a fellow sleuth told me about another editor linked to paper mills, Masoud Afrand, who had not been included in our list. He had actually been removed from the list of editors in March 2022, but then reappeared at some point in the summer of 2024 (as verified by the Wayback Machine). The Editor-in-Chief, Rafal Marszalek, explained this as a "clerical error". In email correspondence I pointed out to him that this was a pretty odd kind of clerical error, and that it did nothing to assuage growing concerns that there may be someone at a senior level at Scientific Reports who was actively working with paper millers.  I did not get a reply.

My experience as a reviewer for MDPI: guest post by René Aquarius

In this post, René describes the fate of a paper that he agreed to review for the Journal of Clinical Medicine, published by MDPI. The paper had numerous methodological flaws, including lack of a control group and discrepancies between the registered protocol and the final study, and René recommended rejection. What then followed was a bizarre series of exchanges with MDPI editors, who encouraged resubmission by the authors, then attempted to block René from re-reviewing the paper, after he'd agreed to do so. Eventually the paper was transferred to  another MDPI journal, Geriatrics, where he was again asked to review it. When he pointed out that the paper was largely unchanged from the original submission, it was withdrawn - but he then found it published a few weeks later in yet another MDPI journal.

This blogpost attracted a lot of comments, not least from people who had had similar experiences with MDPI. I have to say that these experiences, coupled with other evidence such as this and this make me very dubious about standards of peer review at MDPI. The impression is that hard-pressed editorial staff are expected to ensure peer review of submissions is achieved in 2-3 weeks, and they accordingly come to rely on anyone who will do a speedy job, without regard for quality. Indeed, high-quality reviews that raise difficult issues are problematic for them, because they will slow up the process. Accordingly these are sidelined or quietly forgotten about, in favour of minimalist 'reviews' such as those you can see here in reviews 1, 3 and 4.

Will anything change? Hopefully, René's post has helped raise awareness of the problematic aspects of peer review at MDPI journals. I doubt the publisher will do anything to change their peer review model unless the reputational damage from such revelations starts to hit their bottom line. Interestingly, in my last post of 2024, I noted that just before the Christmas break, Finland's Publication Forum (JUFO) downgraded the status of 187 MDPI journals in their index, a move that will disincentivize Finnish researchers from publishing there. However, a day later, MDPI announced that it had secured a national publishing agreement with ZB Med, which offered substantial discounts to authors from over 100 German Universities.

Epilogue

In each of these cases, we can see that a blogpost has raised the profile of an issue that is of relevance to academic scientists. Concrete impacts are harder to demonstrate - where changes happen, it's not clear if they are related to the blogpost, or just coincidental. And for most topics, it's a case of two steps forward and one step back, at best. But if I've learned one thing from my many years writing about such things, it's that you can't trust the people in positions of power to actually do the right thing unless they are prodded repeatedly and publicly (and sometimes not even then). So I hope to continue blogging through 2025, even though it may at times seem futile.

Happy New Year, and Illegitimi non carborundum.

Comments are moderated. Those that are in scope and from an identified person will be published after a delay. 

Finland vs. Germany: the case of MDPI

It's been an odd week for the academic publisher MDPI. On 16th December, Finland's Publication Forum (known as JUFO) announced that from January 2025 it was downgrading its classification of 271 open access journals to the lowest level, zero. By my calculations, this includes 187 journals from MDPI and 82 from Frontiers, plus 2 others. This is likely to deter Finnish researchers from submitting work to these journals, as the rating indicates they are of low quality. As explained in an article in the Times Higher Education, JUFO justified its decision on the grounds that these journals “aim to increase the number of publications with the minimum time spend for editorial work and quality assessment”. 

Then the very next day, 17th December, MDPI announced that it had secured a national publishing agreement with ZB Med, which offered substantial discounts to authors from over 100 German Universities publishing in MDPI journals. On Bluesky, this news was not greeted with the universal joy that ZB Med might have anticipated, with comments such as "an embarrassment", "shocking", and "a catastrophe".

To understand the background, it's worth reading a thread on Bluesky by Mark Hanson, one of the authors of a landmark paper entitled "The Strain on Scientific Publishing". This article showed that MDPI and Frontiers stand out from other publishers in terms of having an exponential growth in number of papers published in recent years, a massive shift to special issues as a vehicle for this increase, and a sharp drop in publication lag from 2016 to 2022.

In their annual report for 2023, MDPI described annualised publication growth of 20.4%. They stated that they have over 295,693 academic editors on editorial boards, and a median lag of 17 days from receipt of a paper to the first editorial decision and 6 weeks from submission to publication. It's not clear whether the 17 day figure includes desk rejections, but even if it does, this is remarkably fast. Of course, you could argue (and I'm sure the publishers will argue) that if you are publishing a lot more and doing it faster, you are just being efficient. However, an alternative view, and one that is apparently held by JUFO, is that this speedy processing goes hand in hand with poor editorial quality control.

The problem here is that anyone who has worked as a journal editor knows that, while a 17 day turnaround might be a good goal to aim for, it is generally not feasible. There is a limited pool of experts who can do a thorough peer review, and often one has to ask as many as 20 people to do a review in order to achieve 2 or 3 reviews. So it can take at least a couple of weeks to secure reviewers, and then it is likely to be another couple of weeks before all reviewers have completed a comprehensive review. Given these constraints, most editors would be happy if they could achieve a median time to first decision of 34 days - i.e. double that reported by MDPI. So the sheer speed of decision making - regarded by MDPI as a selling point for authors - is a red flag.

It seems that speed is achieved by adopting a rather unorthodox process of assigning peer reviewers, where the involvement of an academic editor is optional: "At least two review reports are collected for each submitted article. The academic editor can suggest reviewers during pre-check. Alternatively, MDPI editorial staff will use qualified Editorial Board members, qualified reviewers from our database, or new reviewers identified by web searches for related articles." The impression is that, in order to meet targets, editorial staff will select peer reviewers who can be relied upon to respond immediately to requests to review.

A guest post on this blog by René Aquarius supported these suspicions and further suggested that reviewers who are critical may be sidelined. After writing an initial negative review, René had promptly agreed to review a revision of a paper, but was then told his review was not needed - this is quite remarkable, given that most editors would be delighted if a competent reviewer agreed to do a timely re-review. It's worth looking not just at René's blogpost but also the comments from others, which indicate his experience is not unique.

A further issue concerns the fate of papers receiving negative reviews. René found that the paper he had rejected popped up as a new submission in another MDPI journal, and after negative reviews there, it was published in a third journal. This raises questions about MDPI's reported rejection rate of around 50%. If each of these resubmissions was counted as a new submission, the rejection rate would appear to be 66%, but given that the same paper was recycled from journal to journal before eventual acceptance, the actual rate was 0%. 

One good thing about MDPI is that it gives authors the option of making peer review open. However, analysis of published peer reviews further dents confidence in the rigour of the peer review process. A commentary in Science covering the work of Maria Ángeles Oviedo García noted how some MDPI peer reviews contained repetitive phrases that suggested they were generated by a template. They were superficial and did not engage seriously with the content of the article. In some cases the reviewer freely admitted a lack of expertise in the topic of the article, and in others, there was evidence of coercive citation (i.e., authors being told to cite the work of a specific individual, sometimes the editor).

Comments on Reddit cannot, of course, be treated as hard evidence, but they raise further questions about the peer review process at MDPI. Several commenters described having recommended rejection as a peer reviewer only to find that the paper was published without their comments. If negative peer review comments are selectively suppressed in the public record, this would be a serious breach of ethical standards by the publisher.

Lack of competent peer review and/or editorial malpractice is also suggested by the publication of papers in MDPI that fall well below the threshold of acceptable science. Of course, quality judgements are subjective, and it's common for researchers to complain "How did this get past peer review?" But in the case of MDPI journals, one finds articles that are so poor that they suggest the editor was either corrupt or asleep on the job. I have covered some examples in previous blogposts, here, and here.

The Finns are not alone in being concerned about research quality in MDPI journals. The Swiss National Science Foundation did not mention specific publishers by name, but in November 2023 they withdrew funding for articles published in special issues. Since 2023, the Directory of Open Access Journals has withdrawn 19 MDPI journals from its index for "Not adhering to best practice". Details are not provided, but these tend to be journals where guest editors have used special issues to publish a high volume of articles by themselves and close collaborators - another red flag for scientific quality. Yet another issue is citation manipulation, where editors or peer reviewers demand inclusion of specific references in a revision of an article in order to boost their own citation count. In February 2024, China released its latest Early Warning List of journals that are deemed to be "untrustworthy, predatory or not serving the Chinese research community’s interests". This included four MDPI journals listed for citation manipulation.

A final red flag about MDPI is that it seems remarkably averse to retracting articles. Hindawi publishing, which was bought by Wiley in 2021, was heavily criticised for allowing a flood of paper-milled articles to be published, but it did subsequently retract over 13,000 of them (just under 5% of 267K articles), before the closure of the brand. A search on Web of Science for documents classified as "retracted publication" or "retraction" and published by "MDPI or MDPI Ag" turned up a mere 582 retractions since 2010, which amounts to 0.04% of the 1.4 million articles listed on the database.

I've heard various arguments against JUFO's action, such as: many papers published in MDPI journals are fine; you should judge an article by its content, not where it is published; authors should be free to prefer speed over scrutiny if they wish. The reason why I support JUFO, and think ZB Med is rash to sign an agreement, is because if the peer review process is not managed by experienced and respected academic editors with specialist subject knowledge, then we need to consider the impact, not just on individual authors, but on the scientific literature as a whole. If we allow trivia, fatally flawed studies or pseudoscience to be represented as "peer-reviewed" this contaminates the research literature, with adverse consequences for everyone.

I don't care about journal impact factors but I do care about visibility

There's been a fair bit of discussion about Clarivate's decision to pause inclusion of eLife publications on the Science Citation Index (e.g. on Research Professional).  What I find exasperating is that most of the discussion focuses on a single consequence - loss of eLife's impact factor.  For authors, there are graver consequences.   

I've reviewed for eLife but never published there; however, I have published a lot in Wellcome Open Research, which is another journal that aimed to disrupt the traditional publishing model, and has some similarities with eLife.  Wellcome Open Research has never been included in Science Citation Index, despite the fact that it uses peer review.  Wellcome Open Research has an unconventional model whereby submitted papers are immediately published, as a kind of pre-print prior to peer review, and then updated after peer review.  It is true that some papers don't get sufficient approval to proceed to the peer-reviewed stage; the distinction between those that do and do not pass peer review is clearly flagged on the article.  In addition to peer review, Wellcome Open Research maintains some quality control by limiting eligibility to researchers funded by Wellcome.  

 

When Wellcome Open Research started up, all Wellcome-funded researchers were encouraged to publish there.  As someone committed to Open Research, this seemed a great idea.  There were no publication charges, and everything was open: access to the publication, data, and peer review. Peer reviewers even get DOIs for their reviews, some of which are worth citing in their own right.  I was increasingly adopting open practices, and I think some of my best peer-reviewed work is published there. 

 

I was shocked when I discovered that the journal wasn't included in Web of Science. I remember preparing a progress report for Wellcome and using Web of Science to check I hadn't omitted any publications.  I was puzzled that I seemed to have published far less than I remembered. Then it became clear: everything in Wellcome Open Research was missing. 

 

I was on the Advisory Board for Wellcome Open Research at the time, and raised this with them. They were shocked that I was upset.  "We thought you of all people didn't care about impact factors", they said. This, of course, was true. But I did care a lot about my work being visible.  I was also aware that any WOS-based H-index would exclude all the papers listed below: not a big deal for me, but potentially harmful to junior authors.  

 

The reply I got was similar to the argument being made by eLife  - well, the articles are indexed in Google Scholar and PubMed.  That was really little consolation to me, given that I had relied heavily on Web of Science in my own literature searches, believing that it screened out dodgy journals. (This belief turns out to be false - there are many journals featured in WoS that are very low quality, which just rubs salt into the wound).  

 

I have some criticisms of eLife's publishing model, but I would like them to succeed. We urgently need alternatives to the traditional journal model operated by the big commercial publishers.  Their response to the open access movement has been to monetise it, with catastrophic consequences for science, as an unlimited supply of shoddy and fake work gets published - often in journals that are indexed in Web of Science.

 

I agree that we need an index of published academic work that has some quality control.  Whether alternatives like OpenAlex will do the job remains to be seen. 

 

Papers that aren't indexed on Web of Science

Bishop, D. V. M., & Bates, T. C. (2020). Heritability of language laterality assessed by functional transcranial Doppler ultrasound: A twin study. Wellcome Open Research, 4, 161. https://doi.org/10.12688/wellcomeopenres.15524.3

Bishop, D. V. M., Brookman-Byrne, A., Gratton, N., Gray, E., Holt, G., Morgan, L., Morris, S., Paine, E., Thornton, H., & Thompson, P. A. (2019). Language phenotypes in children with sex chromosome trisomies. Wellcome Open Research, 3, 143. https://doi.org/10.12688/wellcomeopenres.14904.2

Bishop, D. V. M., Grabitz, C. R., Harte, S. C., Watkins, K. E., Sasaki, M., Gutierrez-Sigut, E., MacSweeney, M., Woodhead, Z. V. J., & Payne, H. (2021). Cerebral lateralisation of first and second languages in bilinguals assessed using functional transcranial Doppler ultrasound. Wellcome Open Research, 1, 15. https://doi.org/10.12688/wellcomeopenres.9869.2

Frizelle, P., Thompson, P. A., Duta, M., & Bishop, D. V. M. (2019). The understanding of complex syntax in children with Down syndrome. Wellcome Open Research, 3, 140. https://doi.org/10.12688/wellcomeopenres.14861.2

Newbury, D. F., Simpson, N. H., Thompson, P. A., & Bishop, D. V. M. (2018). Stage 1 Registered Report: Variation in neurodevelopmental outcomes in children with sex chromosome trisomies: protocol for a test of the double hit hypothesis. Wellcome Open Research, 3, 10. https://doi.org/10.12688/wellcomeopenres.13828.2

Newbury, D. F., Simpson, N. H., Thompson, P. A., & Bishop, D. V. M. (2021). Stage 2 Registered Report: Variation in neurodevelopmental outcomes in children with sex chromosome trisomies: testing the double hit hypothesis. Wellcome Open Research, 3, 85. 

https://doi.org/10.12688/wellcomeopenres.14677.4

Pritchard, V. E., Malone, S. A., Burgoyne, K., Heron-Delaney, M., Bishop, D. V. M., & Hulme, C. (2019). Stage 2 Registered Report: There is no appreciable relationship between strength of hand preference and language ability in 6- to 7-year-old children. Wellcome Open Research, 4, 81. https://doi.org/10.12688/wellcomeopenres.15254.1

Thompson, P. A., Bishop, D. V. M., Eising, E., Fisher, S. E., & Newbury, D. F. (2020). Generalized Structured Component Analysis in candidate gene association studies: Applications and limitations. Wellcome Open Research, 4, 142. https://doi.org/10.12688/wellcomeopenres.15396.2

Wilson, A. C., & Bishop, D. V. M. (2019). ‘If you catch my drift...’: Ability to infer implied meaning is distinct from vocabulary and grammar skills. Wellcome Open Research, 4, 68. https://doi.org/10.12688/wellcomeopenres.15210.3

Wilson, A. C., King, J., & Bishop, D. V. M. (2019). Autism and social anxiety in children with sex chromosome trisomies: An observational study. Wellcome Open Research, 4, 32. https://doi.org/10.12688/wellcomeopenres.15095.2

Woodhead, Z. V. J., Rutherford, H. A., & Bishop, D. V. M. (2020). Measurement of language laterality using functional transcranial Doppler ultrasound: A comparison of different tasks. Wellcome Open Research, 3, 104. https://doi.org/10.12688/wellcomeopenres.14720.3

 

 

 

Prodding the behemoth with a stick

 

Like many academics, I was interested to see an announcement on social media that a US legal firm had filed a federal antitrust lawsuit against six commercial publishers of academic journals: (1) Elsevier B.V.; (2) Wolters Kluwer N.V.; (3) John Wiley & Sons, Inc.; (4) Sage Publications, Inc.; (5) Taylor and Francis Group, Ltd.; and (6) Springer Nature AG & Co, on the grounds that "In violation of Section 1 of the Sherman Act, the Publisher Defendants conspired to unlawfully appropriate billions of dollars that would have otherwise funded scientific research".   

 

So far, so good.  I've been writing about the avaricious practices of academic publishers for over 12 years, and there's plenty of grounds for a challenge. 

 

However, when I saw the case being put forward, I was puzzled.  From my perspective, the arguments just don't stack up.  In particular, three points are emphasised in the summary (quoted verbatim here from the website): 

 

• First, an agreement to fix the price of peer review services at zero that includes an agreement to coerce scholars into providing their labor for nothing by expressly linking their unpaid labor with their ability to get their manuscripts published in the defendants’ preeminent journals.

 

But it's not true that there is an express link between peer review and publishing papers in the pre-eminent journals.  In fact, many journal editors complain that some of the most prolific authors never do any peer review - gaining an advantage by not adopting the "good citizen" behaviour of a peer reviewer.  I think this point can be rapidly thrown out.

 

• Second, the publisher defendants agreed not to compete with each other for manuscripts by requiring scholars to submit their manuscripts to only one journal at a time, which substantially reduces competition by removing incentives to review manuscripts promptly and publish meritorious research quickly. 

 

This implies that the rationale for not allowing multiple submissions is to reduce competition between publishers.  But if there were no limit on how many journals you could simultaneously submit to, then the number of submissions to each journal would grow massively, increasing the workload for editors and peer reviewers - and much of their time would be wasted. This seems like a rational requirement, not a sinister one.

 

• Third, the publisher defendants agreed to prohibit scholars from freely sharing the scientific advancements described in submitted manuscripts while those manuscripts are under peer review, a process that often takes over a year. As the complaint notes, “From the moment scholars submit manuscripts for publication, the Publisher Defendants behave as though the scientific advancements set forth in the manuscripts are their property, to be shared only if the Publisher Defendant grants permission. Moreover, when the Publisher Defendants select manuscripts for publication, the Publisher Defendants will often require scholars to sign away all intellectual property rights, in exchange for nothing. The manuscripts then become the actual property of the Publisher Defendants, and the Publisher Defendants charge the maximum the market will bear for access to that scientific knowledge.” 

Again, I would question the accuracy of this account.  For a start, in most science fields, peer review is a matter of weeks or months, not "over a year".  But also, most journals these days allow authors to post their articles as preprints, prior to, or at the point of submission. In fact, this is encouraged by many institutions, as it means that a Green Open Access version of the publication is available, even if the work is subsequently published in a pay-to-read version.  

 

In all, I am rather dismayed by this case, especially when there are very good grounds on which academic publishers can be challenged.  For instance:

 

1. Academic publishers claim to ensure quality control of what gets published, but some of them fail to do the necessary due diligence in selecting editors and reviewers, with the result that the academic literatureis flooded with weak and fraudulent material, making it difficult to distinguish valuable from pointless work, and creating an outlet for damaging material, such as pseudoscience.  This has become a growing problem with the advent of paper mills.

 

2. Many publishers are notoriously slow at responding to credible evidence of serious problems in published papers. It can take years to get studies retracted, even when they have important real world consequences.

 

3. Perhaps the only point in common with the case by Leiff Cabraser, Heimann and Bernstein concerns the issue of retention of intellectual property rights.  It is the case that publishers have traditionally required authors to sign away copyright of their works.  In the UK, at least, there has been a movement to fight this requirement, which has had some success, but undoubtedly more could be done. 

 

If I can find time I will add some references to support some of the points above - this is a hasty response to discussion taking place on social media, where many people seem to think it's great that someone is taking on the big publishers. I never thought I would find myself in a position of defending them, but I think if you are going to attack a behemoth, you need to do so with good weapons.  

 

 

Postscript

Comments on this post are welcome - there is moderation so they don't appear immediately.

 Nick Wise attempted unsuccessfully to add a comment (sorry, Blogger can be weird), providing this helpful reference on typical duration of peer review.  Very field-dependent and may be a biased sample, I suspect, but it gives us a rough idea.

PPS. 5th October 2024.

Before I wrote this blogpost, I contacted the legal firm involved, Leiff Cabraser, Heimann & Bernstein, via their website, to raise the same points.  Yesterday I received a reply from them, explaining that "Because you are located abroad, unfortunately you are not a member of this class suit".  This suggests they don't read correspondence sent to them. Not impressed.  

My experience as a reviewer for MDPI

 

Guest post by 

René Aquarius, PhD

Department of Neurosurgery

Radboud University Medical Center, Nijmegen, The Netherlands

 

After a recent zoom-call where Dorothy and I discussed several research-related topics, she invited me to write a guest blogpost about the experience I had as a peer-reviewer for MDPI. As I think transparency in research is important, I was happy to accept this invitation.  

 

Mid November 2023 I received a request to peer-review a manuscript for a special issue on subarachnoid hemorrhage for the Journal of Clinical Medicine, published by MDPI. This blog post summarizes that process. I hope it will give some insight on the nitty-gritty of the peer-review process for MDPI.

 

I decided to review the manuscript two days after receiving the invitation and what I found was a study like many others in the field: a single-center, retrospective analysis of a clinical case series. I ended up recommending rejection of the paper two days after accepting to review. My biggest gripes were that the authors claimed that data were collected prospectively, but their protocol was registered at the very end of the period in which they included patients. In addition, I discovered some important discrepancies between protocol and the final study. Target sample size according to the protocol was 50% bigger than what was used in their study. The minimum age for patients also differed between the protocol and the manuscript. I also had problems with their statistical analysis as they used more than 20 t-tests to test variables, which creates a high probability of Type I errors. The biggest problem was the lack of a control group, which made it impossible to establish whether changes in a physiological parameter could really predict intolerance for a certain drug in a small subset of patients.

 

When filling out the reviewer form for MDPI, certain aspects struck me as peculiar. There are four options for Overall Recommendation:

• Accept in present form
• Accept after minor revision (correction to minor methodological errors and text editing)
• Reconsider after major revision (control missing in some experiments)
• Reject (article has serious flaws, additional experiments needed, research not conducted correctly)

 

Regardless of which of the last two options you select, the response is: "If we ask the authors to revise the manuscript, the revised manuscript will be sent to you for further evaluation". 

 

Although reviewer number 2 is often jokingly referred to as "the difficult one" it couldn’t be further from the truth in this case. The reviewer liked the paper and recommended accept after minor revision. So with a total of two reviews, the paper got the editorial decision of rejected, with a possibility of resubmission after extensive revisions only one day after I handed in my peer review report.

 

Revisions were quite extensive, as you will discover below, and arrived only two days after the initial rejection. I agreed to review the revised manuscript. But before I could start my review of the revision, just four days after receiving the invitation, I received a response from the editorial office that my review was no longer needed because they already had enough peer-reviewers for the manuscript. I politely ignored this request, because I wanted to know if the manuscript had improved. What happened next was quite a bit of a surprise, but not in a good way. 

 

The manuscript had indeed undergone extensive revisions. The biggest change, however, was also the biggest red flag. Without any explanation the study had lost almost 20% of its participants. An additional problem was that all the issues I had raised in my previous review report remained unaddressed. I sent my newly written feedback report the same day, exactly one week after my initial rejection.

 

When I handed in my second review report, I understood why I initially got an email that my review was not needed anymore. One peer reviewer had also rejected the manuscript and had concerns similar to mine. Two other reviewers, however, accepted the manuscript. One with minor revisions (English needed some improvement) and one in present form, so without any suggested revisions. This means that if I had followed the advice of the editorial office of MDPI, the paper would probably have been accepted in its current form. But because my vote was now also cast and the paper received two rejections, the editor couldn’t do much more than to reject the manuscript, which happened three days after I handed in my review report.  

 

Fifteen days after receiving my first invitation to review, the manuscript had already seen two full rounds of peer-review by at least four different peer-reviewers.

 

This is not where the story ends.  

 

In December, about a month later, I received an invitation to review a manuscript for the MDPI journal Geriatrics. You’ve guessed it by now: it was the same manuscript. It's reasonable to assume this was shifted internally through MDPI's transfer service, summarised in this figure.  I can only speculate as to why I was still attached to the manuscript as a peer-reviewer, but I guess somebody forgot to remove my name from it.

from: https://www.mdpi.com/authors/transfer-service

The manuscript had, again, transformed. It was now very similar to the very first version I reviewed. Almost word-for-word similar. That also meant that the number of included patients was restored to the initial number. However, the registered protocol that was previously mentioned in the methods section (which had led to some of the most difficult to refute critiques) was now completely left out. The icing on the cake was that, for a reason that was not explained, another author was added to the manuscript. There was no mention in this invitation of the previous reviews and rejections of the same manuscript.   Although one might wonder whether MDPI editors were aware of this, it would be strange if they were not, since they pride themselves on their Susy manuscript submission system where "editors can easily track concurrent and previous submissions from the same authors".

 

Because the same issues were still present in the manuscript, I rejected it for a third time on the same day I agreed to review it. In an accompanying message to the editor, I clearly articulated my problems with the manuscript and the review process.

 

The week after, I received a message that the editor had decided to withdraw the manuscript in consultation with the authors.

 

Late January 2024, the manuscript was published in the MDPI journal Medicina. I was not attached to the manuscript any more as a reviewer. There was no indication on the website of the name of the acting editor who accepted it. 

Note from Dorothy Bishop

Comments on this blog are moderated so there may be some delay before they appear, but legitimate, on-topic contributions are welcomed. We would be particularly interested to hear from anyone else who has experiences, good or bad, as a reviewer for MDPI journals.

 

Postscript by Dorothy Bishop: 19 Aug 2024 

Here's an example of a paper that was published with the reviews visible. Two were damning and one was agreeable.  https://www.mdpi.com/2079-6382/9/12/868.  Thanks to @LymeScience for drawing our attention to this, and noting the important clinical consequences when those promoting an alternative, non-evidenced treatment have a "peer-reviewed" study to refer to.