Guest post: Stealth corrections are still a threat to scientific integrity

Authors

René Aquarius, Floris Schoeters, Alex Glynn, Guillaume Cabanac

 

An update on stealth corrections

Last year, we published an article describing stealth corrections, a phenomenon in which a publisher makes at least one post-publication change to a scientific article, without providing a correction note or any other indicator that the publication was temporarily or permanently altered.

 

Now, we have expanded our database with newly identified stealth corrections. We also wrote a freely accessible COSIG guide describing how to report stealth corrections in a transparent fashion.

 

Difficult to pinpoint

Stealth corrections are, by nature, extremely difficult to track down and most stealth corrections are identified by science sleuths who might notice a mismatch between different versions of an article. It is impossible to provide a comprehensive overview and one must assume that we have only identified a small minority of these issues.

 

For this update we applied the same pragmatic approach in documenting stealth retractions as previously: registering stealth corrections on PubPeer ourselves, asking around within the science sleuthing community and searching the PubPeer database for terms as “no erratum”, “no corrigendum”, or “stealth” (repeat the search yourself).

 

Stealth corrections were further categorized into the following types:

• Changes in author information (addition or removal of authors, changes in author affiliation, etc.);
• Changes in content (figures, data or text, etc.);
• Changes in the record of editorial process (editor name, date of submission, acceptance or publication, etc.);
• Changes in additional information (ethics statements, conflicts of interest statements, funding information, etc.).

New cases

We found 32 published articles that were affected by stealth corrections in addition to the 131 we had identified last year. An overview of all stealth corrections (#1-163) can be found in the online database, which also contains the links to all accompanying PubPeer posts for additional detail. Table 1 shows the type of correction per publisher for the 32 new cases.

 

Table 1. Type of correction per publisher for newly identified stealth corrections. 

	Changes in additional information	Changes in author information	Changes in content	Changes in the record of editorial process
ACM	3	0	0	0
Am Phytopathological Soc	0	0	1	0
CV Literasi Indonesia	0	0	1	0
Elsevier	0	0	3	0
Impact Journals	0	0	1	0
Int Soc Computational Biology	1	0	0	0
MDPI	0	0	0	19
Oxford University Press	0	1	0	0
Springer Nature	0	0	1	0
Taylor & Francis	0	0	1	0

 

 

Why are stealth corrections still a thing?

Last year we wrote “post-publication amendments that are made silently, without a visible correction note, will give rise to questions regarding the ethics and integrity of the specific journal, editors and publisher, and might undermine the validity of the published literature as a whole”. Again, we have identified stealth corrections that might be used as a shortcut to ‘repair’ more serious issues. The three publishers with the most stealth corrections in this update were: MDPI, Elsevier, and the Association of Computing Machinery (ACM).

MDPI was involved in 19 new stealth corrections. Sixteen cases (#141, #144-158) were registered in August of 2024, too late for our initial pre-print and subsequent article on stealth corrections. All of these involved moving articles out of a ‘special issue’ and into a ‘section’. What stands out for all these 16 articles, is that the special issue editor was also an author on all of these papers. The Directory of Open Access Journals (DOAJ) has dictated that the number of articles co-authored by a special issue editor needs to be below lower than the 25% for each special issue. When it is higher than 25%, the DOAJ can delist the journal for not adhering to best practice, as detailed on their change log. Thus, by moving these articles silently out of special issues, MDPI is retroactively lowering this percentage to adhere to the rules of the DOAJ and therefore preventing potential delisting of their journals. In September 2024 -after publication of our pre-print- MDPI refuted “that removing a Special Issue article from the digital SI website can be considered a ‘stealth correction’”. Possibly, the updated correction process (which now includes ‘minor corrections’) facilitated a complete stop of this practice by MDPI. We have not identified any recent cases, which is an encouraging sign.

In the remaining three cases (#135-137), the name of a peer reviewer was suddenly set to anonymous, while the contents of the peer review reports did not change. According to MDPI, this was done to adhere to GDPR requirements. However, this only happened after the peer reviewer was identified as being part of a review mill. The reviewer claims on PubPeer that they were not involved in writing the peer review report. These cases prove that a request for anonymity might hamper the desire for transparency and strengthening research integrity.

Elsevier was involved in 3 new stealth corrections. All of them involved changes in content. In 3 cases an image was silently replaced (#133, #139-141) according to PubPeer reports that were posted between December 2024 and May 2025. In response to our pre-print, Elsevier stated that they “do not correct articles without a formal notice”. However, in this update we -again- present clear evidence of major changes to the scientific record that went through without any formal acknowledgement in the form of a correction notice. This directly contradicts earlier statements from Elsevier. Eventually, all of these articles have been retracted, but only 4-12 months after the stealth correction was noticed, meaning there was a substantial window of time that allowed for interaction with these flawed articles, without any proper indication that there might have been a problem.

ACM silently made multiple changes to the introduction from three conference proceedings written by the conference chair (#159-161). References were removed and in one case the text was heavily altered. In all three cases, a notice of concern was also published to indicate that the peer review process had been compromised and the publisher strongly urged people not to cite the conference papers. It seems as if the ACM retroactively tried to erase the citations to the conference papers, but they did it by secretly making all kinds of alterations to the documents, which is far from ideal.

This update shows that some scientific publishers continue to use stealth corrections as a way to change the scientific record. Stealth corrections can undermine the entire enterprise of science; at the level of the individual article, the lack of a transparent correction minimizes the likelihood of those who read or cited the original version being informed of the change; on the macro level, the integrity of the published literature as a whole is compromised as readers never know for certain whether an article has been silently corrected or not. Meanwhile, there is still no consensus on issuing corrections.

 

Conclusion and recommendations

Stealth corrections are still problematic as they are sometimes used as a shortcut to ‘repair’ other integrity issues. Again, we stress that stealth corrections are notoriously difficult to find and that this update likely only shows chance findings by science sleuths. Correct documentation and transparency are of the utmost importance to uphold scientific integrity and the trustworthiness of science.

We still recommend:

• Tracking of all changes to the published record by all publishers in an open, uniform and transparent manner, preferably by online submission systems that log every change publicly, making stealth corrections impossible.
• Clear definitions and guidelines on all types of corrections.
• Sustained vigilance of the scientific community to publicly register stealth corrections. Now made easier by using our COSIG guide.

 

Acknowledgements

We thank Dorothy Bishop for hosting this update on her blog and we thank all (anonymous) science sleuths who have found and reported stealth corrections: your work is much appreciated.

Note from DVMB: Comments are moderated on this blog.  They are usually approved if they are on topic and non-anonymous. 

New publishing models will only work if authors embrace them

Complaints about the broken academic publishing system have been around for years and are getting louder. A common theme is that with the rise of open access publishing, commercial publishers have grasped the opportunity to grow their profits from article-processing-charges (APCs). Whereas in the past, journals competed to be the most highly respected outlet, now they compete to publish on the grounds of speed and quantity of publications (see e.g. Timmis et al, 2025).

In response, various new initiatives have arisen. My focus here is on the F1000 publishing model, which was adopted by the Wellcome Trust in 2016 for their journal Wellcome Open Research. In this model, the author deposits an article on the platform (in effect as a preprint), which then is updated to a published version if two positive peer reviews are obtained. The only editorial input is from office staff who check that the submission meets basic criteria and that the selected peer reviewers are appropriate. This system could quickly get overwhelmed with low quality submissions, but aims to avoid that by restricting submissions to authors funded by the Wellcome Trust, which agrees to pay their APCs. I've always been a fan of open access, and have published several papers in Wellcome Open Research, encouraged by the prospect of straightforward and free open access publication.

A check on the Dimensions platform shows that Wellcome Open Research has grown in popularity over the years, and now dominates outputs from Wellcome-funded researchers.

Figure 1. Plot of the top 5 journals, publications funded by Wellcome Trust between 2016-2025.

My funding by Wellcome Trust came to an end and in 2016 I took up an ERC Advanced Grant. Towards the end of that grant, in March 2021, the European Commission (EC) announced that they were setting up a new journal, Open Research Europe that adopted the same F1000 model and offered free open access publication for EC-funded researchers. In contrast to Wellcome Open Research, Open Research Europe has not been enthusiastically embraced. A search on Dimensions showed that a large proportion of EC-funded research is published open access with for-profit publishers (see Figure 2). Open Research Europe is not shown because the number of publications is relatively small: 213, 251, 290, 374 for the years 2021 to 2024 respectively. It rates 25th among journals used by EC researchers, whose favourite publisher appears to be MDPI.

Figure 2. Plot of the top 5 journals, publications funded by EC between 2021-2025.

This raises two questions: who is paying the APCs, and why don't researchers publish on their funder's platform, which offers them free open access?

Of course, Open Research Europe is relatively young, and its uptake may have been influenced by its launch coinciding with emergence from lockdown. It's possible that communications from the EC encouraging grantholders to publish there haven't been sufficient to raise awareness of this option. I'd be curious if any readers who have EC funding could comment on barriers to uptake. Meanwhile, between 2021-2025 around US$973 million* in APCs has gone into the coffers of publishers - money that could have been used to fund researchers in other ways. On a rough estimate, around US$197 million of this has been paid to the most popular publisher, MDPI.

*To obtain this estimate, I searched Dimensions.ai using search terms Funder = European Commission (EC), Publication Type = Article, and year range from 2021-2025, and then used Analytical views to generate a table of source titles. For the first 30 titles, I manually checked the APC and used a currency converter to convert to $US. For the remaining titles, I estimated the APC as equivalent to the average for the first 30 titles. I coded MDPI as publisher for titles I recognised, but I did not carefully check each one. A csv file with the relevant data can be found here: https://osf.io/rcxd3.

Reference

Timmis, K., et al. (2025). Journals operating predatory practices are systematically eroding the science ethos: A gate and code strategy to minimise their operating space and restore research best practice. Microbial Biotechnology, 18(6), e70180. https://doi.org/10.1111/1751-7915.70180 

 

Comments are moderated to prevent spam, but relevant, on-topic comments are welcome. 

Tomatoes roaming the fields and canaries in the coalmine: another embarrassing paper for MDPI

 

Many publishers are getting nervous about infiltration by paper mills, who can torpedo a journal's reputation when they succeed in publishing papers that are obvious nonsense. In a recent Open Letter, a group of sleuths drew attention to an example in Scientific Reports, published by Springer Nature.

After the Open Letter was published, the paper that instigated our concern was promptly retracted by the journal, but as far as I can tell, not much else has changed. The point about a paper like this is that it is so blatantly bad that it cannot have been through any kind of serious editorial scrutiny or peer review. It acts as a canary in the coalmine: if gobbledegook is published in your journal, it's an indicator that you need to look very carefully at your editorial processes, and act immediately to remove editors who let this stuff in. Sadly, I haven't yet seen much evidence of that happening at Scientific Reports.

This post, however, concerns another publisher, MDPI, who have regularly featured on my blog, and not in a good way. Last month, I commented on the strange state of affairs whereby Finland had downgraded its classification of 187 MDPI journals because of evidence of "minimum time spend for editorial work and quality assessment", at the same time that German universities had secured a national publishing agreement with MDPI. The story I have to tell here may confirm Finland's judgement, and give Germany pause for thought. It concerns this article: Abbas, R., Amran, G. A., Hussain, I., & Ma, S. (2022). A Soft Computing View for the Scientific Categorization of Vegetable Supply Chain Issues. Logistics, 6(3), https://doi.org/10.3390/logistics6030039 

As with the Springer Nature example, the first indication of problems came via the Problematic Paper Screener, the excellent system that checks articles for various red flags, including "tortured phrases". These provide an indicator that a paper has probably been plagiarised but then passed through a process that substitutes synonyms for main words, with the aim of evading plagiarism detection software. So, as noted on PubPeer, in this case we have "fluffy logic" for fuzzy logic, and "unaided ML" for unsupervised machine learning. 

However, example sentences in which tortured phrases were embedded indicated a deeper problem. Most of the text is incomprehensible, and things start to get seriously weird when the authors get on to tomatoes. We are told: 

.... the third creation framework considered for the creation phase is tomatoes. This creation framework is devoted to developing homegrown creatures brought up in rural settings to create vegetables. This can bring domesticated tomatoes likewise to broad or serious frameworks. Broad frameworks include creatures wandering meadows (ordinarily under the oversight of a herder). Differently, serious tomatoes are situated in shut foundations and are outfitted with ICT innovation, which empowers creatures to be observed continuously. Inside these creation frameworks, the most run-of-the-mill issues we run over are meadow observing [75], creature government assistance [76], creature conduct following [77], and tomato creation forecast and enhancement [78,79], as displayed in Figure 3. 

According to a VSC point of view, the formal meanings of these issues are recorded beneath. 

• Field checking: This issue is connected with the exact recognizable proof of meadow inventories to separate between the most reasonable sorts for tomatoes purposes. 

• Tomato government assistance: This is centered around the example arrangement of the dehydration way of behaving in brushing creatures for investigations of creature nourishment, development, and well-being. 

• Tomato growth checking: This depends on the utilization of conduct investigations to recognize early indications of medical problems and advance early negotiation. 

A clue to the origin of this material comes from the cited references, which are about pigs and cattle. Anonymous PubPeer commenter Nerita vitiensis found that a substantial part of the text was adapted from a previous work by different authors, but with the topics of "livestock and fish" changed to "tomatoes and cruciferous vegetables". This explains the description of tomatoes as "creatures" under the oversight of a herder. 

The authors of this piece seem seriously out of their depth, as evidenced by the bland comments apparently written by Chat GPT that they provided on PubPeer. 

Now, one very good thing about MDPI is that it generally identifies the academic editor who handled a paper, and it sometimes also makes public the reviewer reports. This should mean that when a major foul-up like this occurs, it should be possible to identify and purge those responsible for accepting the work. 

The academic editors who accepted this article are Xue-Ming Yuan, who is currently soliciting papers for a special issue in the MDPI journal Mathematics, and Anrong Xue.

The MDPI website shows reports from three named reviewers. 

The first reviewer, Edyta Kardas, was concerned about the use of first-person language, and punctuation, but not apparently about statements about animated tomatoes. She reviewed 8 papers for MDPI journals in 2024.

The second reviewer, Alejandro Vega-Muñoz focused solely on the structure of the article, but apparently did not look at the content. He has edited two special issues for other MDPI journals. 

 The third reviewer, Francesco Barreca, attempted a synopsis of the article (which I could not understand) and then had just two suggestions:

"The work is well done but I have some remarks:

• The figures should be review, the dimension are variable

 • Moderate English changes are required"

The "moderate English changes" were unspecified. Barreca has a track record of editing a special issue of another MDPI journal.

 
Last week, I contacted Publication Ethics at MDPI to draw their attention to this article, noting the dereliction of duty by reviewers and editors, and suggesting that as well as retracting the paper, they should remove the editors and peer reviewers from their database. They replied to say: 

"We confirm that the Editorial Office is investigating the concerns related to this paper following the guidelines of the Committee on Publication Ethics https://publicationethics.org/ of which we are a member and our policy https://www.mdpi.com/ethics#_bookmark29.

We would like to inform you that this case is a priority for us, and we are actively working to resolve it. We will update you on the outcome of this investigation as soon as possible."

I await developments with interest. It is widely recognised that COPE guidelines are not well-suited for dealing with this kind of situation: they make the default assumption that authors should be consulted to give their perspective when criticisms are raised - a reasonable assumption in many cases, but not when there is such blatant evidence of fakery.

The most serious case of infestation of a publisher by nonsense occurred in 2022-3, when the publisher Hindawi (owned by Wiley) was targeted by paper mills who, among other things, generated numerous papers that I labelled as AI gobbledegook sandwiches. Eventually, the publisher withdrew literally thousands of papers and closed the Hindawi brand, after complaints by shareholders started impacting profits.

Like many of the sleuths who track down paper mills, I have become cynical about the commitment to research integrity that is claimed by many publishers, including MDPI. But I do believe they will act when it is in their interests to do so. As the amount of nonsense and disinformation in the scientific literature increases, I think we'll enter a new phase where trustworthiness of journal contents will start to have much higher value. If you want to be taken seriously as a peer-reviewed journal, you just cannot continue to pump out articles accompanied by superficial verbiage from "peer reviewers" that makes no real contact with the subject matter. Publishers will need to act now to clean up their editorial boards if they want to stay in business.   

Update 7th February 2025

 

I am pleased to report that I have now heard from MDPI as follows:

Following a thorough investigation of this paper and according to the recommendations of the journal's Editorial Board, we have decided to retract this publication, in line with MDPI’s retraction policy: https://www.mdpi.com/ethics#_bookmark30

Further details regarding this retraction can be found at: https://www.mdpi.com/2305-6290/9/1/20

However, as far as I can tell, no action has been taken against the editors and reviewers who were responsible for accepting the article.  The only thing I notice is that the peer reviewer comments are no longer linked to the main page for the article on the journal website (though they are still available here).  I have written to MDPI Publication Ethics to ask for clarification as to whether any action will be taken to replace the editors, and to remove the peer reviewers from their register.  Unless a robust approach is taken to removing those who admit such nonsense into the journal, readers and potential authors can have no confidence in the integrity of MDPI's editorial processes.

Update 12th February 2025

I have now received a response from MDPI Publication Ethics to my query about editors and peer reviewers.  The plan is to educate them rather than remove them.  

We are happy to provide additional details regarding our approach to this situation. 

In line with the recommendations from COPE for ethical and transparent scholarly publishing (https://publicationethics.org/guidance/guideline/principles-transparency-and-best-practice-scholarly-publishing), we are adopting an educational approach. 
This implies sending notifications to the reviewers and the editor to inform them of this retraction, clarify their responsibilities in the review process, and outline the potential consequences of a superficial review, according to our guidelines for reviewers (https://www.mdpi.com/reviewers#_bookmark11) and information for editors (https://www.mdpi.com/editors). 

Given the significant responsibility that comes with the role of peer reviewers, which directly impacts the credibility of both the journal and the broader scientific literature, we take proactive steps to offer additional guidance and resources beyond our standard guidelines. An example of this initiative can be found on our MDPI Blog: https://blog.mdpi.com/2025/01/22/reviewer-responsibilities/

Moving forward, we will closely monitor the activities of both the editor and reviewers to ensure adherence to the highest standards of academic publishing.

We remain at your disposal for any further information.

Papermillers must be feeling very cheerful right now.

Note: Comments on this blog are moderated, so there is a delay before they appear.  Anonymous or off-topic comments are not accepted. 

Retrospective look at blog highlights of 2024: What happened next?

It's always interesting for a blogger to look back to see which posts have garnered most attention. In 2024 there were three standout items, my resignation from the Royal Society, an Open Letter about editorial failings at the journal Scientific Reports, and a guest post by René Aquarius about his experiences as a reviewer for MDPI. For each of these blogposts, it's interesting to consider not just the number of hits, but also the broader impact. Blogging is a great way to let off steam, but does it actually achieve anything?

Resignation from the Royal Society 

As I said at the time of writing the post, I don't expect my resignation to change much at the Royal Society, but I was surprised at the amount of interest in the matter. The story was picked up by national media (e.g. here and here), and even got covered in France and the Netherlands. I had anticipated I might be attacked by a swarm of trolls, but there was no more than a handful of rude emails, saying the Royal Society was better off without me. What was more remarkable was the outpouring of positive reactions. My inbox was swamped with people thanking me for taking this step - not just academics, but people from all walks of life. It's customary when elected to the Royal Society for colleagues to send congratulatory emails: the number of congratulations I received for resigning was about five times as many as I had for being elected. I felt I had struck a chord with many people who were fed up with unsuitable individuals being honoured. British people in particular were outraged that the Royal Society was honouring someone who was spreading misinformation with the apparent aim of undermining our democracy.

I've seen only one serious attempt to challenge the arguments in my blogpost - by Toby Young in the Spectator. I can imagine that the comms team at the Royal Society may not have been overjoyed at his main line of argument, which was to say that there had been many Fellows of the Royal Society with rebarbative opinions who hadn't been thrown out, so why should they start now? He described me as a left-wing Torquemada, which I rather enjoyed, except that his implication was that my objections to Musk were political. Had he read my blogpost, he'd see that politics didn't come into it - it is the lack of scientific integrity that is the issue.

As a delightful twist to the story, Toby Young, who is notorious for his foul-mouthed comments on women and disabled people, has just been appointed to the House of Lords, showing that the Royal Society is not alone among the British establishment in honouring those who represent the antithesis of their core values. I can only assume that Kemi Badenoch, who was responsible for his elevation, did it to own the libs.

Open letter regarding Scientific Reports

This letter, cosigned by 22 scientific sleuths, noted that laughably awful papers were getting published in the Springer Nature journal Scientific Reports, which is generally regarded as a serious journal. It looked to us as if the journal was being infiltrated by paper mills who were using it as an outlet for fraudulent or low quality work. For this to happen, there need to be editors who are either turning a blind eye to such articles, or were actively working with a paper mill to accommodate them. A quick analysis of the editors listed on the journal website turned up 28 who had records on PubPeer indicative of involvement in research misconduct.

So what has happened since? We received a reply from Chris Graf, the Research Integrity Director at Springer Nature, thanking us for our letter and emphasising the extent to which the publisher was putting resources into tackling research fraud and paper mills. The article that had sparked off the Open Letter was retracted fairly promptly.

But when I rechecked the Editorial Board list today, 26 the 28 editors we'd listed were still in position - even though we'd given links to PubPeer entries that specified problematic behaviours of all of them. The two who are no longer listed are Ilyas Khan, and Achyut Shankar. I have no idea whether their disappearance from editorial roles at Scientific Reports has anything to do with the Open Letter. (Rule 1 of sleuthing: don't expect anyone in power to tell you if they have acted on information you provided).

And a final wrinkle to the story: a fellow sleuth told me about another editor linked to paper mills, Masoud Afrand, who had not been included in our list. He had actually been removed from the list of editors in March 2022, but then reappeared at some point in the summer of 2024 (as verified by the Wayback Machine). The Editor-in-Chief, Rafal Marszalek, explained this as a "clerical error". In email correspondence I pointed out to him that this was a pretty odd kind of clerical error, and that it did nothing to assuage growing concerns that there may be someone at a senior level at Scientific Reports who was actively working with paper millers.  I did not get a reply.

My experience as a reviewer for MDPI: guest post by René Aquarius

In this post, René describes the fate of a paper that he agreed to review for the Journal of Clinical Medicine, published by MDPI. The paper had numerous methodological flaws, including lack of a control group and discrepancies between the registered protocol and the final study, and René recommended rejection. What then followed was a bizarre series of exchanges with MDPI editors, who encouraged resubmission by the authors, then attempted to block René from re-reviewing the paper, after he'd agreed to do so. Eventually the paper was transferred to  another MDPI journal, Geriatrics, where he was again asked to review it. When he pointed out that the paper was largely unchanged from the original submission, it was withdrawn - but he then found it published a few weeks later in yet another MDPI journal.

This blogpost attracted a lot of comments, not least from people who had had similar experiences with MDPI. I have to say that these experiences, coupled with other evidence such as this and this make me very dubious about standards of peer review at MDPI. The impression is that hard-pressed editorial staff are expected to ensure peer review of submissions is achieved in 2-3 weeks, and they accordingly come to rely on anyone who will do a speedy job, without regard for quality. Indeed, high-quality reviews that raise difficult issues are problematic for them, because they will slow up the process. Accordingly these are sidelined or quietly forgotten about, in favour of minimalist 'reviews' such as those you can see here in reviews 1, 3 and 4.

Will anything change? Hopefully, René's post has helped raise awareness of the problematic aspects of peer review at MDPI journals. I doubt the publisher will do anything to change their peer review model unless the reputational damage from such revelations starts to hit their bottom line. Interestingly, in my last post of 2024, I noted that just before the Christmas break, Finland's Publication Forum (JUFO) downgraded the status of 187 MDPI journals in their index, a move that will disincentivize Finnish researchers from publishing there. However, a day later, MDPI announced that it had secured a national publishing agreement with ZB Med, which offered substantial discounts to authors from over 100 German Universities.

Epilogue

In each of these cases, we can see that a blogpost has raised the profile of an issue that is of relevance to academic scientists. Concrete impacts are harder to demonstrate - where changes happen, it's not clear if they are related to the blogpost, or just coincidental. And for most topics, it's a case of two steps forward and one step back, at best. But if I've learned one thing from my many years writing about such things, it's that you can't trust the people in positions of power to actually do the right thing unless they are prodded repeatedly and publicly (and sometimes not even then). So I hope to continue blogging through 2025, even though it may at times seem futile.

Happy New Year, and Illegitimi non carborundum.

Comments are moderated. Those that are in scope and from an identified person will be published after a delay. 

Using PubPeer to screen editors

 

2023 was the year when academic publishers started to take seriously the threat that paper mills posed to their business. Their research integrity experts have penned various articles about the scale of the problem and the need to come up with solutions (e.g., here and here).  Interested parties have joined forces in an initiative called United2Act. And yet, to outsiders, it looks as though some effective actions are being overlooked. It's hard to tell whether this is the result of timidity, poor understanding, or deliberate footdragging from those who have a strong financial conflict of interest.

As I have emphasised before, the gatekeepers to journals are editors. Therefore it is crucial that they are people of the utmost integrity and competence. The growth of mega-journals with hundreds of editors has diluted scrutiny of who gets to be an editor. This has been made worse by the bloating of journals with hundreds of special issues, each handled by "guest editors". We know that paper millers will try to bribe existing editors, and to place their own operatives as editors or guest editors, use fake reviewers, and stuff articles with irrelevant citations. Stemming this tide of corruption would be one effective way to reduce the contamination of the research literature. Here are two measures I suggest that publishers should take if they seriously want to clean up their journals.

1. Three strikes and you are out. Any editor who has accepted three or more paper milled papers should be debarred from acting as an editor, and all papers that they have been responsible for accepting should be regarded as suspect. This means retrospectively cleaning-up the field by scrutinising the suspect papers and retracting any from authors associated with paper mills, or which are characterised by features suggestive of paper mills, such as tortured phrases, citation stacking, gobbledegook content, fake reviews from reviewers suggested by authors, invalid author email domains, or co-authors who are known to be part of a paper mill ring. All of these are things that any competent editor should be able to detect. I anticipate this would lead to a large number of retractions, particularly from journals with many Special Issues. As well as these simple indicators, we are told that publishers are working hard to develop AI-based checks. They should use these not only to screen new submissions, and to retract published papers, but also to identify editors who are allowing this to happen on their watch. It also goes without saying that nobody who has co-authored a paper-milled paper should act as an editor.

2. All candidates for roles as Editor or Guest Editor at a journal should be checked against the post-publication peer review website PubPeer, and rejected if this reveals evidence of papers that have had credible criticisms suggesting of data fabrication or falsification. This is a far from perfect indicator: only a tiny fraction of authors receive PubPeer comments, and these may comment on trivial or innocent aspects of a paper. But, as I shall demonstrate, using such a criterion can reveal cases of editorial misconduct.

I will illustrate how this might work in practice, using the example of the MDPI journal Electronics. This journal came to my attention because it has indicators that all is not well with its Special Issues programme. 

First, in common with nearly all MDPI journals, Electronics has regularly broken the rule that specifies that no more than 25% of articles should be authored by a Guest Editor. As mentioned in a previous post, this is a rule that has come and gone in the MDPI guidelines, but which is clearly stated as a requirement for inclusion in the Directory of Open Access Journals (DOAJ). 13% of Special Issues in Electronics completed in 2023-4 broke this rule**. DOAJ have withdrawn some MDPI journals from their directory for this reason, and I live in hope that they will continue to implement this policy rigorously - which would entail delisting from their Directory the majority of MDPI journals. Otherwise, there is nothing to stop publishers claiming to be adhering to rigorous standards while failing to implement them, making listing in DOAJ an irrelevance.  

Even more intriguing, for around 11% of the 517 Special issues of Electronics published in 2023-4, the Guest Editor doesn't seem to have done any editing We can tell this because Special Issues are supposed to list who has acted as Academic Editor for each paper. MDPI journals vary in how rigorously they implement that rule - some journals have no record of who was the Academic Editor. But most do, and in most Special Issues, as you might expect, the Guest Editor is the Academic Editor, except for any papers where there is conflict of interest (e.g. if authors are Guest Editors or are from the same institution as the Guest Editor). Where the Guest Editor cannot act as Academic Editor, the MDPI guidelines state that this role will be taken by a member of the Editorial Board. But, guess what? Sometimes that doesn't happen. As someone with a suspicious frame of mind, and a jaundiced view of how paper mills operate, this is a potential red flag for me.

Accordingly, I decided to check PubPeer comments for individuals in three editorial roles at Electronics for the years 2023-4:

• Those listed as being in a formal editorial role on the journal website. 
• Those acting a Guest Editors 
• Those acting as Academic Editors, despite not being in the other two categories.

For Editors, a PubPeer search by name revealed 213/931 that had one or more comments. That sounds alarming, but cannot be taken at face value, because there are many innocent reasons for this result. The main one is namesakes: this is particularly common with Chinese names, which tend to be less distinctive than Western names. It is therefore important to match PubPeer comments on affiliations as well as names. Using this approach, it was depressingly easy to find instances of Editors who appeared associated with paper mills. I will mention just three, to illustrate the kind of evidence that PubPeer provides, but remember, there are many others deserving of scrutiny. 

• As well as being a section board member of Electronics, Danda B Rawat (Department of Electrical Engineering and Computer Science, Howard University, Washington, DC 20059, USA) is Editor-in-Chief of Journal of Cybersecurity and Privacy, and a section board member of two further MDPI journals: Future Internet and Sensors. A PubPeer search reveals him to be co-author of one paper with tortured phrases, and another where equations make no sense. He is listed as Editor of three MDPI Special Issues: Multimodal Technologies and Interaction: Human Computer Communications and Internet of Things Sensors: Frontiers in Mobile Multimedia Communications Journal of Cybersecurity and Privacy: Applied Cryptography.
• Aniello Castiglione  (Department of Management & Innovation Systems, University of Salerno, Italy) is Section Board Member of three journals: Electronics, Future Internet, and Journal of Cybersecurity and Privacy, and an Editorial Board member of Sustainability. PubPeer reveals he has co-authored one paper that was recently retracted because of compromised editorial processing, and that his papers are heavily cited in several other articles that appear to be used as vehicles for citation stacking. 
•  Natalia Kryvinska (Department of Information Systems, Faculty of Management, Comenius University in Bratislava, Slovakia) is a Section Board Member of Electronics. She has co-authored several articles with tortured phrases.

Turning to the 1326 Guest Editors of Special Issues, there were 500 with at least one PubPeer comment, but as before, note that in many cases name disambiguation is difficult, so this will overestimate the problem. Once again, while it may seem invidious to single out specific individuals, it seems important to show the kinds of issues that can be found among those who are put in this important gatekeeping role. 

• Heena Rathore (Department of Computer Science, Texas State Uni, San Marcos, Texas, USA) is a Guest Editor of this Special Issue. She has been lead author on a paper containing tortured phrases.
  
• Aimin Yang (College of Science, North China University of Science and Technology, Tangshan 063000) is one of five Chinese Guest Editors of a Special Issue on Intelligent Analysis and Security Calculation of Multisource Data. It is unclear what he did in this role: he was not mentioned as Academic Editor for any of the 12 papers the Special Issue. A PubPeer search reveals indicators of citation stacking (i.e. citing of irrelevant references) in his articles.  
• Joseph Bamidele (Department of Computer Science, University of Ilorin, Ilorin, Nigeria) is a Guest Editor of a Special Issue on Artificial Intelligence in Cybersecurity for Industry 4.0. A PubPeer search identifies him as lead author on an article with tortured phrases.
  

Finally, let's look at the category of Academic Editors who aren't listed as journal Editors. It's unclear how they are selected and who approves their selection. Again, among those with PubPeer comments, there's a lot to choose from. I'll focus here on three who have been exceptionally busy doing editorial work on several special issues. 

• Gwanggil Jeon (Incheon National University, Korea) has acted as Academic Editor for 18 Special Issues in Electronics. He is not on the Editorial Board of the journal, but he has been Guest Editor for two special issues in Remote Sensing, and one in Sensors.  PubPeer comments note recycled figures and irrelevant references in papers that he has co-authored, as well as a problematic Special Issue that he co-edited for Springer Nature, which led to several retractions.
• Hamid Reza Karimi (Department of Mechanical Engineering, Politecnico di Milano, Milan, Italy) has acted as Academic Editor for 12 Special Issues in Electronics. He was previously Guest Editor for two Special Issues of Electronics, one of Sensors, one of Micromachines, and one of Machines.  In 2022, he was specifically called out by the IEEE for acting "in violation of the IEEE Principles of Ethical Publishing by artificially inflating the number of citations" for several articles. 
  
• Finally, Juan M. Corchado (University of Salamanca, Spain) has acted as Academic Editor for 29 Special Issues. He was picked up by my search as he is not currently listed as being an Editor for Electronics, but that seems to be a relatively recent change: when searching for information, I found this interview from 2023. Thus his role as Academic Editor seems legitimate. Furthermore, as far as PubPeer is concerned, I found only one old comment, concerned with duplicate publication. However, he is notorious for boosting citations to his work by unorthodox means, as described in this article.* I guess we could regard his quiet disappearance from the Editorial Board as a sign that MDPI are genuinely concerned about editors who try to game the system. If so, we can only hope that they employ some experts who can do the kinds of cross-checking that I have described here at scale. If I can find nine dubious editors of one journal in a couple of hours searching, then surely the publisher, with all its financial resources, could uncover many more if they really tried.
  

Note that many of the editors featured here have quite substantial portfolios of publications. This makes me dubious about MDPI's latest strategy for improving integrity - to use an AI tool to select potential reviewers "from our internal databases with extensive publication records". That seems like an excellent way to keep paper millers in control of the system. 

Although the analysis presented here just scratches the surface of the problem, it would not have been possible without the help of sleuths who made it straightforward to extract the information I needed from the internet. My particular thanks to Pablo Gómez Barreiro, Huanzi and Sholto David.

I want to finish by thanking the sleuths who attempt to decontaminate the literature by posting comments to PubPeer. Without their efforts it would be much harder to keep track of paper millers. The problem is large and growing. Publishers are going to need to invest seriously in employing those with the expertise to tackle this issue. 

 *As I was finalising this piece, this damning update from El Pais appeared. It seems that many retractions of Corchado papers are imminent.  

 I can't keep up.... here's today's news. 

** P.S. 25th Sept 2024. DOAJ inform me that Electronics was removed from their directory in June of this year. 

*** P.P.S. 26th Sept 2024.  Guillaume Cabanac pointed me to this journal-level report on PubPeer, where he noted a high rate of Electronics papers picked up by the Problematic Paper Screener.

Now you see it, now you don't: the strange world of disappearing Special Issues at MDPI

 

There is growing awareness that Special Issues have become a menace in the world of academic publishing, because they provide a convenient way for large volumes of low quality work to be published in journals that profit from a healthy article processing charge. There has been a consequent backlash against Special Issues, with various attempts to rein them in. Here I'll describe the backstory and show how such attempts are being subverted. 

Basically, when it became normative for journals to publish open access papers in exchange for an article processing charge, many publishers saw an opportunity to grow their business by expanding the number of articles they published. There was one snag: to maintain quality standards, one requires academic editors to oversee the peer review process and decide what to publish. The solution was to recruit large numbers of temporary "guest editors", each of whom could invite authors to submit to a Special Issue in their area of expertise; this cleverly solved two problems at once: it provided a way to increase the number of submissions to the journal, and it avoided overloading regular academic editors. In addition, if an eminent person could be persuaded to act as guest editor, this would encourage researchers to submit their work to a Special Issue.

Problems soon became apparent though. Dubious individuals, including those running paper mills, seized the opportunity to volunteer as guest editors, and then proceeded to fill Special Issues with papers that were at best low quality and at worse fraudulent. As described in this blogpost, the publisher Wiley, was badly hit by fallout from the Special Issues programme, with its Hindawi brand being 'sunsetted' in 2023 . In addition, the Swiss National Science Foundation declared they would not fund APCs for articles in Special Issues, on the grounds that the increase in the number of special issues was associated with shorter processing times and lower rejection rates, suggestive of rushed and superficial peer review. Other commentators noted the reputational risks of overreliance on Special Issues.

Some publishers that had adopted the same strategy for growth looked on nervously, but basically took the line that the golden goose should be tethered rather than killed, introducing various stringent conditions around how Special Issues operated. The publisher MDPI, one of those that had massive growth in Special Issues in recent years, issued detailed guidelines.

One of these concerned guest editors publishing in their own special issues. These guidelines have undergone subtle changes over time, as evidenced by these comparisons of different versions (accessed via Wayback Machine):

JUNE 2022: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited to 20%, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue.... Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

 21 JAN 2023: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue. Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

2 JAN 2024: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited to 25%, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue. Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

3 MAY 2024: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue. Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

The May 2024 version of guidelines is nonspecific but problematic, because it is out of alignment with criteria for accreditation by the Directory of Open Access Journals (DOAJ) , who state "Papers submitted to a special issue by the guest editor(s) must be handled under an independent review process and make up no more than 25% of the issue's total". Most of MDPI's journals are listed on DOAJ, which is a signal of trustworthiness.

So, how well is MDPI doing in terms of the DOAJ criteria? I was first prompted to ask this question when writing about an article in a Special Issue of Journal of Personalized Medicine that claimed to "reverse autism symptoms". You can read my critique of that article here; one question it raised was how on earth did it ever get published? I noted that the paper was handled by a guest editor, Richard E. Frye, who had coauthored 7 of the 14 articles in the Special Issue. I subsequently found that between 2021 and 2024 he had published 30 articles in Journal of Personalized Medicine, most in three special issues where he was guest editor. I'm pleased to say that DOAJ have now delisted the journal from their Directory. But this raises the question of how well MDPI is regulating their guest editors to prevent them going rogue and using a Special Issue as a repository for papers by themselves and their cronies.

To check up on this, I took a look at Special Issues published in 2023-2024 in 28 other MDPI journals*, focusing particularly on those with implications for public health. What I found was concerning at four levels. 

1. Every single journal I looked at had Special Issues that broke the DOAJ rule of no more than 25% papers co-authored by guest editors (something DOAJ refer to as "endogeny").  Some of these can be found on PubPeer, flagged with the term "stuffedSI". 
2. A minority of Special Issues conformed to the description of a "successful Special Issue" envisaged by the MDPI guidelines: "Normally, a successful Special Issue consists of 10 or more papers, in addition to an editorial (optional) written by the Guest Editor(s)." For the journals I looked at around 60% of Special Issues had fewer than 10 articles. 
3. Quite often, the listed guest editors did not actually do any editing. One can check this by comparing the Action Editor listed for each article. Here's one example, where a different editor was needed for three of the nine papers to avoid conflict of interest, because they were co-authored by the guest editors;  but the guest editors are not listed as action editors for any of the other six papers in the special issue. 
   
4. As I did this analysis, I became aware that some articles changed status. For instance, Richard E. Frye, mentioned above, had additional articles in the Journal of Personalized Medicine that were originally part of a Special Issue that are now listed as just belonging to a Section. see https://pubpeer.com/publications/BA21B22CA3FED62B6D3F679978F591#1.This change was not transparent, but was evident when earlier versions of the website were accessed using the Wayback Machine. Some of these are flagged with the term "stealth correction" on PubPeer.

This final observation was particularly worrying, because it indicated that the publisher could change the Special Issue status of articles post-publication. The concern is that lack of oversight of guest editors has created a mechanism whereby authors can propose a Special Issue, get taken on as a guest editor, and then have papers accepted there (either their own, or from friends, which could include papermillers), after which the Special Issue status is removed. In fact, given the growing nervousness around Special Issues, removal of Special Issue status could be an advantage.

When I have discussed with colleagues these and other issues around MDPI practices, I find that credible researchers tell me that there are some excellent journals published by MDPI. It seems unfortunate that, in seeking rapid growth via the mechanism of Special Issues, the publisher has risked its reputation by giving editorial responsibility to numerous guest editors without adequate oversight, and encouraged quantity over quality. Furthermore, the lack of transparency demonstrated by the publisher covertly removing Special Issue status from articles by guest editors does not appear consistent with their stated commitment to ethical policies. 

 *The code for this analysis and a summary chart for the 28 journals can be found on Github.

My experience as a reviewer for MDPI

 

Guest post by 

René Aquarius, PhD

Department of Neurosurgery

Radboud University Medical Center, Nijmegen, The Netherlands

 

After a recent zoom-call where Dorothy and I discussed several research-related topics, she invited me to write a guest blogpost about the experience I had as a peer-reviewer for MDPI. As I think transparency in research is important, I was happy to accept this invitation.  

 

Mid November 2023 I received a request to peer-review a manuscript for a special issue on subarachnoid hemorrhage for the Journal of Clinical Medicine, published by MDPI. This blog post summarizes that process. I hope it will give some insight on the nitty-gritty of the peer-review process for MDPI.

 

I decided to review the manuscript two days after receiving the invitation and what I found was a study like many others in the field: a single-center, retrospective analysis of a clinical case series. I ended up recommending rejection of the paper two days after accepting to review. My biggest gripes were that the authors claimed that data were collected prospectively, but their protocol was registered at the very end of the period in which they included patients. In addition, I discovered some important discrepancies between protocol and the final study. Target sample size according to the protocol was 50% bigger than what was used in their study. The minimum age for patients also differed between the protocol and the manuscript. I also had problems with their statistical analysis as they used more than 20 t-tests to test variables, which creates a high probability of Type I errors. The biggest problem was the lack of a control group, which made it impossible to establish whether changes in a physiological parameter could really predict intolerance for a certain drug in a small subset of patients.

 

When filling out the reviewer form for MDPI, certain aspects struck me as peculiar. There are four options for Overall Recommendation:

• Accept in present form
• Accept after minor revision (correction to minor methodological errors and text editing)
• Reconsider after major revision (control missing in some experiments)
• Reject (article has serious flaws, additional experiments needed, research not conducted correctly)

 

Regardless of which of the last two options you select, the response is: "If we ask the authors to revise the manuscript, the revised manuscript will be sent to you for further evaluation". 

 

Although reviewer number 2 is often jokingly referred to as "the difficult one" it couldn’t be further from the truth in this case. The reviewer liked the paper and recommended accept after minor revision. So with a total of two reviews, the paper got the editorial decision of rejected, with a possibility of resubmission after extensive revisions only one day after I handed in my peer review report.

 

Revisions were quite extensive, as you will discover below, and arrived only two days after the initial rejection. I agreed to review the revised manuscript. But before I could start my review of the revision, just four days after receiving the invitation, I received a response from the editorial office that my review was no longer needed because they already had enough peer-reviewers for the manuscript. I politely ignored this request, because I wanted to know if the manuscript had improved. What happened next was quite a bit of a surprise, but not in a good way. 

 

The manuscript had indeed undergone extensive revisions. The biggest change, however, was also the biggest red flag. Without any explanation the study had lost almost 20% of its participants. An additional problem was that all the issues I had raised in my previous review report remained unaddressed. I sent my newly written feedback report the same day, exactly one week after my initial rejection.

 

When I handed in my second review report, I understood why I initially got an email that my review was not needed anymore. One peer reviewer had also rejected the manuscript and had concerns similar to mine. Two other reviewers, however, accepted the manuscript. One with minor revisions (English needed some improvement) and one in present form, so without any suggested revisions. This means that if I had followed the advice of the editorial office of MDPI, the paper would probably have been accepted in its current form. But because my vote was now also cast and the paper received two rejections, the editor couldn’t do much more than to reject the manuscript, which happened three days after I handed in my review report.  

 

Fifteen days after receiving my first invitation to review, the manuscript had already seen two full rounds of peer-review by at least four different peer-reviewers.

 

This is not where the story ends.  

 

In December, about a month later, I received an invitation to review a manuscript for the MDPI journal Geriatrics. You’ve guessed it by now: it was the same manuscript. It's reasonable to assume this was shifted internally through MDPI's transfer service, summarised in this figure.  I can only speculate as to why I was still attached to the manuscript as a peer-reviewer, but I guess somebody forgot to remove my name from it.

from: https://www.mdpi.com/authors/transfer-service

The manuscript had, again, transformed. It was now very similar to the very first version I reviewed. Almost word-for-word similar. That also meant that the number of included patients was restored to the initial number. However, the registered protocol that was previously mentioned in the methods section (which had led to some of the most difficult to refute critiques) was now completely left out. The icing on the cake was that, for a reason that was not explained, another author was added to the manuscript. There was no mention in this invitation of the previous reviews and rejections of the same manuscript.   Although one might wonder whether MDPI editors were aware of this, it would be strange if they were not, since they pride themselves on their Susy manuscript submission system where "editors can easily track concurrent and previous submissions from the same authors".

 

Because the same issues were still present in the manuscript, I rejected it for a third time on the same day I agreed to review it. In an accompanying message to the editor, I clearly articulated my problems with the manuscript and the review process.

 

The week after, I received a message that the editor had decided to withdraw the manuscript in consultation with the authors.

 

Late January 2024, the manuscript was published in the MDPI journal Medicina. I was not attached to the manuscript any more as a reviewer. There was no indication on the website of the name of the acting editor who accepted it. 

Note from Dorothy Bishop

Comments on this blog are moderated so there may be some delay before they appear, but legitimate, on-topic contributions are welcomed. We would be particularly interested to hear from anyone else who has experiences, good or bad, as a reviewer for MDPI journals.

 

Postscript by Dorothy Bishop: 19 Aug 2024 

Here's an example of a paper that was published with the reviews visible. Two were damning and one was agreeable.  https://www.mdpi.com/2079-6382/9/12/868.  Thanks to @LymeScience for drawing our attention to this, and noting the important clinical consequences when those promoting an alternative, non-evidenced treatment have a "peer-reviewed" study to refer to.