Tomatoes roaming the fields and canaries in the coalmine: another embarrassing paper for MDPI

 

Many publishers are getting nervous about infiltration by paper mills, who can torpedo a journal's reputation when they succeed in publishing papers that are obvious nonsense. In a recent Open Letter, a group of sleuths drew attention to an example in Scientific Reports, published by Springer Nature.

After the Open Letter was published, the paper that instigated our concern was promptly retracted by the journal, but as far as I can tell, not much else has changed. The point about a paper like this is that it is so blatantly bad that it cannot have been through any kind of serious editorial scrutiny or peer review. It acts as a canary in the coalmine: if gobbledegook is published in your journal, it's an indicator that you need to look very carefully at your editorial processes, and act immediately to remove editors who let this stuff in. Sadly, I haven't yet seen much evidence of that happening at Scientific Reports.

This post, however, concerns another publisher, MDPI, who have regularly featured on my blog, and not in a good way. Last month, I commented on the strange state of affairs whereby Finland had downgraded its classification of 187 MDPI journals because of evidence of "minimum time spend for editorial work and quality assessment", at the same time that German universities had secured a national publishing agreement with MDPI. The story I have to tell here may confirm Finland's judgement, and give Germany pause for thought. It concerns this article: Abbas, R., Amran, G. A., Hussain, I., & Ma, S. (2022). A Soft Computing View for the Scientific Categorization of Vegetable Supply Chain Issues. Logistics, 6(3), https://doi.org/10.3390/logistics6030039 

As with the Springer Nature example, the first indication of problems came via the Problematic Paper Screener, the excellent system that checks articles for various red flags, including "tortured phrases". These provide an indicator that a paper has probably been plagiarised but then passed through a process that substitutes synonyms for main words, with the aim of evading plagiarism detection software. So, as noted on PubPeer, in this case we have "fluffy logic" for fuzzy logic, and "unaided ML" for unsupervised machine learning. 

However, example sentences in which tortured phrases were embedded indicated a deeper problem. Most of the text is incomprehensible, and things start to get seriously weird when the authors get on to tomatoes. We are told: 

.... the third creation framework considered for the creation phase is tomatoes. This creation framework is devoted to developing homegrown creatures brought up in rural settings to create vegetables. This can bring domesticated tomatoes likewise to broad or serious frameworks. Broad frameworks include creatures wandering meadows (ordinarily under the oversight of a herder). Differently, serious tomatoes are situated in shut foundations and are outfitted with ICT innovation, which empowers creatures to be observed continuously. Inside these creation frameworks, the most run-of-the-mill issues we run over are meadow observing [75], creature government assistance [76], creature conduct following [77], and tomato creation forecast and enhancement [78,79], as displayed in Figure 3. 

According to a VSC point of view, the formal meanings of these issues are recorded beneath. 

• Field checking: This issue is connected with the exact recognizable proof of meadow inventories to separate between the most reasonable sorts for tomatoes purposes. 

• Tomato government assistance: This is centered around the example arrangement of the dehydration way of behaving in brushing creatures for investigations of creature nourishment, development, and well-being. 

• Tomato growth checking: This depends on the utilization of conduct investigations to recognize early indications of medical problems and advance early negotiation. 

A clue to the origin of this material comes from the cited references, which are about pigs and cattle. Anonymous PubPeer commenter Nerita vitiensis found that a substantial part of the text was adapted from a previous work by different authors, but with the topics of "livestock and fish" changed to "tomatoes and cruciferous vegetables". This explains the description of tomatoes as "creatures" under the oversight of a herder. 

The authors of this piece seem seriously out of their depth, as evidenced by the bland comments apparently written by Chat GPT that they provided on PubPeer. 

Now, one very good thing about MDPI is that it generally identifies the academic editor who handled a paper, and it sometimes also makes public the reviewer reports. This should mean that when a major foul-up like this occurs, it should be possible to identify and purge those responsible for accepting the work. 

The academic editors who accepted this article are Xue-Ming Yuan, who is currently soliciting papers for a special issue in the MDPI journal Mathematics, and Anrong Xue.

The MDPI website shows reports from three named reviewers. 

The first reviewer, Edyta Kardas, was concerned about the use of first-person language, and punctuation, but not apparently about statements about animated tomatoes. She reviewed 8 papers for MDPI journals in 2024.

The second reviewer, Alejandro Vega-Muñoz focused solely on the structure of the article, but apparently did not look at the content. He has edited two special issues for other MDPI journals. 

 The third reviewer, Francesco Barreca, attempted a synopsis of the article (which I could not understand) and then had just two suggestions:

"The work is well done but I have some remarks:

• The figures should be review, the dimension are variable

 • Moderate English changes are required"

The "moderate English changes" were unspecified. Barreca has a track record of editing a special issue of another MDPI journal.

 
Last week, I contacted Publication Ethics at MDPI to draw their attention to this article, noting the dereliction of duty by reviewers and editors, and suggesting that as well as retracting the paper, they should remove the editors and peer reviewers from their database. They replied to say: 

"We confirm that the Editorial Office is investigating the concerns related to this paper following the guidelines of the Committee on Publication Ethics https://publicationethics.org/ of which we are a member and our policy https://www.mdpi.com/ethics#_bookmark29.

We would like to inform you that this case is a priority for us, and we are actively working to resolve it. We will update you on the outcome of this investigation as soon as possible."

I await developments with interest. It is widely recognised that COPE guidelines are not well-suited for dealing with this kind of situation: they make the default assumption that authors should be consulted to give their perspective when criticisms are raised - a reasonable assumption in many cases, but not when there is such blatant evidence of fakery.

The most serious case of infestation of a publisher by nonsense occurred in 2022-3, when the publisher Hindawi (owned by Wiley) was targeted by paper mills who, among other things, generated numerous papers that I labelled as AI gobbledegook sandwiches. Eventually, the publisher withdrew literally thousands of papers and closed the Hindawi brand, after complaints by shareholders started impacting profits.

Like many of the sleuths who track down paper mills, I have become cynical about the commitment to research integrity that is claimed by many publishers, including MDPI. But I do believe they will act when it is in their interests to do so. As the amount of nonsense and disinformation in the scientific literature increases, I think we'll enter a new phase where trustworthiness of journal contents will start to have much higher value. If you want to be taken seriously as a peer-reviewed journal, you just cannot continue to pump out articles accompanied by superficial verbiage from "peer reviewers" that makes no real contact with the subject matter. Publishers will need to act now to clean up their editorial boards if they want to stay in business.   

Update 7th February 2025

 

I am pleased to report that I have now heard from MDPI as follows:

Following a thorough investigation of this paper and according to the recommendations of the journal's Editorial Board, we have decided to retract this publication, in line with MDPI’s retraction policy: https://www.mdpi.com/ethics#_bookmark30

Further details regarding this retraction can be found at: https://www.mdpi.com/2305-6290/9/1/20

However, as far as I can tell, no action has been taken against the editors and reviewers who were responsible for accepting the article.  The only thing I notice is that the peer reviewer comments are no longer linked to the main page for the article on the journal website (though they are still available here).  I have written to MDPI Publication Ethics to ask for clarification as to whether any action will be taken to replace the editors, and to remove the peer reviewers from their register.  Unless a robust approach is taken to removing those who admit such nonsense into the journal, readers and potential authors can have no confidence in the integrity of MDPI's editorial processes.

Update 12th February 2025

I have now received a response from MDPI Publication Ethics to my query about editors and peer reviewers.  The plan is to educate them rather than remove them.  

We are happy to provide additional details regarding our approach to this situation. 

In line with the recommendations from COPE for ethical and transparent scholarly publishing (https://publicationethics.org/guidance/guideline/principles-transparency-and-best-practice-scholarly-publishing), we are adopting an educational approach. 
This implies sending notifications to the reviewers and the editor to inform them of this retraction, clarify their responsibilities in the review process, and outline the potential consequences of a superficial review, according to our guidelines for reviewers (https://www.mdpi.com/reviewers#_bookmark11) and information for editors (https://www.mdpi.com/editors). 

Given the significant responsibility that comes with the role of peer reviewers, which directly impacts the credibility of both the journal and the broader scientific literature, we take proactive steps to offer additional guidance and resources beyond our standard guidelines. An example of this initiative can be found on our MDPI Blog: https://blog.mdpi.com/2025/01/22/reviewer-responsibilities/

Moving forward, we will closely monitor the activities of both the editor and reviewers to ensure adherence to the highest standards of academic publishing.

We remain at your disposal for any further information.

Papermillers must be feeling very cheerful right now.

Note: Comments on this blog are moderated, so there is a delay before they appear.  Anonymous or off-topic comments are not accepted. 

Finland vs. Germany: the case of MDPI

It's been an odd week for the academic publisher MDPI. On 16th December, Finland's Publication Forum (known as JUFO) announced that from January 2025 it was downgrading its classification of 271 open access journals to the lowest level, zero. By my calculations, this includes 187 journals from MDPI and 82 from Frontiers, plus 2 others. This is likely to deter Finnish researchers from submitting work to these journals, as the rating indicates they are of low quality. As explained in an article in the Times Higher Education, JUFO justified its decision on the grounds that these journals “aim to increase the number of publications with the minimum time spend for editorial work and quality assessment”. 

Then the very next day, 17th December, MDPI announced that it had secured a national publishing agreement with ZB Med, which offered substantial discounts to authors from over 100 German Universities publishing in MDPI journals. On Bluesky, this news was not greeted with the universal joy that ZB Med might have anticipated, with comments such as "an embarrassment", "shocking", and "a catastrophe".

To understand the background, it's worth reading a thread on Bluesky by Mark Hanson, one of the authors of a landmark paper entitled "The Strain on Scientific Publishing". This article showed that MDPI and Frontiers stand out from other publishers in terms of having an exponential growth in number of papers published in recent years, a massive shift to special issues as a vehicle for this increase, and a sharp drop in publication lag from 2016 to 2022.

In their annual report for 2023, MDPI described annualised publication growth of 20.4%. They stated that they have over 295,693 academic editors on editorial boards, and a median lag of 17 days from receipt of a paper to the first editorial decision and 6 weeks from submission to publication. It's not clear whether the 17 day figure includes desk rejections, but even if it does, this is remarkably fast. Of course, you could argue (and I'm sure the publishers will argue) that if you are publishing a lot more and doing it faster, you are just being efficient. However, an alternative view, and one that is apparently held by JUFO, is that this speedy processing goes hand in hand with poor editorial quality control.

The problem here is that anyone who has worked as a journal editor knows that, while a 17 day turnaround might be a good goal to aim for, it is generally not feasible. There is a limited pool of experts who can do a thorough peer review, and often one has to ask as many as 20 people to do a review in order to achieve 2 or 3 reviews. So it can take at least a couple of weeks to secure reviewers, and then it is likely to be another couple of weeks before all reviewers have completed a comprehensive review. Given these constraints, most editors would be happy if they could achieve a median time to first decision of 34 days - i.e. double that reported by MDPI. So the sheer speed of decision making - regarded by MDPI as a selling point for authors - is a red flag.

It seems that speed is achieved by adopting a rather unorthodox process of assigning peer reviewers, where the involvement of an academic editor is optional: "At least two review reports are collected for each submitted article. The academic editor can suggest reviewers during pre-check. Alternatively, MDPI editorial staff will use qualified Editorial Board members, qualified reviewers from our database, or new reviewers identified by web searches for related articles." The impression is that, in order to meet targets, editorial staff will select peer reviewers who can be relied upon to respond immediately to requests to review.

A guest post on this blog by René Aquarius supported these suspicions and further suggested that reviewers who are critical may be sidelined. After writing an initial negative review, René had promptly agreed to review a revision of a paper, but was then told his review was not needed - this is quite remarkable, given that most editors would be delighted if a competent reviewer agreed to do a timely re-review. It's worth looking not just at René's blogpost but also the comments from others, which indicate his experience is not unique.

A further issue concerns the fate of papers receiving negative reviews. René found that the paper he had rejected popped up as a new submission in another MDPI journal, and after negative reviews there, it was published in a third journal. This raises questions about MDPI's reported rejection rate of around 50%. If each of these resubmissions was counted as a new submission, the rejection rate would appear to be 66%, but given that the same paper was recycled from journal to journal before eventual acceptance, the actual rate was 0%. 

One good thing about MDPI is that it gives authors the option of making peer review open. However, analysis of published peer reviews further dents confidence in the rigour of the peer review process. A commentary in Science covering the work of Maria Ángeles Oviedo García noted how some MDPI peer reviews contained repetitive phrases that suggested they were generated by a template. They were superficial and did not engage seriously with the content of the article. In some cases the reviewer freely admitted a lack of expertise in the topic of the article, and in others, there was evidence of coercive citation (i.e., authors being told to cite the work of a specific individual, sometimes the editor).

Comments on Reddit cannot, of course, be treated as hard evidence, but they raise further questions about the peer review process at MDPI. Several commenters described having recommended rejection as a peer reviewer only to find that the paper was published without their comments. If negative peer review comments are selectively suppressed in the public record, this would be a serious breach of ethical standards by the publisher.

Lack of competent peer review and/or editorial malpractice is also suggested by the publication of papers in MDPI that fall well below the threshold of acceptable science. Of course, quality judgements are subjective, and it's common for researchers to complain "How did this get past peer review?" But in the case of MDPI journals, one finds articles that are so poor that they suggest the editor was either corrupt or asleep on the job. I have covered some examples in previous blogposts, here, and here.

The Finns are not alone in being concerned about research quality in MDPI journals. The Swiss National Science Foundation did not mention specific publishers by name, but in November 2023 they withdrew funding for articles published in special issues. Since 2023, the Directory of Open Access Journals has withdrawn 19 MDPI journals from its index for "Not adhering to best practice". Details are not provided, but these tend to be journals where guest editors have used special issues to publish a high volume of articles by themselves and close collaborators - another red flag for scientific quality. Yet another issue is citation manipulation, where editors or peer reviewers demand inclusion of specific references in a revision of an article in order to boost their own citation count. In February 2024, China released its latest Early Warning List of journals that are deemed to be "untrustworthy, predatory or not serving the Chinese research community’s interests". This included four MDPI journals listed for citation manipulation.

A final red flag about MDPI is that it seems remarkably averse to retracting articles. Hindawi publishing, which was bought by Wiley in 2021, was heavily criticised for allowing a flood of paper-milled articles to be published, but it did subsequently retract over 13,000 of them (just under 5% of 267K articles), before the closure of the brand. A search on Web of Science for documents classified as "retracted publication" or "retraction" and published by "MDPI or MDPI Ag" turned up a mere 582 retractions since 2010, which amounts to 0.04% of the 1.4 million articles listed on the database.

I've heard various arguments against JUFO's action, such as: many papers published in MDPI journals are fine; you should judge an article by its content, not where it is published; authors should be free to prefer speed over scrutiny if they wish. The reason why I support JUFO, and think ZB Med is rash to sign an agreement, is because if the peer review process is not managed by experienced and respected academic editors with specialist subject knowledge, then we need to consider the impact, not just on individual authors, but on the scientific literature as a whole. If we allow trivia, fatally flawed studies or pseudoscience to be represented as "peer-reviewed" this contaminates the research literature, with adverse consequences for everyone.

What is going on at the Journal of Psycholinguistic Research?

Last week this blog focussed on problems affecting Scientific Reports, a mega-journal published by Springer Nature. This week I look at a journal at the opposite end of the spectrum, the Journal of Psycholinguistic Research (JPR), a small, specialist journal which has published just 2187 papers since it was founded in 1971. This is fewer than Scientific Reports publishes in one year. It was brought to my attention by Anna Abalkina because it shows every sign of having been targeted by one or more Eastern European paper mills.

Now, this was really surprising to me. JPR was founded in 1971 by Robert Rieber, whose obituaries in the New York Times  and the American Psychologist confirm he had a distinguished career (though both misnamed JPR!). The Advisory and Editorial boards of the journal are peppered with names of famous linguists and psychologists, starting with Noam Chomsky. So there is a sense that if this can happen to JPR, no journal is safe.

Coincidentally, last week Anna and I submitted revisions for a commentary on paper mills coauthored with Pawel Matusz. (You can read the preprint here). Pawel is editor of the journal Mind, Brain & Education (MBE), which experienced an attack by the Tanu.pro paper mill involving papers published in 2022-3. In the commentary, we discussed characteristics of the paper mill, which are rather distinctive and quite different from what is seen in basic biomedical or physical sciences. A striking feature is that the IMRaD structure (Introduction, Methods, Results and Discussion) is used, but in a clueless fashion, with these headings being inserted in what is otherwise a rambling and discursive piece of text, that typically has little or no empirical content. Insofar as there are any methods described, they don't occur in the methods section, and they are too vague for the research to be replicable.

Reading these papers rapidly turns my brain to mush, but in the interest of public service I did wade through five of them and left comments on Pubpeer:  

Yeleussizkyzy, M., Zhiyenbayeva, N., Ushatikova, I. et al. E-Learning and Flipped Classroom in Inclusive Education: The Case of Students with the Psychopathology of Language and Cognition. J Psycholinguist Res 52, 2721–2742 (2023). https://doi.org/10.1007/s10936-023-10015-y  

https://pubpeer.com/publications/7C9B4538A07EE8777082D63B2944B9 

Snezhko, Z., Yersultanova, G., Spichak, V. et al. Effects of Bilingualism on Students’ Linguistic Education: Specifics of Teaching Phonetics and Lexicology. J Psycholinguist Res 52, 2693–2720 (2023). https://doi.org/10.1007/s10936-023-10016-x

https://pubpeer.com/publications/09F57870A7D20BF5F77C53F414449F 

Nurakenova, A., Nagymzhanova, K. A Study of Psychological Features Related to Creative Thinking Styles of University Students. J Psycholinguist Res 53, 1 (2024). https://doi.org/10.1007/s10936-024-10042-3

https://pubpeer.com/publications/A1FB870AD77F1D07D34B9BF9AEEFF3 

Auganbayeva, M., Turguntayeva, G., Anafinova, M. et al.Linguacultural and Cognitive Peculiarities of Linguistic Universals. J Psycholinguist Res 53, 3 (2024). https://doi.org/10.1007/s10936-024-10050-3

https://pubpeer.com/publications/06DD63C96B4D7C74DAC8AB72F18A34 

Shalkarbek, A., Kalybayeva, K., Shaharman, G. et al. Cognitive Linguistic Analysis of Hyperbole-based Phraseological Expressions in Kazakh and English Languages. J Psycholinguist Res 53, 4 (2024). https://doi.org/10.1007/s10936-024-10052-1

https://pubpeer.com/publications/D52E26EE9DE5F83EA07B6B9917A645#1

My experience with the current batch of papers suggests that a relatively quick way of screening a submitted paper would be to look at the Methods section. This should contain an account of methods that would indicate what was done and how, at a level of detail sufficient for others to replicate the work. Obviously, this is not appropriate for theoretical papers, but for those purporting to report empirical work, it would work well, at least for the papers I looked at in JPR.   

All of these papers have authors from Kazakhstan, sometimes with co-authors from the Russian Federation. This led me to look at the geographic distribution of authors in the journal over time. The top countries represented by JPR authors in 2020 onwards are China (113), United States (68), Iran (52), Germany (28), Saudi Arabia (22) and Kazakhstan (19). However, these composite numbers mask striking trends. All the Kazakhstan authored papers are in 2023-2024. There's also a notable fall-off in papers authored by USA-based authors in the same time period, with only 11 cases in total. This is quite remarkable for a journal that had a striking USA dominance in authors up until around 2015, as shown in the attached figure (screenshot from Dimensions.ai).

 

Number of papers in JPR from five top countries: 2005-2024

Exported: October 20, 2024

Criteria: Source Title is Journal of Psycholinguistic Research.

© 2024 Digital Science and Research Solutions Inc. All rights reserved. 

Non-commercial redistribution / external re-use of this work is permitted subject to appropriate acknowledgement. 

This work is sourced from Dimensions® at www.dimensions.ai.

Whenever a paper mill infestation is discovered, it raises the question of how it happened. Surely the whole purpose of peer review is to prevent low quality or fraudulent material entering the literature? In other journals where this has happened it has been found that the peer review process was compromised, with fake peer reviewers being used. Even so, one would have hoped that an editor would scrutinize papers and realise something was amiss. As mentioned in the previous blogpost, it would be much easier to track down the ways in which fraudulent papers get into mainstream journals if the journal reported information about the editor who handled the paper, and published open peer review.

Whatever the explanation, it is saddening to see a fine journal brought so low. In 2021, at the 50th anniversary of the founding of the journal, the current editor, Rafael Art. Javier, wrote a tribute to his predecessor, Robert Rieber:

"His expectation, as stated in that first issue, was that manuscripts accepted 'must add to knowledge in some way, whether they are in the form of experimental reports, review papers, or theoretical papers...and studies with negative results,' provided that they are of sufficiently high quality to make an original contribution."

Let us hope that the scourge of paper mills can be banished from the journal to allow it to be restored to the status it once had, and for Robert Rieber's words to once more be applicable.

 

An open letter regarding Scientific Reports

16th October 2024 

to: Mr Chris Graf

Research Integrity Director, Springer Nature and Chair Elect of the World Conference on Research Integrity Foundation Governing Board.

 

Dear Mr Graf,

We are a group of sleuths and forensic meta-scientists who are concerned that Springer Nature is failing in its duty to protect the scientific literature from fraudulent and low quality work. We are aware that, as noted in the 2023 Annual Report, you are committed to maintaining research integrity. We agree with the statement: “To solve the world’s biggest challenges, we all need research that’s reliable, trustworthy and can be built on by scientists and innovators. As a leading global research publisher, we have a pivotal role to play.” It is encouraging to hear that the Springer Nature research integrity group doubled in size in 2023. Nevertheless, we have a growing sense that all is not well concerning the mega journal Scientific Reports.

Some of the work that has been published is so seriously flawed that it is not credible that it underwent any meaningful form of peer review. In other cases, when we have reported flawed papers to the editor or integrity team, the response has been inadequate. A striking example cropped up last week when a “corrected” version of an article was published in Scientific Reports. This article had been flagged up by Guillaume Cabanac as containing numerous “tortured phrases” that are indicative of fraudulent authors attempting to bypass plagiarism checks; the authors were allowed to “correct” the article by merely removing some (not all) of the tortured phrases. This led some of us to look more closely at the article. As is evident from comments on PubPeer, it turned out to be a kind of case study of all the red flags for fraud that we look for. As well as (still uncorrected) tortured phrases, it contained irrelevant content, irrelevant citations, meaningless gibberish, a nonsensical figure, and material recycled from other publications.

This is perhaps the most flagrant example, but we argue that it indicates problems with your editorial processes that are not going to be fixed by AI. The only ways an article like this can have been published are either through editorial negligence or outright malpractice. For it to be negligence would require a remarkable degree of professional incompetence from a handling editor. The possibility of malpractice, would mean there is a corrupt handling editor who bypasses the peer review process entirely or willingly appoints corrupt peer reviewers to approve the manuscript. We appreciate that some papers that we and others have reported have been retracted, but in other cases blatantly fraudulent papers can take years to be retracted or to receive any appropriate editorial action.

We have some specific suggestions for actions that Springer Nature could take to address these issues.

1. Employ a task force of people with the necessary expertise to carry out an urgent audit of all editors of Scientific Reports. We have looked at the editors on your website, and it is clear that this is an enormous task, given that there are over 13,000 of them, and they are not listed with disambiguating information such as Orcid IDs. Even so, in a few hours, by cross-checking this list against PubPeer, it was possible to identify the 28 cases listed below, covering a range of disciplines, and all, in our view, with pretty clear-cut evidence of problems. Four are members of the Editorial Board. We stress, this is just the low-hanging fruit which was fairly easy to detect.
2. The list of problematic articles appended below or tabulated on the Problematic Paper Screener might provide an alternative route to identify editors who should never have been given a gatekeeping role in academic publishing. As well as checking the papers we list below, we recommend that all other articles accepted by the same editors should be scrutinised.
3. Detection of problematic articles and editors could be helped by requiring open peer review for all journals, and ensuring that the name and Orcid ID of the handling editor is included with the published meta-data for all articles.

We hope these suggestions will be helpful in ensuring that research published in Scientific Reports is reliable and trustworthy.

Yours sincerely

Dorothy Bishop

Guillaume Cabanac

François-Xavier Coudert

René Aquarius

Nick Wise

Lonni Besançon

Simon A.J. Kimber

Anna Abalkina

Rickard Carlsson

Samuel J Westwood

Patricia Murray

Nicholas J. L. Brown

Smut Clyde

Leonid Schneider

Ian Hussey

Tu Duong

Gustav Nilsonne

Jamie Cummins

Alexander Magazinov

Elisabeth Bik

Mu Yang

Corrado Viotti

Sholto David

 

Appendices

1. Some examples of editors with concerning PubPeer entries

Editorial board Ghulam Md Ashraf

Editorial board Eun Bo Shim

Editorial board Ajay Goel

Editorial board Rasoul Kowsar

AGEING Vittorio Calabrese

AGRICULTURE Sudip Mitra

ANALYTICAL CHEMISTRY Syed Ghulam Musharraf

CELL BIOLOGY Gabriella Dobrowolny

CHEMICAL ENGINEERING Enas Taha Sayed

CIVIL ENGINEERING Manoj Khandelwal

CLINICAL ONCOLOGY Marcello Maggiolini

COMPUTATIONAL SCIENCE Praveen Kumar Reddy Maddikunta

DRUG DISCOVERY Salvatore Cuzzocrea

ENDOCRINOLOGY Sihem Boudina

ENERGY Karthik Rajendran

ENVIRONMENTAL ENGINEERING Rama Rao Karri

ENVIRONMENTAL SCIENCE Mayeen Uddin Khandaker

FOOD Herbert Marini

GASTROENTEROLOGY AND HEPATOLOGY Sharon DeMorrow

IMMUNOLOGY Marcin Wysoczynski

INFECTIOUS DISEASES Fatah Kashanchi

MATHEMATICAL PHYSICS Ilyas Khan

MICROBIOLOGY Massimiliano Galdiero

NETWORKS AND COMPLEX SYSTEMS Achyut Shankar

NEUROLOGY Yvan Torrente

RESPIRATORY MEDICINE Soni Savai Pullamsetti

STRUCTURAL AND MOLECULAR BIOLOGY Stefania Galdiero

SURGERY Alessandro Gonfiotti

2. Some examples of problematic papers

https://pubpeer.com/publications/8455A9FB49E592CD51CD2D07247E10#7

https://pubpeer.com/publications/46DB67305BCBC074E8D7BF91CEB86D

https://pubpeer.com/publications/F62D726F7B33CC905114E4342234AE

https://pubpeer.com/publications/FA9961B4B9CA7257D9C506BD4BDEDF

https://pubpeer.com/publications/047DBE5B115545CAB5C5C482FDC9C2

https://pubpeer.com/publications/FD5E495851DDB3A2B910D8AC2788A7#1

https://pubpeer.com/publications/8FAE406379730D89C30FBA44022B7B#2

https://pubpeer.com/publications/42901FD2901EC917E3EE54B8DBD749#4 (authors claim a correction is underway, but none published for 2 years)

https://pubpeer.com/publications/01FE09F1127DF0598985987677A101 (part of a list of many flagged papers from this author group. Corrected rather than retracted)

https://pubpeer.com/publications/931607614D960AFFC95DB8B0839A74

https://pubpeer.com/publications/29480A863001CFE6B2AFC8B36C9A25

https://pubpeer.com/publications/69EDBAECD50F31B051ECECCD1DF346 (notified on 31-3-2023 about this paper, no action so far)

https://pubpeer.com/publications/F8A1AD2B165888A06C18B28C860E7B. EiC contacted Nov. 22 with authorship concerns, responded that he would investigate. No action taken so far.

https://pubpeer.com/publications/286F83F9553D29F82CD4281309A1E4. Has had EoC for authorship irregularities since July 22, no action taken since.

https://pubpeer.com/publications/7913EB1475AD969B06A3D8C2AD8D00

https://pubpeer.com/publications/7A2615D16AA5BBF22F5039AAF0F8A7

https://pubpeer.com/publications/9CDCC9C72FF9EB84E20E2A37A518BC

https://pubpeer.com/publications/8A2CF2E1EBFD3ADCD835ADB91DDFE8

https://pubpeer.com/publications/5BEDDDA9CF92B9CDDD2AB1AA796271 (blatantly nonsensical paper reported to publisher in June 2024; no action as yet)

https://pubpeer.com/publications/37B87CAC48DE4BC98AD40E00330143 (various corrections since 2022, and in Feb 2023 readers were told “conclusions of this article are being considered by the Editors. A further editorial response will follow the resolution of these issues”. 19 Months later we are still waiting.)

3. Some examples of journal-level reports posted on PubPeer

Scientific Reports

other Springer Nature journals:

Chemosphere

Environmental Science and Pollution Research

Journal of Ambient Intelligence and Humanized Computing

Multimedia Tools and Applications

Optical and Quantum Electronics

Wireless Personal Communications

We need to talk about editors

Editoris spivia

The role of journal editor is powerful: you decide what is accepted or rejected for publication. Given that publications count as an academic currency – indeed in some institutions they are literally fungible – a key requirement for editors is that they are people of the utmost integrity. Unfortunately, there are few mechanisms in place to ensure editors are honest – and indeed there is mounting evidence that many are not. I argue here that we can no longer take editorial honesty for granted, and systems need to change to weed out dodgy editors if academic publishing is to survive as a useful way of advancing science. In particular, the phenomenon of paper mills has shone a spotlight on editorial malpractice.

Questionable editorial practices

Back in 2010, I described a taxonomy of journal editors based on my own experience as an author over the years. Some were negligent, others were lordly, and others were paragons – the kind of editor we all want, who is motivated solely by a desire for academic excellence, who uses fair criteria to select which papers are published, who aims to help an author improve their work, and provides feedback in a timely and considerate fashion. My categorisation omitted another variety of editor that I have sadly become acquainted with in the intervening years: the spiv. The spiv has limited interest in academic excellence: he or she sees the role of editor as an opportunity for self-advancement. This usually involves promoting the careers of self or friends by facilitating publication of their papers, often with minimal reviewing, and in some cases may go as far as working hand in glove with paper mills to receive financial rewards for placing fraudulent papers.

When I first discovered a publication ring that involved journal editors scratching one another’s backs, in the form of rapid publication of each other’s papers, I assumed this was a rare phenomenon. After I blogged about this, one of the central editors was replaced, but others remained in post. 

I subsequently found journals where the editor-in-chief authored an unusually high percentage of the articles published in the journal. I drew these to the attention of integrity advisors of the publishers that were involved, but did not get the impression that they regarded this as particularly problematic or were going to take any action about it. Interestingly, there was one editor, George Marcoulides, who featured twice in a list of editors who authored at least 15 articles in their own journal over a five year period. Further evidence that he equates his editorial role with omnipotence came when his name cropped up in connection with a scandal where a reviewer, Fiona Fidler, complained after she found her positive report on a paper had been modified by the editor to justify rejecting the paper: see this Twitter thread for details. It appears that the publishers regard this as acceptable: Marcoulides is still editor-in-chief at the Sage journal Educational and Psychological Measurement, and at Taylor and Francis’ Structural Equation Modeling, though his rate of publishing in both journals has declined since 2019; maybe someone had a word with him to explain that publishing most of your papers in a journal you edit is not a good look.

Scanff et al (2021) did a much bigger investigation of what they termed “self-promotion journals” - those that seemed to be treated as the personal fiefdom of editors, who would use the journal as an outlet for their own work. This followed on from a study by Locher et al (2021), which found editors who were ready to accept papers by a favoured group of colleagues with relatively little scrutiny. This had serious consequences when low-quality studies relating to the Covid-19 pandemic appeared in the literature and subsequently influenced clinical decisions. Editorial laxness appears in this case to have done real harm to public health.

So, it's doubtful that all editors are paragons. And this is hardly surprising: doing a good job as editor is hard and often thankless work. On the positive side, an editor may obtain kudos for being granted an influential academic role, but often there is little or no financial reimbursement for the many hours that must be dedicated to reading and evaluating papers, assigning reviewers, and dealing with fallout from authors who react poorly to having their papers rejected. Even if an editor starts off well, they may over time start to think “What’s in this for me?” and decide to exploit the opportunities for self-advancement offered by the position. The problem is that there seems little pressure to keep them on the straight and narrow; it's like when a police chief is corrupt. Nobody is there to hold them to account. 

Paper mills

Many people are shocked when they read about the phenomenon of academic paper mills – defined in a recent report by the Committee on Publication Ethics (COPE) and the Association of Scientific, Tehcnical and Medical Publishers (STM) as “the process by which manufactured manuscripts are submitted to a journal for a fee on behalf of researchers with the purpose of providing an easy publication for them, or to offer authorship for sale.” The report stated that “the submission of suspected fake research papers, also often associated with fake authorship, is growing and threatens to overwhelm the editorial processes of a significant number of journals.” It concluded with a raft of recommendations to tackle the problem from different fronts: changing the incentives adopted by institutions, investment in tools to detect paper mill publications, education of editors and reviewers to make them aware of paper mills, introduction of protocols to impede paper mills succeeding, and speeding up the process of retraction by publishers.

However, no attention was given to the possibility that journal editors may contribute to the problem: there is talk of “educating” them to be more aware of paper mills, but this is not going to be effective if the editor is complicit with the paper mill, or so disengaged from editing as to not care about them. 

It’s important to realise that not all paper mill papers are the same. Many generate outputs that look plausible. As Byrne and Labbé (2017) noted, in biomedical genetic studies, fake papers are generated from a template that is based on a legitimate paper, and just vary in terms of the specific genetic sequence and/or phenotype that is studied. There are so many genetic sequences and phenotypes, that the number of possible combinations of these is immense. In such cases, a diligent editor may get tricked into accepting a fake paper, because the signs of fakery are not obvious and aren’t detected by reviewers. But at the other extreme, some products of paper mills are clearly fabricated. The most striking examples are those that contain what Guillaume Cabanac and colleagues term “tortured phrases”. These appear to be generated by taking segments of genuine articles and running them through an AI app that will use a thesaurus to alter words, with the goal of evading plagiarism detection software. In other cases, the starting point appears to be text from an essay mill. The results are often bizarre and so incomprehensible that one only needs read a few sentences to know that something is very wrong. Here’s an example from Elsevier’s International Journal of Biological Macromolecules, which those without access can pay $31.50 for (see analysis on Pubpeer, here).

"Wound recuperating camwood a chance to be postponed due to the antibacterial reliance of microorganisms concerning illustration an outcome about the infection, wounds are unable to mend appropriately, furthermore, take off disfiguring scares [150]. Chitin and its derivatives go about as simulated skin matrixes that are skilled should push a fast dermal redesign after constantly utilized for blaze treatments, chitosan may be wanton toward endogenous enzymes this may be a fundamental preference as evacuating those wound dressing camwood foundation trauma of the wounds and harm [151]. Chitin and its derivatives would make a perfect gas dressing. Likewise, they dampen the wound interface, are penetrability will oxygen, furthermore, permit vaporous exchange, go about as a boundary with microorganisms, and are fit about eliminating abundance secretions"

And here’s the start of an Abstract from a Springer Nature collection called Modern Approaches in Machine Learning and Cognitive Science (see here for some of the tortured phrases that led to detection of this article). The article can be yours for £19.95:

“Asthma disease are the scatters, gives that influence the lungs, the organs that let us to inhale and it’s the principal visit disease overall particularly in India. During this work, the matter of lung maladies simply like the trouble experienced while arranging the sickness in radiography are frequently illuminated. There are various procedures found in writing for recognition of asthma infection identification. A few agents have contributed their realities for Asthma illness expectation. The need for distinguishing asthma illness at a beginning period is very fundamental and is an exuberant research territory inside the field of clinical picture preparing. For this, we’ve survey numerous relapse models, k-implies bunching, various leveled calculation, characterizations and profound learning methods to search out best classifier for lung illness identification. These papers generally settlement about winning carcinoma discovery methods that are reachable inside the writing.”

These examples are so peculiar that even a layperson could detect the problem. In more technical fields, the fake paper may look superficially normal, but is easy to spot by anyone who knows the area, and who recognises that the term “signal to noise” does not mean “flag to commotion”, or that while there is such a thing as a “Swiss albino mouse” there is no such thing as a “Swiss pale-skinned person mouse”. These errors are not explicable as failures of translation by someone who does not speak good English. They would be detected by any reviewer with expertise in the field. Another characteristic of paper mill outputs, featured in this recent blogpost, are fake papers that combine tables and figures from different publications in nonsensical contexts.

Sleuths who are interested in unmasking paper mills have developed automated methods for identifying such papers, and the number is both depressing and astounding. As we have seen, though some of these outputs appear in obscure sources, many crop up in journals or edited collections that are handled by the big scientific publishing houses, such as Springer Nature, Elsevier and Wiley. When sleuths find these cases, they report the problems on the website PubPeer, and this typically raises an incredulous response as to how on earth did this material get published. It’s a very good question, and the answer has to be that somehow an editor let this material through. As explained in the COPE&STM report, sometimes a nefarious individual from a paper mill persuades a journal to publish a “special issue” and the unwitting journal is then hijacked and turned into a vehicle for publishing fraudulent work. If the special issue editor poses as a reputable scientist, using a fake email address that looks similar to the real thing, this can be hard to spot.

But in other cases, we see clearcut instances of paper mill outputs that have apparently been approved by a regular journal editor. In a recent preprint, Anna Abalkina and I describe finding putative paper mill outputs in a well-established Wiley journal, the Journal of Community Psychology. Anna identified six papers in the journal in the course of a much larger investigation of papers that came from faked email addresses. For five of them the peer review and editorial correspondence was available on Publons. The papers,  from addresses in Russia or Kazakhstan, were of very low quality and frequently opaque. I had to read and re-read to work out what the paper was about, and still ended up uncertain. The reviewers, however, suggested only minor corrections. They used remarkably similar language to one another, giving the impression that the peer review process had been compromised. Yet the Editor-in-Chief, Michael B. Blank, accepted the papers after minor revisions, with a letter concluding: “Thank you for your fine contribution”. 

There are two hypotheses to consider when a journal publishes incomprehensible or trivial material: either the editor was not doing their job of scrutinising material in the journal, or they were in cahoots with a paper mill. I wondered whether the editor was what I have previously termed an automaton – one who just delegates all the work to a secretary. After all, authors are asked to recommend reviewers, so all that is needed is for someone to send out automated requests to review, and then keep going until there are sufficient recommendations to either accept or reject. If that were the case, then maybe the journal would accept a paper by us. Accordingly, we submitted our manuscript about paper mills to the Journal of Community Psychology. But it was desk rejected by the Editor in Chief with a terse comment: “This a weak paper based on a cursory review of six publications”. So we can reject hypothesis 1 – that the editor is an automaton. But that leaves hypothesis 2 – that the editor does read papers submitted to his journal, and had accepted the previous paper mill outputs in full knowledge of their content. This raises more questions than it answers. In particular, why would he risk his personal reputation and that of his journal by behaving that way? But perhaps rather than dwelling on that question, we should think positively about how journals might protect themselves in future from attacks by paper mills.

A call for action

My proposal is that, in addition to the useful suggestions from the COPE&STM report, we need additional steps to ensure that those with editorial responsibility are legitimate and are doing their job. Here are some preliminary suggestions:

1. Appointment to the post of editor should be made in open competition among academics who meet specified criteria.
2. It should be transparent who is responsible for final sign-off for each article that is published in the journal.
3. Journals where a single editor makes the bulk of editorial decisions should be discouraged. (N.B. I looked at the 20 most recent papers in Journal of Community Psychology that featured on Publons and all had been processed by Michael B. Blank).
   
4. There should be an editorial board consisting of reputable people from a wide range of institutional backgrounds, who share the editorial load, and meet regularly to consider how the journal is progressing and to discuss journal business.
5.  Editors should be warned about the dangers of special issues and should not delegate responsibility for signing off on any papers appearing in a special issue.
6. Editors should be required to follow COPE guidelines about publishing in their own journal, and publishers should scrutinise the journal annually to check whether the recommended procedures were followed.
7. Any editor who allows gibberish to be published in their journal should be relieved of their editorial position immediately.
   

Many journals run by academic societies already adopt procedures similar to these. Particular problems arise when publishers start up new journals to fill a perceived gap in the market, and there is no oversight by academics with expertise in the area. The COPE&STM report has illustrated how dangerous that can be – both for scientific progress and for the reputation of publishers.

Of course, when one looks at this list of requirements, one may start to wonder why anyone would want to be an editor. Typically there is little financial renumeration, and the work is often done in a person’s “spare time”. So maybe we need to rethink how that works, so that paragons with a genuine ability and interest in editing are rewarded more adequately for the important work they do.

P.S. Comment moderation is enabled for this blog to prevent it being overwhelmed by spam, but I welcome comments, and will check for these in the weeks following the post, and admit those that are on topic. 

 

Comment by Jennifer Byrne, 9th Sept 2022 

(this comment by email, as Blogger seems to eat comments by Jennifer for some reason, while letting through weird spammy things!).

This is a fantastic list of suggestions to improve the important contributions of journal editors. I would add that journal editors should be appointed for defined time periods, and their contributions regularly reviewed. If for any reason it becomes apparent that an editor is not in a position to actively contribute to the journal, they should be asked to step aside. In my experience, editorial boards can include numerous inactive editors. These can provide the appearance of a large, active and diverse editorial board, when in practice, the editorial work may be conducted by a much smaller group, or even one person. Journals cannot be run successfully without a strong editorial team, but such teams require time and resources to establish and maintain.

Universities vs Elsevier: who has the upper hand?

 

The academic publisher Elsevier is currently negotiating a deal with UK universities. In Oxford, as in other universities, there have been extensive discussions about the proposed deal; the goals are to reduce costs to sustainable levels and to provide full and immediate open access to UK research. I have a nasty feeling that we could end up in the situation of those at the COP summit: with a deal that where the publishers feel they are giving away a huge amount, while the consumers are still unsatisfied. 

In the print era, publishers already had a large profit margin.  I was a journal editor when electronic publishing first came in, and I remember discussions with the publisher - who was clearly very nervous about how this might damage their bottom line. With some clever business practices, such as bundling, they managed to keep going, making more rather than less money.

Other new developments - the advance of metrics and requirements for open access publishing - also pose challenges to them, but their response, like any savvy business, is to take control of these new developments and find a way to profit from them as well.

So should we go along with this? The discussions have largely centred around money, and this is a real concern. Publishers charge massively inflated subscription charges, taking as profit money that our libraries could otherwise put to good use. But the problems go way beyond that, including blocking open access to authors' own work, and giving poor quality and even fraudulent material a veneer of respectability. The publisher currently has considerable power over what gets published but takes little responsibility when things go wrong.

But we also have power, and I think it's time we started to wield it. I have two proposals: one radical and the other less so, both of which go beyond what is currently being considered in the JISC discussions.

The radical option: move publishing in-house  

In an ideal world, we would not have any deals with for-profit publishers. Universities would take control of academic publishing. A model which I have found works well is the Wellcome Trust journal Wellcome Open Research (WOR). I served on its Editorial Advisory Board in the first years of its operation and I have published eight papers in WOR. 

Quality control is maintained by requiring that work published in WOR has been Wellcome-Trust funded. The platform is paid for by the Wellcome Trust, and all material is Open Access, with no charge to authors.

Universities could use this as a model for developing their own platforms for publishing work of their researchers. As with Wellcome Trust, researchers would be encouraged rather than required to publish there. Setting up and maintaining the platform would cost money, but this might be covered by savings from stepping back from deals with big publishers. This is not a perfect solution, of course; we would still want access to past material published by traditional publishers. But I feel this option should be given serious consideration.

The less radical option. An agreement with Elsevier that reflects what we want and need 

If we are to retain a working relationship with Elsevier, then we need to ensure that we are getting value for money, and that they are delivering what we want.

One way in which they fail to do this is through the copyright agreements that they require authors to sign. I agree with Sally Rumsey that any publisher that does not allow an author to retain rights to their own work is problematic. Furthermore, there is concern that even if Elsevier were to say they agreed to author rights retention, they would continue to adopt practices that would mislead authors into signing away their rights.

Another issue is how publishers deal with matters related to misbehaviour by editors and authors. We have a number of serious problems with the academic publishing system that have been developing for years, and have been thrown into stark relief by the pandemic. Publications in journals are a kind of academic currency, with potentially huge rewards in terms of promotion, pay and tenure of academic staff. Unfortunately, this means that some people will go to great lengths to procure publications, and this can include fraudulent means. A reputable publisher should recognise and indeed anticipate such problems and take steps to deal with them. Elsevier has been notably poor in doing so. I will give a handful of examples.

Clearly erroneous/fraudulent work remains in the literature In recent years, a massive problem has been discovered with so-called paper mills - a form of industrialised cheating that involves generation of literally thousands of fake papers.  This phenomenon was discussed at a meeting this summer, the Computational Research Integrity Conference (CRI-Conf), which brought together some of the scientists who uncovered paper mills and representatives of publishers (including Elsevier) and university integrity officers. A repeated complaint of the 'data sleuths' (scientists who uncovered the paper mills) was that when the problems were drawn to the attention of the editors or publishers, typically nothing was done.  Some of these papers are highly cited and have medical implications - see, for instance work by Jennifer Byrne and Cyril Labbé on cancer biology, or by Elisabeth Bik on image manipulations.

To its credit, Elsevier was recently cited as acting promptly when a cache of nonsense papers in special issues was uncovered, but one still has to ask how these papers, some of which were complete gibberish, originally got through the supposedly rigorous peer-review process. For another recent example, see here.

Part of the problem seems to be that there is very little oversight of editors.  I first became aware of Elsevier's lax attitude to research integrity when I discovered a ring of editors. There was overwhelming evidence that a set of editors were publishing one another's papers in their journals, with publication lags so brief as to indicate there had not been peer review. Emails to Elsevier received no reply, and so I started reporting the events on my blog. This created such publicity that Elsevier then posted a reply, but initially they denied that any papers had been published without peer review. Eventually, as the evidence mounted up, they were forced to acknowledge the problem, and claimed to be investigating it, but at no point did they communicate with me, and their investigation went on for years and has never led to any consequences. One of the offending editors, Johnny Matson, quietly disappeared without comment and is now editing another journal for another publisher. The unreviewed papers, many of which are concerned with intervention for children with developmental disorders, remain in the literature. 

More recently I did a bibliometric analysis of editors of psychology journals who publish prolifically in their own journals, and broke down the results by publisher. Elsevier had the highest number of editors-in-chief in this category,

A related case in another field is that of Didier Raoult, who was senior author on a paper advocating hydroxychloroquine as a treatment for Covid-19 that appeared in 2020 in an Elsevier journal for which he was Editor-in-Chief. It has subsequently been shown to be riddled with methodological errors, but, although Elsevier claimed to be investigating it in April 2020, it has not been retracted.

At the CRI-Conf publishers said that they were doing their best, but the problem of fake and fraudulent practices was so huge they could not be expected to tackle all of it. Yet we know publishers make enormous profits, so why aren't they putting more effort into checking the integrity of the work they publish - and dealing promptly with the problems that are drawn to their attention? Elsevier is not the only large publisher to be affected, but if it wants to be seen as a leader in the field, it should recognise that integrity of published material is of supreme importance - especially in clinical fields where people's health and wellbeing can be affected (as in the cases of Raoult and Matson noted above).

If UK Universities do strike a deal with Elsevier, then they should take the opportunity to include in any future contract conditions requirements that: 

(a) Elsevier allows authors the retention of the rights to their publications and  

(b) it provides annually an open report on the number of integrity issues that have been raised about its publications and how these have been dealt with. 

This would go some way toward ensuring that we get value for money, and are not simply adding to the profits of a company whose underlying ethos is not aligned with promoting scientific quality.

Universities have historically been extremely meek in accepting the terms and conditions imposed by big publishers, who have in turn treated academics as a cash cow. We on the one hand produce for free the product that they sell, and then pay to publish and read it. They cannot survive without the research created in Universities: we should start recognising that we need not bow down to their demands.

DISCLAIMER: I am writing in a personal capacity and not as a representative of the University of Oxford.

14/11/21

P.S. Some further examples of integrity problems with Elsevier have been pointed out to me on Twitter:

Geoff Frampton (@Geoff_Frampton) noted 'Elsevier had the largest market share among the publishers of retracted Covid-19 articles in our study' 

Patricia Murray (@PMurray_65) tweeted: Elsevier have failed to retract this 2008 Lancet paper that is known to be fraudulent and poses a risk to patients: https://thelancet.com/journals/lancet/article/PIIS0140-6736(08)61598-6/references… The paper stated the so-called 'bioengineered' airway had "a normal appearance and mechanical properties at 4 months." 

It is particularly shocking that the publisher is still making this article available (at a cost of $31) given that the author had a prison sentence for faked research, and conducted unethical experiments on seriously ill patients: see this report. 

PPS - it gets worse. There have been strenuous attempts to get the journal to retract the paper, to no avail. This has made the national news in the UK.

 

Comments on this blog are welcome, but may only appear after a delay as they are moderated to avoid spam.