The dangers of using bibliometrics with polluted data

 

This week I attended a Webinar organised by Clarivate on the topic of “Eugene Garfield Centenary Celebration: Past, Present and Future of Scientometrics”. This covered the early history of the late Eugene Garfield’s work as well as current developments and future trends. The historical sessions were fascinating, describing the remarkable innovations that Garfield made in his quest to understand the body of scientific information as a network. He realised that similarities between articles could be identified by shared citations, and, in the 1950s he devised systems for capturing this information using punched cards. I am old enough to remember going to the library in the 1970s to consult Science Citation Index, which could not only point me to important articles in my field, but often led me in strange directions as I stumbled upon other fascinating topics. 

Garfield is known as the father of the Journal Impact Factor, which is regarded by many as an abomination that distorts publishing behaviour because of its connotations of prestige. It was, however, originally conceived as an index that would help librarians decide which journals to purchase, and only later repurposed into a metric that people used as a proxy for the status of the researchers publishing in those journals. 

I enjoyed hearing about Garfield, who sounds like a delightful and humane polymath, who recognised the value of the information in indexes and found ingenious ways to synthesise it. I can recommend an archive of his work maintained by the University of Pennsylvania.

The later speakers in the webinar focused on novel developments in the use of scientometrics to evaluate research quality. Giovanni Abramo noted how Italian science had been affected by favouritism, because of an exclusive reliance on subjective peer review to evaluate researchers and their institutions. His view is that use of metrics improves research evaluation by making it fairer and more objective. He noted that while metrics may not be an option in some areas of arts and humanities, for disciplines where outputs generally appear in indexed journals, bibliometrics were invaluable, concluding that “bibliometrics are to research assessment what diagnostic imaging is to medicine”- i.e. a key source of objective information. 

Funnily enough, I would have agreed with him 12 years ago, when I suggested a simple bibliometric index (departmental H-index) could achieve very similar results to the complex and time-consuming peer review process adopted in the British Research Excellence Framework. At the time I was writing, I thought that Goodhart’s Law ("When a measure becomes a target, it ceases to be a good measure") wouldn’t apply to a citation-based metric, because citations were not controlled by authors, so it would be difficult to game. 

It turns out I was naïve. The crudest method of gaming is excessive self-citation, but there are also citation rings (you cite my paper and I’ll cite yours). This year Maria Ángeles Oviedo-García, René Aquarius and I described a more sophisticated version, a “review mill”, where a group of Italian medics used positions as peer reviewers to coerce citations to work of the group. We suggested that the change in Italian research evaluation, which had been implemented with the best of intentions, had led to cynical gaming of peer review. One might respond by saying that this activity, though disturbing, affects only a tiny proportion of articles and so would not have a detectable effect. Again, I would have agreed a decade ago. But now, with an explosion in publications that seems driven by publishers who are more focused on income than quality (see Hanson et al, 2024), and extraordinarily lax editorial standards,  that may no longer be true. The key point about review mills is that we saw evidence of their activity because they used generic templates for peer reviews, but these can be detected only for journals that publish open peer review – a tiny minority. The most prolific member of the review mill was a journal editor who had nearly 3000 verified peer reviews listed on Web of Science, only a handful of which we could access. 

So I fear that bibliometrics is more like a diagnostic image that has blurred together data from several patients - there is some valid information in there, but it's distorted by error. 

The final presentation by Valentin Bogorov described the future of Scientometrics, where AI would be harnessed to give much more fine-grained and nuanced information about societal impacts of research. But I felt he was ignoring the elephant of fraud that had lumbered into the bibliometric databases. Review mills are one issue for the validity of citation data, but paper mills are a much bigger problem. Whereas review mills rely on the self-organisation of dubious research groups to burnish their reputations, many paper mills are run by outside organisations whose sole motivation is financial profit (Parker et al., 2024). They sell authorships and citations for a price that depends on the Impact Factor of the journal - Eugene Garfield would be turning in his grave. They were first detected about 12 years ago  but have multiplied like a virus and are seriously infecting whole bodies of research. Sometimes they are first recognised when a researcher with subject-domain expertise finds anomalous or fraudulent articles when trying to review the field (see, e.g., Aquarius et al, 2025). 

Paper mills thrive in a warm, soggy environment, where corrupt or incompetent editors will wave through articles that contain clear breaches of scientific method, or even are evidently patched together from various plagiarised articles. The hope of publishers is that AI will provide ways to detect fraudulent papers and remove them before they enter the literature, but paper millers have proved skilled at mutating to evade detection. Unfortunately, the very areas where AI and big data seem to hold most promise, such as databases linking genes, proteins, molecules and biomarkers, already are contaminated.  The fear is that the papermillers will themselves increasingly use AI to create ever more plausible articles. 

I am not opposed to bibliometrics or AI in principle, but I find the optimism about its application to research evaluation concerning, especially since no mention was made of the problems that will emerge if the database interrogated by AI is polluted. Any method of evaluation will have costs, benefits and unforeseen consequences. My concern is that if we focus only on the benefits, we could end up with a system that encourages fraudsters and rewards those who are most skilled at gaming the system rather than the best scientists.  

Tomatoes roaming the fields and canaries in the coalmine: another embarrassing paper for MDPI

 

Many publishers are getting nervous about infiltration by paper mills, who can torpedo a journal's reputation when they succeed in publishing papers that are obvious nonsense. In a recent Open Letter, a group of sleuths drew attention to an example in Scientific Reports, published by Springer Nature.

After the Open Letter was published, the paper that instigated our concern was promptly retracted by the journal, but as far as I can tell, not much else has changed. The point about a paper like this is that it is so blatantly bad that it cannot have been through any kind of serious editorial scrutiny or peer review. It acts as a canary in the coalmine: if gobbledegook is published in your journal, it's an indicator that you need to look very carefully at your editorial processes, and act immediately to remove editors who let this stuff in. Sadly, I haven't yet seen much evidence of that happening at Scientific Reports.

This post, however, concerns another publisher, MDPI, who have regularly featured on my blog, and not in a good way. Last month, I commented on the strange state of affairs whereby Finland had downgraded its classification of 187 MDPI journals because of evidence of "minimum time spend for editorial work and quality assessment", at the same time that German universities had secured a national publishing agreement with MDPI. The story I have to tell here may confirm Finland's judgement, and give Germany pause for thought. It concerns this article: Abbas, R., Amran, G. A., Hussain, I., & Ma, S. (2022). A Soft Computing View for the Scientific Categorization of Vegetable Supply Chain Issues. Logistics, 6(3), https://doi.org/10.3390/logistics6030039 

As with the Springer Nature example, the first indication of problems came via the Problematic Paper Screener, the excellent system that checks articles for various red flags, including "tortured phrases". These provide an indicator that a paper has probably been plagiarised but then passed through a process that substitutes synonyms for main words, with the aim of evading plagiarism detection software. So, as noted on PubPeer, in this case we have "fluffy logic" for fuzzy logic, and "unaided ML" for unsupervised machine learning. 

However, example sentences in which tortured phrases were embedded indicated a deeper problem. Most of the text is incomprehensible, and things start to get seriously weird when the authors get on to tomatoes. We are told: 

.... the third creation framework considered for the creation phase is tomatoes. This creation framework is devoted to developing homegrown creatures brought up in rural settings to create vegetables. This can bring domesticated tomatoes likewise to broad or serious frameworks. Broad frameworks include creatures wandering meadows (ordinarily under the oversight of a herder). Differently, serious tomatoes are situated in shut foundations and are outfitted with ICT innovation, which empowers creatures to be observed continuously. Inside these creation frameworks, the most run-of-the-mill issues we run over are meadow observing [75], creature government assistance [76], creature conduct following [77], and tomato creation forecast and enhancement [78,79], as displayed in Figure 3. 

According to a VSC point of view, the formal meanings of these issues are recorded beneath. 

• Field checking: This issue is connected with the exact recognizable proof of meadow inventories to separate between the most reasonable sorts for tomatoes purposes. 

• Tomato government assistance: This is centered around the example arrangement of the dehydration way of behaving in brushing creatures for investigations of creature nourishment, development, and well-being. 

• Tomato growth checking: This depends on the utilization of conduct investigations to recognize early indications of medical problems and advance early negotiation. 

A clue to the origin of this material comes from the cited references, which are about pigs and cattle. Anonymous PubPeer commenter Nerita vitiensis found that a substantial part of the text was adapted from a previous work by different authors, but with the topics of "livestock and fish" changed to "tomatoes and cruciferous vegetables". This explains the description of tomatoes as "creatures" under the oversight of a herder. 

The authors of this piece seem seriously out of their depth, as evidenced by the bland comments apparently written by Chat GPT that they provided on PubPeer. 

Now, one very good thing about MDPI is that it generally identifies the academic editor who handled a paper, and it sometimes also makes public the reviewer reports. This should mean that when a major foul-up like this occurs, it should be possible to identify and purge those responsible for accepting the work. 

The academic editors who accepted this article are Xue-Ming Yuan, who is currently soliciting papers for a special issue in the MDPI journal Mathematics, and Anrong Xue.

The MDPI website shows reports from three named reviewers. 

The first reviewer, Edyta Kardas, was concerned about the use of first-person language, and punctuation, but not apparently about statements about animated tomatoes. She reviewed 8 papers for MDPI journals in 2024.

The second reviewer, Alejandro Vega-Muñoz focused solely on the structure of the article, but apparently did not look at the content. He has edited two special issues for other MDPI journals. 

 The third reviewer, Francesco Barreca, attempted a synopsis of the article (which I could not understand) and then had just two suggestions:

"The work is well done but I have some remarks:

• The figures should be review, the dimension are variable

 • Moderate English changes are required"

The "moderate English changes" were unspecified. Barreca has a track record of editing a special issue of another MDPI journal.

 
Last week, I contacted Publication Ethics at MDPI to draw their attention to this article, noting the dereliction of duty by reviewers and editors, and suggesting that as well as retracting the paper, they should remove the editors and peer reviewers from their database. They replied to say: 

"We confirm that the Editorial Office is investigating the concerns related to this paper following the guidelines of the Committee on Publication Ethics https://publicationethics.org/ of which we are a member and our policy https://www.mdpi.com/ethics#_bookmark29.

We would like to inform you that this case is a priority for us, and we are actively working to resolve it. We will update you on the outcome of this investigation as soon as possible."

I await developments with interest. It is widely recognised that COPE guidelines are not well-suited for dealing with this kind of situation: they make the default assumption that authors should be consulted to give their perspective when criticisms are raised - a reasonable assumption in many cases, but not when there is such blatant evidence of fakery.

The most serious case of infestation of a publisher by nonsense occurred in 2022-3, when the publisher Hindawi (owned by Wiley) was targeted by paper mills who, among other things, generated numerous papers that I labelled as AI gobbledegook sandwiches. Eventually, the publisher withdrew literally thousands of papers and closed the Hindawi brand, after complaints by shareholders started impacting profits.

Like many of the sleuths who track down paper mills, I have become cynical about the commitment to research integrity that is claimed by many publishers, including MDPI. But I do believe they will act when it is in their interests to do so. As the amount of nonsense and disinformation in the scientific literature increases, I think we'll enter a new phase where trustworthiness of journal contents will start to have much higher value. If you want to be taken seriously as a peer-reviewed journal, you just cannot continue to pump out articles accompanied by superficial verbiage from "peer reviewers" that makes no real contact with the subject matter. Publishers will need to act now to clean up their editorial boards if they want to stay in business.   

Update 7th February 2025

 

I am pleased to report that I have now heard from MDPI as follows:

Following a thorough investigation of this paper and according to the recommendations of the journal's Editorial Board, we have decided to retract this publication, in line with MDPI’s retraction policy: https://www.mdpi.com/ethics#_bookmark30

Further details regarding this retraction can be found at: https://www.mdpi.com/2305-6290/9/1/20

However, as far as I can tell, no action has been taken against the editors and reviewers who were responsible for accepting the article.  The only thing I notice is that the peer reviewer comments are no longer linked to the main page for the article on the journal website (though they are still available here).  I have written to MDPI Publication Ethics to ask for clarification as to whether any action will be taken to replace the editors, and to remove the peer reviewers from their register.  Unless a robust approach is taken to removing those who admit such nonsense into the journal, readers and potential authors can have no confidence in the integrity of MDPI's editorial processes.

Update 12th February 2025

I have now received a response from MDPI Publication Ethics to my query about editors and peer reviewers.  The plan is to educate them rather than remove them.  

We are happy to provide additional details regarding our approach to this situation. 

In line with the recommendations from COPE for ethical and transparent scholarly publishing (https://publicationethics.org/guidance/guideline/principles-transparency-and-best-practice-scholarly-publishing), we are adopting an educational approach. 
This implies sending notifications to the reviewers and the editor to inform them of this retraction, clarify their responsibilities in the review process, and outline the potential consequences of a superficial review, according to our guidelines for reviewers (https://www.mdpi.com/reviewers#_bookmark11) and information for editors (https://www.mdpi.com/editors). 

Given the significant responsibility that comes with the role of peer reviewers, which directly impacts the credibility of both the journal and the broader scientific literature, we take proactive steps to offer additional guidance and resources beyond our standard guidelines. An example of this initiative can be found on our MDPI Blog: https://blog.mdpi.com/2025/01/22/reviewer-responsibilities/

Moving forward, we will closely monitor the activities of both the editor and reviewers to ensure adherence to the highest standards of academic publishing.

We remain at your disposal for any further information.

Papermillers must be feeling very cheerful right now.

Note: Comments on this blog are moderated, so there is a delay before they appear.  Anonymous or off-topic comments are not accepted. 

What is going on at the Journal of Psycholinguistic Research?

Last week this blog focussed on problems affecting Scientific Reports, a mega-journal published by Springer Nature. This week I look at a journal at the opposite end of the spectrum, the Journal of Psycholinguistic Research (JPR), a small, specialist journal which has published just 2187 papers since it was founded in 1971. This is fewer than Scientific Reports publishes in one year. It was brought to my attention by Anna Abalkina because it shows every sign of having been targeted by one or more Eastern European paper mills.

Now, this was really surprising to me. JPR was founded in 1971 by Robert Rieber, whose obituaries in the New York Times  and the American Psychologist confirm he had a distinguished career (though both misnamed JPR!). The Advisory and Editorial boards of the journal are peppered with names of famous linguists and psychologists, starting with Noam Chomsky. So there is a sense that if this can happen to JPR, no journal is safe.

Coincidentally, last week Anna and I submitted revisions for a commentary on paper mills coauthored with Pawel Matusz. (You can read the preprint here). Pawel is editor of the journal Mind, Brain & Education (MBE), which experienced an attack by the Tanu.pro paper mill involving papers published in 2022-3. In the commentary, we discussed characteristics of the paper mill, which are rather distinctive and quite different from what is seen in basic biomedical or physical sciences. A striking feature is that the IMRaD structure (Introduction, Methods, Results and Discussion) is used, but in a clueless fashion, with these headings being inserted in what is otherwise a rambling and discursive piece of text, that typically has little or no empirical content. Insofar as there are any methods described, they don't occur in the methods section, and they are too vague for the research to be replicable.

Reading these papers rapidly turns my brain to mush, but in the interest of public service I did wade through five of them and left comments on Pubpeer:  

Yeleussizkyzy, M., Zhiyenbayeva, N., Ushatikova, I. et al. E-Learning and Flipped Classroom in Inclusive Education: The Case of Students with the Psychopathology of Language and Cognition. J Psycholinguist Res 52, 2721–2742 (2023). https://doi.org/10.1007/s10936-023-10015-y  

https://pubpeer.com/publications/7C9B4538A07EE8777082D63B2944B9 

Snezhko, Z., Yersultanova, G., Spichak, V. et al. Effects of Bilingualism on Students’ Linguistic Education: Specifics of Teaching Phonetics and Lexicology. J Psycholinguist Res 52, 2693–2720 (2023). https://doi.org/10.1007/s10936-023-10016-x

https://pubpeer.com/publications/09F57870A7D20BF5F77C53F414449F 

Nurakenova, A., Nagymzhanova, K. A Study of Psychological Features Related to Creative Thinking Styles of University Students. J Psycholinguist Res 53, 1 (2024). https://doi.org/10.1007/s10936-024-10042-3

https://pubpeer.com/publications/A1FB870AD77F1D07D34B9BF9AEEFF3 

Auganbayeva, M., Turguntayeva, G., Anafinova, M. et al.Linguacultural and Cognitive Peculiarities of Linguistic Universals. J Psycholinguist Res 53, 3 (2024). https://doi.org/10.1007/s10936-024-10050-3

https://pubpeer.com/publications/06DD63C96B4D7C74DAC8AB72F18A34 

Shalkarbek, A., Kalybayeva, K., Shaharman, G. et al. Cognitive Linguistic Analysis of Hyperbole-based Phraseological Expressions in Kazakh and English Languages. J Psycholinguist Res 53, 4 (2024). https://doi.org/10.1007/s10936-024-10052-1

https://pubpeer.com/publications/D52E26EE9DE5F83EA07B6B9917A645#1

My experience with the current batch of papers suggests that a relatively quick way of screening a submitted paper would be to look at the Methods section. This should contain an account of methods that would indicate what was done and how, at a level of detail sufficient for others to replicate the work. Obviously, this is not appropriate for theoretical papers, but for those purporting to report empirical work, it would work well, at least for the papers I looked at in JPR.   

All of these papers have authors from Kazakhstan, sometimes with co-authors from the Russian Federation. This led me to look at the geographic distribution of authors in the journal over time. The top countries represented by JPR authors in 2020 onwards are China (113), United States (68), Iran (52), Germany (28), Saudi Arabia (22) and Kazakhstan (19). However, these composite numbers mask striking trends. All the Kazakhstan authored papers are in 2023-2024. There's also a notable fall-off in papers authored by USA-based authors in the same time period, with only 11 cases in total. This is quite remarkable for a journal that had a striking USA dominance in authors up until around 2015, as shown in the attached figure (screenshot from Dimensions.ai).

 

Number of papers in JPR from five top countries: 2005-2024

Exported: October 20, 2024

Criteria: Source Title is Journal of Psycholinguistic Research.

© 2024 Digital Science and Research Solutions Inc. All rights reserved. 

Non-commercial redistribution / external re-use of this work is permitted subject to appropriate acknowledgement. 

This work is sourced from Dimensions® at www.dimensions.ai.

Whenever a paper mill infestation is discovered, it raises the question of how it happened. Surely the whole purpose of peer review is to prevent low quality or fraudulent material entering the literature? In other journals where this has happened it has been found that the peer review process was compromised, with fake peer reviewers being used. Even so, one would have hoped that an editor would scrutinize papers and realise something was amiss. As mentioned in the previous blogpost, it would be much easier to track down the ways in which fraudulent papers get into mainstream journals if the journal reported information about the editor who handled the paper, and published open peer review.

Whatever the explanation, it is saddening to see a fine journal brought so low. In 2021, at the 50th anniversary of the founding of the journal, the current editor, Rafael Art. Javier, wrote a tribute to his predecessor, Robert Rieber:

"His expectation, as stated in that first issue, was that manuscripts accepted 'must add to knowledge in some way, whether they are in the form of experimental reports, review papers, or theoretical papers...and studies with negative results,' provided that they are of sufficiently high quality to make an original contribution."

Let us hope that the scourge of paper mills can be banished from the journal to allow it to be restored to the status it once had, and for Robert Rieber's words to once more be applicable.

 

Using PubPeer to screen editors

 

2023 was the year when academic publishers started to take seriously the threat that paper mills posed to their business. Their research integrity experts have penned various articles about the scale of the problem and the need to come up with solutions (e.g., here and here).  Interested parties have joined forces in an initiative called United2Act. And yet, to outsiders, it looks as though some effective actions are being overlooked. It's hard to tell whether this is the result of timidity, poor understanding, or deliberate footdragging from those who have a strong financial conflict of interest.

As I have emphasised before, the gatekeepers to journals are editors. Therefore it is crucial that they are people of the utmost integrity and competence. The growth of mega-journals with hundreds of editors has diluted scrutiny of who gets to be an editor. This has been made worse by the bloating of journals with hundreds of special issues, each handled by "guest editors". We know that paper millers will try to bribe existing editors, and to place their own operatives as editors or guest editors, use fake reviewers, and stuff articles with irrelevant citations. Stemming this tide of corruption would be one effective way to reduce the contamination of the research literature. Here are two measures I suggest that publishers should take if they seriously want to clean up their journals.

1. Three strikes and you are out. Any editor who has accepted three or more paper milled papers should be debarred from acting as an editor, and all papers that they have been responsible for accepting should be regarded as suspect. This means retrospectively cleaning-up the field by scrutinising the suspect papers and retracting any from authors associated with paper mills, or which are characterised by features suggestive of paper mills, such as tortured phrases, citation stacking, gobbledegook content, fake reviews from reviewers suggested by authors, invalid author email domains, or co-authors who are known to be part of a paper mill ring. All of these are things that any competent editor should be able to detect. I anticipate this would lead to a large number of retractions, particularly from journals with many Special Issues. As well as these simple indicators, we are told that publishers are working hard to develop AI-based checks. They should use these not only to screen new submissions, and to retract published papers, but also to identify editors who are allowing this to happen on their watch. It also goes without saying that nobody who has co-authored a paper-milled paper should act as an editor.

2. All candidates for roles as Editor or Guest Editor at a journal should be checked against the post-publication peer review website PubPeer, and rejected if this reveals evidence of papers that have had credible criticisms suggesting of data fabrication or falsification. This is a far from perfect indicator: only a tiny fraction of authors receive PubPeer comments, and these may comment on trivial or innocent aspects of a paper. But, as I shall demonstrate, using such a criterion can reveal cases of editorial misconduct.

I will illustrate how this might work in practice, using the example of the MDPI journal Electronics. This journal came to my attention because it has indicators that all is not well with its Special Issues programme. 

First, in common with nearly all MDPI journals, Electronics has regularly broken the rule that specifies that no more than 25% of articles should be authored by a Guest Editor. As mentioned in a previous post, this is a rule that has come and gone in the MDPI guidelines, but which is clearly stated as a requirement for inclusion in the Directory of Open Access Journals (DOAJ). 13% of Special Issues in Electronics completed in 2023-4 broke this rule**. DOAJ have withdrawn some MDPI journals from their directory for this reason, and I live in hope that they will continue to implement this policy rigorously - which would entail delisting from their Directory the majority of MDPI journals. Otherwise, there is nothing to stop publishers claiming to be adhering to rigorous standards while failing to implement them, making listing in DOAJ an irrelevance.  

Even more intriguing, for around 11% of the 517 Special issues of Electronics published in 2023-4, the Guest Editor doesn't seem to have done any editing We can tell this because Special Issues are supposed to list who has acted as Academic Editor for each paper. MDPI journals vary in how rigorously they implement that rule - some journals have no record of who was the Academic Editor. But most do, and in most Special Issues, as you might expect, the Guest Editor is the Academic Editor, except for any papers where there is conflict of interest (e.g. if authors are Guest Editors or are from the same institution as the Guest Editor). Where the Guest Editor cannot act as Academic Editor, the MDPI guidelines state that this role will be taken by a member of the Editorial Board. But, guess what? Sometimes that doesn't happen. As someone with a suspicious frame of mind, and a jaundiced view of how paper mills operate, this is a potential red flag for me.

Accordingly, I decided to check PubPeer comments for individuals in three editorial roles at Electronics for the years 2023-4:

• Those listed as being in a formal editorial role on the journal website. 
• Those acting a Guest Editors 
• Those acting as Academic Editors, despite not being in the other two categories.

For Editors, a PubPeer search by name revealed 213/931 that had one or more comments. That sounds alarming, but cannot be taken at face value, because there are many innocent reasons for this result. The main one is namesakes: this is particularly common with Chinese names, which tend to be less distinctive than Western names. It is therefore important to match PubPeer comments on affiliations as well as names. Using this approach, it was depressingly easy to find instances of Editors who appeared associated with paper mills. I will mention just three, to illustrate the kind of evidence that PubPeer provides, but remember, there are many others deserving of scrutiny. 

• As well as being a section board member of Electronics, Danda B Rawat (Department of Electrical Engineering and Computer Science, Howard University, Washington, DC 20059, USA) is Editor-in-Chief of Journal of Cybersecurity and Privacy, and a section board member of two further MDPI journals: Future Internet and Sensors. A PubPeer search reveals him to be co-author of one paper with tortured phrases, and another where equations make no sense. He is listed as Editor of three MDPI Special Issues: Multimodal Technologies and Interaction: Human Computer Communications and Internet of Things Sensors: Frontiers in Mobile Multimedia Communications Journal of Cybersecurity and Privacy: Applied Cryptography.
• Aniello Castiglione  (Department of Management & Innovation Systems, University of Salerno, Italy) is Section Board Member of three journals: Electronics, Future Internet, and Journal of Cybersecurity and Privacy, and an Editorial Board member of Sustainability. PubPeer reveals he has co-authored one paper that was recently retracted because of compromised editorial processing, and that his papers are heavily cited in several other articles that appear to be used as vehicles for citation stacking. 
•  Natalia Kryvinska (Department of Information Systems, Faculty of Management, Comenius University in Bratislava, Slovakia) is a Section Board Member of Electronics. She has co-authored several articles with tortured phrases.

Turning to the 1326 Guest Editors of Special Issues, there were 500 with at least one PubPeer comment, but as before, note that in many cases name disambiguation is difficult, so this will overestimate the problem. Once again, while it may seem invidious to single out specific individuals, it seems important to show the kinds of issues that can be found among those who are put in this important gatekeeping role. 

• Heena Rathore (Department of Computer Science, Texas State Uni, San Marcos, Texas, USA) is a Guest Editor of this Special Issue. She has been lead author on a paper containing tortured phrases.
  
• Aimin Yang (College of Science, North China University of Science and Technology, Tangshan 063000) is one of five Chinese Guest Editors of a Special Issue on Intelligent Analysis and Security Calculation of Multisource Data. It is unclear what he did in this role: he was not mentioned as Academic Editor for any of the 12 papers the Special Issue. A PubPeer search reveals indicators of citation stacking (i.e. citing of irrelevant references) in his articles.  
• Joseph Bamidele (Department of Computer Science, University of Ilorin, Ilorin, Nigeria) is a Guest Editor of a Special Issue on Artificial Intelligence in Cybersecurity for Industry 4.0. A PubPeer search identifies him as lead author on an article with tortured phrases.
  

Finally, let's look at the category of Academic Editors who aren't listed as journal Editors. It's unclear how they are selected and who approves their selection. Again, among those with PubPeer comments, there's a lot to choose from. I'll focus here on three who have been exceptionally busy doing editorial work on several special issues. 

• Gwanggil Jeon (Incheon National University, Korea) has acted as Academic Editor for 18 Special Issues in Electronics. He is not on the Editorial Board of the journal, but he has been Guest Editor for two special issues in Remote Sensing, and one in Sensors.  PubPeer comments note recycled figures and irrelevant references in papers that he has co-authored, as well as a problematic Special Issue that he co-edited for Springer Nature, which led to several retractions.
• Hamid Reza Karimi (Department of Mechanical Engineering, Politecnico di Milano, Milan, Italy) has acted as Academic Editor for 12 Special Issues in Electronics. He was previously Guest Editor for two Special Issues of Electronics, one of Sensors, one of Micromachines, and one of Machines.  In 2022, he was specifically called out by the IEEE for acting "in violation of the IEEE Principles of Ethical Publishing by artificially inflating the number of citations" for several articles. 
  
• Finally, Juan M. Corchado (University of Salamanca, Spain) has acted as Academic Editor for 29 Special Issues. He was picked up by my search as he is not currently listed as being an Editor for Electronics, but that seems to be a relatively recent change: when searching for information, I found this interview from 2023. Thus his role as Academic Editor seems legitimate. Furthermore, as far as PubPeer is concerned, I found only one old comment, concerned with duplicate publication. However, he is notorious for boosting citations to his work by unorthodox means, as described in this article.* I guess we could regard his quiet disappearance from the Editorial Board as a sign that MDPI are genuinely concerned about editors who try to game the system. If so, we can only hope that they employ some experts who can do the kinds of cross-checking that I have described here at scale. If I can find nine dubious editors of one journal in a couple of hours searching, then surely the publisher, with all its financial resources, could uncover many more if they really tried.
  

Note that many of the editors featured here have quite substantial portfolios of publications. This makes me dubious about MDPI's latest strategy for improving integrity - to use an AI tool to select potential reviewers "from our internal databases with extensive publication records". That seems like an excellent way to keep paper millers in control of the system. 

Although the analysis presented here just scratches the surface of the problem, it would not have been possible without the help of sleuths who made it straightforward to extract the information I needed from the internet. My particular thanks to Pablo Gómez Barreiro, Huanzi and Sholto David.

I want to finish by thanking the sleuths who attempt to decontaminate the literature by posting comments to PubPeer. Without their efforts it would be much harder to keep track of paper millers. The problem is large and growing. Publishers are going to need to invest seriously in employing those with the expertise to tackle this issue. 

 *As I was finalising this piece, this damning update from El Pais appeared. It seems that many retractions of Corchado papers are imminent.  

 I can't keep up.... here's today's news. 

** P.S. 25th Sept 2024. DOAJ inform me that Electronics was removed from their directory in June of this year. 

*** P.P.S. 26th Sept 2024.  Guillaume Cabanac pointed me to this journal-level report on PubPeer, where he noted a high rate of Electronics papers picked up by the Problematic Paper Screener.

The world of Poor Things at MDPI journals

At the weekend, the Observer ran a piece by Robin McKie entitled "‘The situation has become appalling’: fake scientific papers push research credibility to crisis point". I was one of those interviewed for the article, describing my concerns about a flood of dodgy papers that was polluting the scientific literature.

Two days later I received an email from the editorial office of MDPI publishers with the header "[Children] (IF: 2.4, ISSN 2227-9067): Good Paper Sharing on the Topic of" (sic) that began:

Greetings from the Children Editorial Office!

We recently collected 10 highly cited papers in our journal related to Childhood Autism. And we sincerely invite you to visit and read these papers, because you are an excellent expert in this field of study.

Who could resist such a flattering invitation? MDPI is one of those publishers that appears to be encouraging publication of low quality work, with a massive growth in special issues where papers are published with remarkably rapid turnaround times. Only last week it was revealed that the journal is affected by fake peer review that appears to be generated by AI. So I was curious to take a look.

The first article, by Frolli et al (2022a) was weird. It reported a comparison of two types of intervention designed to improve emotion recognition in children with autism, one of which used virtual reality. The first red flag was the sample size: two groups each of 30 children, all originally from the city of Caserta. I checked Wikipedia, which told me the population of Caserta was around 76,000 in 2017. Recruiting participants for intervention studies is typically slow and laborious and this is a remarkable sample size to recruit from such a small region. But credibility is then stretched to breaking point on hearing that the selection criteria required that the children were all aged between 9 and 10 years and had IQs of 97 or above. No researcher in their right mind would impose unnecessary constraints on recruitment, and both the age and IQ criteria are far tighter than would usually be adopted. I wondered whether there might be a typo in this account, but we then hear that the IQ range of the sample is indeed remarkably narrow: 

"The first experimental group (Gr1) was composed of 30 individuals with a mean age of 9.3 (SD 0.63) and a mean IQ of 103.00 (SD 1.70). ...... The second experimental group (Gr2) was composed of 30 individuals with a mean age of 9.4 (SD 0.49) and mean IQ of 103.13 (SD 2.04)...."

Most samples for studies using Wechsler IQ scales have SD of at least 8, even if cutoffs are applied as selection criteria, so this is unbelievably low.

This dubious paper prompted me to look at others by the first author. It was rather like pulling a thread on a hole in a sweater - things started to unravel fast. A paper published by Frolli et al (2023a) in the MDPI journal Behavioral Sciences claimed to have studied eighty 18-year-olds recruited from four different high schools. The selection criteria were again unbelievably stringent: IQ assessed on the WAIS-IV fell between 95-105 "to ensure that participants fell within the average range of intellectual functioning, minimizing the impact of extreme cognitive variations on our analyses". The lower IQ range selected here corresponds to z-score of -0.33 or 37th percentile. If the population of students covered the full range of IQ, then only around 25% would meet the criterion (between 37th and 63rd centile), so to obtain a sample of 80 it would be necessary to test over 300 potential participants. Furthermore, there are IQ screening tests that can be used in this circumstance that are relatively quick to administer, but the WAIS-IV is not one of them. We are told all participants were given the full test, which requires individual administration by a qualified psychologist and takes around one hour to complete. So who did all this testing, and where? The article states: "The data were collected and analyzed at the FINDS Neuropsychiatry Outpatient Clinic by licensed psychologists in collaboration with the University of International Studies of Rome (UNINT)." So we are supposed to believe that hundreds of 18-year-olds trekked to a neuropsychiatry outpatient clinic for a full IQ screening which most of them would not have passed. I cannot imagine a less efficient way of conducting such a study. I could not find any mention of compensation for participants, which is perhaps unsurprising as the research received no external funding. All of this is described as happening remarkably fast, with ethics approval in January 2023, and submission of the article in October 2023.

Another paper in Children in 2023 focused on ADHD, and again reported recruiting two groups of 30 children for an intervention that lasted 5 months (Frolli et al., 2023b). The narrow IQ selection criteria were again used, with WISC-IV IQs in the range 95-105, and the mean IQs were 96.48 (SD =1.09) and 98.44 (SD = 1.12) for groups 1 and 2 respectively. Again, the research received no external funding. The report of ethics approval is scanty "The study was conducted in accordance with the Declaration of Helsinki. The study was approved by the Ethics Committee and the Academic Senate of the University of International Studies of Rome."

The same first author published a paper on the impact of COVID-19 on cognitive development and executive functioning in adolescents in 2021 (Frolli et al, 2021). I have not gone over it in detail, but a quick scan revealed some very odd statistical reporting. There were numerous F-ratios, but they were all negative, which is impossible, as F is a ratio between two positive numbers. Furthermore, the reported p-values and degrees of freedom didn't always correspond to the F-ratio, even if the sign was ignored.

At this point I was running out of steam, but a quick look at Frolli et al (2022a) on Executive Functions and Foreign Language Learning suggested yet more problems, with the sentence "Significance at the level of 5% (α < 0.001) has been accepted" featuring at least twice. It is hard to believe that a human being wrote this sentence, or that any human author, editor or reviewer read it without comment.

If anyone is interested in pulling at other related threads, I suspect it would be of interest to look at articles accepted for a Special Issue of the MDPI journal Disabilities co-edited by Frolli.

In his brilliant film Poor Things, Yorgos Lanthimos distorts familiar objects and places just enough to be disturbing. Lisbon looks like what I imagine Lisbon would be in the Victorian age, except that the colours are unusually vivid, there are strange flying cars in the sky, and nobody seems concerned at the central character wandering around only partially clothed (see, e.g., this review).  The combined impression is that MDPI publishes papers from that universe, where everything looks superficially like genuine science but with jarring features that tell you something is amiss. The difference is that Poor Things has a happy ending.

References 

Frolli, A.; Ricci, M.C.; Di Carmine, F.; Lombardi, A.; Bosco, A.; Saviano, E.; Franzese, L. The Impact of COVID-19 on Cognitive Development and Executive Functioning in Adolescents: A First Exploratory Investigation. Brain Sci. 2021, 11, 1222. https://doi.org/10.3390/brainsci11091222

Frolli, A.; Savarese, G.; Di Carmine, F.; Bosco, A.; Saviano, E.; Rega, A.; Carotenuto, M.; Ricci, M.C. Children on the Autism Spectrum and the Use of Virtual Reality for Supporting Social Skills. Children 2022a, 9, 181. https://doi.org/10.3390/children9020181

Frolli, A.; Cerciello, F.; Esposito, C.; Ciotola, S.; De Candia, G.; Ricci, M.C.; Russo, M.G. Executive Functions and Foreign Language Learning. Pediatr. Rep. 2022b, 14, 450-456. https://doi.org/10.3390/pediatric14040053

Frolli, A.; Cerciello, F.; Ciotola, S.; Ricci, M.C.; Esposito, C.; Sica, L.S. Narrative Approach and Mentalization. Behav. Sci. 2023a, 13, 994. https://doi.org/10.3390/bs13120994

Frolli, A.; Cerciello, F.; Esposito, C.; Ricci, M.C.; Laccone, R.P.; Bisogni, F. Universal Design for Learning for Children with ADHD. Children 2023b, 10, 1350. https://doi.org/10.3390/children10081350

Is Hindawi “well-positioned for revitalization?”

Guest post by Huanzi Zhang*

 Screenshot from https://www.wiley.com/en-us/network/publishing/research-publishing/open-access/wiley-acquires-hindawi-qa-with-liz-ferguson

 

Over the past year, special issues of dozens of Hindawi journals have been exposed as being systematically manipulated, resulting in the delisting of more than 20 Hindawi journals from major journal databases, as well as the retraction of more than 2,700 papers by the publisher. This "unexpected event" at Hindawi also led to a slump in profits for the parent company, John Wiley & Sons. However, in a recent statement, the president, CEO & director of Wiley, Brian Napack, stated that Hindawi was now ready for revitalization and reinstatement of the special issue program. In my opinion, Wiley has not dealt adequately with the integrity issues that led to the problem, but appears focused on growth  through the medium of special issues. This raises questions as to whether Hindawi’s operation is sustainable in the long term. 

Napack’s statement can be read here. He stated:

 “Fiscal '24 (starts on May 1, 2023) will be a year of revitalization for Hindawi with positive signs already emerging. We've now named a new leader of Hindawi, a talented Wiley veteran with deep expertise in the area. We've restarted the special issues program and we will be ramping it up throughout the year. We're working through the large article backlog, and we are executing our journal growth plans.”

One might, however, be forgiven for being a bit sceptical about this upbeat message, since just a year before, Napack said:

 “Hindawi performed at a very high level this year, delivering strong double-digit organic revenue growth and 36% article output growth on a pro forma basis, and has achieved this with exceptional margins. We have now completed the integration and we are benefiting significantly from Hindawi's industry leading open publishing practices and its highly efficient systems.” 

So what is the problem with Hindawi journals that suddenly got Wiley into trouble? Is Hindawi really well-positioned for revitalization? 

Wiley’s acquisition of Hindawi

Wiley announced the acquisition of Hindawi on January 5, 2021. The person who pushed Wiley to buy Hindawi was Judy Verses, Wiley's Executive Vice President, who left Wiley to join Elsevier a few months after the acquisition was completed. On January 11, 2021, in an interview with Verses and Wiley's Senior Vice President Liz Ferguson, we learned that Wiley expected that Hindawi journals would publish many articles by Chinese authors, expanding their market in China:

 “It has surpassed the US in recent years, and in an increasing number of disciplines is undoubtedly the global leader. Hindawi had the foresight to launch and develop journals that reflect strengths in the China research space. Similar to Wiley, Hindawi identified early on how important it was to serve the needs of China-based researchers. Bringing together our two teams now means we have an even stronger position to be able to work to the needs of those researchers.”

The Publishing Perspective's interview with Jay Flynn, Senior Vice President and Chief Product Offer of Wiley Research, published on January 5, 2021, and Wiley's internal interview with Ferguson, published on February 23, 2021, mentioned the importance of China. The new niche market for Hindawi journals seemed to respond soundly to this appointment, and the number of submissions began to increase significantly (Figure 1), while no such increase had been seen in the preceding months.  In September 2021, Jay Flynn was promoted to Executive Vice President of Wiley Research to replace Judy Verses.

 

Figure 1. Number of submissions received per month for 24 journals that subsequently had papers retracted by Hindawi. Data from Hindawi Journal Report.

The papermill problem 

The first concerns were raised by research integrity communities and individuals, who since 2021 posted thousands of comments on the PubPeer journal club relating to papers published in hundreds of special issues of Hindawi journals. Many comments were made by anonymous sleuths Rhipidura albiventris, Hoya camphorifolia and Parashorea tomentella, whose contributions sometimes exceeded 100 per day.  Problems were not limited to individual special issues or even individual journals. There appeared to be systematic manipulation of the publishing process, especially affecting special issues, indicating activities of so-called “paper mills” – fraudulent organisations that will sell authorship and/or citations of papers, often faked, for a fee. Leonid Schneider, who runs the blog For Better Science, assisted David Bimler and others in posting their findings from Hindawi journals. Nick Wise discussed “What is going on in Hindawi special issues?” on October 12, 2022. These sleuths noted that many supposedly ‘peer-reviewed’ manuscripts had incoherent or unintelligible content, and corresponding authors used email addresses from other institutions; furthermore, a large number of papers cited irrelevant references, presumably to boost citation counts; some paper mills used the article processing charge (APC) waiver policy of Hindawi to make more profit. The pattern of abnormal citations confirmed that the fraud was not bounded by journals, and so publication-based investigations made it difficult to expose specific paper mills. 

It soon became clear that the special issues that were such a lucrative source of income for Hindawi were wide open to corrupt "guest editors" who, once appointed, could use fake peer review and flood the journal with fraudulent papers and irrelevant citations. In the gold open access model, Wiley earns APCs for every article published in Hindawi journals, whether in a special issue or not, so their incentive to exert quality control is compromised. Some examples are so extreme that nobody could take them seriously, such as an article on the ideological and political education of the Chinese Communist Party in a special issue "Exploration of Human Cognition using Artificial Intelligence in Healthcare", which was submitted, peer-reviewed and received on the same day. There are thousands of other papers which may be genuine but whose subject falls well outside the scope of the special issue where they are published, indicating that the journal is out of editorial control. 

Many of the authors and guest editors of the problematic papers mentioned by Bimler came from Asia. Ruihang Huang, Chunjiong Zhang, and Hanlie Cheng, PhD students at Donghua University, Tongji University and China University of Geosciences, Beijing, respectively, were beneficiaries of the citation manipulation and participants in the manipulation of special issues. Another example is Kaifa Zhao, who approved many nonsense manuscripts for publication as a guest editor at two journals: Computational Intelligence and Neuroscience and Journal of Environmental and Public Health. TigerBB8 identified Zhao as a PhD student at the Hong Kong Polytechnic University, and Dorothy Bishop requested an investigation by the university. As reported in Retraction Watch, their report claimed that Zhao's identity had been stolen by Yizhang Jiang, Zhao's master's program advisor. 

"According to Mr Zhao, he was not aware of relevant emails from Hindawi and has never responded to emails that are related to the two special issues"

 Hindawi takes action (slowly)

After a year of rapid growth in the Chinese market (Figure 2), Wiley acted. On September 28, 2022, Ferguson announced that 511 papers would be retracted from Hindawi journals. Intrestingly, no mention was made of the comments on PubPeer and Bimler's blog post; instead it was stated that these retractions were based on the findings of the Hindawi Research Integrity Team. The first retractions were seen in mid-November with concentrated releases during the Lunar New Year.

 

Figure 2. Number of articles and reviews published in 14 Hindawi journals 2019-2022. Data from Scopus.

It is possible that mass retractions were delayed because Wiley did not want to disrupt their agenda at the 74th Frankfurter Buchmesse (October 19 to 23, 2022). There was no indication that Wiley shared Hindawi's problems at the book fair. Instead, they were busy with other things. Flynn announced the creation of Wiley Partners Solution to meet the "scholarly publishing needs at scale" on October 17. Ferguson participated in a forum entitled "How the Article-Based Economy is Transforming Research Publishing" on October 19. Intriguingly, an essay posted by Chemistry World on April 24, 2023, citing Flynn, noted that Wiley had convened a meeting of publishers at that book fair, invited Clarivate, the owner of major journal database Web of Science (WoS) Core Collection, and disclosed to them the problems with Hindawi journals. We do not know the outcome of this meeting, but change did occur. On the one hand, Hindawi began issuing retractions on November 16, 2022. Nevertheless, in October 2022, special issues of Hindawi journals were still being published with many compromised articles, though from December 2022 onwards, the number of papers published in special issues decreased.  

Delisting of Hindawi journals

The public information prompted journal databases to re-evaluate whether Hindawi journals should continue to be indexed. In February 2023, DOAJ (Directory of Open Access Journals) delisted thirteen Hindawi journals. Then Scopus discontinued the indexing of six Hindawi journals. On March 20, 2023, Clarivate delisted nineteen Hindawi journals from WoS Core Collection. The fact that Education Research International was delisted suggested Clarivate conducted an independent investigation, as this journal had not been mentioned in relevant sources.  

As shown in Figure 3, the actions of the publisher and journal databases did not always involve the same journals. 

Figure 3. Twenty-six problematic Hindawi journals. WoS Core Collection: The journal was delisted from WoS Core Collection in March 2023. DOAJ: The journal was delisted from DOAJ in February 2023. Scopus: The journal was delisted from Scopus in the first half of 2023. Ferguson 500+: Papers in the journal were retracted and Liz Ferguson's statement was cited in the retraction statement. Flynn 2200+: Papers in the journal were retracted by Wiley using similar statements after May 2023. 

Clarivate did not publish the reasons for the delisting of each journal, nor did they delist more Hindawi journals before the release of 2023 Journal Citation Reports on June 28, 2023. Compared to MDPI, whose mega-journal the International Journal of Environmental Research and Public Health was delisted, Wiley's public response was subdued. In a mildly worded statement on March 22, 2023, on their WeChat Official Account, Hindawi said they were "disappointed" that their journals were delisted by Clarivate but did not offer any defence. In another post on April 5, 2023, Hindawi stated they would not appeal the delisting and suggested that the authors submit their manuscripts to Wiley journals. A guest post by Flynn in the Scholarly Kitchen on April 4, 2023, mentioned that:

 “At Wiley we take full responsibility for the quality of the content we publish across our portfolio.” 

He also announced a further 1,200 retractions to be issued by Hindawi journals. Flynn disclosed to Chemistry World how they selected which publications to retract, specifically that he deployed 200 people from his editorial staff to conduct "a manual review of every single paper that we thought may have been compromised'".

Impact on authors

Many authors published in Hindawi journals because they had the cachet of being listed in scholarly databases. One author of an article published in February 2023 in Oxidative Medicine and Cellular Longevity distributed email templates she drafted to others via an instant messaging software, encouraging them to ask Hindawi to work with Clarivate to index papers with publication dates before March 19, 2023. Anonymous sources described the chaos of Hindawi's customer service in late March 2023. Many people complained that Hindawi never responded to their emails. On the other hand, one author received a response from Oxidative Medicine and Cellular Longevity (omcl@hindawi.com), even though his complaint was about an article in another journal. Some authors of accepted manuscripts complained that Hindawi delayed their requests to withdraw their submissions, and feared that manuscripts would be accidentally published. 

Other authors turned on the sleuths who had exposed paper mill activity on PubPeer, describing their activities as "social media-related PubPeer extortion". Jincheng Wang from the Nanjing Drum Tower Hospital, who had published in a compromised special issue, suggested that the intent of those who posted comments was to blackmail the authors, under the threat of translating publicly available comments into Chinese and then posting them on social media in China. He encouraged authors to report these comments to the moderators.

Impact of delisting on Hindawi’s business 

Wiley did not inform investors about the retractions in Hindawi journals in their 2nd quarter report (August 1 to October 31, 2022) published on December 7, 2022. In the 3rd quarter report (November 1, 2022, to January 31, 2023) on March 9, 2023, Napack confronted the issue head-on: 

“Upon discovery, the Wiley team responded quickly, suspending the Hindawi special issues program and fixing the source of the problem by purging the external bad actors and by implementing measures to prevent this from happening again. To date, these actions include increasing editorial controls and introducing new AI-based screening tools into the editorial process. We've also been scrubbing the archive and publicly retracting any compromised articles.”

 And, 

“We put the fixes in place. We feel very good about what we've done. We are reopening the programs. And we are moving forward to clear the backlog and drive forward with our publishing program.” 

However, the statistics on publications showed that publications in special issues continued through November 2022 to January 2023. Despite the claim that the problem had been resolved, more than 200 articles published in special issues of 34 Hindawi journals in 2023 received comments on PubPeer relating to concerns about the publishing process. The release of retraction statements was also delayed. As of July 20, 2023,  retraction statements were issued on six dates, on May 24, June 21, June 28, June 29, July 12, and July 19, with 112, 559, 514, 1, 521, and 510 retractions, respectively. The total number of retracted papers, approximately 2700 including the initial 500 from 2022, substantially exceeds the 1200 mentioned by Flynn. 

An interesting development has been the involvement of law firms who specialize in shareholder rights litigation, such as Rosen Law Firm, Kirby McInerney LLP, Schall Law Firm, Glancy Prongay & Murray LLP, the Law Offices of Frank R. Cruz and the Law Offices of Howard G. Smith. All of these firms recently advertised that they are investigating whether Wiley issued misleading information about Hindawi to the investing public. 

In the 4th quarterly report for 2023, Napack stated that they had remedied all the problems, and were ready for “revitalization” 

“As discussed in Q3, we suspended the fast-growing special issues program after identifying a research integrity issue. This issue was the result of external misconduct by non-Wiley editors and reviewers. Essentially, Wiley decided to take a short-term hit to preserve the integrity of our journals and the value of our highly respected Wiley brand. This industry-wide issue has been widely reported on, and we believe that we now have it fully remediated in Wiley.” 

In the same report, Napack was still expressing satisfaction with Hindawi's performance since the acquisition: 

“Our expectation for Hindawi was a couple-fold. One, it would accelerate our position in that market, which it has; and that it would provide significant growth, which it has, and it will provide the ability to provide significant cascade across our portfolio that we could find homes for the many hundreds of thousands of articles that we get every year that are not published. Our expectations are the same going forward. We expect that over the next 12 to 18 months, we will be fully ramping back up. So, by '25, we're back on course with our volume growth and it should drop to the bottom line at/or we're close to the margin -- very healthy margin that it always has in our -- across all of our Open Access, but certainly across the Hindawi asset. So, the -- relative to our initial expectations, this acquisition has outperformed if you just can look aside for a minute against a very short-term thing that happened to us. But we're going to lead our way -- lead the industry out of it, and we feel very, very good about the future of our overall Open Access program.” 

So the paper mill debacle, which led to thousands of fraudulent papers being published in Hindawi journals over at least two years, is described as “a very short-term thing that happened to us” and is now “fully remediated”, was blamed on “external misconduct by non-Wiley editors and reviewers”. This leaves hanging the question of how those non-Wiley editors and reviewers not only achieved powerful positions determining what was published in Hindawi journals, but also continued to do so long after attention had been drawn to the problem by sleuths.

Regaining the market? 

Wiley is a mighty, major international publisher, and they have the potential to achieve a Hindawi revitalization, if revitalization is defined as a significant rebound in the number of papers published in Hindawi journals. The question is, which niche market does Hindawi intend to regain? Are they well-positioned to do so? Do they have any appreciation of the tension between their goal of publishing as much as possible, and the reputational costs of publishing papers that are low-quality at best and fraudulent at worst?

The sleuth Parashorea tomentella described the evolution of the niche of special issues of Hindawi journals after its acquisition in China. There is a highly competitive "first-tier" niche of authors from research universities and institutions, who have many manuscripts. China also has an extended, "second-tier" niche of authors from community colleges, polytechnics, and non-teaching hospitals. There are many, many of these potential customers, but they lack manuscripts on the one hand, and desire publications for promotion on the other. Paper mills fill the need of authors in this niche to publish and the needs of publishers to make money. It would be difficult for Hindawi to regain the first-tier niche, because most authors and institutions care about the reputation of the publisher, and even if they aren’t concerned about integrity they are spooked by unprecedented large-scale retractions. 

Hindawi is, however, in a strong position to regain the second-tier niche. First, despite all the problems, they still have many journals that are recognized by the Chinese authorities (typically those listed in databases such as WoS Core Collection and Engineering Village). Wiley has close links with those who maintain databases and may have an advantage in avoiding their journals being blacklisted. Wiley's longstanding commitment and partnership with Chinese research institutes and government stakeholders has brought them particularly close to the National Science Library, Chinese Academy of Sciences (NSL/CAS), a bureaucracy that compiles a blacklist which is recognized by many Chinese institutions. On June 16, 2023, Wiley and NSL/CAS announced the establishment of a Joint Laboratory on Scientific and Technical Journal Innovation. The press release mentioned that an important topic for the joint lab is research integrity, and Liying Yang, director of journal evaluation in NSL/CAS, introduced what NSL/CAS can do, including updating their Early Warning Journal List, which aimed to target paper mills. 

For reasons unknown, NSL/CAS has been particularly kind to Hindawi journals. NSL/CAS released their controversial Early Warning Journal List in December 2020, December 2021 and January 2023. In the latest version, the Hindawi journals Biomed Research International, Complexity, Advances in Civil Engineering, Shock and Vibration, Scientific Programming and Journal of Mathematics, which had been on the blacklist, were reinstated. Other problematic Hindawi journals were never on the blacklist. In contrast, the blacklist compiled by another Chinese bureaucracy, the Institute of Scientific and Technical Information of China (ISTIC), does not show undue goodwill toward Hindawi journals. In January and February 2023, some Chinese institutions, such as Zhejiang Gongshang University and Anhui Provincial Hospital (The First Affiliated Hospital of University of Science and Technology of China), told their employees not to submit to Hindawi journals. NSL/CAS was sending a different signal from other Chinese institutions and encouraged authors to continue submitting to Hindawi journals, although this effect was offset by Clarivate's delisting of nineteen Hindawi journals three months later. 

Hindawi’s determination to retain the second-tier niche may explain why they have continued to publish their customers' manuscripts from known paper mills. For instance, an article (now retracted) was published on March 11, 2023, long after the guest editor Kaifa Zhao had been proven to be an impostor. To take another example, the Hindawi Research Integrity Team retracted nine articles from a special issue of BioMed Research International on “Minimally Invasive Treatment Protocols in Clinical Dentistry” between November 22, 2022, and February 14, 2023, but subsequently published new articles in the same special issue on topics out of scope.

 

Figure 6. A special issue (https://www.hindawi.com/journals/bmri/si/652179/ ) of BioMed Research International published articles on a topic out of scope in between retraction statements

 

Even more alarmingly, special issues of four journals (Computational and Mathematical Methods in Medicine, Journal of Healthcare Engineering, Journal of Environmental and Public Health, and Computational Intelligence and Neuroscience) continued to publish questionable articles after Hindawi announced that the journals had closed on May 2, 2023, see e.g., https://pubpeer.com/publications/1F1263D5537A0EF96588929A60D15B ). In one weird case, an article that had been accepted 604 days previously was published in Journal of Healthcare Engineering on July 7, 2023. Perhaps this manuscript had been blocked by production processes or the Hindawi Research Integrity Team, but it was eventually published after the journal was closed. 

Reasons for pessimism 

The reason I am pessimistic is that so far Wiley's proposals to improve the publishing process for Hindawi journals have focused on the use of AI-based screening tools. Wiley has not committed to hiring more editors for Hindawi journals. As the number of submissions increases, the situation will only get worse if peer-review processes hosted by guest editors are assigned to overworked in-house editors to oversee. 

Wiley still has a chance to fix things. The first thing they should do is stop manuscripts received by external bad actors from continuing to be published. The second is to issue more retractions. I'm glad to see that in June 2023, Catriona MacCallum, the director of Open Science of Hindawi shared their approach to scaling up retractions, including focusing on manipulation of the process rather than author wrongdoing. Publisher retractions are painful for the publisher, journals, and authors, but they are necessary, and there are not enough of them. The third is to investigate the internal bad actors in an open and transparent manner. I would also encourage them to recruit more in-house editors, release the identities of the external bad actors they have found, and document the details of how internal controls failed so lessons can be learned. 

Most of us value our reputation for its own sake. As Shakespeare said in Othello: 

“Good name in man and woman, dear my lord, 

Is the immediate jewel of their souls: 

Who steals my purse steals trash; ’tis something, nothing; ’twas mine, ’tis his, and has been slave to thousands; 

But he that filches from me my good name 

Robs me of that which not enriches him, 

And makes me poor indeed” 

Indeed, as the Hindawi story shows, for a commercial organization, reputation is not just a desirable feel-good factor – it has huge financial implications. If an academic publisher like Wiley becomes known for boosting their profits by publishing screeds of arrant nonsense, their bottom line will ultimately suffer. Reputable researchers will not want their name associated with a publisher who behaves this way. If Wiley are not willing to control fraud because it is the right thing to do, they should at least recognize the importance of integrity for retaining the confidence of the academic institutions on whom they depend. 

 

Footnote 

* The author declares that there is no potential conflict of interest. The author uses a pseudonym because he/she lives in an authoritarian state and fears facing unpredictable political reprisals.