Now you see it, now you don't: the strange world of disappearing Special Issues at MDPI

 

There is growing awareness that Special Issues have become a menace in the world of academic publishing, because they provide a convenient way for large volumes of low quality work to be published in journals that profit from a healthy article processing charge. There has been a consequent backlash against Special Issues, with various attempts to rein them in. Here I'll describe the backstory and show how such attempts are being subverted. 

Basically, when it became normative for journals to publish open access papers in exchange for an article processing charge, many publishers saw an opportunity to grow their business by expanding the number of articles they published. There was one snag: to maintain quality standards, one requires academic editors to oversee the peer review process and decide what to publish. The solution was to recruit large numbers of temporary "guest editors", each of whom could invite authors to submit to a Special Issue in their area of expertise; this cleverly solved two problems at once: it provided a way to increase the number of submissions to the journal, and it avoided overloading regular academic editors. In addition, if an eminent person could be persuaded to act as guest editor, this would encourage researchers to submit their work to a Special Issue.

Problems soon became apparent though. Dubious individuals, including those running paper mills, seized the opportunity to volunteer as guest editors, and then proceeded to fill Special Issues with papers that were at best low quality and at worse fraudulent. As described in this blogpost, the publisher Wiley, was badly hit by fallout from the Special Issues programme, with its Hindawi brand being 'sunsetted' in 2023 . In addition, the Swiss National Science Foundation declared they would not fund APCs for articles in Special Issues, on the grounds that the increase in the number of special issues was associated with shorter processing times and lower rejection rates, suggestive of rushed and superficial peer review. Other commentators noted the reputational risks of overreliance on Special Issues.

Some publishers that had adopted the same strategy for growth looked on nervously, but basically took the line that the golden goose should be tethered rather than killed, introducing various stringent conditions around how Special Issues operated. The publisher MDPI, one of those that had massive growth in Special Issues in recent years, issued detailed guidelines.

One of these concerned guest editors publishing in their own special issues. These guidelines have undergone subtle changes over time, as evidenced by these comparisons of different versions (accessed via Wayback Machine):

JUNE 2022: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited to 20%, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue.... Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

 21 JAN 2023: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue. Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

2 JAN 2024: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited to 25%, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue. Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

3 MAY 2024: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue. Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

The May 2024 version of guidelines is nonspecific but problematic, because it is out of alignment with criteria for accreditation by the Directory of Open Access Journals (DOAJ) , who state "Papers submitted to a special issue by the guest editor(s) must be handled under an independent review process and make up no more than 25% of the issue's total". Most of MDPI's journals are listed on DOAJ, which is a signal of trustworthiness.

So, how well is MDPI doing in terms of the DOAJ criteria? I was first prompted to ask this question when writing about an article in a Special Issue of Journal of Personalized Medicine that claimed to "reverse autism symptoms". You can read my critique of that article here; one question it raised was how on earth did it ever get published? I noted that the paper was handled by a guest editor, Richard E. Frye, who had coauthored 7 of the 14 articles in the Special Issue. I subsequently found that between 2021 and 2024 he had published 30 articles in Journal of Personalized Medicine, most in three special issues where he was guest editor. I'm pleased to say that DOAJ have now delisted the journal from their Directory. But this raises the question of how well MDPI is regulating their guest editors to prevent them going rogue and using a Special Issue as a repository for papers by themselves and their cronies.

To check up on this, I took a look at Special Issues published in 2023-2024 in 28 other MDPI journals*, focusing particularly on those with implications for public health. What I found was concerning at four levels. 

1. Every single journal I looked at had Special Issues that broke the DOAJ rule of no more than 25% papers co-authored by guest editors (something DOAJ refer to as "endogeny").  Some of these can be found on PubPeer, flagged with the term "stuffedSI". 
2. A minority of Special Issues conformed to the description of a "successful Special Issue" envisaged by the MDPI guidelines: "Normally, a successful Special Issue consists of 10 or more papers, in addition to an editorial (optional) written by the Guest Editor(s)." For the journals I looked at around 60% of Special Issues had fewer than 10 articles. 
3. Quite often, the listed guest editors did not actually do any editing. One can check this by comparing the Action Editor listed for each article. Here's one example, where a different editor was needed for three of the nine papers to avoid conflict of interest, because they were co-authored by the guest editors;  but the guest editors are not listed as action editors for any of the other six papers in the special issue. 
   
4. As I did this analysis, I became aware that some articles changed status. For instance, Richard E. Frye, mentioned above, had additional articles in the Journal of Personalized Medicine that were originally part of a Special Issue that are now listed as just belonging to a Section. see https://pubpeer.com/publications/BA21B22CA3FED62B6D3F679978F591#1.This change was not transparent, but was evident when earlier versions of the website were accessed using the Wayback Machine. Some of these are flagged with the term "stealth correction" on PubPeer.

This final observation was particularly worrying, because it indicated that the publisher could change the Special Issue status of articles post-publication. The concern is that lack of oversight of guest editors has created a mechanism whereby authors can propose a Special Issue, get taken on as a guest editor, and then have papers accepted there (either their own, or from friends, which could include papermillers), after which the Special Issue status is removed. In fact, given the growing nervousness around Special Issues, removal of Special Issue status could be an advantage.

When I have discussed with colleagues these and other issues around MDPI practices, I find that credible researchers tell me that there are some excellent journals published by MDPI. It seems unfortunate that, in seeking rapid growth via the mechanism of Special Issues, the publisher has risked its reputation by giving editorial responsibility to numerous guest editors without adequate oversight, and encouraged quantity over quality. Furthermore, the lack of transparency demonstrated by the publisher covertly removing Special Issue status from articles by guest editors does not appear consistent with their stated commitment to ethical policies. 

 *The code for this analysis and a summary chart for the 28 journals can be found on Github.

So do we need editors?

It’s been an interesting week in world politics, and I’ve been distracting myself by pondering the role of academic editors. The week kicked off with a rejection of a preprint written with co-author Anna Abalkina, who is an expert sleuth who tracks down academic paper mills – organisations that will sell you a fake publication in an academic journal. Our paper describes a paper mill that had placed six papers in the Journal of Community Psychology, a journal which celebrated its 50th anniversary in 2021. We had expected rejection, as we submitted the paper to the Journal of Community Psychology, as a kind of stress test to see whether the editor, Michael B. Blank, actually reads papers that he accepts for the journal. I had started to wonder, because you can read his decision letters on Publons, and they are identical for every article he accepts. I suspected he may be an instance of Editoris Machina, or automaton, one who just delegates editorial work to an underling, waits until reviewer reports converge on a recommendation, and then accepts or rejects accordingly without actually reading the paper. I was wrong, though. He did read our paper, and rejected it with the comment that it was a superficial analysis of six papers. We immediately posted it as a preprint and plan to publish it elsewhere.

Although I was quite amused by all of this, it has a serious side. As we note in our preprint, when paper mills succeed in breaching the defences of a journal, this is not a victimless crime. First, it gives competitive advantage to the authors who paid the paper mill – they do this in order to have a respectable-looking publication that will help their career. I used to think this was a minor benefit, but when you consider that the paper mills can also ensure that the papers they place are heavily cited, you start to realise that authors can edge ahead on conventional indicators of academic prestige, while their more honest peers trail behind. The second set of victims are those who publish in the journal in good faith. Once its reputation is damaged by the evidence that there is no quality control, then all papers appearing in the journal are tainted by association. The third set of victims are busy academics who are trying to read and integrate the literature, who can get tangled up in the weeds as they try to navigate between useful and useless information. And finally, we need to be concerned about the cynicism induces in the general public when they realise that for some authors and editors, the whole business of academic publishing is a game, which is won not by doing good science, but by paying someone to pretend you have done so.

Earlier this week I shared my thoughts on the importance of ensuring that we have some kind of quality control over journal editors. They are, after all, the gatekeepers of science. When I wrote my taxonomy of journal editors back in 2010, I was already concerned at the times I had to deal with editors who were lazy or superficial in their responses to authors. I had not experienced ‘hands off’ editors in the early days of my research career, and I wondered how far this was a systematic change over time, or whether it was related to subject area. In the 1970s and 1980s, I mostly published in journals that dealt with psychology and/or language, and the editors were almost always heavily engaged with the paper, adding their own comments and suggesting how reviewer comments might be addressed. That’s how I understood the job when I myself was an editor. But when I moved to publishing work in journals that were more biological (genetics, neuroscience) things seemed different, and it was not uncommon to find editors who really did nothing more than collate peer reviews.

The next change I experienced was when, as a Wellcome-funded researcher, I started to publish in Wellcome Open Research (WOR), which adopts a very different publication model, based on that initiated by F1000. In this model, there is no academic editor. Instead, the journal employs staff who check that the paper complies with rigorous criteria: the proposed peer reviewers much have a track record of publishing and be clear from conflict of interest. Data and other materials must be openly available so that the work can be reproduced. And the peer review is published. The work is listed on PubMed if and when peer reviewers agree that it meets a quality threshold: otherwise the work remains visible but with status shown as not approved by peer review. 

The F1000/WOR model shows that editors are not needed, but I generally prefer to publish in journals that do have academic editors – provided the editor is engaged and does their job properly. My papers have benefitted from input from a wise and experienced editor on many occasions. In a specialist journal, such an editor will also know who are the best reviewers – those who have the expertise to give a detailed and fair appraisal of the work. However, in the absence of an engaged editor, I prefer the F1000/WOR model, where at least everything is transparent. The worst of all possible worlds is when you have an editor who doesn’t do more than collate peer reviews, but where everything is hidden: the outside world cannot know who the editor was, how decisions were made, who did the reviews, and what they said. Sadly, this latter situation seems to be pretty common, especially in the more biological realms of science. To test my intuitions, I ran a little Twitter poll for different disciplines, asking, for instance: 

 

 

 Results are below

% respondents stating Not Read, Read Superfially, or Read in Depth

 

Such polls of course have to be taken with a pinch of salt, as the respondents are self-selected, and the poll allows only very brief questions with no nuance. It is clear that within any one discipline, there is wide variability in editorial engagement. Nevertheless, I find it a matter of concern that in all areas, some respondents had experienced a journal editor who did not appear to have read the paper they had accepted, and in areas of biomedicine, neuroscience, and genetics, and also in mega journals, this was as high as around 25-33%

So my conclusion is that it is not surprising that we are seeing phenomena like paper mills, because the gatekeepers of the publication process are not doing their job. The solution would be either to change the culture for editors, or, where that is not feasible, to accept that we can do without editors. But if we go down that route, we should move to a model such as F1000 with much greater quality control over reviewers and COI, and much more openness and transparency.

 As usual comments are welcome: if you have trouble getting past comment moderation, please email me.

My response to the EPA's 'Strengthening Transparency in Regulatory Science'

Incredible things have happened at the US Environmental Protection Agency since Donald Trump was elected. The agency is responsible for creating standards and laws that promote the health of individuals and the environment. During previous administrations it has overseen laws concerned with controlling pollution and regulating carbon emissions. Now, under Administrator Scott Pruitt, the voice of industry and climate scepticism is in the ascendant. 

A new rule that purports to 'Strengthen Transparency in Regulatory Science' has now been proposed - ironically, at a time when the EPA is being accused of a culture of secrecy regarding its own inner workings. Anyone can comment on the rule here: I have done so, but my comment appears to be in moderation, so I am posting it here.

Dear Mr Pruitt

re: Regulatory Science- Docket ID No. EPA-HQ-OA-2018-0259

The proposed rule, ‘Strengthening transparency in regulatory science’ brings together two strands of contemporary scientific activity. On the one hand, there is a trend to make policy more evidence-based and transparent. On the other hand, there has, over the past decade, been growing awareness of problems with how science is being done, leading to research that is not always reproducible (the same results achieved by re-analysis of the data) or replicable (similar results when an experiment is repeated). The proposed rule by the Environmental Protection Agency (EPA) brings these two strands together by proposing that policy should only be based on research that has openly available public data. While this may on the surface sound like a progressive way of integrating these two strands, it rests on an over-simplified view of how science works and has considerable potential for doing harm.

I am writing in a personal capacity, as someone at the forefront of moves to improve reproducibility and replication of science in the UK. I chaired a symposium at the Academy of Medical Sciences on this topic in 2015; this was jointly organised with UK major funders: Wellcome Trust, Medical Research Council and Biotechnology and Biological Science Research Council (https://acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research). I am involved in training early career researchers in methods to improve reproducibility, and I am a co-author of Munafò, M. R et al  (2017). A manifesto for reproducible science. Nature Human Behavior, 1(1: 0021). doi:10.1038/s41562-016-0021. I would welcome any move by the US Government that would strengthen research by encouraging adoption of methods to improve science, including making analysis scripts and data open when this is not in conflict with legal/ethical issues. Unfortunately, this proposal will not do that. Instead, it will weaken science by drawing a line in the sand that effectively disregards scientific discoveries prior to the first part of the 21st century when the importance of open data started to be increasingly recognised.

The proposal ignores a key point about how scientific research works: the time scale. Most studies that would be relevant to EPA take years to do, and even longer to filter through to affect policy. Consequences for people and the planet that are relevant to environmental protection are often not immediately obvious: if they were, we would not need research. Recognition, for instance, of the dangers of asbestos, took years because the impacts on health were not immediate. Work demonstrating the connection occurred many years ago and I doubt that the data are anywhere openly available, yet the EPA’s proposed rule would imply that it could be disregarded. Similarly, I doubt there is open data demonstrating the impact of lead in paint or exhaust fumes, or of pesticides such as DDT: does this mean that manufacturers would be free to reintroduce these?

A second point is that scientific advances never depend on a single study: having open scripts and data is one way of improving our ability to check findings, but it is a relatively recent development, and it is certainly not the only way to validate science. The growth of knowledge has always depended on converging evidence from different sources, replications by different scientists and theoretical understanding of mechanisms. Scientific facts become established when the evidence is overwhelming. The EPA proposal would throw out the mass of accumulated scientific evidence from the past, when open practices were not customary – and indeed often not practical before computers for big data were available.

Contemporary scientific research is far from perfect, but the solution is not to ignore it, but to take steps to improve it and to educate policy-makers in how to identify strong science; government needs advisors who have scientific expertise and no conflict of interest, who can integrate existing evidence with policy implications. The ‘Strengthening Transparency’ proposal is short-sighted and dangerous and appears to have been developed by people with little understanding of science. It puts citizens at risk of significant damage – both to health and prosperity -  and it will make the US look scientifically illiterate to the rest of the world.

Yours sincerely

D. V. M. Bishop FMedSci, FBA, FRS