Prodding the behemoth with a stick

 

Like many academics, I was interested to see an announcement on social media that a US legal firm had filed a federal antitrust lawsuit against six commercial publishers of academic journals: (1) Elsevier B.V.; (2) Wolters Kluwer N.V.; (3) John Wiley & Sons, Inc.; (4) Sage Publications, Inc.; (5) Taylor and Francis Group, Ltd.; and (6) Springer Nature AG & Co, on the grounds that "In violation of Section 1 of the Sherman Act, the Publisher Defendants conspired to unlawfully appropriate billions of dollars that would have otherwise funded scientific research".   

 

So far, so good.  I've been writing about the avaricious practices of academic publishers for over 12 years, and there's plenty of grounds for a challenge. 

 

However, when I saw the case being put forward, I was puzzled.  From my perspective, the arguments just don't stack up.  In particular, three points are emphasised in the summary (quoted verbatim here from the website): 

 

• First, an agreement to fix the price of peer review services at zero that includes an agreement to coerce scholars into providing their labor for nothing by expressly linking their unpaid labor with their ability to get their manuscripts published in the defendants’ preeminent journals.

 

But it's not true that there is an express link between peer review and publishing papers in the pre-eminent journals.  In fact, many journal editors complain that some of the most prolific authors never do any peer review - gaining an advantage by not adopting the "good citizen" behaviour of a peer reviewer.  I think this point can be rapidly thrown out.

 

• Second, the publisher defendants agreed not to compete with each other for manuscripts by requiring scholars to submit their manuscripts to only one journal at a time, which substantially reduces competition by removing incentives to review manuscripts promptly and publish meritorious research quickly. 

 

This implies that the rationale for not allowing multiple submissions is to reduce competition between publishers.  But if there were no limit on how many journals you could simultaneously submit to, then the number of submissions to each journal would grow massively, increasing the workload for editors and peer reviewers - and much of their time would be wasted. This seems like a rational requirement, not a sinister one.

 

• Third, the publisher defendants agreed to prohibit scholars from freely sharing the scientific advancements described in submitted manuscripts while those manuscripts are under peer review, a process that often takes over a year. As the complaint notes, “From the moment scholars submit manuscripts for publication, the Publisher Defendants behave as though the scientific advancements set forth in the manuscripts are their property, to be shared only if the Publisher Defendant grants permission. Moreover, when the Publisher Defendants select manuscripts for publication, the Publisher Defendants will often require scholars to sign away all intellectual property rights, in exchange for nothing. The manuscripts then become the actual property of the Publisher Defendants, and the Publisher Defendants charge the maximum the market will bear for access to that scientific knowledge.” 

Again, I would question the accuracy of this account.  For a start, in most science fields, peer review is a matter of weeks or months, not "over a year".  But also, most journals these days allow authors to post their articles as preprints, prior to, or at the point of submission. In fact, this is encouraged by many institutions, as it means that a Green Open Access version of the publication is available, even if the work is subsequently published in a pay-to-read version.  

 

In all, I am rather dismayed by this case, especially when there are very good grounds on which academic publishers can be challenged.  For instance:

 

1. Academic publishers claim to ensure quality control of what gets published, but some of them fail to do the necessary due diligence in selecting editors and reviewers, with the result that the academic literatureis flooded with weak and fraudulent material, making it difficult to distinguish valuable from pointless work, and creating an outlet for damaging material, such as pseudoscience.  This has become a growing problem with the advent of paper mills.

 

2. Many publishers are notoriously slow at responding to credible evidence of serious problems in published papers. It can take years to get studies retracted, even when they have important real world consequences.

 

3. Perhaps the only point in common with the case by Leiff Cabraser, Heimann and Bernstein concerns the issue of retention of intellectual property rights.  It is the case that publishers have traditionally required authors to sign away copyright of their works.  In the UK, at least, there has been a movement to fight this requirement, which has had some success, but undoubtedly more could be done. 

 

If I can find time I will add some references to support some of the points above - this is a hasty response to discussion taking place on social media, where many people seem to think it's great that someone is taking on the big publishers. I never thought I would find myself in a position of defending them, but I think if you are going to attack a behemoth, you need to do so with good weapons.  

 

 

Postscript

Comments on this post are welcome - there is moderation so they don't appear immediately.

 Nick Wise attempted unsuccessfully to add a comment (sorry, Blogger can be weird), providing this helpful reference on typical duration of peer review.  Very field-dependent and may be a biased sample, I suspect, but it gives us a rough idea.

 

Now you see it, now you don't: the strange world of disappearing Special Issues at MDPI

 

There is growing awareness that Special Issues have become a menace in the world of academic publishing, because they provide a convenient way for large volumes of low quality work to be published in journals that profit from a healthy article processing charge. There has been a consequent backlash against Special Issues, with various attempts to rein them in. Here I'll describe the backstory and show how such attempts are being subverted. 

Basically, when it became normative for journals to publish open access papers in exchange for an article processing charge, many publishers saw an opportunity to grow their business by expanding the number of articles they published. There was one snag: to maintain quality standards, one requires academic editors to oversee the peer review process and decide what to publish. The solution was to recruit large numbers of temporary "guest editors", each of whom could invite authors to submit to a Special Issue in their area of expertise; this cleverly solved two problems at once: it provided a way to increase the number of submissions to the journal, and it avoided overloading regular academic editors. In addition, if an eminent person could be persuaded to act as guest editor, this would encourage researchers to submit their work to a Special Issue.

Problems soon became apparent though. Dubious individuals, including those running paper mills, seized the opportunity to volunteer as guest editors, and then proceeded to fill Special Issues with papers that were at best low quality and at worse fraudulent. As described in this blogpost, the publisher Wiley, was badly hit by fallout from the Special Issues programme, with its Hindawi brand being 'sunsetted' in 2023 . In addition, the Swiss National Science Foundation declared they would not fund APCs for articles in Special Issues, on the grounds that the increase in the number of special issues was associated with shorter processing times and lower rejection rates, suggestive of rushed and superficial peer review. Other commentators noted the reputational risks of overreliance on Special Issues.

Some publishers that had adopted the same strategy for growth looked on nervously, but basically took the line that the golden goose should be tethered rather than killed, introducing various stringent conditions around how Special Issues operated. The publisher MDPI, one of those that had massive growth in Special Issues in recent years, issued detailed guidelines.

One of these concerned guest editors publishing in their own special issues. These guidelines have undergone subtle changes over time, as evidenced by these comparisons of different versions (accessed via Wayback Machine):

JUNE 2022: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited to 20%, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue.... Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

 21 JAN 2023: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue. Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

2 JAN 2024: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited to 25%, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue. Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

3 MAY 2024: The special issue may publish contributions from the Guest Editor(s), but the number of such contributions should be limited, to ensure the diversity and inclusiveness of authorship representing the research area of the Special Issue. Any article submitted by a Guest Editor will be handled by a member of the Editorial Board.

The May 2024 version of guidelines is nonspecific but problematic, because it is out of alignment with criteria for accreditation by the Directory of Open Access Journals (DOAJ) , who state "Papers submitted to a special issue by the guest editor(s) must be handled under an independent review process and make up no more than 25% of the issue's total". Most of MDPI's journals are listed on DOAJ, which is a signal of trustworthiness.

So, how well is MDPI doing in terms of the DOAJ criteria? I was first prompted to ask this question when writing about an article in a Special Issue of Journal of Personalized Medicine that claimed to "reverse autism symptoms". You can read my critique of that article here; one question it raised was how on earth did it ever get published? I noted that the paper was handled by a guest editor, Richard E. Frye, who had coauthored 7 of the 14 articles in the Special Issue. I subsequently found that between 2021 and 2024 he had published 30 articles in Journal of Personalized Medicine, most in three special issues where he was guest editor. I'm pleased to say that DOAJ have now delisted the journal from their Directory. But this raises the question of how well MDPI is regulating their guest editors to prevent them going rogue and using a Special Issue as a repository for papers by themselves and their cronies.

To check up on this, I took a look at Special Issues published in 2023-2024 in 28 other MDPI journals*, focusing particularly on those with implications for public health. What I found was concerning at four levels. 

1. Every single journal I looked at had Special Issues that broke the DOAJ rule of no more than 25% papers co-authored by guest editors (something DOAJ refer to as "endogeny").  Some of these can be found on PubPeer, flagged with the term "stuffedSI". 
2. A minority of Special Issues conformed to the description of a "successful Special Issue" envisaged by the MDPI guidelines: "Normally, a successful Special Issue consists of 10 or more papers, in addition to an editorial (optional) written by the Guest Editor(s)." For the journals I looked at around 60% of Special Issues had fewer than 10 articles. 
3. Quite often, the listed guest editors did not actually do any editing. One can check this by comparing the Action Editor listed for each article. Here's one example, where a different editor was needed for three of the nine papers to avoid conflict of interest, because they were co-authored by the guest editors;  but the guest editors are not listed as action editors for any of the other six papers in the special issue. 
   
4. As I did this analysis, I became aware that some articles changed status. For instance, Richard E. Frye, mentioned above, had additional articles in the Journal of Personalized Medicine that were originally part of a Special Issue that are now listed as just belonging to a Section. see https://pubpeer.com/publications/BA21B22CA3FED62B6D3F679978F591#1.This change was not transparent, but was evident when earlier versions of the website were accessed using the Wayback Machine. Some of these are flagged with the term "stealth correction" on PubPeer.

This final observation was particularly worrying, because it indicated that the publisher could change the Special Issue status of articles post-publication. The concern is that lack of oversight of guest editors has created a mechanism whereby authors can propose a Special Issue, get taken on as a guest editor, and then have papers accepted there (either their own, or from friends, which could include papermillers), after which the Special Issue status is removed. In fact, given the growing nervousness around Special Issues, removal of Special Issue status could be an advantage.

When I have discussed with colleagues these and other issues around MDPI practices, I find that credible researchers tell me that there are some excellent journals published by MDPI. It seems unfortunate that, in seeking rapid growth via the mechanism of Special Issues, the publisher has risked its reputation by giving editorial responsibility to numerous guest editors without adequate oversight, and encouraged quantity over quality. Furthermore, the lack of transparency demonstrated by the publisher covertly removing Special Issue status from articles by guest editors does not appear consistent with their stated commitment to ethical policies. 

 *The code for this analysis and a summary chart for the 28 journals can be found on Github.

Optimizing research integrity investigations: the need for evidence

 

An article was published last week by Caron et al (2024) entitled "The PubPeer conundrum: Administrative challenges in research misconduct proceedings". The authors present a perspective on research misconduct from a viewpoint that is not often heard: three of them are attorneys who advise higher education institutions on research misconduct matters, and the other has served as a Research Integrity Officer at a hospital. 

The authors conclude that the bar for research integrity investigations should be raised, requiring a complaint to reach a higher evidential standard in order to progress, and using a statute of limitations to provide a cutoff date beyond which older research would not usually be investigated. This amounts to saying that the current system is expensive and has bad consequences, so let's change it to do fewer investigations - this will cost less and fewer bad consequences will happen.  The tldr; version of this blogpost is that the argument fails because on the one hand the authors give no indication of the frequency of bad consequences, and on the other hand, they ignore the consequences of failing to act.

How we handle misconduct allegations can be seen as an optimization problem; to solve it, we need two things: data on frequency of different outcomes, and an evaluation of how serious different outcomes are.

We can draw an analogy with a serious medical condition that leads to a variety of symptoms, and which can only be unambiguously diagnosed by an invasive procedure which is both unpleasant and expensive. In such a case, the family doctor will base the decision whether to refer for invasive testing on the basis of information such as physical symptoms or blood test results, and refer the patient for specialist investigations only if the symptoms exceed some kind of threshold. 

The invasive procedure may confirm that the disease is really present, a true positive, or that it is absent, a false positive. Those whose symptoms do not meet a cutoff do not progress to the invasive procedure, but may nevertheless have the disease, i.e., false negatives, or they may be free from the disease, true negatives. The more lenient the cutoff, the more true positives, but the price we pay will be to increase the rate of false positives. Conversely, with a stringent cutoff, we will reduce false positives, but will also miss true cases (i.e. have more false negatives).

Optimization is not just a case of seeking to maximize correct diagnoses - it must also take into account costs and benefits of each outcome. For some common conditions, it is deemed more serious to miss a true case of disease (false negative) than to send someone for additional testing unnecessarily (false positive). Many people feel they would put up with inconvenience, embarrassment, or pain rather than miss a fatal tumour. But some well-established medical screening programmes have been queried or even abandoned on the grounds that they may do more harm than good by creating unnecessary worry or leading to unwarranted medical interventions in people who would be fine left untreated. 

So, how does this analogy relate to research misconduct? The paper by Caron et al emphasizes the two-stage nature of the procedure that is codified in the US by the Office of Science and Technology Policy (OSTP), which is mandatory for federal agencies that conduct or support research. When an allegation of research misconduct is presented to a research institution, it is rather like a patient presenting themselves to a physician: symptoms of misconduct are described, and the research integrity officers must decide whether to proceed to a full investigation - a procedure which is both costly and stressful.

Just as patients will present with symptoms that are benign or trivial, some allegations of misconduct can readily be discounted. They may concern minor matters or be obviously motivated by malice. But there comes a point when the allegations can't be dismissed without a deeper investigation - equivalent to referring the patient for specialist testing. The complaint of Caron et al is that the bar for starting an investigation is specified by the regulator, and is set too low, leading to a great deal of unnecessary investigation. They make it sound rather like the situation that arose with prostate screening in the UK: use of a rather unreliable blood test led to a situation where there was overdiagnosis and overtreatment: in other words, the false positive rate was far too high. The screening programme was eventually abandoned.

My difficulty with this argument is that at no point do Caron et al indicate what the false positive rate is for investigations of misconduct. They emphasize that the current procedures for investigation of misconduct are onerous, both on the institution and on the person under investigation. They note the considerable damage that can be done when a case proves to be a false positive, where an aura of untrustworthiness may hang around the accused, even if they are exonerated. Their conclusion is that the criteria for undertaking an investigation should be made more stringent. This would undoubtedly reduce the rate of false positives, but it would also decrease the true positive detection rate.

One rather puzzling aspect of Caron et al's paper was their focus on the post-publication peer review website PubPeer as the main source of allegations of research misconduct. The impression they gave is that PubPeer has opened the floodgates to accusation of misconduct, many of which have little substance, but which the institutions are forced to respond to because of ORI regulations. This is the opposite of what most research sleuths experience, which is that it is extremely difficult to get institutions to take reports of possible research misconduct seriously, even when the evidence looks strong.

Given these diametrically opposed perspectives, what is needed is hard data on how many reported cases of misconduct proceed to a full investigation, and how many subsequently are found to be justified. And, given the authors' focus on PubPeer, it would be good to see those numbers for allegations that are based on PubPeer comments versus other sources.

There's no doubt that the volume of commenting on PubPeer has increased, but the picture presented by Caron et al seems misleading in implying that most complaints involve concerns such as "a single instance of image duplication in a published paper". Most sleuths who regularly report on PubPeer know that such a single instance is unlikely to be taken seriously; they also know that a researcher who commits research misconduct is often a serial offender, with a pattern of problems across multiple papers. Caron et al note the difficulties that arise when concerns are raised about papers that were published many years ago, where it is unlikely that original data still exist. That is a valid point, but I'd be surprised if research integrity officers receive many allegations via PubPeer based solely on a single paper from years ago; the reason that older papers come to attention is typically because a researcher's more recent work has come into question, which triggers a sleuth to look at other cases. I accept I could be wrong, though. I tend to focus on cases where there is little doubt that misconduct has occurred, and, like many sleuths, I find it frustrating when concerns are not taken seriously, so maybe I underestimate the volume of frivolous or unfounded allegations. If Caron et al want to win me over, they'd have to provide hard data showing how much investigative time is spent on cases that end up being dismissed.

Second, and much harder to estimate, what is the false negative rate: how often are cases of misconduct missed? The authors focus on the sad plight of the falsely accused researcher but say nothing about the negative consequences when a researcher gets away with misconduct. 

Here, the medical analogy may be extended further, because in one important respect, misconduct is less like cancer and more like an infectious disease. It affects all who work with the researcher, particularly younger researchers who will be trained to turn a blind eye to inconvenient data and to "play the game" rather than doing good research. The rot spreads even further: huge amounts of research funding are wasted by others trying to build on noncredible research, and research syntheses are corrupted by the inclusion of unreliable or even fictitious findings. In some high-stakes fields, medical practice or government policy may be influenced by fraudulent work. If we simply make it harder to investigate allegations of misconduct, we run the risk of polluting academic research. And the research community at large can develop a sense of cynicism when they see fraudsters promoted and given grants while honest researchers are neglected.

So, we have to deal with the problem that, currently, fraud pays. Indeed, it is so unlikely to be detected that, for someone with a desire to succeed uncoupled from ethical scruples, it is a more sensible strategy to make up data than to collect it. Research integrity officers may worry now that they are confronted with more accusations of misconduct than they can handle, but if institutions focus on raising the bar for misconduct investigations, rather than putting resources in to tackle the problem, it will only get worse.

In the UK, universities sign up to a Concordat to Support Research Integrity which requires them to report on the number and outcome of research misconduct investigations every year. When it was first introduced, the sense was that institutions wanted to minimize the number of cases reported, as it might be a source of shame.  Now there is growing recognition that fraud is widespread, and the shame lies in failing to demonstrate a robust and efficient approach to tackling it. 

Reference

Caron, M. M., Lye, C. T., Bierer, B. E., & Barnes, M. (2024). The PubPeer conundrum: Administrative challenges in research misconduct proceedings. Accountability in Research, 1–19. https://doi.org/10.1080/08989621.2024.2390007.

My experience as a reviewer for MDPI

 

Guest post by 

René Aquarius, PhD

Department of Neurosurgery

Radboud University Medical Center, Nijmegen, The Netherlands

 

After a recent zoom-call where Dorothy and I discussed several research-related topics, she invited me to write a guest blogpost about the experience I had as a peer-reviewer for MDPI. As I think transparency in research is important, I was happy to accept this invitation.  

 

Mid November 2023 I received a request to peer-review a manuscript for a special issue on subarachnoid hemorrhage for the Journal of Clinical Medicine, published by MDPI. This blog post summarizes that process. I hope it will give some insight on the nitty-gritty of the peer-review process for MDPI.

 

I decided to review the manuscript two days after receiving the invitation and what I found was a study like many others in the field: a single-center, retrospective analysis of a clinical case series. I ended up recommending rejection of the paper two days after accepting to review. My biggest gripes were that the authors claimed that data were collected prospectively, but their protocol was registered at the very end of the period in which they included patients. In addition, I discovered some important discrepancies between protocol and the final study. Target sample size according to the protocol was 50% bigger than what was used in their study. The minimum age for patients also differed between the protocol and the manuscript. I also had problems with their statistical analysis as they used more than 20 t-tests to test variables, which creates a high probability of Type I errors. The biggest problem was the lack of a control group, which made it impossible to establish whether changes in a physiological parameter could really predict intolerance for a certain drug in a small subset of patients.

 

When filling out the reviewer form for MDPI, certain aspects struck me as peculiar. There are four options for Overall Recommendation:

• Accept in present form
• Accept after minor revision (correction to minor methodological errors and text editing)
• Reconsider after major revision (control missing in some experiments)
• Reject (article has serious flaws, additional experiments needed, research not conducted correctly)

 

Regardless of which of the last two options you select, the response is: "If we ask the authors to revise the manuscript, the revised manuscript will be sent to you for further evaluation". 

 

Although reviewer number 2 is often jokingly referred to as "the difficult one" it couldn’t be further from the truth in this case. The reviewer liked the paper and recommended accept after minor revision. So with a total of two reviews, the paper got the editorial decision of rejected, with a possibility of resubmission after extensive revisions only one day after I handed in my peer review report.

 

Revisions were quite extensive, as you will discover below, and arrived only two days after the initial rejection. I agreed to review the revised manuscript. But before I could start my review of the revision, just four days after receiving the invitation, I received a response from the editorial office that my review was no longer needed because they already had enough peer-reviewers for the manuscript. I politely ignored this request, because I wanted to know if the manuscript had improved. What happened next was quite a bit of a surprise, but not in a good way. 

 

The manuscript had indeed undergone extensive revisions. The biggest change, however, was also the biggest red flag. Without any explanation the study had lost almost 20% of its participants. An additional problem was that all the issues I had raised in my previous review report remained unaddressed. I sent my newly written feedback report the same day, exactly one week after my initial rejection.

 

When I handed in my second review report, I understood why I initially got an email that my review was not needed anymore. One peer reviewer had also rejected the manuscript and had concerns similar to mine. Two other reviewers, however, accepted the manuscript. One with minor revisions (English needed some improvement) and one in present form, so without any suggested revisions. This means that if I had followed the advice of the editorial office of MDPI, the paper would probably have been accepted in its current form. But because my vote was now also cast and the paper received two rejections, the editor couldn’t do much more than to reject the manuscript, which happened three days after I handed in my review report.  

 

Fifteen days after receiving my first invitation to review, the manuscript had already seen two full rounds of peer-review by at least four different peer-reviewers.

 

This is not where the story ends.  

 

In December, about a month later, I received an invitation to review a manuscript for the MDPI journal Geriatrics. You’ve guessed it by now: it was the same manuscript. It's reasonable to assume this was shifted internally through MDPI's transfer service, summarised in this figure.  I can only speculate as to why I was still attached to the manuscript as a peer-reviewer, but I guess somebody forgot to remove my name from it.

from: https://www.mdpi.com/authors/transfer-service

The manuscript had, again, transformed. It was now very similar to the very first version I reviewed. Almost word-for-word similar. That also meant that the number of included patients was restored to the initial number. However, the registered protocol that was previously mentioned in the methods section (which had led to some of the most difficult to refute critiques) was now completely left out. The icing on the cake was that, for a reason that was not explained, another author was added to the manuscript. There was no mention in this invitation of the previous reviews and rejections of the same manuscript.   Although one might wonder whether MDPI editors were aware of this, it would be strange if they were not, since they pride themselves on their Susy manuscript submission system where "editors can easily track concurrent and previous submissions from the same authors".

 

Because the same issues were still present in the manuscript, I rejected it for a third time on the same day I agreed to review it. In an accompanying message to the editor, I clearly articulated my problems with the manuscript and the review process.

 

The week after, I received a message that the editor had decided to withdraw the manuscript in consultation with the authors.

 

Late January 2024, the manuscript was published in the MDPI journal Medicina. I was not attached to the manuscript any more as a reviewer. There was no indication on the website of the name of the acting editor who accepted it. 

Note from Dorothy Bishop

Comments on this blog are moderated so there may be some delay before they appear, but legitimate, on-topic contributions are welcomed. We would be particularly interested to hear from anyone else who has experiences, good or bad, as a reviewer for MDPI journals.

 

Postscript by Dorothy Bishop: 19 Aug 2024 

Here's an example of a paper that was published with the reviews visible. Two were damning and one was agreeable.  https://www.mdpi.com/2079-6382/9/12/868.  Thanks to @LymeScience for drawing our attention to this, and noting the important clinical consequences when those promoting an alternative, non-evidenced treatment have a "peer-reviewed" study to refer to. 

Collapse of scientific standards at MDPI journals: a case study

 

"Here's one for you", said my husband, as he browsed the online Daily Telegraph*: "Severe autism can be reversed, groundbreaking study suggests".  My heart did not lift up at this news, which was also covered in the Daily Mail; it is a near certainty that any study making such claims is flawed. But I found it hard to believe just how flawed it turned out to be.  

 

The article, by D'Adamo et al.,  was published in a special issue of an MDPI journal, the Journal of Personalized Medicine.  The special issue "A Personalized Medicine Approach to the Diagnosis and Management of Autism Spectrum Disorder: Beyond Genetic Syndromes" appears largely to be a vehicle for papers by the guest editor Richard E. Frye, who co-authored 3/4 editorials, 3/9 articles and 1/1 review in this collection.  He was editor but not an author on the paper by D'Adamo et al, which is categorised as a "case report".

 

Essentially, this is a description of development of a pair of non-identical twin girls who were diagnosed with autism at 20 months of age, along with various gastric and motor conditions, and were subsequently subjected to a remarkable list of interventions, viz:

·      Parents worked with an autism parent coach who informed them about the link between "total allostatic load" and developmental disorders, a concept promoted by the Documenting Hope Project that one of the authors is affiliated with

·      Parents accessed resources, including free webinars, through Epidemic Answers

·      Parents took part in a parent forum called "Healing Together"

·      Applied Behavior Analysis (ABA) from 22 mo to 33 mo

·      Speech Therapy starting at 24 mo

·      Rigorous diet and nutrition intervention eliminating sources of glutamate, following Reduced Excitatory Inflammatory Diet

·      A strict gluten-free, casein-free diet that was low in sugar and additives

·      Dietary supplements, including omega-3 fatty acids, multivitamins, vitamin D, carnitine, 5-methyltetrahydrofolate and "bio-individualized homeopathic remedies"

·      Family consulted a naturopathic doctor who used IntellxxDNA genomics tool to recommend diet rich in tryptophan, vitamins B12, B6 and folate, betaine and choline, lion's mane mushroom and resolvins, as well as some dietary variants specific to each twin.

·      Neuro-sensory motor reflex integration given by an occupational therapist

·      Environmental evaluation of the home for air quality, mould and moisture, culminating in a visit by a Building Biology Environmental Consultant who identified possible water damage.

·      Cranial osteopathy, as recommended by a developmental optometrist.

 

When first diagnosed, the girls were given an assessment called the Autism Treatment Evaluation Checklist, which was repeated after 18 months. Scores declined (i.e. improved) for both girls, with one twin improving substantially, and the other less so.

 

I haven't been able to find any norms for the ATEC, but there's a bit of data on trajectories in a study by Mahapatra et al (2018) and this shows that scores normally improve between the age of 2 and 3 years.  We may also note that the ATEC is completed by a parent, and so may be influenced by parental expectations of improvement.

 

The authors conclude the study demonstrates "the clear environmental and lifestyle influences on ASD" and that there is "comparatively greater impact of these types of factors than genetics".  Neither conclusion is warranted. We can't know whether these changes would have occurred in the absence of the numerous interventions that the children were given.  If the interventions did have an effect, it would be impossible to tell which were the key ingredients, because all were given together.

 

The authors grudgingly state "...while effective at reversing ASD diagnoses, the comprehensive approach that was employed in this case may not yet be widely generalizable".  But they didn't show their approach was effective - at least not by any conventional standards of evidence-based intervention. 

 

Another limitation they noted was that this was an expensive regimen that might be out of reach for parents with limited resources.

 

Should the journalists who gave this study coverage be criticised? I think the answer is yes: science journalists can't be expected to be experts in all areas of science, but they should be aware of some basic principles, such as the need for adequate sample sizes and control groups in intervention studies, and the need to be careful in assuming causality. They can, if uncertain, ask the Science Media Centre to put them in touch with knowledgeable experts who could advise. This is particularly important for a sensitive subject matter such as autism, where publicity for unfounded interventions that portray autism as a "disease" requiring "healing" has potential to do harm.  At minimum, they might have noted the ethics statement that with only two children involved, "this is not considered a systematic investigation designed to contribute to generalizable knowledge."  So how on earth did this come to be emblazoned across national newspapers as a "groundbreaking study"?

 

Having said that, though, my strongest criticism is for the MDPI publishers, who have encouraged an explosion in "special Issues" of their journals, with scant scrutiny of the quality of published articles - each of which brings in an Article Processing Charge of CHF 2600 (around £2,200, or US $2,900). The proliferation of anecdotal reports in the literature gives ammunition to those who wish to promote all kinds of unevidenced treatments: they can point to these "peer reviewed" papers as evidence of scientific respectability.

 

In the longer term, the lax editorial standards that admit pseudoscience into the research literature will further damage the already tarnished MDPI brand, and we can only hope that papers published in the Journal of Personalized Medicine will be discounted by serious journalists. But meanwhile, every practitioner with a dubious product to sell will see this journal as a perfect outlet for their promotional material. 

 

*The headline has been modified and now reads "Autism can be reversed, scientists discover", but it still describes the study as a 'trial', which is isn't.

 

 

References

D’Adamo, C. R., Nelson, J. L., Miller, S. N., Rickert Hong, M., Lambert, E., & Tallman Ruhm, H. (2024). Reversal of autism symptoms among dizygotic twins through a personalized lifestyle and environmental modification approach: A case report and review of the literature. Journal of Personalized Medicine, 14(6), Article 6. https://doi.org/10.3390/jpm14060641

 

Mahapatra, S., Vyshedsky, D., Martinez, S., Kannel, B., Braverman, J., Edelson, S. M., & Vyshedskiy, A. (2018). Autism Treatment Evaluation Checklist (ATEC) Norms: A “Growth Chart” for ATEC score changes as a function of age. Children, 5(2), Article 2. https://doi.org/10.3390/children5020025

Whistleblowing, research misconduct, and mental health

 

Book Review

Carl Elliott. The Occasional Human Sacrifice: Medical Experimentation and the Price of Saying No.  W. W. Norton, 2024. ISBN: 978-1-324-06550-0

 

This book should be required reading for three groups of people. First, every member of university or hospital administration, and those who sit on ethics committees (Institutional Review Boards) in such institutions. Second, people interested in the psychology and sociology of moral behaviour. And finally, anyone who has experience of being a whistleblower regarding academic malpractice, and anyone who is thinking of acting in that role.

 There are many books on bioethics, and readers may be familiar with at least some of the seven cases described in The Occasional Human Sacrifice:

• The Markingson case, where a severely disturbed man was persuaded to take part in a drug trial against the wishes of his mother, leading, as she had predicted, to his death by suicide.
•  The Tuskegee study where black men were coerced to take part in a study of the consequences of untreated syphilis.
• Willowbrook School, a residential institution where children with intellectual disability were deliberately infected with hepatitis.
•  Protocol 126 at the Fred Hutchinson Cancer Center, which involved patients being given an experimental bone marrow transplant treatment that proved to be ineffective, without adequate informed consent.
• The Radiation Effects study at Cincinnati Medical Center, where cancer patients were exposed to whole body irradiation, not as treatment, but in order to study its effects. 
•  "The Unfortunate Experiment" in Dunedin, New Zealand, an observational study that involved monitoring rather than treating precancerous abnormalities of the cervix, without sufficient explanation to the woman concerned.
• The case of Paolo Macchiarini, who performed experimental stem cell transplants on the trachea in humans without having done adequate animal studies to justify the approach, with dire outcomes for most patients.

The details of each of these cases makes for sobering reading: patients who trusted their doctors to look after them were experimented on without proper consent procedures, in some cases over a period of years. Most accounts of these events focus on the doctors who allowed this to happen: what motivated them, why did they ignore conventional medical ethics, and why didn't they listen to those who expressed misgivings? Elliott's primary focus, however, is on a different aspect of these cases - the whistleblowers who drew attention to the problems, their motivations, and the impact of their whistleblowing on their lives.

Elliott speaks from personal knowledge - he was the whistleblower in the Markingson case, and his account makes for uncomfortable reading. When he drew attention to the unethical practices in his department of psychiatry he was first ignored and then ostracised. He became obsessed with the case, with a major toll on his mental health. In the concluding chapter he notes:

I am the last person to advise anyone to ignore the demands of conscience, but to encourage potential whistleblowers to speak out without noting the long odds of success and the grim personal costs would be like advising them to leap blindfolded off a cliff. (p 290).

All but one of the whistleblowers featured in this book had a particularly difficult time of it because they were speaking out about ethical abuses within their own institutions. As Elliott noted, the line between a traitor and a whistleblower is a fine one, relating to the perceived motivation of the person involved. A whistleblower may feel they are acting on the purest of motives, to do the right thing, yet others will interpret it as disloyalty to the institution, or attribute their behaviour to a sense of self-importance and moral superiority. Even when, as in several of these cases, the whistleblower is eventually vindicated, they are not thanked, there is seldom restitution for victims, and the mental scars of their battles with authority stay with them.

My own forays into challenging dodgy research are nothing like as serious as the cases covered by Elliott: no lives were at stake in the instances of fraud and malpractice that I've focused on, all of which have been outside my own department. Nevertheless, I resonated strongly with what Elliott had to say about the transformation that occurs in whistleblowers when they first try to report a problem. 

Unlike most people, who are used to the darker side of human behavior, whistleblowers are genuinely shocked when they discover dishonesty, greed, or corruption. They naively assume that if they blow the whistle, others will be just as outraged as they are. They might even imagine they'll be thanked." (p. 158).

 And

 "...if becoming a whistleblower requires a certain naivete, then surviving the experience demands a hard shell of cynicism." (p 159). 

I've definitely been on that journey, after years of observing the delays and obfuscations that routinely occur when a sleuth reports a fraudulent paper to a publisher, or when an institution manages to exonerate a researcher despite ample evidence of misconduct.

I've also been thinking a lot lately about parallels with political life. Here in the UK we've had a number of scandals where citizens were harmed, and sometimes killed, by people in power who were at best incompetent and at worst corrupt. Politicians handed lucrative contracts for personal protective equipment to their cronies who failed to deliver, leaving health professionals without protection; NHS patients were given contaminated blood products long after a problem had been identified; A horrendous fire in a tower block in West London arose after contractors ignored regulations and installed inappropriate cladding on the building - in each of these cases the stories came out, there was a public outcry, and a sense that "something must be done" - but the process gets mired in red tape and nothing actually is done, nobody gets punished, and those affected aren't compensated. This is terrible not just because of the impact on the individuals who were directly harmed, but also because it destroys people's trust in the institutions that we assume will look after us. I see the failure of institutions to deal adequately with academic misconduct as just one part of this broader picture.

So is there anything that can be done to challenge wrongdoing without crashing out over the cliff? A major point made by Elliott is that whistleblowers should not go it alone: that way madness lies. One option that he raised in a recent talk on this topic was to consult a lawyer with expertise in whistleblowing cases: they will be able to advise on the likelihood of success, and may be able to bring a complaint directly to an organisation, so the whistleblower can fly under the radar. There is also the option of going to the press, but, though Elliott describes some cases where this was effective, he notes it can be a two-edged sword, because journalists will want a good story, and it may not coincide with the whistleblower's viewpoint. In my experience, it's worth forming links with responsible journalists: institutions who have been inert for years may spring into action when confronted with the possibility of adverse publicity. Perhaps the most important point is that you need to have allies. They can restore confidence that there are other people who share one's values, who appreciate what one is doing and provide moral support. Although Elliott doesn't say this, I think the internet has made it easier to find one's tribe in this regard.

Where, perhaps, I take issue with Elliott is in the implication that many people aren't troubled by the issues that worry whistleblowers. When I give talks about reproducible research practices, I reliably get a question along the lines of "What advice can you give an early-career researcher whose supervisor is encouraging them to adopt questionable research practices?" The questioner clearly finds it troublesome but is being told that their career is at risk if they don't cut corners, and that "everyone is doing it". I reply that this is an issue that has serious consequences for mental health: doing what you know to be wrong causes internal conflict that can eat away at you, leading to a clash between your model of a good person and your current behaviour.

For this reason I advise those starting out in research to read this piece by Sarah Kendzior. Her advice is intended to help people survive under the threat of authoritarianism, but many of the points apply to academic survival in an age of cynicism:

Write down what you value; what standards you hold for yourself and for others.... Write a list of things you would never do. .... 

But most of all, never lose sight of who you are and what you value. If you find yourself doing something that feels questionable or wrong a few months or years from now, find that essay you wrote on who you are and read it. Ask if that version of yourself would have done the same thing.

 And if the answer is no? Don’t do it.

 

 

Does Elsevier's negligence pose a risk to public health?

 

Yesterday, Retraction Watch published a piece about a notorious 2020 article by Gautret et al that had promoted the idea that hydroxychloroquine could treat COVID-19. Despite numerous concerns, the article has not been retracted from the International Journal for Antimicrobial Agents, a journal that is co-owned by the publisher, Elsevier, and The International Society of Antimicrobial Chemotherapy (ISAC).

On June 3rd, a group of concerned scientists wrote to Elsevier asking for the paper to be retracted in light of the evidence that the paper was flawed and had serious consequences for public health. The fact that the article pops up in the Web of Science database with a trophy symbol denoting it as a 'highly cited paper' just adds to concerns that it is still being taken seriously, giving credence to those who continue to promote an ineffective drug, hydroxychloroquine, to treat COVID-19.

Elsevier's response? They are "reopening the investigation" into the paper. This is the investigation that previously decided nothing needed to be done, despite a damning report by a group of respected experts. In their reply to the letter they stated:

"We cannot currently provide further detail as to the particulars of this investigation as it remains ongoing, and is necessarily confidential while we allow time to discuss our concerns with the authors as part of due process and in line with COPE recommendations. I will however be more then glad to keep you appraised of any final decisions in due course."

As one experienced in these communications, I can confirm that this is the kind of reply you get by a publisher who is kicking the issue into the long grass in the hope that you will forget about it and go away. The wonderful phrase "it remains ongoing" typically refers to something that goes on, and on, and on, and "due course" can take years.

A pertinent question is what COPE recommendations are being followed: there are a whole range of flowcharts provided by COPE for different circumstances. The most relevant in this case would appear to be this one on "Concerns about risk in published data".

Early on in the flowchart, the decision tree asks "Based on the initial assessment, should the dataset be removed or restricted during the investigation to mitigate potential risk?" If the answer is YES, then the appropriate action is "Remove public access to the dataset while following up on the concerns."

Given that the article is still freely available on the web, we have to ask ourselves, is Elsevier being negligent here? Have they decided that there is no risk, despite the evidence that we and others have provided that public health is endangered by publicising misleading data on an ineffective drug? Or do they agree that there is risk, but deem it inconvenient for them to take action?

Are commitments to open data policies worth the paper they are written on?

 

As Betteridge's law of headlines states: "Any headline that ends in a question mark can be answered by the word no."  So you know where I am going with this.  

 

I'm a longstanding fan of open data - in fact, I first blogged about this back in 2015. So I've been gratified to see the needle shift on this, in the sense that over the past decade, in a rush to present themselves as good guys, various institutions and publishers have published policies supporting open data. The problem is that when you actually ask them to implement those policies, they back down.   

 

I discussed arguments for and against data-sharing in a Commentary article in 2016. I divided the issues according to whether they focused on the impact of data-sharing on researchers or on research participants. Table 1 from that article, entitled "Conflict between interests of researchers and advancement of science" is reproduced here:

 

Argument	Counter-argument
1. Lack of time to curate data.	Unless adequately curated, data will over time become unusable, including by the original researcher.
2. Personal investment—reluctance to give data to freeloaders.	Reuse of data increases its value and the researcher benefits from additional citations. There is also an ethical case for maximizing use of data obtained via public funding.
3. Concerns about being scooped before the analysis is complete.	This is a common concern though there are few attested cases. A time-limited period of privileged use by the study team can be specified to avoid scooping.
4. Fear of errors being found in the data.	Culture change is needed to recognize errors are inevitable in any large dataset and should not be a reason for reputational damage. Data-sharing allows errors to be found and corrected.

 

I then went on to discuss two other concerns which focused on implications of data-sharing for human participants, viz:

5.  Ethical concerns about confidentiality of personal data, especially in the context of clinical research

6.  Possibility that others with a different agenda may misuse the data, e.g. perform selective analyses that misrepresent the findings.

 

These last two issues raise complex concerns and there's plenty to discuss on how address them, but I'll put that to one side for now, as the case I want to comment on concerns a simple dataset where there is limited scope for secondary analyses and where no human participants are involved.

 

My interest was piqued by comments on PubPeer about a paper entitled "Magnetic field screening in hydrogen-rich high-temperature superconductors ".  The thread on PubPeer starts with this extraordinary comment by J. E. Hirsch:

 

I requested the underlying data for Figs. 3a, 3e, 3b, 3f of this paper on Jan 11, 2023. This is because the published data for Figs. 3a and 3e, as well as for Figs. 3b and 3f, are nominally the same but incompatible with each other, and I would like to understand why that is. I asked the authors to explain, but they did not provide an explanation. Neither did they supply the data. The journal told me that it had received the data from the authors but will not share them with me because they are "confidential". I requested that the journal posts an Editor Note informing readers that data are unavailable to readers. The journal responded that because data were share with editors they "cannot write an editorial note on the published article stating the data is unavailable as this would be factually incorrect".

 

Pseudonymous commenter Orchestes quercus drew attention to the Data Availability statement in the article: "The data that support the findings of this study are available from the corresponding authors upon reasonable request".

 

J. E. Hirsch then added a further comment: 

 

The underlying data are still not available, the editor says the author deems the request "unreasonable" but it cannot divulge the reasoning behind it, nor can the journal publish an editor note that there are restrictions on data availability because the data were provided to the journal.  Springer Nature's Research Integrity Director wrote to me in September 2023 that "we recognize the right of the authors to not share the data with you, in line with the authors’ chosen data availability statement", and that "As Springer Nature considers the correspondence with the authors confidential, we cannot share with you any further details.

 

Now, I know nothing whatsoever about superconductors or J. E. Hirsch, but I think the editors, publisher and the authors are making themselves look very silly, and indeed suspicious, by refusing to share the data.  They can't plead patient confidentiality or ethical restrictions - it seems they are just refusing to comply because they don't want to.  

 

To up the ante, Orchestes quercus extracted data from the figures and did further analyses, which confirmed that J. E. Hirsch had a point - the data did not appear to be internally consistent.

 

Meanwhile, I had joined the PubPeer thread, pointing out

 

The authors and editor appear to be in breach of the policy of Nature Portfolio journals, stated here:
https://www.nature.com/nature-portfolio/editorial-policies/reporting-standards, viz:

An inherent principle of publication is that others should be able to replicate and build upon the authors' published claims. A condition of publication in a Nature Portfolio journal is that authors are required to make materials, data, code, and associated protocols promptly available to readers without undue qualifications. Any restrictions on the availability of materials or information must be disclosed to the editors at the time of submission. Any restrictions must also be disclosed in the submitted manuscript.

After publication, readers who encounter refusal by the authors to comply with these policies should contact the chief editor of the journal. In cases where editors are unable to resolve a complaint, the journal may refer the matter to the authors' funding institution and/or publish a formal statement of correction, attached online to the publication, stating that readers have been unable to obtain necessary materials to replicate the findings.

I also noted that two of the authors are based at a Max Planck Institute. The Max Planck Gesellschaft is a signatory to the BerlinDeclaration on Open Access to Knowledge in the Sciences and Humanities.  On the website it states:

the Max Planck Society (MPG) is committed to the goal of providing free and open access to all publications and data from scholarly research (my emphasis).

 

Well, the redoubtable J. E. Hirsch had already thought of that, and in a subsequent PubPeer comment made public various exchanges he had had with luminaries from the Max Planck Institutes.

Jos Lelieveld, director of the Max Planck Institute for Chemistry responded to my request for data with:

"Our institute fully supports the work of Mikhail Eremets and his team.”

Claudia Felser, Vice President of the MPG, responded to my request for data with:

"First of all, may I point out that the data on which the publication is based are available at the Institute. At the same time, I can confirm that, in response to your inquiry, it has been verified that the data support the results presented in the publication.”

and

"I do not consider myself responsible for your request for access to the raw data.”

and

"The President has asked me, as the responsible Vice-President of the Section, to give you a final answer on this matter….We had already informed you that Mr Eremets had expressed concerns about making his research data available. We believe that Mr Eremets has sufficiently justified these concerns. Given that the editors of Nature Communications, and therefore independent third parties, have also considered your concerns and have not granted you access to the data, we see no need to appoint a further panel of external experts to assess the situation.”

and

"In the absence of concrete evidence of misconduct, we urge you, in the spirit of fair science, to refrain from making allegations that the authors have transformed and manipulated the measured data”

Patrick Cramer, President of the MPG, did not respond to my emails.

 

All I can say to the Max Planck Gesellschaft is that this is not a good look. Hirsch has noted an inconsistency in the published figures.  This has been confirmed by another reader and needs to be explained. The longer people dig in defensively, attacking the person making the request rather than just showing the raw data, the more it looks as if something fishy is going on here.

 

Why am I so hung up on data-sharing? The reason is simple. The more I share my own data, or use data shared by others, the more I appreciate the value of doing so. Errors are ubiquitous, even when researchers are careful, but we'll never know about them if data are locked away.

 

Furthermore, it is a sad reality that fraudulent papers are on the rise, and open data is one way of defending against them. It's not a perfect defence: people can invent raw data as well as summary data, but realistic data are not so easy to fake, and requiring open data would slow down the fraudsters and make them easier to catch.

 

Having said that, asking for data is not tantamount to accusing researchers of fraud: it should be accepted as normal scientific practice to make data available in order that others can check the reproducibility of findings. If someone treats such a request as an accusation, or deems it "unreasonable", then I'm afraid it just makes me suspicious.  

 

And if organisations like Springer Nature and Max Planck Gesellschaft won't back up their policies with action, then I think they should delete them from their websites. They are presenting themselves as champions of open, reproducible science, while acting as defenders of non-transparent, secret practices. As we say in the UK, fine words butter no parsnips.   

 

 P.S. 27th May:  A comprehensive account of the superconductivity affair has just appeared on the website For Better Science.  This suggests things are even worse than I thought.   

 

In addition, you can see Jorge Hirsch explain his arduous journey in attempting to access the data here.    

 

NOTE ON COMMENTS: Many thanks to those who have commented. Comments are moderated to prevent spam, so there is a delay before they appear, but I will accept on-topic comments in due course.