Collapse of scientific standards at MDPI journals: a case study

 

"Here's one for you", said my husband, as he browsed the online Daily Telegraph*: "Severe autism can be reversed, groundbreaking study suggests".  My heart did not lift up at this news, which was also covered in the Daily Mail; it is a near certainty that any study making such claims is flawed. But I found it hard to believe just how flawed it turned out to be.  

 

The article, by D'Adamo et al.,  was published in a special issue of an MDPI journal, the Journal of Personalized Medicine.  The special issue "A Personalized Medicine Approach to the Diagnosis and Management of Autism Spectrum Disorder: Beyond Genetic Syndromes" appears largely to be a vehicle for papers by the guest editor Richard E. Frye, who co-authored 3/4 editorials, 3/9 articles and 1/1 review in this collection.  He was editor but not an author on the paper by D'Adamo et al, which is categorised as a "case report".

 

Essentially, this is a description of development of a pair of non-identical twin girls who were diagnosed with autism at 20 months of age, along with various gastric and motor conditions, and were subsequently subjected to a remarkable list of interventions, viz:

·      Parents worked with an autism parent coach who informed them about the link between "total allostatic load" and developmental disorders, a concept promoted by the Documenting Hope Project that one of the authors is affiliated with

·      Parents accessed resources, including free webinars, through Epidemic Answers

·      Parents took part in a parent forum called "Healing Together"

·      Applied Behavior Analysis (ABA) from 22 mo to 33 mo

·      Speech Therapy starting at 24 mo

·      Rigorous diet and nutrition intervention eliminating sources of glutamate, following Reduced Excitatory Inflammatory Diet

·      A strict gluten-free, casein-free diet that was low in sugar and additives

·      Dietary supplements, including omega-3 fatty acids, multivitamins, vitamin D, carnitine, 5-methyltetrahydrofolate and "bio-individualized homeopathic remedies"

·      Family consulted a naturopathic doctor who used IntellxxDNA genomics tool to recommend diet rich in tryptophan, vitamins B12, B6 and folate, betaine and choline, lion's mane mushroom and resolvins, as well as some dietary variants specific to each twin.

·      Neuro-sensory motor reflex integration given by an occupational therapist

·      Environmental evaluation of the home for air quality, mould and moisture, culminating in a visit by a Building Biology Environmental Consultant who identified possible water damage.

·      Cranial osteopathy, as recommended by a developmental optometrist.

 

When first diagnosed, the girls were given an assessment called the Autism Treatment Evaluation Checklist, which was repeated after 18 months. Scores declined (i.e. improved) for both girls, with one twin improving substantially, and the other less so.

 

I haven't been able to find any norms for the ATEC, but there's a bit of data on trajectories in a study by Mahapatra et al (2018) and this shows that scores normally improve between the age of 2 and 3 years.  We may also note that the ATEC is completed by a parent, and so may be influenced by parental expectations of improvement.

 

The authors conclude the study demonstrates "the clear environmental and lifestyle influences on ASD" and that there is "comparatively greater impact of these types of factors than genetics".  Neither conclusion is warranted. We can't know whether these changes would have occurred in the absence of the numerous interventions that the children were given.  If the interventions did have an effect, it would be impossible to tell which were the key ingredients, because all were given together.

 

The authors grudgingly state "...while effective at reversing ASD diagnoses, the comprehensive approach that was employed in this case may not yet be widely generalizable".  But they didn't show their approach was effective - at least not by any conventional standards of evidence-based intervention. 

 

Another limitation they noted was that this was an expensive regimen that might be out of reach for parents with limited resources.

 

Should the journalists who gave this study coverage be criticised? I think the answer is yes: science journalists can't be expected to be experts in all areas of science, but they should be aware of some basic principles, such as the need for adequate sample sizes and control groups in intervention studies, and the need to be careful in assuming causality. They can, if uncertain, ask the Science Media Centre to put them in touch with knowledgeable experts who could advise. This is particularly important for a sensitive subject matter such as autism, where publicity for unfounded interventions that portray autism as a "disease" requiring "healing" has potential to do harm.  At minimum, they might have noted the ethics statement that with only two children involved, "this is not considered a systematic investigation designed to contribute to generalizable knowledge."  So how on earth did this come to be emblazoned across national newspapers as a "groundbreaking study"?

 

Having said that, though, my strongest criticism is for the MDPI publishers, who have encouraged an explosion in "special Issues" of their journals, with scant scrutiny of the quality of published articles - each of which brings in an Article Processing Charge of CHF 2600 (around £2,200, or US $2,900). The proliferation of anecdotal reports in the literature gives ammunition to those who wish to promote all kinds of unevidenced treatments: they can point to these "peer reviewed" papers as evidence of scientific respectability.

 

In the longer term, the lax editorial standards that admit pseudoscience into the research literature will further damage the already tarnished MDPI brand, and we can only hope that papers published in the Journal of Personalized Medicine will be discounted by serious journalists. But meanwhile, every practitioner with a dubious product to sell will see this journal as a perfect outlet for their promotional material. 

 

*The headline has been modified and now reads "Autism can be reversed, scientists discover", but it still describes the study as a 'trial', which is isn't.

 

 

References

D’Adamo, C. R., Nelson, J. L., Miller, S. N., Rickert Hong, M., Lambert, E., & Tallman Ruhm, H. (2024). Reversal of autism symptoms among dizygotic twins through a personalized lifestyle and environmental modification approach: A case report and review of the literature. Journal of Personalized Medicine, 14(6), Article 6. https://doi.org/10.3390/jpm14060641

 

Mahapatra, S., Vyshedsky, D., Martinez, S., Kannel, B., Braverman, J., Edelson, S. M., & Vyshedskiy, A. (2018). Autism Treatment Evaluation Checklist (ATEC) Norms: A “Growth Chart” for ATEC score changes as a function of age. Children, 5(2), Article 2. https://doi.org/10.3390/children5020025

Whistleblowing, research misconduct, and mental health

 

Book Review

Carl Elliott. The Occasional Human Sacrifice: Medical Experimentation and the Price of Saying No.  W. W. Norton, 2024. ISBN: 978-1-324-06550-0

 

This book should be required reading for three groups of people. First, every member of university or hospital administration, and those who sit on ethics committees (Institutional Review Boards) in such institutions. Second, people interested in the psychology and sociology of moral behaviour. And finally, anyone who has experience of being a whistleblower regarding academic malpractice, and anyone who is thinking of acting in that role.

 There are many books on bioethics, and readers may be familiar with at least some of the seven cases described in The Occasional Human Sacrifice:

• The Markingson case, where a severely disturbed man was persuaded to take part in a drug trial against the wishes of his mother, leading, as she had predicted, to his death by suicide.
•  The Tuskegee study where black men were coerced to take part in a study of the consequences of untreated syphilis.
• Willowbrook School, a residential institution where children with intellectual disability were deliberately infected with hepatitis.
•  Protocol 126 at the Fred Hutchinson Cancer Center, which involved patients being given an experimental bone marrow transplant treatment that proved to be ineffective, without adequate informed consent.
• The Radiation Effects study at Cincinnati Medical Center, where cancer patients were exposed to whole body irradiation, not as treatment, but in order to study its effects. 
•  "The Unfortunate Experiment" in Dunedin, New Zealand, an observational study that involved monitoring rather than treating precancerous abnormalities of the cervix, without sufficient explanation to the woman concerned.
• The case of Paolo Macchiarini, who performed experimental stem cell transplants on the trachea in humans without having done adequate animal studies to justify the approach, with dire outcomes for most patients.

The details of each of these cases makes for sobering reading: patients who trusted their doctors to look after them were experimented on without proper consent procedures, in some cases over a period of years. Most accounts of these events focus on the doctors who allowed this to happen: what motivated them, why did they ignore conventional medical ethics, and why didn't they listen to those who expressed misgivings? Elliott's primary focus, however, is on a different aspect of these cases - the whistleblowers who drew attention to the problems, their motivations, and the impact of their whistleblowing on their lives.

Elliott speaks from personal knowledge - he was the whistleblower in the Markingson case, and his account makes for uncomfortable reading. When he drew attention to the unethical practices in his department of psychiatry he was first ignored and then ostracised. He became obsessed with the case, with a major toll on his mental health. In the concluding chapter he notes:

I am the last person to advise anyone to ignore the demands of conscience, but to encourage potential whistleblowers to speak out without noting the long odds of success and the grim personal costs would be like advising them to leap blindfolded off a cliff. (p 290).

All but one of the whistleblowers featured in this book had a particularly difficult time of it because they were speaking out about ethical abuses within their own institutions. As Elliott noted, the line between a traitor and a whistleblower is a fine one, relating to the perceived motivation of the person involved. A whistleblower may feel they are acting on the purest of motives, to do the right thing, yet others will interpret it as disloyalty to the institution, or attribute their behaviour to a sense of self-importance and moral superiority. Even when, as in several of these cases, the whistleblower is eventually vindicated, they are not thanked, there is seldom restitution for victims, and the mental scars of their battles with authority stay with them.

My own forays into challenging dodgy research are nothing like as serious as the cases covered by Elliott: no lives were at stake in the instances of fraud and malpractice that I've focused on, all of which have been outside my own department. Nevertheless, I resonated strongly with what Elliott had to say about the transformation that occurs in whistleblowers when they first try to report a problem. 

Unlike most people, who are used to the darker side of human behavior, whistleblowers are genuinely shocked when they discover dishonesty, greed, or corruption. They naively assume that if they blow the whistle, others will be just as outraged as they are. They might even imagine they'll be thanked." (p. 158).

 And

 "...if becoming a whistleblower requires a certain naivete, then surviving the experience demands a hard shell of cynicism." (p 159). 

I've definitely been on that journey, after years of observing the delays and obfuscations that routinely occur when a sleuth reports a fraudulent paper to a publisher, or when an institution manages to exonerate a researcher despite ample evidence of misconduct.

I've also been thinking a lot lately about parallels with political life. Here in the UK we've had a number of scandals where citizens were harmed, and sometimes killed, by people in power who were at best incompetent and at worst corrupt. Politicians handed lucrative contracts for personal protective equipment to their cronies who failed to deliver, leaving health professionals without protection; NHS patients were given contaminated blood products long after a problem had been identified; A horrendous fire in a tower block in West London arose after contractors ignored regulations and installed inappropriate cladding on the building - in each of these cases the stories came out, there was a public outcry, and a sense that "something must be done" - but the process gets mired in red tape and nothing actually is done, nobody gets punished, and those affected aren't compensated. This is terrible not just because of the impact on the individuals who were directly harmed, but also because it destroys people's trust in the institutions that we assume will look after us. I see the failure of institutions to deal adequately with academic misconduct as just one part of this broader picture.

So is there anything that can be done to challenge wrongdoing without crashing out over the cliff? A major point made by Elliott is that whistleblowers should not go it alone: that way madness lies. One option that he raised in a recent talk on this topic was to consult a lawyer with expertise in whistleblowing cases: they will be able to advise on the likelihood of success, and may be able to bring a complaint directly to an organisation, so the whistleblower can fly under the radar. There is also the option of going to the press, but, though Elliott describes some cases where this was effective, he notes it can be a two-edged sword, because journalists will want a good story, and it may not coincide with the whistleblower's viewpoint. In my experience, it's worth forming links with responsible journalists: institutions who have been inert for years may spring into action when confronted with the possibility of adverse publicity. Perhaps the most important point is that you need to have allies. They can restore confidence that there are other people who share one's values, who appreciate what one is doing and provide moral support. Although Elliott doesn't say this, I think the internet has made it easier to find one's tribe in this regard.

Where, perhaps, I take issue with Elliott is in the implication that many people aren't troubled by the issues that worry whistleblowers. When I give talks about reproducible research practices, I reliably get a question along the lines of "What advice can you give an early-career researcher whose supervisor is encouraging them to adopt questionable research practices?" The questioner clearly finds it troublesome but is being told that their career is at risk if they don't cut corners, and that "everyone is doing it". I reply that this is an issue that has serious consequences for mental health: doing what you know to be wrong causes internal conflict that can eat away at you, leading to a clash between your model of a good person and your current behaviour.

For this reason I advise those starting out in research to read this piece by Sarah Kendzior. Her advice is intended to help people survive under the threat of authoritarianism, but many of the points apply to academic survival in an age of cynicism:

Write down what you value; what standards you hold for yourself and for others.... Write a list of things you would never do. .... 

But most of all, never lose sight of who you are and what you value. If you find yourself doing something that feels questionable or wrong a few months or years from now, find that essay you wrote on who you are and read it. Ask if that version of yourself would have done the same thing.

 And if the answer is no? Don’t do it.