Collapse of scientific standards at MDPI journals: a case study

 

"Here's one for you", said my husband, as he browsed the online Daily Telegraph*: "Severe autism can be reversed, groundbreaking study suggests".  My heart did not lift up at this news, which was also covered in the Daily Mail; it is a near certainty that any study making such claims is flawed. But I found it hard to believe just how flawed it turned out to be.  

 

The article, by D'Adamo et al.,  was published in a special issue of an MDPI journal, the Journal of Personalized Medicine.  The special issue "A Personalized Medicine Approach to the Diagnosis and Management of Autism Spectrum Disorder: Beyond Genetic Syndromes" appears largely to be a vehicle for papers by the guest editor Richard E. Frye, who co-authored 3/4 editorials, 3/9 articles and 1/1 review in this collection.  He was editor but not an author on the paper by D'Adamo et al, which is categorised as a "case report".

 

Essentially, this is a description of development of a pair of non-identical twin girls who were diagnosed with autism at 20 months of age, along with various gastric and motor conditions, and were subsequently subjected to a remarkable list of interventions, viz:

·      Parents worked with an autism parent coach who informed them about the link between "total allostatic load" and developmental disorders, a concept promoted by the Documenting Hope Project that one of the authors is affiliated with

·      Parents accessed resources, including free webinars, through Epidemic Answers

·      Parents took part in a parent forum called "Healing Together"

·      Applied Behavior Analysis (ABA) from 22 mo to 33 mo

·      Speech Therapy starting at 24 mo

·      Rigorous diet and nutrition intervention eliminating sources of glutamate, following Reduced Excitatory Inflammatory Diet

·      A strict gluten-free, casein-free diet that was low in sugar and additives

·      Dietary supplements, including omega-3 fatty acids, multivitamins, vitamin D, carnitine, 5-methyltetrahydrofolate and "bio-individualized homeopathic remedies"

·      Family consulted a naturopathic doctor who used IntellxxDNA genomics tool to recommend diet rich in tryptophan, vitamins B12, B6 and folate, betaine and choline, lion's mane mushroom and resolvins, as well as some dietary variants specific to each twin.

·      Neuro-sensory motor reflex integration given by an occupational therapist

·      Environmental evaluation of the home for air quality, mould and moisture, culminating in a visit by a Building Biology Environmental Consultant who identified possible water damage.

·      Cranial osteopathy, as recommended by a developmental optometrist.

 

When first diagnosed, the girls were given an assessment called the Autism Treatment Evaluation Checklist, which was repeated after 18 months. Scores declined (i.e. improved) for both girls, with one twin improving substantially, and the other less so.

 

I haven't been able to find any norms for the ATEC, but there's a bit of data on trajectories in a study by Mahapatra et al (2018) and this shows that scores normally improve between the age of 2 and 3 years.  We may also note that the ATEC is completed by a parent, and so may be influenced by parental expectations of improvement.

 

The authors conclude the study demonstrates "the clear environmental and lifestyle influences on ASD" and that there is "comparatively greater impact of these types of factors than genetics".  Neither conclusion is warranted. We can't know whether these changes would have occurred in the absence of the numerous interventions that the children were given.  If the interventions did have an effect, it would be impossible to tell which were the key ingredients, because all were given together.

 

The authors grudgingly state "...while effective at reversing ASD diagnoses, the comprehensive approach that was employed in this case may not yet be widely generalizable".  But they didn't show their approach was effective - at least not by any conventional standards of evidence-based intervention. 

 

Another limitation they noted was that this was an expensive regimen that might be out of reach for parents with limited resources.

 

Should the journalists who gave this study coverage be criticised? I think the answer is yes: science journalists can't be expected to be experts in all areas of science, but they should be aware of some basic principles, such as the need for adequate sample sizes and control groups in intervention studies, and the need to be careful in assuming causality. They can, if uncertain, ask the Science Media Centre to put them in touch with knowledgeable experts who could advise. This is particularly important for a sensitive subject matter such as autism, where publicity for unfounded interventions that portray autism as a "disease" requiring "healing" has potential to do harm.  At minimum, they might have noted the ethics statement that with only two children involved, "this is not considered a systematic investigation designed to contribute to generalizable knowledge."  So how on earth did this come to be emblazoned across national newspapers as a "groundbreaking study"?

 

Having said that, though, my strongest criticism is for the MDPI publishers, who have encouraged an explosion in "special Issues" of their journals, with scant scrutiny of the quality of published articles - each of which brings in an Article Processing Charge of CHF 2600 (around £2,200, or US $2,900). The proliferation of anecdotal reports in the literature gives ammunition to those who wish to promote all kinds of unevidenced treatments: they can point to these "peer reviewed" papers as evidence of scientific respectability.

 

In the longer term, the lax editorial standards that admit pseudoscience into the research literature will further damage the already tarnished MDPI brand, and we can only hope that papers published in the Journal of Personalized Medicine will be discounted by serious journalists. But meanwhile, every practitioner with a dubious product to sell will see this journal as a perfect outlet for their promotional material. 

 

*The headline has been modified and now reads "Autism can be reversed, scientists discover", but it still describes the study as a 'trial', which is isn't.

 

 

References

D’Adamo, C. R., Nelson, J. L., Miller, S. N., Rickert Hong, M., Lambert, E., & Tallman Ruhm, H. (2024). Reversal of autism symptoms among dizygotic twins through a personalized lifestyle and environmental modification approach: A case report and review of the literature. Journal of Personalized Medicine, 14(6), Article 6. https://doi.org/10.3390/jpm14060641

 

Mahapatra, S., Vyshedsky, D., Martinez, S., Kannel, B., Braverman, J., Edelson, S. M., & Vyshedskiy, A. (2018). Autism Treatment Evaluation Checklist (ATEC) Norms: A “Growth Chart” for ATEC score changes as a function of age. Children, 5(2), Article 2. https://doi.org/10.3390/children5020025