Collapse of scientific standards at MDPI journals: a case study

 

"Here's one for you", said my husband, as he browsed the online Daily Telegraph*: "Severe autism can be reversed, groundbreaking study suggests".  My heart did not lift up at this news, which was also covered in the Daily Mail; it is a near certainty that any study making such claims is flawed. But I found it hard to believe just how flawed it turned out to be.  

 

The article, by D'Adamo et al.,  was published in a special issue of an MDPI journal, the Journal of Personalized Medicine.  The special issue "A Personalized Medicine Approach to the Diagnosis and Management of Autism Spectrum Disorder: Beyond Genetic Syndromes" appears largely to be a vehicle for papers by the guest editor Richard E. Frye, who co-authored 3/4 editorials, 3/9 articles and 1/1 review in this collection.  He was editor but not an author on the paper by D'Adamo et al, which is categorised as a "case report".

 

Essentially, this is a description of development of a pair of non-identical twin girls who were diagnosed with autism at 20 months of age, along with various gastric and motor conditions, and were subsequently subjected to a remarkable list of interventions, viz:

·      Parents worked with an autism parent coach who informed them about the link between "total allostatic load" and developmental disorders, a concept promoted by the Documenting Hope Project that one of the authors is affiliated with

·      Parents accessed resources, including free webinars, through Epidemic Answers

·      Parents took part in a parent forum called "Healing Together"

·      Applied Behavior Analysis (ABA) from 22 mo to 33 mo

·      Speech Therapy starting at 24 mo

·      Rigorous diet and nutrition intervention eliminating sources of glutamate, following Reduced Excitatory Inflammatory Diet

·      A strict gluten-free, casein-free diet that was low in sugar and additives

·      Dietary supplements, including omega-3 fatty acids, multivitamins, vitamin D, carnitine, 5-methyltetrahydrofolate and "bio-individualized homeopathic remedies"

·      Family consulted a naturopathic doctor who used IntellxxDNA genomics tool to recommend diet rich in tryptophan, vitamins B12, B6 and folate, betaine and choline, lion's mane mushroom and resolvins, as well as some dietary variants specific to each twin.

·      Neuro-sensory motor reflex integration given by an occupational therapist

·      Environmental evaluation of the home for air quality, mould and moisture, culminating in a visit by a Building Biology Environmental Consultant who identified possible water damage.

·      Cranial osteopathy, as recommended by a developmental optometrist.

 

When first diagnosed, the girls were given an assessment called the Autism Treatment Evaluation Checklist, which was repeated after 18 months. Scores declined (i.e. improved) for both girls, with one twin improving substantially, and the other less so.

 

I haven't been able to find any norms for the ATEC, but there's a bit of data on trajectories in a study by Mahapatra et al (2018) and this shows that scores normally improve between the age of 2 and 3 years.  We may also note that the ATEC is completed by a parent, and so may be influenced by parental expectations of improvement.

 

The authors conclude the study demonstrates "the clear environmental and lifestyle influences on ASD" and that there is "comparatively greater impact of these types of factors than genetics".  Neither conclusion is warranted. We can't know whether these changes would have occurred in the absence of the numerous interventions that the children were given.  If the interventions did have an effect, it would be impossible to tell which were the key ingredients, because all were given together.

 

The authors grudgingly state "...while effective at reversing ASD diagnoses, the comprehensive approach that was employed in this case may not yet be widely generalizable".  But they didn't show their approach was effective - at least not by any conventional standards of evidence-based intervention. 

 

Another limitation they noted was that this was an expensive regimen that might be out of reach for parents with limited resources.

 

Should the journalists who gave this study coverage be criticised? I think the answer is yes: science journalists can't be expected to be experts in all areas of science, but they should be aware of some basic principles, such as the need for adequate sample sizes and control groups in intervention studies, and the need to be careful in assuming causality. They can, if uncertain, ask the Science Media Centre to put them in touch with knowledgeable experts who could advise. This is particularly important for a sensitive subject matter such as autism, where publicity for unfounded interventions that portray autism as a "disease" requiring "healing" has potential to do harm.  At minimum, they might have noted the ethics statement that with only two children involved, "this is not considered a systematic investigation designed to contribute to generalizable knowledge."  So how on earth did this come to be emblazoned across national newspapers as a "groundbreaking study"?

 

Having said that, though, my strongest criticism is for the MDPI publishers, who have encouraged an explosion in "special Issues" of their journals, with scant scrutiny of the quality of published articles - each of which brings in an Article Processing Charge of CHF 2600 (around £2,200, or US $2,900). The proliferation of anecdotal reports in the literature gives ammunition to those who wish to promote all kinds of unevidenced treatments: they can point to these "peer reviewed" papers as evidence of scientific respectability.

 

In the longer term, the lax editorial standards that admit pseudoscience into the research literature will further damage the already tarnished MDPI brand, and we can only hope that papers published in the Journal of Personalized Medicine will be discounted by serious journalists. But meanwhile, every practitioner with a dubious product to sell will see this journal as a perfect outlet for their promotional material. 

 

*The headline has been modified and now reads "Autism can be reversed, scientists discover", but it still describes the study as a 'trial', which is isn't.

 

 

References

D’Adamo, C. R., Nelson, J. L., Miller, S. N., Rickert Hong, M., Lambert, E., & Tallman Ruhm, H. (2024). Reversal of autism symptoms among dizygotic twins through a personalized lifestyle and environmental modification approach: A case report and review of the literature. Journal of Personalized Medicine, 14(6), Article 6. https://doi.org/10.3390/jpm14060641

 

Mahapatra, S., Vyshedsky, D., Martinez, S., Kannel, B., Braverman, J., Edelson, S. M., & Vyshedskiy, A. (2018). Autism Treatment Evaluation Checklist (ATEC) Norms: A “Growth Chart” for ATEC score changes as a function of age. Children, 5(2), Article 2. https://doi.org/10.3390/children5020025

An index of neighbourhood advantage from English postcode data

Screenshot from http://dclgapps.communities.gov.uk/imd/idmap.html

Densely packed postcodes appear grey: you need to expand the map to see colours

-->
The Ministry of Housing, Communities and Local Government has a website which provides an ‘index of multiple deprivation’ for every postcode in England.  This is a composite index based on typical income, employment, education, health, crime, housing and living environment for each of 32,844 postcodes in 2015. You can also extract indices for the component factors that contribute to the index, which are explained further here. And there is a fascinating interactive website where you can explore the indices on a map of England.

Researchers have used the index of multiple deprivation as an overall measure of environmental factors that might affect child development, but it has one major drawback. The number that the website gives you is a rank from 1 to 32,844. This means it is not normally distributed, and not easy to interpret. You are also given decile bands, but these are just less precise versions of the ranks – and like ranks, have a rectangular, rather than a normal distribution (with each band containing 10% of the postcodes). If you want to read more about why rectangularly distributed data are problematic, see this earlier blogpost.

I wanted to use this index, but felt it would make sense to convert the ranks into z-scores. This is easily done, as z-scores are simply rescaled proportions. Here’s what you do:

Use the website to convert the postcode to an index of deprivation: in fact, it’s easiest to paste in a list of postcodes and you then get a set of indices for each one, which you can download either as .csv or .xlsx file. The index of multiple deprivation is given in the fifth column.

To illustrate, I put in the street address where I grew up, IG38NP, which corresponds to a multiple deprivation index of 12596.

In Excel, you can just divide the multiple deprivation index by 32844, to get a value of .3835, which you can then convert to a z-score using the NORMSINV function. Or, to do this in one step, if you have your index of multiple deprivation in cell A2, you type
 =normsinv(A2/32844)

This gives a value of -0.296, which is the corresponding z-score. I suggest calling it the ‘neighbourhood advantage score’ – so it’s clear that a high score is good and a low score is bad.

If you are working in R, you can just use the command:
neighbz = qnorm(deprivation_index/depmax)
where neighbz is the neighbourhood advantage score,  depmax has been assigned to 32844 and deprivation_index is the index of multiple deprivation.

Obviously, I’ve presented simplified commands here, but in either Excel or R it is easy to convert a whole set of postcodes in one go.

It is, of course, important to keep in mind that this is a measure of the neighbourhood a person lives in, and not of the characteristics of the individual. Postcode indicators may be misleading in mixed neighbourhoods, e.g. where gentrification has occurred, so rich and poor live side by side. And the different factors contributing to the index may be dissociated. Nevertheless, I think this index can be useful for providing an indication of whether a sample of individuals is representative of the population of England. In psychology studies, volunteers tend to come from more advantaged backgrounds, and this provides one way to quantify this effect.

Parent talk and child language

© www.CartoonStock.com

There's been a lot in the media lately about the impacts of parental talk on children's language development. Some of it has been opinion, as in this piece in the Daily Telegraph, in which the headline proclaimed that children were "starting school unable to speak". This reflected the views of a head teacher, who claimed that the proportion of children with poor language skills had increased in his lifetime, and that this was the fault of parents who did not have time to talk to their children any more. There is nothing new here: versions of this story pop up every few years or so (here's one from 2003,  and a blogpost on another case from 2011): Editors know that stories about feckless parents sell newspapers: readers love the sense of complacency and moral superiority they induce.

But there is also more evidence-based stuff. Some children do have serious difficulties mastering spoken language, and there is research demonstrating links between parent talk and child language outcomes. We've known since the influential study of Hart and Risley (1995) that there is massive variation in the amount of language children are exposed to at home, and this is predicted by socio-economic status. There are many subsequent studies showing positive associations between aspects of the language that babies and toddlers hear and the rate and complexity of their language development.

When the Guardian ran a piece last week on the latest of these studies, someone tweeted "do we really need a study to demonstrate that?" – to most people it's blindingly obvious that children's language development will be determined by the language that they hear at home. This assumption is shared by many professionals in the field of language development; for instance, in a recent review, Leffel and Suskind (2013) describe poor attainment of children from disadvantaged homes and unambiguously state: "Parent linguistic input lies at the heart of the problem".

Except that it's not so simple. And the complexities become apparent when we look at the type of evidence that we have, which is mostly correlational. Students learn in Psychology #101 that correlation does not equal causation, yet when a causal interpretation seems so obvious to most people, this can get forgotten. I have lost count of the number of times I've seen a study showing that parent talk predicts child language development, where the conclusion drawn by the authors (and press offices and the media) is that limited parental language causes child language problems. No other explanation is even countenanced. Yet if we were well taught in Psychology #101, we would realise that we need to consider alternative explanations for the observed association. The figure below shows three possible causal models; these are not mutually exclusive and so all could play a role.

Different Models to account for association between parent talk and child language

Model A is the one that is typically assumed by most people: parent talk to children boosts their language development, and accordingly, if a child has poor language skills, this is likely to be caused by inadequate talk from parents.

In Model B, the association goes in the other direction. Poor language in the child leads to less talk from the parent. This could occur if, for instance, parents are discouraged from talking to a child who is unresponsive and appears not to understand. Consider too, this recent study looking at outcomes of infants in a special care baby unit . Children who were exposed to more adult language in hospital had better language outcomes; however, as the authors noted, "It could be that parents and caregivers have more opportunity to talk to infants who are less sick."

Model C explains the association without postulating a direct link from parental talk to child language. Instead it sees both of these as outcomes of some other cause. This could be an environmental factor, such as poor diet, or a genetic risk that is shared by parents and their children.

It is the job of researchers to try and find evidence to establish the relative importance of these different causal routes. In the case of child language, this is not just a theoretical exercise: it potentially makes a difference to the kinds of intervention that are likely to be effective in helping children. In particular, if model A is the main explanation for the association, then we should be able to boost poor child language by encouraging reticent parents to interact more like talkative parents. This is unlikely to be effective if model B explains the association. And if model C applies, then we would need to either modify the third factor (X) itself, or clarify how it operated in order to alter its association with poor outcomes in children.

I am concerned about the near-universal acceptance of model A as the sole explanation, because there are two lines of evidence that go against it. First, we can to some extent disentangle the impact of socioeconomic disadvantage and parental talk if we study children whose parents produce little spoken language input because they have a congenital hearing impairment. Some profoundly deaf parents have children with normal hearing. In the past there was concern about such children: how would they learn spoken language if their parents produced little intelligible speech? In fact, the studies that were done obtained unexpectedly positive results, leading to the conclusion that although young children clearly need some exposure to spoken language in order to learn to speak, they could develop normal language on the basis of exposure to other adults outside the home and language on TV (Schiff-Myers, 1988).

The second line of evidence comes from studies that disentangle genetic and environmental influences by considering language development in twins. If parental talk is an important determinant of child language, then we would expect twins growing up together in the same home to resemble each other. However, if model A is all-important, we would not expect the genetic relationship between the twins to have any effect. But it does make a difference, and on many language measures this effect is quite substantial. So we find that twins do resemble each other in general, but that resemblance is quite a bit higher if the twins are genetically identical (monozygotic) than if they are fraternal (dizygotic, and sharing around half their DNA for genes that vary between people).

I remember being struck when I first did twin studies of children's language difficulties at how different two twins growing up in the same family could be – provided they were non-identical. It was, however, unusual to find identical twin pairs where one had a significant language problem and the other was unaffected. The overall pattern of results tells us that the child's genetic makeup plays a role in determining their language development (Bishop, 2006).

So what has this to do with models A, B and C? Quite simply, the twin data support a version of model C: given that genes affect language development, we expect parents (who share around half their genes with their children) to resemble their children. We already know that parents of children with language impairments are more likely than other parents to have some kind of language or literacy problem themselves (Barry et al, 2007). This doesn't affect everyone: of course there are many literate and articulate parents whose children have language difficulties. But on balance, these kinds of difficulties run through generations, and we therefore expect to see an association between limited language ability in parents and language difficulties in their children. Note that a genetic account will also predict that language difficulties in children will predominate among those of lower social-economic status: parents who themselves are language-impaired are likely to have low levels of educational attainment and poor occupational prospects.

This kind of genetic explanation for parent-child similarities has a lot of evidential support, but people are very reluctant to accept it. If you propose that genes may play a role in children's developmental difficulties, people will tend to assume that you have a political agenda aligned with the Third Reich, with a goal of identifying a genetic underclass who should not be helped because they are just 'made that way'. This reflects a wrong-headed genetic determinism that is at odds with contemporary understanding of how genes work. Genes do not determine your fate: their impact is likely to vary according to the environment, and by modifying environments we may alter outcomes. Unlike in model A, though, model C predicts that sensitivity to specific environments may depend on one's genes. The arguments have been cogently put in a recent book by Asbury and Plomin (2013), who lament the way in which genetic influences on children's development have been ignored in favour of a political stance that blames educational and developmental difficulties on either poor parenting or poor teaching. If, as has been repeatedly shown, there is evidence that genes are important in influencing children's language development, then we may be squandering our intervention resources by ignoring this fact.

The bottom line is that we need more research. Well-conducted randomized controlled trials on the impact of modifying parent input have been thin on the ground to date, and have not generated impressive evidence of efficacy (see my earlier blogpost) . Obviously, it's early days, and I'd cheer on others who are attempting such research. Results may depend on the nature of the intervention, the aspects of language that are assessed, and the type of population the intervention is used with. My suggestion is that rather than denying the reality of genetic effects, we should be conducting research to find out what kinds of input are most effective for children who are at genetic risk. It is possible that rather than more language input, they may do best with a different kind of language input, specifically tailored to take into account their cognitive strengths and weaknesses. We are a long way from understanding how best to do this, and meanwhile, ingenious and dedicated practitioners are working hard to tackle the very real problems that some children experience. My message is simply that to lay the blame for these difficulties at the door of parents, and to anticipate that problems can be readily overcome by encouraging parents to talk more to their children may be oversimplistic.

To finish, I cannot resist adding my favourite quote from Richard Dawkins, which focuses on mathematics rather than language learning, but gets to the nub of inappropriate concerns about genetic explanations:

People seem to have little difficulty in accepting the modifiability of "environmental" effects on human development. If a child has had bad teaching in mathematics, it is accepted that the resulting deficiency can be remedied by extra good teaching the following year. But any suggestion that the child's mathematical deficiency might have a genetic origin is likely to be greeted with something approaching despair: if it is in the genes "it is written", it is "determined" and nothing can be done about it: you might as well give up attempting to teach the child mathematics. This is pernicious rubbish on an almost astrological scale ..... What did genes do to deserve their sinister juggernaut-like reputation? Why do we not make a similar bogey out of, say, nursery education or confirmation classes? Why are genes thought to be so much more fixed and inescapable in their effects than television, nuns, or books? 

Richard Dawkins (1982) The extended phenotype, Oxford University Press (p. 13) 

References 
Asbury, K., & Plomin, R. (2013). G is for genes: The impact of genetics on education and achievement. Chichester: Wiley Blackwell.
Barry, J. G., Yasin, I., & Bishop, D. V. M. (2007). Heritable risk factors associated with language impairments. Genes, Brain and Behavior, 6, 66-76.
Bishop, D. V. M. (2006). What causes specific language impairment in children? Current Directions in Psychological Science, 15, 217-221.
Caskey, M., Stephens, B., Tucker, R., & Vohr, B. (2014). Adult talk in the NICU with preterm infants and developmental outcomes Pediatrics DOI: 10.1542/peds.2013-0104
Hart, B., & Risley, T. R. (1995). Meaningful differences in the everyday experience of young American children. Baltimore, MD: Paul H. Brookes Publishing Co.
Leffel, K., & Suskind, D. (2013). Parent-directed approaches to enrich the early language environments of children living in poverty. Seminars in Speech and Language, 34(4), 267-277. doi: 10.1055/s-0033-1353443
Schiff-Myers, N. (1988). Hearing children of deaf parents. In D. Bishop & K. Mogford (Eds.), Language development in exceptional circumstances (pp. 47-61). Edinburgh: Churchill Livingstone.

This article (Figshare version) can be cited as:
Bishop, Dorothy V M (2014): Parent talk and child language. figshare.
http://dx.doi.org/10.6084/m9.figshare.1030407