Sowing seeds of doubt: how Gilbert et al’s critique of the reproducibility project has played out

In Merchants of Doubt, Eric Conway and Naomi Oreskes describe how raising doubt can be used as an effective weapon against inconvenient science. On topics such as the effects of tobacco on health, climate change and causes of acid rain, it has been possible to delay or curb action to tackle problems by simply emphasising the lack of scientific consensus. This is always an option, because science is characterised by uncertainty, and indeed, we move forward by challenging one another’s findings: only a dead science would have no disagreements. But those raising concerns wield a two-edged sword: spurious and discredited criticisms can disrupt scientific progress, especially if the arguments are complex and technical: people will be left with a sense that they cannot trust the findings, even if they don’t fully understand the matters under debate.

The parallels with Merchants of Doubt occurred to me as I re-read the critique by Gilbert et al of the classic paper by the Open Science Collaboration (OSC) on ‘Estimating the reproducibility of psychological science’. I was prompted to do so because we were discussing the OSC paper in a journal club* and inevitably the question arose as to whether we needed to worry about reproducibility, in the light of the remarkable claim by Gilbert et al:  ‘We show that OSC's article contains three major statistical errors and, when corrected, provides no evidence of a replication crisis. Indeed, the evidence is also consistent with the opposite conclusion -- that the reproducibility of psychological science is quite high and, in fact, statistically indistinguishable from 100%.’

The Gilbert et al critique has, in turn, been the subject of considerable criticism, as well as a response by a subset of the OSC group. I summarise the main points of contention in Table 1: at times they seem to be making a defeatist argument that we don’t need to worry because replication in psychology is bound to be poor: something I have disputed.

But my main focus in this post is simply to consider the impact of the critique on the reproducibility debate by looking at citations of the original article and the critique. A quick check on Web of Science found 797 citations of the OSC paper, 67 citations of Gilbert et al, and 33 citations of the response by Anderson et al.

The next thing I did, admittedly in a very informal fashion, was to download the details of the articles citing Gilbert et al and code them according to the content of what they said, as either supporting Gilbert et al’s view, rejecting the criticism, or being neutral. I discovered I needed a fourth category for papers where the citation seemed wrong or so vague as to be unclassifiable. I discarded any papers where the relevant information could not be readily accessed – I can access most journals via Oxford University but a few were behind paywalls, others were not in English, or did not appear to cite Gilbert et al. This left 44 citing papers that focused on the commentary on the OSC study. Nine of these were supportive of Gilbert et al, two noted problems with their analysis, but 33 were categorised as ‘neutral’, because the citation read something like this: 

“Because of the current replicability crisis in psychological science (e.g., Open Science Collaboration, 2015; but see Gilbert, King, Pettigrew, & Wilson, 2016)….”

The strong impression was that the authors of these papers lacked either the appetite or the ability to engage with the detailed arguments in the critique, but had a sense that there was a debate and felt that they should flag this up. That’s when I started to think about Merchants of Doubt: whether intentionally or not, Gilbert et al had created an atmosphere of uncertainty to suggest there is no consensus on whether or not psychology has a reproducibility problem - people are left thinking that it's all very complicated and depends on arguments that are only of interest to statisticians. This makes it easier for those who are reluctant to take action to deal with the issue.

Fortunately, it looks as if Gilbert et al’s critique has been less successful than might have been expected, given the eminence of the authors. This may in part be because the arguments in favour of change are founded not just on demonstrations such as the OSC project, but also on logical analyses of statistical practices and publication biases that have been known about for years (see slides 15-20 of my presentation here). Furthermore, as evidenced in the footnotes to Table 1, social media allows a rapid evaluation of claims and counter-claims that hitherto was not possible when debate was restricted to and controlled by journals. The publication this week of three more big replication studies  just heaps on further empirical evidence that we have a problem that needs addressing. Those who are saying ‘nothing to see here, move along’ cannot retain any credibility.

    Table 1

Criticism	Rejoinder
‘many of OSC’s replication studies drew their samples from different populations than the original studies did’	·     ‘Many’ implies the majority. No attempt to quantify – just gives examples ·     Did not show that this feature affected replication rate
‘many of OSC’s replication studies used procedures that differed from the original study’s procedures in substantial ways.’	·     ‘Many’ implies the majority. No attempt to quantify – just gives examples ·     OSC showed that this did not affect replication rate ·     Most striking example used by Gilbert et al is given detailed explanation by Nosek (1)
‘How many of their replication studies should we expect to have failed by chance alone? Making this estimate requires having data from multiple replications of the same original study.’ Used data from pairwise comparisons of studies from the Many Labs project to argue a low rate of agreement is to be expected.	·     Ignores publication bias impact on original studies (2, 3) ·     G et al misinterpret confidence intervals (3, 4) ·     G et al fail to take sample size/power into account, though this is crucial determinant of confidence interval (3, 4) ·      ‘Gilbert et al.’s focus on the CI measure of reproducibility neither addresses nor can account for the facts that the OSC2015 replication effect sizes were about half the size of the original studies on average, and 83% of replications elicited smaller effect sizes than the original studies.’ (2)
Results depended on whether original authors endorsed the protocol for the replication: ‘This strongly suggests that the infidelities did not just introduce random error but instead biased the replication studies toward failure.	·     Use of term ‘the infidelities’ assumes the only reason for lack of endorsement is departure from original protocol. (2) ·     Lack of endorsement included non-response from original authors (3)

(1)   Nosek: https://retractionwatch.com/2016/03/07/lets-not-mischaracterize-replication-studies-authors/

(2)   Anderson et al: http://science.sciencemag.org/content/351/6277/1037.3

(3)   Srivastava:  https://thehardestscience.com/2016/03/03/evaluating-a-new-critique-of-the-reproducibility-project/

(4)   Lakens: http://daniellakens.blogspot.co.uk/2016/03/the-statistical-conclusions-in-gilbert.html

References

Anderson, C. J., Bahnik, S., Barnett-Cowan, M., & et al. (2016). Response to Comment on "Estimating the reproducibility of psychological science". Science, 351(6277).

Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on "Estimating the reproducibility of psychological science". Science, 351(6277).

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Nature, 349(6251). doi:10.1126/science.aac4716

*Thanks to the enthusiastic efforts of some of our grad students, and the support of Reproducible Research Oxford, we’ve had a series of Reproducibilitea journal clubs in our department this term.  I can recommend this as a great – and relatively cheap and easy - way of raising awareness of issues around reproducibility in a department: something that is sorely needed if a recent Twitter survey by Dan Lakens is anything to go by.

Autism diagnosis in cultural context

A review of Isabel’s World by Roy Richard Grinker

In 1993, when Roy Grinker’s daughter, Isabel, was two years old she did not talk or gesture, flapped her hands and arms, and did not make eye contact. At 32 months, she had mastered about 70 words, which she spoke clearly, but all were nouns and the list did not include ‘mommy’ or ‘daddy’. She didn’t say ‘yes’ or ‘no’ and pulled someone to the refrigerator when she was hungry. Nowadays it’s hard to imagine a paediatrician would fail to recognise classic symptoms of autism, but for the Grinkers the process of getting a diagnosis was protracted and painful. At best they met with ignorance, and at worst with professionals who were imbued with the ideas of Bruno Bettelheim and attributed Isabel’s problems to the fact that her mother went out to work. Once a diagnosis was obtained, the Grinkers still had to fight for appropriate educational provision. They came up against people who had no experience of autism and were unwilling or unable to engage with Isabel. Gradually, though, the battle was won. Isabel, whose high nonverbal ability was eventually recognised, was able to attend a mainstream school with support. Now in her teens, she still has major problems with communication and social interaction, and her parents accept she will not able to live independently. She has, however, found a niche in her community where she has friends, can enjoy her interests in music and animals, and is accepted for who she is.

Grinker’s book is more than a parent’s first-hand account of his daughter’s autism: it is also influenced by the fact that he is a cultural anthropologist, with a special interest in attitudes to health and disability in different parts of the world. He uses his professional insights to comment on two related issues: the rise in autism diagnosis in the USA, and attitudes to autism in other cultures, particularly France, Korea, South Africa and India.

Figure 1

The rise in autism diagnosis is a hotly debated topic. In general, there is agreement about the facts: the number of children receiving an autism diagnosis in the USA has risen sharply since the 1970s. The data in Figure 1 were taken from a website relating to the Individuals with Disability Education Act (IDEA), and they show a more than four-fold increase in children identified with autism in the ten year period between 1997 and 2006. I have previously noted in a PLOS One article that research on autism, as assessed by funding and publications, has skyrocketed over the same period.


The debate is over the reason for the increase: is it due to some environmental factor that causes autism, or can it be explained simply in terms of changes in diagnostic criteria and related factors? Grinker comes down solidly in favour of the latter explanation, and a substantial part of the book is devoted to tracing the history of autism diagnosis and related socio-cultural factors. A major turning point was the publication in 1987 of the third revision of the diagnostic and statistical manual of the American Psychiatric Association, DSM-III-R. Grinker cites a study of 194 children suspected of autism which found that whereas 51% met criteria on DSM-III, 91% met criteria on DSM-III-R. He also reports a typographic error that was made in the publication of DSM-IV in 1994 that led to the ‘autism spectrum’ disorder of PDDNOS being defined on the basis of a child having impairments in social interaction or verbal/nonverbal communication skills, when it should have required impairment in both domains.

Grinker tackles a question often asked by those who think the autism epidemic is genuine: if it’s just due to a change in diagnostic practices, where were all the undiagnosed autistic children in the past? Surely we would have noticed them? On the basis of a follow-up of a small sample of UK children with severe language impairments, our group found cases who would now be regarded as unambiguously autistic, but who were diagnosed as language-impaired when they were seen in the 1980s, prior to DSM-IIIR. So are we just engaging in 'diagnostic substitution', whereby the same child who is now regarded as autistic was previously given a different label? If so, we'd expect to see that as autism diagnosis goes up, language disorder diagnosis goes down. This would be consistent with an analysis of doctor's records in the UK, which found that the proportion of children with diagnoses such as speech/language disorder went down over the same period as autism diagnoses went up. In the IDEA data, however, I found no evidence for such a process: over the 1997-2006 time period, there's actually a slight increase in diagnoses of speech/language diagnoses. Grinker, however, suggests that, in the US, children with autism would, in the past, often have been diagnosed with mental retardation. Consistent with this, the IDEA data do show a corresponding decrease in diagnosis of mental retardation over the same time period as autism diagnoses increase (see Figure 1).

Another point stressed by Grinker is that a diagnosis of autism has consequences for the child’s access to services. A diagnosis may get your child Medicaid waivers so that they can receive a host of interventions at reduced cost. Grinker describes how he lost hundreds of dollars because a speech pathologist who worked with Isabel submitted the bills under the diagnosis of “Mixed Receptive-Expressive Language Disorder”. When the diagnosis was changed to autism, the bills were automatically reimbursed. There is therefore considerable pressure on paediatricians to give a diagnosis of autism rather than some other condition.

The implications of an autism diagnosis for intervention is very different in some other countries. In France, where the legacy of psychoanalysis has been long-lasting, autism is seen as a psychodynamic disorder and there is little educational provision for affected children. In India, the diagnosis is seldom made, even if doctors recognise autism, because there seems no point: there are no facilities for affected children. Grinker has particular interest in South Korea, his wife’s birthplace, where the stigma attached to disability is so great that children with autism will be hidden away, because otherwise their siblings’ marriage prospects will be blighted. Education is seen as the route to success in Korea, and it is normal for children to spend hours after school being coached; a child who struggles at school or who does not conform to expected standards of behaviour brings shame to the family.

Grinker’s book shows that the impact of labels is immense, even if they are worryingly arbitrary. It seems crazy, for instance, that a child with educational difficulties will get insurance cover for interventions, or special help at school on the basis of a diagnosis of autism, whereas a child with equally serious needs who is diagnosed with mental retardation or receptive language disorder gets nothing. But on the positive side, he notes how the growing awareness and acceptance of autism has brought huge benefits to children and their families. Just as in Korea, it used to be common for people in the US to be baffled or even frightened by autism, and for schools to shun children with any kind of disability. The landscape has changed massively over the past twenty years. Grinker notes how Isabel’s schoolmates look out for her, and how her presence in a mainstream classroom has beneficial effects on all pupils. Cultural stereotypes are being challenged, so much so that it is no longer regarded as amazing if a university student tells you they have autism.

This brings Grinker to a final point: which kinds of cultural setting are most supportive for people with autism? He presents some fascinating data showing that rural communities are usually far more positive than urban ones for people with all kinds of disability. In a small community, everyone will know the person with a disability, and see them as an individual. In the context of neurodevelopmental disorders in general, I’ve long been advocating that we need an educational system that doesn’t just try to ‘fix’ children, but rather identifies activities they enjoy and can do well, as a basis for finding them a niche in society.  This view is cogently expressed by Grincher “… in order to help people with autism we don’t always need to fully mainstream them, or pretend that they are not different, and we don’t need to simply reduce stigma. Rather, we need to provide roles in our communities for pepole with autism, some of which they may, in fact, be able to perform better than anyone else…” (p. 342).


Bishop, D., Whitehouse, A., Watt, H., & Line, E. (2008). Autism and diagnostic substitution: evidence from a study of adults with a history of developmental language disorder Developmental Medicine & Child Neurology, 50 (5), 341-345 DOI: 10.1111/j.1469-8749.2008.02057.x