Time for neuroimaging (and PNAS) to clean up its act

© www.CartoonStock.com

There are rumblings in the jungle of neuroscience. There’s been a recent spate of high-profile papers that have drawn attention to methodological shortcomings in neuroimaging studies (e.g., Ioannidis, 2011; Kriegeskorte et al., 2009; Nieuwenhuis et al, 2011) . This is in response to published papers that regularly flout methodological standards that have been established for years. I’ve recently been reviewing the literature on brain imaging in relation to intervention for language impairments and came across this example.
Temple et al (2003) published an fMRI study of 20 children with dyslexia who were scanned both before and after a computerised intervention (FastForword) designed to improve their language. The article in question was published in the Proceedings of the National Academy of Sciences, and at the time of writing has had 270 citations. I did a spot check of fifty of those citing articles to see if any had noted problems with the paper: only one of them did so. The others repeated the authors’ conclusions, namely:

1. The training improved oral language and reading performance.

2. After training, children with dyslexia showed increased activity in multiple brain areas.

3. Brain activation in left temporo-parietal cortex and left inferior frontal gyrus became more similar to that of normal-reading children.

4. There was a correlation between increased activation in left temporo-parietal cortex and improvement in oral language ability.

But are these conclusions valid? I'd argue not, because:

• There was no dyslexic control group. See this blogpost for why this matters. The language test scores of the treated children improved from pre-test to post-test, but where properly controlled trials have been done, equivalent change has been found in untreated controls (Strong et al., 2011). Conclusion 1 is not valid.

• The authors presented uncorrected whole brain activation data. This is not explicitly stated but can be deduced from the z-scores and p-values. Russell Poldrack, who happens to be one of the authors of this paper, has written eloquently on this subject: “…it is critical to employ accurate corrections for multiple tests, since a large number of voxels will generally be significant by chance if uncorrected statistics are used. .. The problem of multiple comparisons is well known but unfortunately many journals still allow publication of results based on uncorrected whole-brain statistics.” Conclusion 2 is based on uncorrected p-values and is not valid.

• To demonstrate that changes in activation for dyslexics made them more like typical children, one would need to demonstrate an interaction between group (dyslexic vs typical) and testing time (pre-training vs post-training). Although a small group of typically-reading children was tested on two occasions, this analysis was not done. Conclusion 3 is based on images of group activations rather than statistical comparisons that take into account within-group variance. It not valid.

• There was no a priori specification of which language measures were primary outcomes, and numerous correlations with brain activation were computed, with no correction for multiple comparisons. The one correlation that the authors focus on (Figure reproduced below) is (a) only significant on a one-tailed test at .05 level; (b) driven by two outliers (encircled), both of whom had a substantial reduction in left temporo-parietal activation associated with a lack of language improvement. Conclusion 4 is not valid. Incidentally, the mean activation change (Y-axis) in this scatterplot is also not significantly different from zero. I'm not sure what this means, as it’s hard to interpret the “effect size” scale, which is described as “the weighted sum of parameter estimates from the multiple regression for rhyme vs. match contrast pre- and post-training.”

Figure 2 from Temple et al. (2003). Data from dyslexic children

.

How is it that this paper has been so influential? I suggest that it is largely because of the image below, summarising results from the study. This was reproduced in a review paper by the senior author that appeared in Science in 2009. This has already had 42 citations. The image is so compelling that it’s also been used in promotional material for a commercial training program other than the one that was used in the study. As McCabe and Castel (2008) have noted, a picture of a brain seems to make people suspend normal judgement.

I don’t like to single out a specific paper for criticism in this way, but feel impelled to do so because the methodological problems were so numerous and so basic. For what it’s worth, every paper I have looked at in this area has had at least some of the same failings. However, in the case of Temple et al (2003) the problem is compounded by the declared interests of two of the authors, Merzenich and Tallal, who co-founded the firm that markets the FastForword intervention. One would have expected a journal editor to subject a paper to particularly stringent scrutiny under these circumstances.

We can also ask why those who read and cite this paper haven’t noted the problems. One reason is that neuroimaging papers are complicated and the methods can be difficult to understand if you don’t work in the area.

Is there a solution? One suggestion is that reviewers and readers would benefit from a simple cribsheet listing the main things to look for in a methods section of a paper in this area. Is there an imaging expert out there who could write such a document, targeted at those like me, who work in this broad area, but aren’t imaging experts? Maybe it already exists, but I couldn’t find anything like that on the web.

Imaging studies are expensive and time-consuming to do, especially when they involve clinical child groups. I'm not one of those who thinks they aren't ever worth doing. If an intervention is effective, imaging may help throw light on its mechanism of action. However, I do not think it is worthwhile to do poorly-designed studies of small numbers of participants to test the mode of action of an intervention that has not been shown to be effective in properly-controlled trials. It would make more sense to spend the research funds on properly controlled trials that would allow us to evaluate which interventions actually work.

References
Gabrieli, J. D. (2009). Dyslexia: a new synergy between education and cognitive neuroscience. Science, 325(5938), 280-283.

Ioannidis, J. P. A. (2011). Excess significance bias in the literature on brain volume abnormalities. Arch Gen Psychiatry, 68(8), 773-780. doi: 10.1001/archgenpsychiatry.2011.28

Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. [10.1038/nn.2303]. Nature Neuroscience, 12(5), 535-540. doi: http://www.nature.com/neuro/journal/v12/n5/suppinfo/nn.2303_S1.html 

McCabe, D., & Castel, A. (2008). Seeing is believing: The effect of brain images on judgments of scientific reasoning Cognition, 107 (1), 343-352 DOI: 10.1016/j.cognition.2007.07.017 

Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. [10.1038/nn.2886]. Nature Neuroscience, 14(9), 1105-1107.

Poldrack, R. A., & Mumford, J. A. (2009). Independence in ROI analysis: where is the voodoo? Social Cognitive and Affective Neuroscience, 4(2), 208-213.

Strong, G. K., Torgerson, C. J., Torgerson, D., & Hulme, C. (2010). A systematic meta-analytic review of evidence for the effectiveness of the ‘Fast ForWord’ language intervention program. Journal of Child Psychology and Psychiatry, in press, doi: 10.1111/j.1469-7610.2010.02329.x.

Temple, E., Deutsch, G. K., Poldrack, R. A., Miller, S. L., Tallal, P., Merzenich, M. M., & Gabrieli, J. D. E. (2003). Neural deficits in children with dyslexia ameliorated by behavioral remediation: Evidence from functional MRI. Proceedings of the National Academy of Sciences of the United States of America, 100(5), 2860-2865. doi: 10.1073/pnas.0030098100

Pioneering treatment or quackery? How to decide

My mother was only slightly older than I am now when she died of emphysema (chronic obstructive pulmonary disease). It’s a progressive condition for which there is no cure, though it can be managed by use of inhalers and oxygen. I am still angry at the discomfort she endured in her last years, as she turned from one alternative practitioner to another. It started with a zealous nutritionist who was a pupil of hers. He had a complicated list of foods she should avoid: I don’t remember much about the details, except that when she was in hospital I protested at the awful meal she’d been given - unadorned pasta and peas - only to be told that this was at her request. Meat, sauces, fats, cheese were all off the menu. My mother was a great cook who enjoyed good food, but she was seriously underweight and the unappetising meals were not helping. In that last year she also tried acupuncture, which she did not enjoy: she told me how it involved lying freezing on a couch having needles prodded into her stick-like body. Homeopathy was another source of hope, and the various remedies stacked up in the kitchen. Strangely enough, spiritual healing was resisted, even though my Uncle Syd was a practitioner. That seemed too implausible for my atheistic mother, whose view was: “If there is a God, why did he make us intelligent enough to question his existence?”

From time to time, friends and relatives of mine have asked my advice about other treatments that are out there. There is, for instance, the Stem Cell Institute in Panama, offering treatment for multiple sclerosis, spinal cord injury, osteoarthritis, rheumatoid arthritis, other autoimmune diseases, autism, and cerebral palsy.  Or nutritional therapist Lucille Leader,  who has a special interest in supporting patients with Parkinson's Disease, Multiple Sclerosis and Inflammatory Bowel Disease. My mother would surely have been interest in AirEnergy, a “compact machine that creates 'energised air' that feeds every cell in your body with oxygen that it can absorb and use more efficiently”.

Another source of queries are parents of the children with neurodevelopmental disorders who are the focus of my research. If you Google for treatments for dyslexia you are confronted by a plethora of options. There is the Dyslexia Treatment Centre, which offers Neurolinguistic Programming and hypnotherapy to help children with dyslexia, dyspraxia or ADHD. Meanwhile the Dore Programme markets a set of “daily physical exercises that aim to improve balance, co-ordination, concentration and social skills” to help those with dyslexia, dyspraxia, ADHD or Asperger’s syndrome. The Dawson Program offers vibrational kinesiology to correct imbalances in the body’s energy fields.  I could go on, and on, and on….

So how on earth can we decide which treatments to trust and which are useless or even fraudulent? There are published lists of warning signs (e.g. ehow Health, Quackwatch), but I wonder how useful they are to the average consumer. For instance, the cartoon by scienceblogs will make skeptics laugh, but I doubt it will be much help for anyone with no science background who is looking for advice. So here’s my twopennyworth. First, a list of things you need to ignore when evaluating a treatment.

1. The sincerity of the practitioner. It’s a mistake to assume all purveyors of ineffective treatments are evil bastards out to make money of the desperate. Many, probably most,  believe honestly in what they are doing. The nutritionist who advised my mother was a charming man who did not charge her a penny - but still did her harm by ensuring her last months were spent on an inadequate and boring diet. The problem is if practitioners don’t adopt scientific methods of evalulating treatments they will convince themselves they are doing good, because some people get better anyway, and they’ll attribute the improvement to their method.

2. The professionalism of the website. Some dodgy treatments have very slick marketing. The Dore Treatment, which I regard as of dubious efficacy, had huge success when it first appeared. Its founder, Wyford Dore was a businessman who had no background in neurodevelopmental disorders but knew a great deal about marketing. He ensured that if you typed ‘dyslexia treatment’ into Google his impressive website was the first thing you’d hit.

3. Fancy-looking credentials. These can be misleading if you aren’t an expert - and sometimes even if you are. My bugbear is ‘Fellow the Royal Society of Medicine’, which sounds very impressive - similar to Fellow the Royal Society (which really is impressive).  In fact, the threshold for fellowship is pretty low, so much so that fellows are told by the RSM that they should not use FRSM on a curriculum vitae. So when you see this on someone’s list of credentials, it means the opposite of what you think: they are likely to be a charlatan. It’s also worth realising that it’s pretty easy to set up your own organisation and offer your own qualifications. I could set up the Society of Skeptical Quackbusters and offer Fellowship to anyone I choose. The letters FSSQ might look good, but carry no guarantee of anything.

4. Testimonials. There is evidence (reviewed here) that humans trust testimonials far more than facts and figures. It’s a tendency that’s hard to overcome, despite scientific training. I still find myself getting swayed if I hear someone tell me of their positive experience with some new nutritional supplement, and thinking, maybe there’s something in it. Advertisers know this: it’s one thing to say that 9 out of 10 cats prefer KittyMunch, but to make it really effective you need a cute cat going ecstatic over the food bowl. If you are deciding whether to go for a treatment you must force yourself to ignore testimonials. For a start, you don’t even know if they are genuine: anyone who regards sick and desperate people as a business opportunity is quite capable of employing actors to pose as satisfied customers. Second, you are given no information about how typical they are. You might be less impressed by the person telling you their dyslexia was cured if you knew that there were a hundred others who paid for the treatment and got no benefit. And the cancer patients who die after a miracle cure are the ones you won’t hear about.

5. Research articles. Practitioners of alternative treatments are finding that the public is getting better educated, and they may be asked about research evidence. So it’s becoming more common to find a link to ‘research’ on websites advertising treatments. The problem is that all too often this is not what it seems. This was recently illustrated by an analysis of research publications from the Burzynski clinic, which offers the opportunity to participate in expensive trials of cancer treatment. I was interested also to see the research listed on the website of FastForword, a company that markets a computerized intervention for children’s language and literacy problems. Under a long list of Foundational Research articles, they list one of my papers that fails to support their theory that phonological and auditory difficulties have common origins. More generally, the reference list contains articles that are relevant to the theory behind the intervention, but don’t necessarily support it. Few people other than me would know that. And a recent meta-analysis of randomized controlled trials of FastForword is a notable omission from the list of references provided. Overall, this website seems to exemplify a strategy that has previously been adopted in other areas such as climate change, impact of tobacco or sex differences, where you create an impression of a huge mass of scientific evidence, which can only be counteracted if painstakingly unpicked by an expert who knows the literature well enough to evaluate what’s been missed out, as well as what’s in there. It’s similar to what Ben Goldacre has termed ‘referenciness’, or the ‘Gish gallop’ technique of creationists. It’s most dangerous when employed by those who know enough about science to make it look believable. The theory behind FastForword is not unreasonable, but the evidence for it is far less compelling than the website would suggest.

So those are the things that can lull you into a false sense of acceptance. What about the red flags, warning signs that suggest you are dealing with a dodgy enterprise? None of these on its own is foolproof, but where several are present together, beware.

1. Is there any theory behind the intervention, and if so is it deemed plausible by mainstream scientists? Don’t be impressed by sciency-sounding theories - these are often designed to mislead. Neuroscience terms are often incorporated to give superficial plausibility: I parodied this in my latest novel, with the invention of Neuropositive Nutrition, which is based on links between nutrients, the thalamus and the immune system. I suspect if I set up a website promoting it, I’d soon have customers. Unfortunately, it can be hard to sort the wheat from the chaff, but NHSChoices is good for objective, evidence-based  information. Most universities have a communications office that may be able to point you to someone who could indicate whether an intervention has any scientific credibility.  
2. How specific is the treatment? A common feature of dodgy treatments is that they claim to work for a wide variety of conditions. Most effective treatments are rather specific in their mode of action.
3. Does the practitioner reject conventional treatments? That’s usually a bad sign, especially if there are effective mainstream approaches.
4. Does the practitioner embrace more than one kind of alternative treatment? I was intriguted when doing my brief research on Fellows of the Royal Society of Medicine to see how alternative interventions tend to cluster together. The same person who is offering chiropractic is often also recommended hypnotherapy, nutritional supplements and homeopathy.  Since modern medical advances have all depended on adopting a scientific stance, anyone who adopts a range of methods that don’t have scientific support is likely to be a bad bet.
5. Are those developing the intervention cautious, and interested in doing proper trials?  Do they know what a randomised controlled trial is? If they aren’t doing them, why not? See this book for an accessible explanation of why this is important.
6. Does it look as though those promoting the intervention are deliberately exploiting people’s gullibility by relying heavily on testimonials? Use of celebrities to promote a product is a technique used by the advertising industry to manipulate people’s judgement. It’s a red flag.
7. Are costs reasonable?  Does the website give you any idea of how much they are, or do you have to phone up for information? (bad sign!). Are people tied in to long-term treatment/payment plans? Are you being asked to pay to take part in a clinical trial? (Very unusual and ethically dubious). Do you get a refund if it doesn’t work? If yes, read the terms and condition very carefully so you understand exactly the circumstances under which you get your money back. For instance, I’ve seen a document from the Dore organisation that promised a money-back guarantee on condition there was ‘no physiological change’. That was interpreted as change on tests of balance and eye movements. These change with age and practice, and don’t necessarily mean a treatment has worked. Failing to improve in reading did not qualify you for the refund.
8. Can the practitioner answer the question of why mainstream medicine/education has not adopted their methods? If the answer refers to others having competing interests, be very, very suspicious. Remember, mainstream practitioners want to make people better, and anyone who can offer effective treatments is going to be more successful than someone who can’t.