Orwellian prize for journalistic misrepresentation: an update

I've been re-reading George Orwell's Nineteen Eighty-Four to get me in the mood for looking at nominations for the Orwellian prize of journalistic misrepresentation. Bizarrely, it cheered me up. I last read it in 1968, and it made me nervous. But here we are in Airstrip One in 2010: we may be heading for mass unemployment, dismantlement of the NHS, the BBC and the Universities, we may be getting increasingly uncomfortable with the state's attitude to controlling dissent, but compared to what Winston Smith went through, this is paradise. New technologies are being used to oppress people the world over, but the internet has also emerged as a tool for fighting oppression. I can expostulate on my blog about things that concern me without the thought police carrying me off.

But, back to the prize. Why, I've been asked, did I choose that name? In Orwell's dystopian world, the press is used to achieve control, such that "nominally free news media are required to present 'balanced' coverage, in which every 'truth' is immediately neutered by an equal and opposite one. Every day public opinion is the target of rewritten history, official amnesia and outright lying, all of which is benevolently termed 'spin', as if it were no more harmful than a ride on a merry-go-round." This captured what I felt when I read the coverage of certain scientific discoveries in the press. I knew that there was an Orwell Prize to celebrate the best of British political journalism; I saw the Orwellian Prize as the inverse – a way of noting the worst of science journalism.

I wasn't so concerned by the fact that journalists sometimes make mistakes. That's inevitable when writing under time pressure on subjects where one does not have expertise. I accept too that journalists, and their editors, are not required to be neutral, and are entitled to promulgate their opinions. I'm concerned, however, when mistakes have implications for people who are vulnerable, who might be misled into adopting an ineffective treatment, or shunning an effective one, by a prominent newspaper article. Or, in the case of major environmental issues, a piece on climate change or the impact of industrial practices might sway public opinion in a direction opposite to that held by informed scientists, and risk making our planet less habitable for future generations.  The aspects of scientific writing that particularly disturb me are when science is misrepresented, and that misrepresentation is done thoughtlessly or even knowingly: sometimes for political ends, but more often to get a 'good story' that will have an eye-catching headline and so sell more newspapers.  The MMR story is the most vivid illustration I'm aware of. It's bad when hard-pressed readers are induced to spend money on ineffective treatments; it's worse when children risk disease or even death as a consequence of irresponsible journalism.

In a brilliant post last September, Martin Robbins lampooned the formulaic treatment of science stories that is so often seen in the media. His was a generic account, but in specific areas of science, one can provide even more detail. Thus, for every new advance in neuroscience or genetics, there seems to be one of two possible conclusions: either (a) if confirmed, the discovery will make it possible treat dementia/dyslexia/autism/Parkinson's disease/depression in future, or more commonly (b), the discovery will lead to better diagnosis of dementia/dyslexia/autism/Parkinson's disease/depression.  Yet most of the research in this area is a long way from translation into clinical practice. We are still learning to interpret tools such as brain imaging and genetic analysis, and the more we do, the more complex it becomes. Researchers become rightly excited when they find a neural or genetic correlate of a disease, which could help our understanding of underlying causal mechanisms. This is a vital step in the direction toward effective clinical procedures, but it's usually unrealistic to imagine these will be available soon. Yet in their desire to appeal to human interest, journalists will exert pressure to make more of a story than is justified, by focusing more on hypothetical rather than actual findings. The impetus behind the Orwellian prize is described in my earlier post, where I dissected an article describing a positive impact of fish oil on concentration in children with attention deficit hyperactivity disorder (ADHD). The study behind the story did not use fish oil, did not include children with ADHD, and did not find any behavioural benefits in the children who were given fatty acids. This article epitomised everything that made me angry about reporting of science in my area, with errors so numerous it was hard to believe they were accidental. I therefore threw down a challenge to others to see if they could find equally worrying examples. As it happens, that article was subsequently the focus of a scathing attack by Ben Goldacre, and was taken off the Observer's website.

Now, one of the things I love about blogging is its interactive nature. A number of commentators made points that made me think more. None of them were apologists for poor reporting, but they noted that journalists are not always to blame for an overhyped article. For a start, they have no control over headlines, which are  typically written by a sub-editor whose job it is to attract the reader's attention. They are also fed information by institutional press officers who may put a spin on a story in the hope that the media would pick it up. And researchers themselves are not immune from wanting their day of glory and being willing to 'accentuate the positive' rather than adopting the cautious, balanced approach that characterises a good scientist.  I have to say that, although I knew this could happen, until I started looking at candidates for the Orwellian, I had always thought that it was only a handful of maverick scientists with personality disorders who would behave in this way. I'd just assumed that for most scientists their reputation with colleagues would be far more important than a brief burst of media attention. I'd also reckoned that press offices would jealously guard the reputation of their institution for impeccably accurate science. But this proved to be naïve. Fame is seductive, and many will compromise their standards for their spot in the limelight.

Since throwing down my challenge, I've had two nominations for the Orwellian. The first was from Cambridge neuroscientist Jon Simons, who true to the spirit of my original post, scored up his submission according to the system I'd proposed, which involved giving points for each statement that was inaccurate when checked against the original source article. The nominated newspaper report was by a Washington Post staff writer and appeared on 9th September under the headline "Scientists can scan brains for maturity, potentially gauging child development". Simons's computations gave it a total of 16 points, putting it on level pegging with the ADHD article. One needs only to look at Figure 1 from the original article, reproduced below, to see one reason why he was exasperated. The prediction from the "functional connectivity Maturation Index" (fcMI), while statistically significant, is far from precise, because of the variation around the average level for each age. Consequently, there is a fair amount of overlap in the range of scores seen for adults and children. If a normal adult can get a brain maturity index of a 8 year old, and a normal 8-year-old can get a maturity index equivalent to an adult, it is questionable just how useful the index would be at detecting abnormality. Also, the regression equation was computed on the combined child and adult data, and nowhere in the paper are data presented on the accuracy of prediction of chronological age from brain measures just within the group of children. As an aside, our research group has tried similar things with the more low-tech methodology of event-related potentials, and we find it is relatively easy to discriminate child brains from adult brains, but not easy to distinguish between a 6-year-old and a 10-year-old (Bishop et al, 2007).

Figure 1 from Dosenbach et al, 2010

I would not be happy, however, giving the Orwellian award to the brain maturity piece, because the journalist does not seem to be at fault. Much of the newspaper article appears to be based on a press release from Oregon Health and Science University.  And furthermore, the Science article reporting the findings is entitled Prediction of individual brain maturity using fMRI and concludes that the method "could one day provide useful information to aid in the screening, diagnosis, and prognosis of individuals with disordered brain function."  I could not find any source for the claim in the newspaper article that the method might tell us "whether teenageers are grown-up enough to be treated as adults", though I did find a Neurolaw blogpost that could be the source of this idea: it stated "in the courtroom, this work could prove useful in determining whether children or adults should be culpable for their actions based on the maturation of the brains" (!).  Overall, I felt that the majority of mistakes in this piece were either misinterpretations that were encouraged by the journal article or press release, or involved extrapolations beyond the data that originated with the authors. And, unlike some other articles focused on developmental disorders, I did not feel this one had much potential for harm.

Another contender for the prize came from Australia, where conservation ecologist Corey Bradshaw was exercised at the way his work on frogs had been misrepresented. In a paper in Conservation Biology entitled Eating frogs to extinction, he and his colleagues called for certification of the trade in frogs' legs, which has the potential to have significant impact on frog populations. I won't attempt to summarise his arguments, which are eloquently stated in his own blogpost. All science journalists should read this – not just because it is witty, but also because it gives some insight into just how angry scientists get when they try hard to explain the science to a journalist who then gets it wrong . But this article, too, didn't seem to merit the prize. Quite apart from the fact it's really too early for the prize, dating back to 2009, there's also a sense of breaking a butterfly on a wheel. The article in question was not in a major newspaper; it was rather on a blog called SlashFood. Admittedly, it's an award-winning blog, described by Time magazine as "a site for people who are serious about what they put in their bodies". But this was clearly a journalist coming at the frogs' legs story from the culinary and not the scientific angle. And as Bradshaw himself says of the journalist, " Now, in all fairness, I think she was trying to do well…" Also, it's hard to see that the article did much harm, beyond raising the blood pressure of a distinguished conservation ecologist to dangerous levels.  Replete with inaccuracies as it was, the overall impact would have been to make environmentally conscious people think twice about eating frogs' legs. Incidentally, journalists at the Guardian will be pleased to hear that Jon Henley's coverage of the same story got a ringing endorsement from Bradshaw.

These were the only nominations I received for the prize, and so I have to announce that it will not be awarded this year, although the two nominators will receive an alcoholic token of my appreciation for their efforts. Although I could give the award to the original Observer article, this does not seem justified given that the newspaper withdrew the piece when it became aware of the scientific criticism.

I will accept  nominations for 2011, and am interested in receiving them even if they don't meet all criteria: I am fascinated by interactions between the media and scientists, in finding out more about what scientists do to irritate journalists, and vice versa. If we are going to improve science reporting, we need to understand one another. I hope that the lack of a serious contender for 2010 is telling us something about improving standards in science journalism - but maybe readers know better.