Science journal editors: a taxonomy

After many years of publishing papers, I have come to recognise wide diversity among journal editors. This variation has major consequences for authors, and it is important that they recognise the creature they are dealing with, if they want their work to be published in as timely and painless a way as possible. I have therefore developed a tripartite system of classification to guide authors.

Taxonomy of the Genus Editoris



Class 1

This species of editor should be avoided at all costs.

The Returning Officer
This humble creature has a very limited brain and is unable to make decisions. It can, however, count, and it therefore uses a strategy of accumulating reviewer reports until a consensus is reached. Typically, it is risk averse, and a single negative report will lead to rejection of a paper, even if other reports are glowing. If you aren’t rejected, an initial communication from a Returning Officer will say “Please address all the comments of the reviewers in your revision”, giving no guidance about how to deal with contradictory recommendations. When you submit your revision, the Returning Officer will send it back to all the reviewers, even if only minor changes were made, leading to unnecessary delay in publication and more toil for overworked reviewers. Since the Returning Officer cannot make a decision unless there is convergence of reviewer opinions, most papers are doomed to a long process with an ultimately negative outcome.

The Automaton
This is a subspecies of Returning Officer which has no human characteristics at all.  It evolved relatively recently with the advent of web-based journal submission systems. It generates letters written in computerese and does not read communications from authors. My most recent experience of an Automaton was with Journal of Neuroscience. The letter from the editor gave a rather ambiguous message, stating that the paper was potentially acceptable, but that major revision was required, and it would need to go back to reviewers. It also included the statement: 

   Violations: -The gender of the species should be mentioned in the methods

The dictionary definition of ‘violation’ includes such phases as “the act of violating, treating with violence, or injuring; the state of being violated. Ravishment; rape; outrage.” I decided it might be unwise to point this out to the editor, but I did explain that the “gender of species” was actually given in a table in the Methods section.  A further round of review took place, and the reviews (which were very useful) were accompanied by another letter from the Automaton. It was identical to the previous letter, gave no indication that the editor had read the paper or my response to reviewers, and simply upped the ante on the violation front, as it now stated:

   Violations: -The species is not mentioned in the abstract;   
   -The gender of the species should be mentioned in the methods
In what, thankfully, proved to be the final round of revision, I put the word “gender” in the text of the Methods. I explained, though, that I was reluctant to put “human children” in the Abstract, as this would be a tautology.

The Vacillator
This is a slightly more evolved form of Returning Officer, which is capable of decision-making, but prone to fits of paralysis when confronted by conflicting information.  The hallmark of a Vacillator is that, rather than waiting for consensus between reviewers, it responds to conflicting opinions by seeking yet more opinions, so that a paper may accumulate as many as four or five reviewers.

A variant known as Vacillator statistica sometimes inhabits the environment of medical journals, where the assumption is made that neither the editor nor the researchers understand statistics, so you are asked at submission whether a statistician was consulted. My experience suggests that if you say no, then after an initial round of review, the paper goes to a statistician if it looks promising.  It would be fine if the journal employed statisticians who could give a rapid response, but in my case, a brief paper sent to Archives of Disease in Childhood sat for months with a statistical reviewer, who eventually concluded that we did indeed know how to compute an odds ratio.

The Sloth
The Sloth has powers of judgement but finds journal editing tedious, so engages with the process only intermittently.  The motivation of the Sloth is often mysterious; it may have become an editor to embellish its curriculum vitae, and is then bewildered when it realises that work is involved. It is important to distinguish the true Sloth, who just can’t summon up the energy to edit a paper, from Crypto-sloths, who may have genuine reasons for tardiness; editors, after all, are beset by life events and health problems just like the rest of us.  Vacillators may also be mistaken for Sloths, because of the slowness of their responding, but their level of activity in soliciting reviews is a key distinguishing feature. Even Paragons (see below) may get unfairly categorised as Sloths, as they are dependent on reviewers, who can delay the editorial process significantly. A true Paragon, however, will be pro-active in informing an author if there are unusual reasons for delays, whereas the distinguishing feature of a Sloth is that it is unresponsive to communications and blithely unconcerned about the impact of delays on authors.

Class 2

Species in class 2 pose less of a threat to an author’s career, but can nevertheless be dangerous to mental health.

The Talent Scout
This species is found in the rarefied habitats of the top high-impact journals, although it starting to spread and may now be found in medium-impact journals who have introduced a triage process. The Talent Scout’s principal concern is whether a research finding has star quality. The species is distinguished from other species by including individuals who are not active researchers:  many are individuals with a doctorate in science who have moved into science journalism. Although it can be depressing to have one’s work judged as too unsexy for publication by someone with no expertise in  your field, the decision-making process is usually mercifully quick, making it possible to regroup and resubmit elsewhere. Although this means that the impact on the author is less severe than for Class 1 editors, it does have potentially worrying implications for science as a whole, because it introduces bias. For instance, it is all too easy to see why Science published a study of a computer-based intervention for language problems in children: the study was headed by a top neuroscientist, the method was innovative, and it demonstrated potential to help children with a common neurodevelopmental disorder. A study like this presses all the buttons for the Talent Scout. However, the methodology was weak and subsequent randomised controlled trials (RCTs) have been disappointing (see review). I don’t know if authors of those RCTs would have tried to publish them in Science, but my guess is that if they did, their papers would have been rejected because it is simply much less interesting to show that something doesn’t work, than to provide evidence that it does (see blog).

The Deity
The Deity is the opposite of the Vacillator: the Deity makes decisions which may strike authors as unfair or subjective, but which are absolute and irreversible. Deities do not engage in correspondence with authors, but delegate this to office staff, as I found on the one occasion when I tried to engage in debate with a Deity from PNAS. I was incensed by a reviewer report that maintained a postdoc and I had been ‘cherrypicking’ results because we’d used an automated artefact removal procedure to remove noisy trials from a study using event-related potentials (ERPs). The reviewer clearly had no expertise in ERP methods and so did not realise that we were following standard practice, and that the idea of cherrypicking was just silly – it would be quicker to re-run the experiment than to go through the thousands of individual trials removing data we didn’t like the look of. In my letter to the editor, I explained that I did not want the paper reconsidered, but I did want an acknowledgement of the fact that I had not been fudging the data. What ensued was a tedious correspondence with a member of editorial staff, whose response was to send the paper back to the reviewer as part of an ‘appeal’ process, and to then inform me that the reviewer still didn’t like the paper.  Nowhere in this process did the Deity descend from the heights to offer any comment. Indeed, I still wonder whether this Deity was really an Automaton. It showed no signs of having any sense of morality.

Another encounter with a Deity was when I sent a paper to New England Journal of Medicine. Since I thought this should have at least warranted review (novel study with important clinical and theoretical implications), I wrote to ask what the reason was for rejecting it without review. The response from editorial staff was classic Deity: they could not give me any feedback as the paper had not been sent out for review.

Class 3

Species in class 3 are typified by their attitude to the job of editor, which is neither as bean-counter, nor as gatekeeper, but as facilitating the communication of high quality research.

The Paragon
The Paragon reads manuscripts and treats reviewer reports as advisory rather than as votes. He or she aims to make decisions fairly, promptly and transparently. Confronted with conflicting reviewer reports, a Paragon makes an honest attempt to adjudicate between them, and explains clearly to the author what needs to be done – or why a paper has been rejected. The Paragon will listen to author complaints, but not be swayed by personal friendship or flattery.  I’ve often heard authors complain about a Paragon who writes such a long decision letter that it is equivalent to a further reviewer report: I don’t see that as cause for complaint. I would sometimes do that myself when I was a journal editor (needless to say, I tried hard to be a Paragon), and I saw it as part of my job to pick up on important points that were missed by reviewers. Paragons write personal letters to authors, and to thank particularly helpful reviewers, rather than relying on computer-generated bureaucratese.

The Obsessive
The Obsessive is a Paragon that has gone over the top. Obsessives are not dangerous like class 1 and 2 editors: they typically damage themselves rather than the authors, to whom they are just irritating.  They essentially take upon themselves the job of copy editor, requiring authors to make minor changes to formatting and punctuation, rather than restricting themselves to matters of content and substance. Journal publishers have got wise to the fact that they can save a lot of money by sacking all the copy editors and requiring the academic editor to do the work instead, and they realise they have hit gold if they can find a natural Obsessive to do this. Academics should be aware of this trap: their training equips them to judge the science, and they should not spend hours looking for extraneous full stops and missing italicisation.  They should remember that they already do work, typically for no reward, for publishers who make a lot of money from journals, and they should demand that the publisher offers appropriate support to them and their authors.  (They should also employ people to assist authors with graphics – see blog).

Summing up

The main problem for authors is that you often don’t know what species of editor you are dealing with until after the event of submitting a paper. In my main field of psychology, I am impressed at how many journals do have Paragons. The APA journals are usually good, in my experience, though Obsessives do make an appearance, and I know of one case where a junior colleague’s career was seriously blighted by a mega-Sloth.  It’s harder to generalise about small moderate-impact journals: many of them are overseen by a dedicated Paragon, but my impression is that you can only be a Paragon for 10 years at most. Editors who have served a longer term than this are liable to transmute into Sloths.  I also publish in the fields of neuroscience and genetics, and here I’ve struck more variability, with Returning Officers, Automatons and Deities being fairly prevalent.  If you want to publish in the really top journals, you have to grapple with Talent Scouts:  my attempts to make my work exciting enough for them have been singularly unsuccessful, and I've given up on them, but it may encourage younger readers to know that I’ve had a happy and successful career all the same.

Note: The author was co-editor of Journal of Child Psychology and Psychiatry from 1990-1993 and Chief Editor from 1994-1997. This year she signed up as an Academic Editor for PLOS One, in support of their Open Access publishing policy.

Science and journalism: an uneasy alliance

“Fish oil helps schoolchildren to concentrate” shouted the headline in the Observer, “US academics discover high doses of omega-3 fish oil combat hyperactivity and attention deficit disorder”.  Previous research on this topic has been decidedly underwhelming (see slides for 7th BDA international conference), so I set off to track down the source article.

Well, here's a surprise:  the study did not include any children with ADHD. It was an experiment with 33 typically-developing boys. And another surprise: on a test of sustained attention, there was no difference between boys who'd been  given supplementation of an omega 3 fatty acid (DHA) for 8 weeks and those given placebo. Indeed, boys given low-dose supplementation made marginally more errors after treatment. So where on earth did this story come from? Well, in a brain scanner, children given DHA supplementation showed a different pattern of brain activity during a concentration task, with greater activation of certain frontal cortical regions than the placebo group. However, the placebo group showed greater activation in other brain regions. It was not possible to conclude that the brains of the treated group were working better, given the large number of brain regions being compared, and the lack of relationship between activation pattern and task performance. 

A day or two later, another article was published, this time in the Guardian, with the headline Male involvement in pregnancy can weaken paternal bond. I tried to track down the research report. I couldn’t find it. I traced the researcher. He told me that the piece was not referring to published research, but rather to views he had expressed in an interview with a journalist. He told me he had not intended to recommend that fathers stay away from antenatal classes. He was also concerned that the article had described him as Director of his research institute - in fact he is a lecturer.

At this point, inspired by the example of the Ig Nobel prize, I announced the Orwellian Prize for Journalistic Misrepresentation, an award for the most inaccurate newspaper report of an academic piece of work, using strict and verifiable criteria. An article would get 3 points for an inaccuracy in the headline, 2 points for inaccuracy in the subtitle, and 1 point for inaccuracy in the body of the article. The fish oil piece totalled 16 points.

Comments on the prize were mostly supportive. I had thought I might attract hordes of journalistic trolls but they did not materialise. Indeed, several journalists responded positively, though they also noted some difficulties for my scoring system. They politely pointed out, for instance, that headlines, to which I gave particular weight in the scoring, are not written by the journalist. Also, it is not unknown for university press officers, who regard it as their job to get their institution mentioned in the media, to give misleading and over-hyped press releases, sometimes endorsed by attention-seeking researchers.

But over in the mainstream media, a fight was brewing up. Ben Goldacre, whose Bad Science column in the Guardian I’ve long regarded as a model of science communication, independently picked up on the fish oil article and gave its author a thorough lambasting.  Jeremy Laurance of the Independent retorted with a piece in which he attacked Goldacre. Laurance made three points: first, science journalism is generally good; second, reporters can’t be expected to check everything they are told (implying that the fault for inaccuracy lay with the researcher in this case), and third, that journalists work under intense pressure and should not be castigated for sometimes making mistakes.

I would be the first to agree with Laurance’s initial point. During occasional trips to Australia and North America, I've found the printed media to be mostly written as if for readers with rather few neurons and no critical faculties. Only when deprived of them do you appreciate British newspapers. They employ many talented people who can write engagingly on a range of issues, including science. Regarding the second point, I am less certain. While I have some sympathy with the dilemma of a science reporter who has to report on a topic without the benefit of expertise, stories of hyped-up press releases and self-publicising but flawed researchers are numerous enough that I think any journalist worth their salt should at least read the abstract of the research paper, or ask a reputable expert for their opinion, rather than taking things on trust. This is particularly important when writing about topics such as developmental disorders that make people’s life a misery. Many parents of children with ADHD would feed their child a diet of caviare if they felt it would improve their chances in life. If they read a piece in a reputable newspaper stating that fish oil will help with concentration, they will go out and buy fish oil.(I've no idea whether fish oil sales spiked in June, but if anyone knows how to check that out, I'd be interested in the answer).  In short, reporting in this area has consequences – it can raise false hopes and make people spend unnecessarily.

On the third point, lack of time, Goldacre’s supporters pointed out that working as a doctor is not exactly a life of leisure, yet Ben manages to do a meticulously researched column every week. Other science bloggers write excellent pieces while holding down a full-time day-job.

It was unfortunate indeed that the following week, Laurance, whom I've always regarded as one of our better science journalists, produced a contender for the Orwellian in an Independent report on a treatment for people with Alzheimer’s disease. Under the title 'Magnets can improve Alzheimer’s symptoms' he described a small-scale trial of a treatment based on repetitive transcranial magnetic simulation, a well-established method for activating or inhibiting neurons by using a rapidly changing strong magnetic field. In this case,  the account of the research seemed accurate enough. The problem was the context in which Laurance placed the story, which was to draw parallels with ‘magnet therapy’ involving the use of bracelets and charms.  Several commentators on the electronic version of the story went on the attack, with one stating “This is not worthy of print and it is absolutely shameful journalism.”

I was recently interviewed for the More or Less radio 4 program about the Orwellian Prize, together with a science journalist who clearly felt I was being unfair in not making allowances for the way journalists work – using arguments similar to those made by Jeremy Laurance. At one point when we were off the air, she said, “But don’t you make loads of mistakes?” I realised when I said no that I was simultanously tempting fate, and giving an impression of arrogance.  Of course I do make mistakes all the time, but  I go to immense lengths of checking and rechecking papers, computations, etc, to avoid making errors in published work. A degree of obsessionality is an essential attribute for a scientist. If our published papers contained ‘loads of’ mistakes we’d be despised by our peers, and probably out of a job.

But is the difference between journalists and scientists just one of accuracy?  My concern is that there is much more to it than that. I did a small experiment with Google to find out how long it would take to find an account of transcranial magnetic stimulation. Answer: less than a minute. Wikipedia gives a straightforward description that makes  it abundantly clear that this treatment has nothing whatever to do with 'magnet therapy'. Laurance may be a busy man, but this is no excuse for his failure to check this out.

So here we come to the nub of the matter, and the reason why scientists tend to get cross about misleading reporting: it is not just down to human error. The errors aren't random: they fall in a particular pattern suggesting that pressure to produce good stories leads to systematic distortion, in a distinctly Orwellian fashion. Dodgy reporting comes in three kinds:

1. Propaganda: the worst case of misleading information, when there is deliberate distortion or manipulation of facts to support the editor’s policy. I think and hope this is pretty rare, though some reporting of climate change science seems to fall in this category. For instance, the Australian, the biggest-selling national daily newspaper in Australia, seems much happier to report on science that queries climate change than on science that provides evidence for it. A similar pattern could be detected in the hysteria surrounding the MMR controversy, where some papers only covered stories that argued for a link between vaccination and autism. It is inconceivable that such bias is just the result of journalists being too inexpert or too busy to check their facts. Another clue to a story being propaganda is when it goes beyond reporting of science to become personal, querying the objectivity, political allegiances and honesty of the scientists. Because scientists are no more perfect than other human beings, it is important that journalists do scrutinise their motives, but the odd thing is that this happens only when scientists are providing inconvenient evidence against an editorial position. The Australian published 85 articles about the 'climategate' leaked emails, in which accusations of dishonesty by scientists were repeated, but they did not cover the report vindicating the scientists at all. 

2. Hype. This typically does not involve actual misrepresentation of the research, but a bending of its conclusions to fit journalistic interests, typically by focusing more on future implications of a study rather than its actual findings. Institutional press officers, and sometimes scientists themselves, may collude with this kind of reporting, because they want to get their story into the papers and realise it needs some kind of spin to be publishable. In my interview with More or Less, I explained how journalists always wanted to know how research could be immediately applied, and this often led to unrealistic claims (see my blog on screening, for examples). The journalist’s response was unequivocal. She was perfectly entitled to ask a scientist what relevance their work was, and if the answer was none, then why were they taking public money to do it? But this reveals a misunderstanding of how research works.  Scientific discoveries proceed incrementally, and the goal of a study is often increased understanding of a phenomenon. This may take years: in terms of research questions, the low-hanging fruit was plucked decades ago, and we are left with the difficult problems.  Of course, if one works on disorders, the ultimate goal is to use that understanding to improve diagnosis or treatment, but the path is a long and slow one.  I discussed the conflict between the nature of scientific progress and the journalists’ need for a ‘breakthrough’ in another blog. So the typical researcher is, on the one hand, being encouraged by their institution to talk to the media, and on the other hand knows that their research will be dismissed as uninteresting (or even pointless) if it can’t be bundled into a juicy sound-bite with a message for the lay person. One of two reactions ensues: many scientists just give up attempting to talk to the media; others are prepared to mould an account of their research into what the journalists want. This means that the less scrupulous academics are more likely to monopolise media attention.

3. Omission: this is harder to pin down, but is nonetheless an aspect of science journalism that can be infuriating. What happens is that the papers go overboard for a story on a particular topic, but totally ignore other research in the same area. So, a few weeks before the fish-oil/ADHD paper was covered, a much larger and well-conducted trial of omega-3 supplementation in school-children was published but ignored by the media. Another striking example was when the salesman Wynford Dore was actively promoting his expensive exercise-based treatment for dyslexia, skilfully using press releases to get media coverage, including a headline item on the BBC News. The story came from a flawed small-scale study published in a specialist journal. While this was given prominence, excellent trials of other more standard interventions went unreported (for just one example, see this link).  I guess it is inevitable: Telling the world that you can cure dyslexia by balancing on a wobble board is newsworthy - it has both novelty and human interest. Telling the world that you can improve reading  with a phonologically-based intervention has a bit of human interest but is less surprising and less newsworthy. Telling the world that balancing on a wobble board has no impact on dyslexia whatsoever is not at all surprising, and is only of interest to those who have paid £3000 for the intervention, so it's totally un-newsworthy.  It's easy to see why this happens: it's just a more extreme form of the publication bias that also tarnishes academic journals whose editors favour 'interesting' research (see also Goldacre on similar issues). Problem is, it has consequences.

For an intelligent analysis of these issues, see Zoe Corbyn’s article in the Times Higher Education, and for some ideas about alternative approaches to science reporting, a blog by Alice Bell. I, meanwhile, am hoping that there won’t be any nominations for the Orwellian Prize that earn more points than the fish oil story, but I’m not all that confident.

P.S. I wanted to link to the original fish oil article, but it is no longer available on the web. The text is on my blog page describing the Orwellian prize.

P.P.S. Ah, I’ve just had a new nomination that gets 17 points, largely because it ignored wise advice tweeted recently by Noah Gray (@noahWG), Senior Editor at Nature: “Journalism Pro Tip: If your piece starts talking more about a study's untested implications rather than what the science showed, start over."

P.P.P.S It has been gently pointed out to me that I erred in original version of this blog, and said the Laurance magnet piece was in the Guardian, when in fact it was in the Independent.  Deeply embarrassing but now corrected.

Genes for optimism, dyslexia and obesity and other mythical beasts

Copyright: www.CartoonStock.com

I recently received an email from a company called mygeneprofile: “By discovering your child's inborn talents & personality traits, it can surely provide a great head start to groom your child in the right way… our Inborn Talent Genetic Test has 99.8% accuracy.” I’d registered to receive information from the company having heard they were offering a genetic test for such diverse traits as optimism, composure, intelligence, and dancing (link).

Despite all the efforts of the Human Genome Project, I was not aware of any genetic test that could reliably predict a child’s personality or ability. I was not therefore surprised when my emails asking for evidence went unanswered, though I continue to receive messages that oscillate between carrots (free gifts! discounts!!) and sticks (without this test “your child will have MISERABLE life (sic))”.

The test company relies on a widespread assumption that people’s psychological attributes are predictable from their genes. So where does this belief come from, and why is it wrong? 

People’s understanding of genetic effects is heavily influenced by the way genetics is taught in schools. Mendel and his wrinkly and smooth peas make a nice introduction to genetic transmission, but the downside is that we go away with the idea that genes have an all-or-none effect on a binary trait.  Some characteristics are inherited this way (more or less), and they tend to be the ones that textbooks focus on: e.g., eye colour, colour-blindness, Huntington’s disease. But most genetic effects are far more subtle and complex than this. Take height, for instance. Genes are important in determining how tall you are, but this is not down to one gene: instead, there is a whole host of genes, each of which nudges height up or down by a small amount (see link).

The expression of a gene may also depend crucially on the environment; for instance, obesity relates both to calorie intake and genetic predisposition, but the effects are not just additive: some people can eat a great deal without gaining weight, whereas in others, body mass depends substantially on food intake (see link). Furthermore, a genetic predisposition to obesity can be counteracted by exercise (see link).Furthermore, genetic influences may interact in complicated ways.  For instance, coat colour in mice is affected by combinations of genes, so that one cannot predict whether a mouse is black, white or agouti (mouse coloured!) just by knowledge of status of one gene.

This means that we get a very different impression of strength of genetic influences on a trait if we look at the impact of a person’s whole genome, compared to looking at individual genes in isolation. The twin study was the traditional method for estimating genetic influences before we had the technology to study genes directly, and it compares how far people’s similarity on a trait depends on their genetic relationship. Researchers measure a trait, such as sensation-seeking, in identical and fraternal twin pairs growing up in the same environment and consider whether the two twin types are equally similar. If both sets of twins resemble each other equally strongly, this indicates that the environment, rather than genes, is critical. And if twins don’t resemble one another at all, this could mean either that the trait is influenced by child-specific experiences, not shared by the co-twin, or that our measure of sensation-seeking is unreliable.  But if identical twins are more similar than fraternal twins, this means genes affect the trait, i.e. it is heritable. There are several niggly criticisms of the twin method; for instance, it can give misleading estimates if identical twins are treated more similarly than fraternal twins, or if twinning itself influences the trait in question. For most traits, however, these don’t seem sufficient to explain away the substantial heritability estimates that are found for traits such as height, reading ability, and sensation-seeking.  But these estimates don’t tell us about the individual genes that influence a trait – they rather indicate how important genes are relative to non-genetic influences.

Interactive effects, either between multiple genes or between genes and environments, will not be detected in a conventional twin study analysis. If a gene is expressed only in a particular environment, twins who have the same version of the gene will usually also have the same environment, and so the expression of the gene will be the same for both. And for an effect that depends on having a particular combination of genes, identical twins will have the same constellation of genetic variants, whereas the likelihood of fraternal twins having an identical gene profile decreases with the number of genes involved.  Heritability estimates depend on comparing similarity of a trait for identical vs fraternal twins, and will be increased if gene-gene interactions are involved.

In contrast,  genome-wide association studies are designed to find individual genes that influence specific traits. They adopt the strategy of looking for associations between DNA variants (alleles) and the trait, either by categorising people, e.g. as dyslexic or not, and comparing the proportions with different alleles, or by seeing whether people who have zero, one or two copies of an allele differ in their average score on a trait such as reading ability.  When these studies started out, many people assumed we would find gene variants that exerted a big effect, and so might reasonably be termed ‘the gene for” dyslexia, optimism, and so on.  However, this has not been the case.

Take personality, for instance, one of the domains that mygeneprofile claims to test for. A few weeks ago, a major study was reported in which  the genes of over 5000 people were investigated but no significant associations were found. Commentators on the research argued that the measurement of personality – typically on the basis of self-report questionnaires - may be the problem.  But the self-same measures yield high estimates of heritability when used in twin studies.  And a similar pattern has been found for other traits: including height, intelligence and obesity, i.e., a mismatch in evidence of genetic influence from twin studies (typically moderate to strong for these traits) and findings of individual genes associated with the trait (with effects that are very small at best). 

This account may surprise readers who have read of recent discoveries of genes for conditions such as dyslexia, where the impression is sometimes given that there are strong effects.  The reason is that reports of molecular genetic studies usually emphasise the p-value, a measure of how probable it is that a result could have arisen by chance. A low p-value indicates that a result is reliable, but it does not mean the effect is large. These studies typically use very large samples precisely because this allows them detect even small effects.  Consider one of the more reliable associations between genes and behaviour: a gene known as KIAA0319 which has been found to relate to reading ability in several different samples. In one study, an overall association was reported with p = .0001, indicating that the likelihood of the association being a fluke is 1 in 10,000. However, this reflected the fact that one gene variant was found in 39% of normal readers and only 25% of dyslexics, with a different variant being seen in 30% of controls and 35% of dyslexics. Some commentators have argued that such small effects are uninteresting.  I disagree: findings like this can pave the way for studies into the neurobiological effects of the gene on brain development (see link), and for studies of gene-gene and gene-environment interactions.  But it does mean that talk of a ‘gene for dyslexia’, or genetic screening for personality or ability are seriously misguided.

The small effect size of individual genes, and interactions with environment or other genes,  are not the only explanations for “missing heritability”. A trait may be influenced by genetic variants that have a large effect but which are individually very rare in the population. These would be very hard to detect using current methods. The role of so-called copy number variants is also a focus of current interest: these are large chunks of DNA which are replicated or deleted and which are surprisingly common in all of us.  These lead to an increase or decrease in gene product, but won’t be found with standard methods that focus just on identifying the DNA sequence. Both mechanisms are thought to be important in the genetics of autism, which is increasingly looking like a highly heterogeneous condition – i.e. there are multiple genetic risk factors and different ones are important in different people. 

What are the implications of all of this for the stories we hear in the media about new genetic discoveries?  The main message is we need to be aware of the small effect of most individual genes on human traits. The idea that we can test for a single gene that causes musical talent, optimism or intelligence is just plain wrong. Even where reliable associations are found, they don’t correspond to the kind of major influences that we learned about in school biology. And we need to realise that twin studies, which consider the total effect of a person’s genetic makeup on a trait, can give different results from molecular studies of individual genes. What makes us individual can’t be reduced to the net effect of a few individual genes.

Background reading

Bishop, D. V. M. (2009). Genes, cognition and communication: insights from neurodevelopmental disorders. The Year in Cognitive Neuroscience: Annals of the New York Academy of Sciences, 1156, 1-18.

Maher, B. (2008). Personal genomes: The case of the missing heritability. Nature, 456, 18-21 doi:10.1038/456018a.

Plomin, R., DeFries, J. C., McClearn, G. E.,  McGuffin, P. (2008). Behavioral Genetics. (5th Edition). New York: Worth Publishers.

Rutter, M. (2006). Genes and Behavior: Nature-Nurture Interplay Explained. Oxford: Blackwell.

Note: this is a slightly extended version of a blog on Guardian Science Blog, 9/9/10

Three ways to improve cognitive test scores without intervention

My research focuses on neurodevelopmental disorders such as autism, specific language impairment and dyslexia, where more or less miraculous cures are heralded at regular intervals. These claims are often unfounded, being based on anecdote or uncontrolled studies. I often feel like a kill-joy, picking holes in reports that offer hope to parents and their children. But I carry on because I don't like to see people being misled, often at considerable personal expense, and sometimes even being encouraged to subject their child to practices that are at best boring and at worst painful or dangerous.

Psychological or educational interventions have much in common with medical treatments, and the same randomised-controlled trial (RCT) methodology is needed to obtain convincing evidence of efficacy. But what is a RCT controlling for?  In drug treatments, the emphasis is on the need to control for (a) spontaneous improvement or (b) placebo effects, i.e. the improvement that you get just from being the focus of medical attention.  In psychological and educational interventions there are three other important sources of potential bias that can give a misleading impression of treatment efficacy unless they are controlled for: developmental change, practice effects and regression to the mean.

Developmental change
 

This is the simplest to understand. Basically, children get better at most things as they get older. If someone gives me a brain training game that is supposed to cure dyslexia, and my child does the exercises every day for a year, he may well be able to read more words at the end of the intervention than he could at the start. However, this would hardly be good evidence for effectiveness of the games, since most children learn more words as they grow older. Indeed, his feet will also be bigger at the end of the intervention, but I would not attribute that to the brain training. This is so obvious that it may seem unnecessary to state it, except for the fact that with some conditions, people assume that development is static.  For instance, the National Autistic Society website states "Autism is a lifelong developmental disability".  Now, this does not mean that children with autism will not change with age, yet it is easy to see how someone might misinterpret it. In fact, diagnostic instruments for autism focus on the child's behaviour at 4 to 5 years of age, precisely because that is when autistic features are most evident.  As the child grows older, these features may become milder or take a different form. See, for instance, the detailed account by Clara Claiborne Park of her autistic daughter, who barely used speech at 4 years, but could produce simple sentences by the age of 8. If your child has some intervention around the age of 4 years and then shows improvements, it is easy to assume that the intervention was responsible.  However, the improvement may well have occurred anyway.  The same is true for children who are late to start talking. This was brought home to me by a study conducted in the 1980s evaluating home-based  therapy with late talkers in an inner city area. The treated children showed a dramatic improvement.  However, so too did an untreated control group.  Without the evidence of the control group it would be easy to be misled into thinking that all late talkers should be given the home-based intervention, but a very different recommendation would follow when the controls are taken into account.  The data suggested that it would be better to adopt a 'watchful waiting' policy, because most children improved spontaneously. I would not want to imply we should not treat children with language difficulties: my own research has been concerned with identifying which children won't catch up by themselves, so we can target intervention appropriately.  But it is easy to underestimate the amount of change that can occur without intervention, especially in very young children.

Practice

On many tests, people get better with practice. Many people will have experienced the WiiFit machine, which gives you a series of tests to estimate your 'fitness age'. These involve activities such as standing on a balance board and trying to keep a dot representing your centre of gravity in a central position, or doing complex memory or calculation tests under time pressure. The first time I did this, my fitness age was estimated at 76. However, just one day later, it had come down to 60 years, and with further practice there was a steep decline. I don't think my ineffectual and limited attempts at WiiFit exercises had reversed 16 years worth of decrepitude: rather, I got better because I had a clearer idea of what I was supposed to be doing.  The first time I had no strategy for performing the tests, and in some cases completely misunderstood what I was supposed to be doing. So it was hardly surprising that I improved dramatically just by my second attempt.  It is worth noting that balance tests are used by Dore centres to evaluate the success of their training program for neurodevelopmental disorders.  Some of these tests show substantial practice effects (Liao et al, 2001)  Parents can see their child's performance improving, and gain the impression that this indicates that important brain changes are occurring. However, without correction for practice effects this is misleading. (As far as I can establish, Dore centres don't correct for developmental changes in balance either, though these are well documented on such tests (e.g. http://www.ncbi.nlm.nih.gov/pubmed/17105679).

Cognitive tests vary in the extent to which they show practice effects, but most will show some benefits of prior exposure. Take a couple of well-known tests of non-verbal ability, matrices and object assembly.  Matrices involves working out the logic behind visual patterns. Once you have worked out how a specific pattern is constructed, it is much easier to do it on the next occasion, because you don't have to entertain a wide range of hypotheses about the correct answer.  Object assembly involves a jig-saw like task where you assemble pieces to make a meaningful figure, but you don't know what the meaningful figure is – at least not the first time you do it. Once you have done it, you are likely to show gains in performance on a second occasion even after some years.  Over a short interval, Rapport et al (1997) showed gains in IQ of 5-10 points after one previous test.  I studied short-term practice effects on a language comprehension test I have developed: children showed significant gains in scores on the second administration, even though I used different test items (but the same format).  McArthur has specifically looked at the extent to which improvements are seen in untreated groups of children on tests of reading and language, and has documented substantial gains.  

Practice effects mean that gains in cognitive test scores after an intervention are usually uninterpretable unless we have comparison information from individuals who were given the same schedule of testing without the intervening intervention.   Again, it sounds obvious, but it is a point that is often missed by researchers, as well as by non-scientists.  A striking example comes from a study of interventions for children with speech-language problems by Gillam and colleagues. This study was designed as a randomised controlled trial, and had four groups: three of them were given specific treatments designed to improve children's language, and the fourth was a control group who were given a  computerised training of general 'academic skills'. Everyone improved, but the amount of improvement was no greater for the three treatment groups than for the control group.  The conventional interpretation of such a result would be to conclude that the treatments were ineffective, because the changes associated with them were no greater than for the control group.  However, impressed by the increase in scores and apparently unaware of the possibility of practice effects, the authors of the study concluded instead that all treatments were effective, and spent some time discussing why the control treatment might be good at boosting language.

One further point: it is sometimes assumed that we don't need to worry about practice effects if a test has good test-retest reliability. But that is not the case. High reliability just means that the rank ordering of a group of children will be stable across testing occasions.  But reliablity indices are insensitive to absolute level of performance. So if we took a group of children's scores at time 1 and gave each of them a score 10 points higher at time 2, then we would have perfect reliability because their rank ordering would be the same. Reliability is, however, an important factor when considering the last topic, regression toward the mean.

Regression toward the mean
 
Regression toward the mean is a horribly difficult concept to understand, even for those with statistical training.  In their masterly introduction to the subject, Campbell and Kenny noted that regression to the mean has confused many people, including Nobel prize winners.   More specifically, they stated that: “Social scientists incorrectly estimate the effects of ameliorative interventions.....and snake-oil peddlers earn a healthy living all because our intuition fails when trying to comprehend regression toward the mean". 

Perhaps the easiest way to get a grasp of what it entails is to suppose we had a group of 10 people and asked them each to throw a dice 10 times and total the score.   Let's then divide them into the 5 people who got the lowest scores and the 5 who got the highest scores and repeat the experiment. What do we expect? Well, assuming we don't believe that anything other than chance affects scores (no supernatural forces or 'winning streaks'), we'd expect the average score of the low-scorers to improve, and the average score of the high scorers to decline.  This is because the probability for any one person is that they will get an average score on any one set of dice throws.  So that's the simplest case, when we have a score that is determined only by chance.

Cognitive test scores are interesting here because they are typically thought of comprising two parts: a 'true' score, which reflects how good you really are at whatever is being measured, and an 'error' score, which reflects random influences. Suppose, for instance, that you test a child's reading ability. In practice, the child is a very good reader, in the top 10% for her age, but the exact number of words she gets right will depend on all sorts of things: the particular words selected for the test (she may know 'catacomb' but not 'pharynx'), the mood she is in on the day, whether she is lucky or unlucky at guessing at words she is uncertain about. All these factors would be implicated in the 'error' score, which is treated just like an additional chance factor or throw of the dice affecting a test score.  A good test is mostly determined by the 'true' score, with only a very small contribution of an 'error' score, and we can identify it by the fact that children will be ranked by the test in a very similar way from one test occasion to the next, i.e. there will be good test-retest reliablity.  In other words, the correlation from time 1 to time 2 will be high.

Simulated test scores for 21 children on tests varying in reliability

These figures show simulated scores for a group of 21 children on tests that vary in test-retest reliability.  In each case individuals are colour-coded depending on whether they fall in the bottom (blue), middle (purple) or top (red) third of scores at time 1. The simulations assume no systematic differences between time 1 and 2 - i.e. no intervention effect, maturation or practice.  Scores are simulated as random numbers at time 1, with time 2 scores then set to give a specific correlation between time 1 and 2, with no change in average score for the group as a whole.

Now suppose we select children because they have a particularly low score at time 1. Insofar as chance contributes to their scores, then at time 2, we would expect the average score of such a group to improve, because chance pushes the group average towards the overall mean score.  The left-hand panel shows the situation when reliability (i.e., correlation between time 1 and time 2 scores) is zero, so scores are determined just by chance, like throwing a dice. The mean scores for the blue, purple and red cases at time 1 are, by definition different (they are selected to be low, medium and high scorers). But at time 2 they are all equivalent.  The upshot is that the mean at time 2 for those with initial low scores (blue) goes up, whereas for those that start out with high scores, the time 2 mean comes down.
 
The middle and right-hand panels show a more realistic situation, where the test score is mixture of true score and error. With very high reliability (right-hand panel) the effect of regression to the mean is small, but with medium reliability (middle panel) it is detectable by eye even for this very small sample.

The key point here is that if we select individuals on the basis of low scores on a test (as is often done, e.g. when identifying children with poor reading scores for a study of a dyslexia treatment), then, unless we have a highly reliable test with a negligible error term, the expectation is that the group's average score will improve on a second test session, for purely statistical reasons.  In general, psychometric tests are designed to have reasonable reliability, but this varies from test to test and is seldom higher than .75-.8.

So regression to the mean is a real issue in longitudinal studies.  It is yet another reason why scores will change over time. Zhang and Tomblin (2003) noted that we can overcome this problem by using different tests to select children for a study and to measure their improvement.  Or we can allow for regression to the mean if our study includes a control group, who will be subject to the same regression to the mean as the intervention group.

So, if you started reading this blog thinking that people who demand control groups for intervention studies were just a kind of methodological Stasi whose obsession with RCTs was misplaced, I hope I have convinced you that the concerns are real. There are just so many ways that we can mislead ourselves into thinking that an intervention has led to improvement, when in fact it is making no difference. It is depressingly hard to demonstrate positive effects of treatment, but the only way to do it is by using properly designed studies that can overcome the biases I have noted here by use of appropriate controls. 



Addition resources on this topic can be found on this section of my website, see esp. 
  7th BDA International Conference – 2008: Dyslexia: Making Links
  2nd UK Paediatric Neuropsychology Symposium
  DysTalk online (video)

And a couple of short articles on how not to do intervention studies:

BISHOP, D. V. M.  2007. Curing dyslexia and ADHD by training motor co-ordination: Miracle or myth? Journal of Paediatrics and Child Health, 43, 653-655.
BISHOP, D. V. M. 2008. Criteria for evaluating behavioural interventions for neurodevelopmental disorders (Letter). Journal of Paediatrics and Child Health, 44, 520-521.

How our current reward structures have distorted and damaged science

copyright www.CartoonStock.com

Two things almost everyone would agree with:

1. Good scientists do research and publish their results, which then have impact on other scientists and the broader community.

2. Science is a competitive business: there is not enough research funding for everyone, not enough academic jobs in science, and not enough space in scientific journals. We therefore need ways of ensuring that the limited resources go to the best people.

When I started in research in the 1970s, research evaluation focused on individuals. If you wrote a grant proposal, applied for a job, or submitted a paper to a journal, evaluation depended on peer review, a process that is known to be flawed and subject to whim and bias, but is nevertheless regarded by many as the best option we have.

What has changed in my lifetime is the increasing emphasis on evaluating institutions rather than individuals. The 1980s saw the introduction of the Research Assessment Exercise, used to evaluate Universities in terms of their research excellence in order to have a more objective and rational basis for allocating central funds (quality weighted research funding or QR) by the national funding council (HEFCE in England). The methods for evaluating institutions evolved over the next 20 years, and are still a matter of debate, but they have subtly influenced the whole process of evaluation of individual academics, because of the need to use standard metrics.

This is inevitable, because the panel evaluating a subject area can't be expected to read all the research produced by staff at an institution, but they would be criticised for operating an 'old boy network', or favouring their own speciality, if they relied just on personal knowledge of who is doing good work – which was what tended to happen before the RAE. Therefore they are forced into using metrics. The two obvious things that can be counted are research income and number of publications. But number of publications was early on recognised as problematic, as it would mean that someone with three parochial reports in a journal of national society would look better than someone with a major breakthrough published in a top journal. There has therefore been an attempt to move from quantity to quality, by taking into account the impact factor of the journals that papers are published in.

Evaluation systems always change the behaviour of those being evaluated, as people attempt to maximise rewards. Recognising that institutional income depends on getting a good RAE score, vice-chancellors and department heads in many institutions now set overt quotas for their staff in terms of expected grant income and number of publications in high impact journals. The jobs market is also affected, as it becomes clear that employability depends on how good one looks on the RAE metrics.

The problem with all of this is that it means that the tail starts to wag the dog. Consider first how the process of grant funding has changed. The motivation to get a grant ought to be that one has an interesting idea and needs money to investigate it. Instead, it has turned into a way of funding the home institution and enhancing employability. Furthermore, the bigger the grant, the more the kudos, and so the pressure is on to do large-scale expensive studies. If individuals were assessed, not in terms of grant income, but in terms of research output relative to grant income, many would change status radically, as cheap, efficient research projects would rise up the scale. In psychology, there has been a trend to bolt on expensive but often unmotivated brain imaging to psychological studies, ensuring that the cost of each experiment is multiplied at least 10-fold. Junior staff are under pressure to obtain a minimum level of research funding, and consequently spend a great deal of time writing grant proposals, and the funding agencies are overwhelmed with applications. In my experience, applications that are written because someone tells you to write one are typically of poor quality, and just waste the time of both applicants and reviewers. The scientist who is successful in meeting their quota is likely to be managing several grants. This may be a good thing if they are really talented, or have superb staff, but in my experience research is done best if the principal investigator puts serious thought and time into the day-to-day running of the project, and that becomes impossible with multiple grants.

Regarding publications, I am old enough to have been publishing before the RAE, and I'm in the fortunate but unusual position of having had full-time research funding for my whole career. In the current system I am relatively safe, and I look good on an RAE return. But most people aren't so fortunate: they are trying to juggle doing research with teaching and administration, raising children and other distractions, yet feel under intense pressure to publish. The worry about the current system is that it will encourage people to cut corners, to favour research that is quick and easy. Sometimes, one is lucky, and a simple study leads to an interesting result that can be published quickly. But the best work typically requires a large investment of time and thought. The studies I am proudest of are ones which have taken years rather than months to complete: in some cases, the time is just on data collection, but in others, the time has involved reading, thinking, and working out ways of analysing and interpreting data. But this kind of paper is getting increasingly rare. As a reviewer, I frequently see piecemeal publication, so if you suggest that a further analysis would strengthen the paper, you are told that it has been done, but is the subject of another paper. Scholarship and contact with prior literature has become extremely rare: prior research is cited without reading it – or not cited at all – and the notion of research building on prior work has been eroded to the point that I sometimes think we are all so busy writing papers that we have no time to read them. There are growing complaints about an 'avalanche' of low-quality publications.

As noted above, in response to this, there has been a move to focus on quality rather than quantity of publications, with researchers being told that their work will only count if it is published in a high-impact journal. Some departments will produce lists of acceptable journals and will discourage staff from publishing elsewhere. In effect, impact factor is being used as a proxy for likelihood that a paper will be cited in future, and I'm sure that is generally true. But just because a paper in a high impact journal is likely to be highly cited, it does not mean that all highly-cited papers appear in high impact journals. In general, my own most highly-cited papers appeared in middle-ranking journals in my field. Moreover, the highest impact journals have several limitations:

1. They only take very short papers. Yes, it is usually possible to put extra information in 'supplementary material', but what you can't do is to take up space putting the work in context or discussing alternative interpretations. When I started in the field, it was not uncommon to publish a short paper in Nature, followed up with a more detailed account in another, lowlier, journal. But that no longer happens .We only get the brief account.

2. Demand for page space outstrips supply. To handle a flood of submissions, these journals operate a triage system, where the editor determines whether the paper should go out for review. This can have the benefit that rejection is rapid, but it puts a lot of power in the hands of editors, who are unlikely to be specialists in the subject area of the paper, and in some cases will explicit in their preference for papers with a 'wow' factor. It also means that one gets no useful feedback from reviewers: viz my recent experience with the New England Journal of Medicine, where I submitted a paper that I thought had all the features they'd find attractive – originality, clinical relevance and a link between genes and behaviour. It was bounced without review, and I emailed, not to appeal, but just to ask if I could have a bit more information about the criteria on which they based their rejection. I was told that they could not give me any feedback as they had not sent it out for review.

3. If the paper does go out for review, the subsequent review process can be very slow. There's an account of the trials and tribulations of dealing with Nature and Science which makes for depressing reading. Slow reviewing is clearly not a problem restricted to high impact journals. My experience is that lower-impact journals can be even worse. But the impression from the comments on FemaleScienceProfessor's blog, is that reviewers can be unduly picky when the stakes are high.

So what can be done? I'd like to see us return to a time when the purpose of publishing was to communicate, and the purpose of research funding was to enable a scientist to pursue interesting ideas. The current methods of evaluation have encouraged an unstoppable tide of publications and grant proposals, many of which are poor quality. Many scientists are spending time on writing doomed proposals and papers when they would be better off engaging in research and scholarship in a less frenetic and more considered manner. But they won't do that so long as the pressures are on them to bring in grants and generate publications. I'll conclude with a few thoughts on how the system might be improved.

1. My first suggestions, regarding publications, are already adopted widely in the UK, but my impression is they may be less common elsewhere. Requiring applicants for jobs or fellowships to specify their five best publications rather than providing a full list rewards those who publish significant pieces of work, and punishes piecemeal publication. Use of the H-index as an evaluation metric rather than either number of publications or journal impact factor is another way to encourage people to focus on producing substantial papers rather than a flood of trivial pieces, as papers with low citations have no impact whatever on the H-index.There are downsides: we have the lag problem, which makes the H-index pretty useless for evaluating junior people, and in its current form the index does not take into account the contribution of authors, thereby encouraging multiple authorship, since anyone who can get their name on a highly-cited paper will boost their H-index, regardless of whether they are a main investigator or freeloader.

2. Younger researchers should be made aware that a sole focus on publishing in very high impact journals may be counter-productive. Rapid publication in an Open Access journals (many of which have perfectly respectable impact factors) may be more beneficial to ones career (http://openaccess.eprints.org/) because the work is widely accessible and so more likely to be cited. A further benefit of the PLOS journals, for instance, is that they don't impose strict length limits, so research can be properly described and put in context, rather than being restricted to the soundbite format required by very high impact journals.

3. Instead of using metrics based on grant income, those doing evaluations should use those based on efficiency, i.e. an input/output function. Two problems here: the lag in output is considerable, and the best metric for measuring output is unclear. The lag means it would be necessary to rely on track record, which can be problematic for those starting out in the field. Nevertheless, a move in this direction would at least encourage applicants and funders to think more about value for money, rather than maximising the size of a grant – a trend that has been exacerbated by Full Economic Costing (don't get me started on that). And it might make grant-holders and their bosses see the value of putting less time and energy into writing multiple proposals and more into getting a project done well, so that it will generate good outputs on a reasonable time scale.

4. The most radical suggestion is that we abandon formal institutional rankings (i.e. the successor to the RAE, the REF). I've been asking colleagues who were around before the RAE, what they think it achieved. The general view was that the first ever RAE was a useful exercise that exposed weaknesses in insitutions and individuals and got everyone to sharpen up their act. But the costs of subsequent RAEs (especially in terms of time) have not been justified by any benefit. I remember a speech given by Prof Colin Blakemore at the British Association for the Advancement of Science some years ago where he made this point, arguing that rankings changed rather little after the first exercise, and certainly not enough to justify the mammoth bureaucratic task involved. When I talk to people who have not known life without an RAE, they find it hard to imagine such a thing, but nobody has put forward a good argument that has convinced me it should be retained. I'd be interested to see what others think.

The difference between p < .05 and a screening test

Here's a fairly standard science story. A researcher compares a group of 30 children with autism/dyslexia/depression/chilblains and a matched control group on something: genotype/reactions to pharamaceuticals/behaviour profile/brain activation. A statistically significant difference is found between the groups. The result is published in a high profile journal.
The media want to know the relevance of this finding for their readers. It is not clear that this will lead to immediate treatment for autism/dyslexia/depression/chilblains, but they do feel justified in saying that the research will be useful in improving screening. After all, we all know that the earlier we can identify something, the easier it is to treat.

What's wrong with this? Well, alas, everything. And before going further, it is important to note that it is not just the media that get it wrong; many scientists blithely assert that their findings will have application in screening with no understanding of what is required to make a good screening test.

The four key considerations for a screening programme are take-up, costs, accuracy and intervention. Take-up can be surprisingly high, even for embarrassing and uncomfortable procedures. In the UK, most women go along with screening for cervical and breast cancer because, hey, having a total stranger introduce your private parts to cold metallic objects is, on balance, less bad than finding too late that you have a terminal disease.

But there are costs, of course. First, to the individual being screened, who has to take time out of a busy life to go along for screening, and second to the NHS, which must pay for the staff, equipment, facilities and infrastructure to do the screening. None of this is going to be worthwhile unless the screening is very accurate. If you are screening for cancer, then you cannot afford to make errors of omission: giving the all-clear when in fact the person has disease. But equally, you will be very unpopular if you make errors of commission – if you tell healthy person they have failed the screen, causing anxiety and leading them to have further tests that then fail to confirm the diagnosis. This has been a problem for the PSA screening test for prostate cancer.

Quite apart from the accuracy of diagnosis, the screening would not be worthwhile if the disorder you are screening for was not readily treatable – or preventable. There would be little point in identifying large numbers of people with disease X, even if you could do accurately, if nothing could be done about X. Indeed, it could be counterproductive.

In this light, let us consider recent stories in the media about using automated analysis of vocalisations to screen for autism. The study, by Oller and colleages, was published in Proceedings of the National Academy of Sciences in July 2010. It is an impressive and innovative study: the researchers had children wearing body-microphones so that all their utterances could be recorded, and they developed software that was able, with a high degree of accuracy, to distinguish the child's own utterances from those of other people and background noise. Using their extensive knowledge of the acoustic characteristics of babble and early language, they derived algorithms to characterise aspects such as pitch and rhythmic features in the vocalisations. They compared results for 106 typically-developing children, 77 children with autism and 49 children with language delay, and demonstrated that on a combination of these indices the groups were well discriminated. They concluded "We interpret this as a proof of concept that automated analysis can now be included in the tool repertoire for research on vocal development and should be expected soon to yield further scientific benefits with clinical implications." Screening is mentioned twice in the paper; in the abstract it states: "automated analysis should soon be able to contribute to screening and diagnosis procedures for early disorders", and in the introduction it is noted that autism is currently diagnosed mainly in terms of negative features, such as joint attention deficits and communication deficits and "abnormal vocal characteristics would constitute a positive marker that might enhance precision of screening or diagnosis." Note that these claims are pretty cautious: the vocal analysis is seen as 'contributing to' and 'enhancing precision' of screening. Nevertheless, the potential of the method as a screening tool has been emphasised in some media reports.

So how accurate is it? The key statistics for any screening instrument are sensitivity and specificity. Sensitivity is the proportion of 'hits', i.e. cases with autism who are correctly identified as having autism. Specificity is the proportion of 'true negatives', i.e, non-autistic children who are correctly identified as non-autistic. These two indices can only be defined in relation to a specific population, and their interpretation will depend crucially on the base rate of disorder in that population.

Oller et al reported identification of autism versus typical development showed sensitivity of 0.75 and specificity of 0.98. This corresponds to the situation shown below: 104/106 (98%) of typical children are accurately diagnosed, and 58/77 (75%) of autistic children are accurately diagnosed.

		Actual diagnosis
		Typical	Autistic	% autistic
Diagnosis	Typical	104	19
from vocal	Autistic	2	58
analysis	Total	106	77	42%

But their study sample was selected to contain a high proportion of children with autism, 42%. In the general population, rates of autism are much lower than this. The most liberal estimate, which includes children with far less severe problems than those included in this study, is around 1% (Baird et al, 2006). If we apply the sensitivity and specificity figures from Oller et al to a population with a base rate of autism of 1% - just by multiplying the number of typical children while keeping specificity constant – then the numbers look like this:

		Actual diagnosis
		Typical	Autistic	% autistic
Diagnosis	Typical	7471	19
from vocal	Autistic	152	58
analysis	Total	7623	77	1%

We see that in this simulated general population sample around three quarters of those who'd be diagnosed as autistic on the basis of vocal analysis are non-autistic. In reality, accuracy is likely to be lower than this in such a sample, because it would include children with milder cases of autism, including those with Asperger syndrome, who were excluded from the study by Oller et al.

It gets worse. A population sample would also include other non-autistic children with poor communication skills, including those with specific language difficulites, or conditions associated with more general developmental delay. Oller et al did include a non-autistic sample with language impairments, and it is clear from the presented data that there was considerable overlap between the autism and language impaired groups; my reading of the plotted data suggests around 43% of children with language impairment fell below the cutoff for abnormality on the vocal measures. The notion that vocal abnormality is a 'unique signature' for autism, as reported by some of the media, is clearly false.

Might there be situations where the vocal analysis could be clinically useful? Potentially, yes. Autism diagnosis is particularly difficult in young, low-functioning children, and even our gold-standard diagnostic tools can have difficulty distinguishing autism from non-autistic mental handicap in children under 18 months of age (Lord et al, 1993). The vocal measures of Oller et al might provide useful supplementary information in such cases, but bear in mind that the data presented so far suggest the vocal abnormalities can occur in conditions other than autism. Another context where the vocal analysis could be of value is in evaluation of very young children at high risk of autism, in particular young siblings of children with autism, around 10-20% of whom will develop autism or a related neurodevelopmental condition. It is possible that vocal analysis will be useful in identifying those likely to have problems. However, as noted by Zwaigenbaum et al (2006), even if we are able to identify such children early on, we then have to confront the fact that little is known about effective early intervention for autism. A further issue in the study of infant sibs is that vocal behaviours could be abnormal in such a sample if children imitate their older autistic sibling. Overall, automated vocal analysis could prove useful in early diagnosis of high risk groups, but we cannot be certain without further research. And even if we did find it could be used in this role, we would want to demonstrate that it provided information that could not be obtained by other simpler means. For instance, a brief parental questionnaire can be pretty effective in identifying autism (Berument et al., 1999).

Despite these caveats, I want to stress that I think the Oller et al study is an excellent piece of science. The method they have developed could provide important insights into the nature of communication problems in young children, and also be used to study how typical children learn to speak. It is a scholarly piece of work built on many years of painstaking research.

I've used the example of the Oller et al study because it provides a recent example of a good study whose clinical implications have been overstated, but there are many, many other instances. Take for instance, media coverage of findings of genes associated with risk for dyslexia. I just did so by Googling 'genes dyslexia screening', and found this article in the New York Times, which is typical of how genetic associations are reported in the media. The newspaper article states " Researchers said a genetic test for dyslexia should be available within a year or less. Children in families that have a history of the disorder could then be tested, with a cheek swab, before they are exposed to reading instruction. If children carry a genetic risk, they could be placed in early intervention programs". Now, I recently reviewed the research on genetic studies of dyslexia (Bishop, 2009). Even though I thought I knew the literature, I was pretty surprised by what I found. I looked at one of the more robust results in the field, on a gene called KIAA0319, described in a paper with the title " Strong evidence that KIAA0319 is a susceptibility gene for developmental dyslexia". The authors compared children with different haplotypes (allele combinations) in relation to dyslexia status. Here's my summary:

"The most common form with alleles 1–1 was equally frequent for affected versus unaffected cases, but two other common forms, 1–2 and 2–1, showed contrasting effects. (Conventionally, 1 is the more frequent allele, and 2 is the less frequent). The 1–2 haplotype was found in 35% of those with dyslexia and 27% of those without, whereas for 2–1, the figures were 24% in those with dyslexia and 36% in unaffected controls.... Extrapolating to the general population (i.e. taking base rates into account), one would expect that most individuals with the 1–2 risk haplotype would not be dyslexic, and most dyslexic individuals would not have the 1– 2 haplotype."

Note that, even without taking base rates into account, the strength of the association with genotypes is considerably lower than the Oller et al association between vocal behaviour and autism. This result causes excitement among geneticists because it has been replicated, and the evidence for association is strong in terms of p-values in a very large sample size. This finding potentially can tell us something about the development of brain regions that lead to risk for dyslexia, and it indicates that biology plays a role in dyslexia, and it is not just the result of poor parenting or poor teaching. But there is a difference between "strong evidence" and "strong effect": in fact, the effect size for individual genotypes is very small – far too small to play a useful role in screening. Since the 1970s we have had reasonably accurate ways of predicting which preschoolers are at risk for reading difficulties, based on simple behavioural tests and family background (see e.g. Satz & Friel, 1978). They are unlikely to be surpassed by any genetic tests in the foreseeable future.

Nevertheless, the belief that genes are destiny and are somehow more accurate than simple behavioural or observational measures in predicting problems is persistent. This encourages disturbing developments, such as firms offering genetic screening to identify your child's talents. Apparently you can pay to have your child tested for genes related to a host of personal characteristics, ranging from intelligence to optimism to dancing. Just one small problem: for many of these characteristics there is no published data linking gene variants to the trait. And for those that exist, the effects are tiny.

The arguments I present here are very old; I was taught about the importance of base rates by the late A. E. Maxwell when training as a clinical psychologist in 1975. However, knowledge of this aspect of epidemiology is not widespread, even among academic psychologists who are otherwise sophisticated users of statistics. I recently asked a group of MSc students if any of them knew about base rates, and they thought I was talking about the Bank of England.

So, the bottom line: just because someone finds a statistically significant difference between two groups, don't assume that will translate into a useful screening test.

(Red underline denotes corrections made 19th August 2010).

Cited references

Baird, G., Simonoff, E., Pickles, A., Chandler, S., Loucas, T., Meldrum, D., et al. (2006). Prevalence of disorders of the autism spectrum in a population cohort of children in South Thames: the Special Needs and Autism Project (SNAP). Lancet, 368 (9531), 210-215.

Berument, S. K., Rutter, M., Lord, C., Pickles, A., & Bailey, A. (1999). Autism screening questionnaire: Diagnostic validity. British Journal of Psychiatry, 175, 444-451.

Bishop, D. V. M. (2009). Genes, cognition and communication: insights from neurodevelopmental disorders. The Year in Cognitive Neuroscience: Annals of the New York Academy of Sciences, 1156, 1-18.

Lord, C., Storoschuk, S., Rutter, M., & Pickles, A. (1993). Using the ADI-R to diagnose autism in preschool children. Infant Mental Health Journal, 14, 234-252.

Satz, P., & Friel, J. (1978). Predictive validity of an abbreviated screening battery. Journal of Learning Disabilities, 11(6), 347-351.

Zwaigenbaum, L., Thurm, A., Stone, W., Baranek, G., Bryson, S., Iverson, J., et al. (2006). Studying the emergence of autism spectrum disorders in high-risk infants: Methodological and practical Issues. Journal of Autism and Developmental Disorders, 37, 466-480.

Further reading

Loong, T. W. (2003). Understanding sensitivity and specificity with the right side of the brain. (10.1136/bmj.327.7417.716). British Medical Journal, 327(7417), 716-719.

Postscript, 12^th August 2010

An article by Ecker et al that was published yesterday in Journal of Neuroscience comprehensively illustrates the points above.  The researchers gathered MRI brain scans on 20 adults with autism, 19 adults with ADHD, and 20 control adults with no neuropsychiatric diagnosis. Previous research has reported various differences between brains of people with autism and controls, but there have been few findings that are sufficiently strong and convincing to discriminate the groups. Indeed, it has been surprisingly difficult to find any consistent abnormalities on brain scanning, even in the case of children with severe impairments. Ecker et al adopted a different approach, by arguing that a constellation of features would be more effective than individual indices in distinguishing autism from control brains.

The three groups did not differ in terms of three measures of brain capacity: intracranial volume, total brain volume, and gray matter volume.

They used five measures to derive a 'classifier': an equation that puts the measures together in a way that maximally distinguishes the autism and control groups. These measures were (as far as I understand the highly technical account):

1. The average convexity or concavity of large-scale features of the brain surface. This measures sulcal depth and gyral height
2. Mean (radial) curvature. This assesses number of smaller convolutions on the surface of the brain.
3. A measure of the degree of cortical folding in terms of metric distortion relative to an average template.
4. Cortical thickness
5. Pial area, a measure of the grey matter surface.

To avoid the bias that can arise when the same data is used to derive and test a classifier, they adopted a 'leave one out' method, whereby one person with autism and one control were omitted when deriving the estimation of the equation. The resulting equation was then applied to classify the left-out cases. This was done separately for the left- and right sides of the brain.  Only the left-sided classifier was effective, giving the following results.

		Actual diagnosis
		Autistic	Control
Diagnosis	Autistic	18	4
from MRI	Control	2	16
classifier	Total	20	20

When the same left-hemisphere classifier was appplied to the ADHD group, the picture was similar. 4 of 19 cases were classified as autistic, and the remainder as non-autistic.

In the discussion of the results, the authors note the potential for clinical application as follows:

"... the existence of an ASD biomarker such as brain anatomy might be useful to facilitate and guide the behavioral diagnosis. This would, however, require further extensive exploration in the clinical setting, particularly with regards to classifier specificity to ASD rather than neurodevelopmental conditions in general. "

In other words, the classifier needs to be validated on other samples, and it is as yet unclear whether it would be as accurate if people with autism were contrasted, say, with people with language impairments. The authors also noted that the adult sample used here was largely composed of high-functioning adults with Asperger syndrome, and it was not clear whether results would generalise to other cases of autism.  In the final sentence of the paper it is stated: "classification values and specific patterns we report must be considered as preliminary".

In sum, the paper illustrates a promising approach to understanding more about brain development in autism, but it does not demonstrate the utility of this approach for differential diagnosis of autism from other conditions, nor for assisting diagnosis in marginal and difficult cases.

Encouraged by an enthusiastic press release from the Medical Research Council, who funded this work, the media picked up the story yesterday.  It was featured prominently on BBC television and radio news, made the front page story on the Independent, and was given extensive coverage in other national newspapers.  The account of the work showed little evidence of the caution and qualification in the published account.  In this case, I don't think the media can be blamed for misrepresenting the work, as much of what was said was taken from the MRC press release.  Here are some statements from the Independent, under the headline: "Brain scan promises to identify the hidden sufferers of autism":

• Autism could in future be diagnosed in 15 minutes from a brain scan

• Scientists ... have devised a method which can distinguish the autistic brain from the normal brain with 90 per cent accuracy.

• The method could lead to the introduction of screening for the disorder in children.

Dr Ecker was quoted as saying:

"The [computer programmme] says yes this is autism or no it is not, and also gives an indication of the severity."  She also contrasted the 15 minutes it took to do an MRI scan with conventional diagnostic procedures which "takes the whole day and involves asking embarrassing questions such as how many friends they have and how they are doing at school"

Dr Ecker seems to be implying that we should stop using diagnostic interviews to diagnose autism, because a brain scan is more accurate.  Since autism is currently defined in terms of a pattern of impairments in different domains of functioning, this is a remarkably bold claim, especially since her method awarded an autism diagnosis to 20% of controls.

Much is also made of the potential of the method for diagnosing autism in children. It's possible this may be effective, but children's brains are very different from adult brains, and one would need a whole new study to evaluate this. Furthermore, although Dr Ecker says of brain scanning "It doesn't hurt and you can even go to sleep", it is not an easy procedure to apply to young children without sedation.  Lying still in a very noisy tube is not something that comes naturally to a child, even if you can explain the procedure to them clearly so they know what to expect. Most children with autism have poor language comprehension and may also become anxious when in an unfamiliar situation. It is not impossible to do brain scanning studies with them, but it certainly is not easy.

A final thought. In recent years, brain research on autism has increased exponentially. There are other MRI datasets for adults and children with autism and other clinical groups as well as controls. It should be possible for Ecker and colleagues to cross-validate their findings fairly rapidly against other datasets. I hope that others will be willing to share their data to make this possible.