Genes for optimism, dyslexia and obesity and other mythical beasts

Copyright: www.CartoonStock.com

I recently received an email from a company called mygeneprofile: “By discovering your child's inborn talents & personality traits, it can surely provide a great head start to groom your child in the right way… our Inborn Talent Genetic Test has 99.8% accuracy.” I’d registered to receive information from the company having heard they were offering a genetic test for such diverse traits as optimism, composure, intelligence, and dancing (link).

Despite all the efforts of the Human Genome Project, I was not aware of any genetic test that could reliably predict a child’s personality or ability. I was not therefore surprised when my emails asking for evidence went unanswered, though I continue to receive messages that oscillate between carrots (free gifts! discounts!!) and sticks (without this test “your child will have MISERABLE life (sic))”.

The test company relies on a widespread assumption that people’s psychological attributes are predictable from their genes. So where does this belief come from, and why is it wrong? 

People’s understanding of genetic effects is heavily influenced by the way genetics is taught in schools. Mendel and his wrinkly and smooth peas make a nice introduction to genetic transmission, but the downside is that we go away with the idea that genes have an all-or-none effect on a binary trait.  Some characteristics are inherited this way (more or less), and they tend to be the ones that textbooks focus on: e.g., eye colour, colour-blindness, Huntington’s disease. But most genetic effects are far more subtle and complex than this. Take height, for instance. Genes are important in determining how tall you are, but this is not down to one gene: instead, there is a whole host of genes, each of which nudges height up or down by a small amount (see link).

The expression of a gene may also depend crucially on the environment; for instance, obesity relates both to calorie intake and genetic predisposition, but the effects are not just additive: some people can eat a great deal without gaining weight, whereas in others, body mass depends substantially on food intake (see link). Furthermore, a genetic predisposition to obesity can be counteracted by exercise (see link).Furthermore, genetic influences may interact in complicated ways.  For instance, coat colour in mice is affected by combinations of genes, so that one cannot predict whether a mouse is black, white or agouti (mouse coloured!) just by knowledge of status of one gene.

This means that we get a very different impression of strength of genetic influences on a trait if we look at the impact of a person’s whole genome, compared to looking at individual genes in isolation. The twin study was the traditional method for estimating genetic influences before we had the technology to study genes directly, and it compares how far people’s similarity on a trait depends on their genetic relationship. Researchers measure a trait, such as sensation-seeking, in identical and fraternal twin pairs growing up in the same environment and consider whether the two twin types are equally similar. If both sets of twins resemble each other equally strongly, this indicates that the environment, rather than genes, is critical. And if twins don’t resemble one another at all, this could mean either that the trait is influenced by child-specific experiences, not shared by the co-twin, or that our measure of sensation-seeking is unreliable.  But if identical twins are more similar than fraternal twins, this means genes affect the trait, i.e. it is heritable. There are several niggly criticisms of the twin method; for instance, it can give misleading estimates if identical twins are treated more similarly than fraternal twins, or if twinning itself influences the trait in question. For most traits, however, these don’t seem sufficient to explain away the substantial heritability estimates that are found for traits such as height, reading ability, and sensation-seeking.  But these estimates don’t tell us about the individual genes that influence a trait – they rather indicate how important genes are relative to non-genetic influences.

Interactive effects, either between multiple genes or between genes and environments, will not be detected in a conventional twin study analysis. If a gene is expressed only in a particular environment, twins who have the same version of the gene will usually also have the same environment, and so the expression of the gene will be the same for both. And for an effect that depends on having a particular combination of genes, identical twins will have the same constellation of genetic variants, whereas the likelihood of fraternal twins having an identical gene profile decreases with the number of genes involved.  Heritability estimates depend on comparing similarity of a trait for identical vs fraternal twins, and will be increased if gene-gene interactions are involved.

In contrast,  genome-wide association studies are designed to find individual genes that influence specific traits. They adopt the strategy of looking for associations between DNA variants (alleles) and the trait, either by categorising people, e.g. as dyslexic or not, and comparing the proportions with different alleles, or by seeing whether people who have zero, one or two copies of an allele differ in their average score on a trait such as reading ability.  When these studies started out, many people assumed we would find gene variants that exerted a big effect, and so might reasonably be termed ‘the gene for” dyslexia, optimism, and so on.  However, this has not been the case.

Take personality, for instance, one of the domains that mygeneprofile claims to test for. A few weeks ago, a major study was reported in which  the genes of over 5000 people were investigated but no significant associations were found. Commentators on the research argued that the measurement of personality – typically on the basis of self-report questionnaires - may be the problem.  But the self-same measures yield high estimates of heritability when used in twin studies.  And a similar pattern has been found for other traits: including height, intelligence and obesity, i.e., a mismatch in evidence of genetic influence from twin studies (typically moderate to strong for these traits) and findings of individual genes associated with the trait (with effects that are very small at best). 

This account may surprise readers who have read of recent discoveries of genes for conditions such as dyslexia, where the impression is sometimes given that there are strong effects.  The reason is that reports of molecular genetic studies usually emphasise the p-value, a measure of how probable it is that a result could have arisen by chance. A low p-value indicates that a result is reliable, but it does not mean the effect is large. These studies typically use very large samples precisely because this allows them detect even small effects.  Consider one of the more reliable associations between genes and behaviour: a gene known as KIAA0319 which has been found to relate to reading ability in several different samples. In one study, an overall association was reported with p = .0001, indicating that the likelihood of the association being a fluke is 1 in 10,000. However, this reflected the fact that one gene variant was found in 39% of normal readers and only 25% of dyslexics, with a different variant being seen in 30% of controls and 35% of dyslexics. Some commentators have argued that such small effects are uninteresting.  I disagree: findings like this can pave the way for studies into the neurobiological effects of the gene on brain development (see link), and for studies of gene-gene and gene-environment interactions.  But it does mean that talk of a ‘gene for dyslexia’, or genetic screening for personality or ability are seriously misguided.

The small effect size of individual genes, and interactions with environment or other genes,  are not the only explanations for “missing heritability”. A trait may be influenced by genetic variants that have a large effect but which are individually very rare in the population. These would be very hard to detect using current methods. The role of so-called copy number variants is also a focus of current interest: these are large chunks of DNA which are replicated or deleted and which are surprisingly common in all of us.  These lead to an increase or decrease in gene product, but won’t be found with standard methods that focus just on identifying the DNA sequence. Both mechanisms are thought to be important in the genetics of autism, which is increasingly looking like a highly heterogeneous condition – i.e. there are multiple genetic risk factors and different ones are important in different people. 

What are the implications of all of this for the stories we hear in the media about new genetic discoveries?  The main message is we need to be aware of the small effect of most individual genes on human traits. The idea that we can test for a single gene that causes musical talent, optimism or intelligence is just plain wrong. Even where reliable associations are found, they don’t correspond to the kind of major influences that we learned about in school biology. And we need to realise that twin studies, which consider the total effect of a person’s genetic makeup on a trait, can give different results from molecular studies of individual genes. What makes us individual can’t be reduced to the net effect of a few individual genes.

Background reading

Bishop, D. V. M. (2009). Genes, cognition and communication: insights from neurodevelopmental disorders. The Year in Cognitive Neuroscience: Annals of the New York Academy of Sciences, 1156, 1-18.

Maher, B. (2008). Personal genomes: The case of the missing heritability. Nature, 456, 18-21 doi:10.1038/456018a.

Plomin, R., DeFries, J. C., McClearn, G. E.,  McGuffin, P. (2008). Behavioral Genetics. (5th Edition). New York: Worth Publishers.

Rutter, M. (2006). Genes and Behavior: Nature-Nurture Interplay Explained. Oxford: Blackwell.

Note: this is a slightly extended version of a blog on Guardian Science Blog, 9/9/10

Three ways to improve cognitive test scores without intervention

My research focuses on neurodevelopmental disorders such as autism, specific language impairment and dyslexia, where more or less miraculous cures are heralded at regular intervals. These claims are often unfounded, being based on anecdote or uncontrolled studies. I often feel like a kill-joy, picking holes in reports that offer hope to parents and their children. But I carry on because I don't like to see people being misled, often at considerable personal expense, and sometimes even being encouraged to subject their child to practices that are at best boring and at worst painful or dangerous.

Psychological or educational interventions have much in common with medical treatments, and the same randomised-controlled trial (RCT) methodology is needed to obtain convincing evidence of efficacy. But what is a RCT controlling for?  In drug treatments, the emphasis is on the need to control for (a) spontaneous improvement or (b) placebo effects, i.e. the improvement that you get just from being the focus of medical attention.  In psychological and educational interventions there are three other important sources of potential bias that can give a misleading impression of treatment efficacy unless they are controlled for: developmental change, practice effects and regression to the mean.

Developmental change
 

This is the simplest to understand. Basically, children get better at most things as they get older. If someone gives me a brain training game that is supposed to cure dyslexia, and my child does the exercises every day for a year, he may well be able to read more words at the end of the intervention than he could at the start. However, this would hardly be good evidence for effectiveness of the games, since most children learn more words as they grow older. Indeed, his feet will also be bigger at the end of the intervention, but I would not attribute that to the brain training. This is so obvious that it may seem unnecessary to state it, except for the fact that with some conditions, people assume that development is static.  For instance, the National Autistic Society website states "Autism is a lifelong developmental disability".  Now, this does not mean that children with autism will not change with age, yet it is easy to see how someone might misinterpret it. In fact, diagnostic instruments for autism focus on the child's behaviour at 4 to 5 years of age, precisely because that is when autistic features are most evident.  As the child grows older, these features may become milder or take a different form. See, for instance, the detailed account by Clara Claiborne Park of her autistic daughter, who barely used speech at 4 years, but could produce simple sentences by the age of 8. If your child has some intervention around the age of 4 years and then shows improvements, it is easy to assume that the intervention was responsible.  However, the improvement may well have occurred anyway.  The same is true for children who are late to start talking. This was brought home to me by a study conducted in the 1980s evaluating home-based  therapy with late talkers in an inner city area. The treated children showed a dramatic improvement.  However, so too did an untreated control group.  Without the evidence of the control group it would be easy to be misled into thinking that all late talkers should be given the home-based intervention, but a very different recommendation would follow when the controls are taken into account.  The data suggested that it would be better to adopt a 'watchful waiting' policy, because most children improved spontaneously. I would not want to imply we should not treat children with language difficulties: my own research has been concerned with identifying which children won't catch up by themselves, so we can target intervention appropriately.  But it is easy to underestimate the amount of change that can occur without intervention, especially in very young children.

Practice

On many tests, people get better with practice. Many people will have experienced the WiiFit machine, which gives you a series of tests to estimate your 'fitness age'. These involve activities such as standing on a balance board and trying to keep a dot representing your centre of gravity in a central position, or doing complex memory or calculation tests under time pressure. The first time I did this, my fitness age was estimated at 76. However, just one day later, it had come down to 60 years, and with further practice there was a steep decline. I don't think my ineffectual and limited attempts at WiiFit exercises had reversed 16 years worth of decrepitude: rather, I got better because I had a clearer idea of what I was supposed to be doing.  The first time I had no strategy for performing the tests, and in some cases completely misunderstood what I was supposed to be doing. So it was hardly surprising that I improved dramatically just by my second attempt.  It is worth noting that balance tests are used by Dore centres to evaluate the success of their training program for neurodevelopmental disorders.  Some of these tests show substantial practice effects (Liao et al, 2001)  Parents can see their child's performance improving, and gain the impression that this indicates that important brain changes are occurring. However, without correction for practice effects this is misleading. (As far as I can establish, Dore centres don't correct for developmental changes in balance either, though these are well documented on such tests (e.g. http://www.ncbi.nlm.nih.gov/pubmed/17105679).

Cognitive tests vary in the extent to which they show practice effects, but most will show some benefits of prior exposure. Take a couple of well-known tests of non-verbal ability, matrices and object assembly.  Matrices involves working out the logic behind visual patterns. Once you have worked out how a specific pattern is constructed, it is much easier to do it on the next occasion, because you don't have to entertain a wide range of hypotheses about the correct answer.  Object assembly involves a jig-saw like task where you assemble pieces to make a meaningful figure, but you don't know what the meaningful figure is – at least not the first time you do it. Once you have done it, you are likely to show gains in performance on a second occasion even after some years.  Over a short interval, Rapport et al (1997) showed gains in IQ of 5-10 points after one previous test.  I studied short-term practice effects on a language comprehension test I have developed: children showed significant gains in scores on the second administration, even though I used different test items (but the same format).  McArthur has specifically looked at the extent to which improvements are seen in untreated groups of children on tests of reading and language, and has documented substantial gains.  

Practice effects mean that gains in cognitive test scores after an intervention are usually uninterpretable unless we have comparison information from individuals who were given the same schedule of testing without the intervening intervention.   Again, it sounds obvious, but it is a point that is often missed by researchers, as well as by non-scientists.  A striking example comes from a study of interventions for children with speech-language problems by Gillam and colleagues. This study was designed as a randomised controlled trial, and had four groups: three of them were given specific treatments designed to improve children's language, and the fourth was a control group who were given a  computerised training of general 'academic skills'. Everyone improved, but the amount of improvement was no greater for the three treatment groups than for the control group.  The conventional interpretation of such a result would be to conclude that the treatments were ineffective, because the changes associated with them were no greater than for the control group.  However, impressed by the increase in scores and apparently unaware of the possibility of practice effects, the authors of the study concluded instead that all treatments were effective, and spent some time discussing why the control treatment might be good at boosting language.

One further point: it is sometimes assumed that we don't need to worry about practice effects if a test has good test-retest reliability. But that is not the case. High reliability just means that the rank ordering of a group of children will be stable across testing occasions.  But reliablity indices are insensitive to absolute level of performance. So if we took a group of children's scores at time 1 and gave each of them a score 10 points higher at time 2, then we would have perfect reliability because their rank ordering would be the same. Reliability is, however, an important factor when considering the last topic, regression toward the mean.

Regression toward the mean
 
Regression toward the mean is a horribly difficult concept to understand, even for those with statistical training.  In their masterly introduction to the subject, Campbell and Kenny noted that regression to the mean has confused many people, including Nobel prize winners.   More specifically, they stated that: “Social scientists incorrectly estimate the effects of ameliorative interventions.....and snake-oil peddlers earn a healthy living all because our intuition fails when trying to comprehend regression toward the mean". 

Perhaps the easiest way to get a grasp of what it entails is to suppose we had a group of 10 people and asked them each to throw a dice 10 times and total the score.   Let's then divide them into the 5 people who got the lowest scores and the 5 who got the highest scores and repeat the experiment. What do we expect? Well, assuming we don't believe that anything other than chance affects scores (no supernatural forces or 'winning streaks'), we'd expect the average score of the low-scorers to improve, and the average score of the high scorers to decline.  This is because the probability for any one person is that they will get an average score on any one set of dice throws.  So that's the simplest case, when we have a score that is determined only by chance.

Cognitive test scores are interesting here because they are typically thought of comprising two parts: a 'true' score, which reflects how good you really are at whatever is being measured, and an 'error' score, which reflects random influences. Suppose, for instance, that you test a child's reading ability. In practice, the child is a very good reader, in the top 10% for her age, but the exact number of words she gets right will depend on all sorts of things: the particular words selected for the test (she may know 'catacomb' but not 'pharynx'), the mood she is in on the day, whether she is lucky or unlucky at guessing at words she is uncertain about. All these factors would be implicated in the 'error' score, which is treated just like an additional chance factor or throw of the dice affecting a test score.  A good test is mostly determined by the 'true' score, with only a very small contribution of an 'error' score, and we can identify it by the fact that children will be ranked by the test in a very similar way from one test occasion to the next, i.e. there will be good test-retest reliablity.  In other words, the correlation from time 1 to time 2 will be high.

Simulated test scores for 21 children on tests varying in reliability

These figures show simulated scores for a group of 21 children on tests that vary in test-retest reliability.  In each case individuals are colour-coded depending on whether they fall in the bottom (blue), middle (purple) or top (red) third of scores at time 1. The simulations assume no systematic differences between time 1 and 2 - i.e. no intervention effect, maturation or practice.  Scores are simulated as random numbers at time 1, with time 2 scores then set to give a specific correlation between time 1 and 2, with no change in average score for the group as a whole.

Now suppose we select children because they have a particularly low score at time 1. Insofar as chance contributes to their scores, then at time 2, we would expect the average score of such a group to improve, because chance pushes the group average towards the overall mean score.  The left-hand panel shows the situation when reliability (i.e., correlation between time 1 and time 2 scores) is zero, so scores are determined just by chance, like throwing a dice. The mean scores for the blue, purple and red cases at time 1 are, by definition different (they are selected to be low, medium and high scorers). But at time 2 they are all equivalent.  The upshot is that the mean at time 2 for those with initial low scores (blue) goes up, whereas for those that start out with high scores, the time 2 mean comes down.
 
The middle and right-hand panels show a more realistic situation, where the test score is mixture of true score and error. With very high reliability (right-hand panel) the effect of regression to the mean is small, but with medium reliability (middle panel) it is detectable by eye even for this very small sample.

The key point here is that if we select individuals on the basis of low scores on a test (as is often done, e.g. when identifying children with poor reading scores for a study of a dyslexia treatment), then, unless we have a highly reliable test with a negligible error term, the expectation is that the group's average score will improve on a second test session, for purely statistical reasons.  In general, psychometric tests are designed to have reasonable reliability, but this varies from test to test and is seldom higher than .75-.8.

So regression to the mean is a real issue in longitudinal studies.  It is yet another reason why scores will change over time. Zhang and Tomblin (2003) noted that we can overcome this problem by using different tests to select children for a study and to measure their improvement.  Or we can allow for regression to the mean if our study includes a control group, who will be subject to the same regression to the mean as the intervention group.

So, if you started reading this blog thinking that people who demand control groups for intervention studies were just a kind of methodological Stasi whose obsession with RCTs was misplaced, I hope I have convinced you that the concerns are real. There are just so many ways that we can mislead ourselves into thinking that an intervention has led to improvement, when in fact it is making no difference. It is depressingly hard to demonstrate positive effects of treatment, but the only way to do it is by using properly designed studies that can overcome the biases I have noted here by use of appropriate controls. 



Addition resources on this topic can be found on this section of my website, see esp. 
  7th BDA International Conference – 2008: Dyslexia: Making Links
  2nd UK Paediatric Neuropsychology Symposium
  DysTalk online (video)

And a couple of short articles on how not to do intervention studies:

BISHOP, D. V. M.  2007. Curing dyslexia and ADHD by training motor co-ordination: Miracle or myth? Journal of Paediatrics and Child Health, 43, 653-655.
BISHOP, D. V. M. 2008. Criteria for evaluating behavioural interventions for neurodevelopmental disorders (Letter). Journal of Paediatrics and Child Health, 44, 520-521.

How our current reward structures have distorted and damaged science

copyright www.CartoonStock.com

Two things almost everyone would agree with:

1. Good scientists do research and publish their results, which then have impact on other scientists and the broader community.

2. Science is a competitive business: there is not enough research funding for everyone, not enough academic jobs in science, and not enough space in scientific journals. We therefore need ways of ensuring that the limited resources go to the best people.

When I started in research in the 1970s, research evaluation focused on individuals. If you wrote a grant proposal, applied for a job, or submitted a paper to a journal, evaluation depended on peer review, a process that is known to be flawed and subject to whim and bias, but is nevertheless regarded by many as the best option we have.

What has changed in my lifetime is the increasing emphasis on evaluating institutions rather than individuals. The 1980s saw the introduction of the Research Assessment Exercise, used to evaluate Universities in terms of their research excellence in order to have a more objective and rational basis for allocating central funds (quality weighted research funding or QR) by the national funding council (HEFCE in England). The methods for evaluating institutions evolved over the next 20 years, and are still a matter of debate, but they have subtly influenced the whole process of evaluation of individual academics, because of the need to use standard metrics.

This is inevitable, because the panel evaluating a subject area can't be expected to read all the research produced by staff at an institution, but they would be criticised for operating an 'old boy network', or favouring their own speciality, if they relied just on personal knowledge of who is doing good work – which was what tended to happen before the RAE. Therefore they are forced into using metrics. The two obvious things that can be counted are research income and number of publications. But number of publications was early on recognised as problematic, as it would mean that someone with three parochial reports in a journal of national society would look better than someone with a major breakthrough published in a top journal. There has therefore been an attempt to move from quantity to quality, by taking into account the impact factor of the journals that papers are published in.

Evaluation systems always change the behaviour of those being evaluated, as people attempt to maximise rewards. Recognising that institutional income depends on getting a good RAE score, vice-chancellors and department heads in many institutions now set overt quotas for their staff in terms of expected grant income and number of publications in high impact journals. The jobs market is also affected, as it becomes clear that employability depends on how good one looks on the RAE metrics.

The problem with all of this is that it means that the tail starts to wag the dog. Consider first how the process of grant funding has changed. The motivation to get a grant ought to be that one has an interesting idea and needs money to investigate it. Instead, it has turned into a way of funding the home institution and enhancing employability. Furthermore, the bigger the grant, the more the kudos, and so the pressure is on to do large-scale expensive studies. If individuals were assessed, not in terms of grant income, but in terms of research output relative to grant income, many would change status radically, as cheap, efficient research projects would rise up the scale. In psychology, there has been a trend to bolt on expensive but often unmotivated brain imaging to psychological studies, ensuring that the cost of each experiment is multiplied at least 10-fold. Junior staff are under pressure to obtain a minimum level of research funding, and consequently spend a great deal of time writing grant proposals, and the funding agencies are overwhelmed with applications. In my experience, applications that are written because someone tells you to write one are typically of poor quality, and just waste the time of both applicants and reviewers. The scientist who is successful in meeting their quota is likely to be managing several grants. This may be a good thing if they are really talented, or have superb staff, but in my experience research is done best if the principal investigator puts serious thought and time into the day-to-day running of the project, and that becomes impossible with multiple grants.

Regarding publications, I am old enough to have been publishing before the RAE, and I'm in the fortunate but unusual position of having had full-time research funding for my whole career. In the current system I am relatively safe, and I look good on an RAE return. But most people aren't so fortunate: they are trying to juggle doing research with teaching and administration, raising children and other distractions, yet feel under intense pressure to publish. The worry about the current system is that it will encourage people to cut corners, to favour research that is quick and easy. Sometimes, one is lucky, and a simple study leads to an interesting result that can be published quickly. But the best work typically requires a large investment of time and thought. The studies I am proudest of are ones which have taken years rather than months to complete: in some cases, the time is just on data collection, but in others, the time has involved reading, thinking, and working out ways of analysing and interpreting data. But this kind of paper is getting increasingly rare. As a reviewer, I frequently see piecemeal publication, so if you suggest that a further analysis would strengthen the paper, you are told that it has been done, but is the subject of another paper. Scholarship and contact with prior literature has become extremely rare: prior research is cited without reading it – or not cited at all – and the notion of research building on prior work has been eroded to the point that I sometimes think we are all so busy writing papers that we have no time to read them. There are growing complaints about an 'avalanche' of low-quality publications.

As noted above, in response to this, there has been a move to focus on quality rather than quantity of publications, with researchers being told that their work will only count if it is published in a high-impact journal. Some departments will produce lists of acceptable journals and will discourage staff from publishing elsewhere. In effect, impact factor is being used as a proxy for likelihood that a paper will be cited in future, and I'm sure that is generally true. But just because a paper in a high impact journal is likely to be highly cited, it does not mean that all highly-cited papers appear in high impact journals. In general, my own most highly-cited papers appeared in middle-ranking journals in my field. Moreover, the highest impact journals have several limitations:

1. They only take very short papers. Yes, it is usually possible to put extra information in 'supplementary material', but what you can't do is to take up space putting the work in context or discussing alternative interpretations. When I started in the field, it was not uncommon to publish a short paper in Nature, followed up with a more detailed account in another, lowlier, journal. But that no longer happens .We only get the brief account.

2. Demand for page space outstrips supply. To handle a flood of submissions, these journals operate a triage system, where the editor determines whether the paper should go out for review. This can have the benefit that rejection is rapid, but it puts a lot of power in the hands of editors, who are unlikely to be specialists in the subject area of the paper, and in some cases will explicit in their preference for papers with a 'wow' factor. It also means that one gets no useful feedback from reviewers: viz my recent experience with the New England Journal of Medicine, where I submitted a paper that I thought had all the features they'd find attractive – originality, clinical relevance and a link between genes and behaviour. It was bounced without review, and I emailed, not to appeal, but just to ask if I could have a bit more information about the criteria on which they based their rejection. I was told that they could not give me any feedback as they had not sent it out for review.

3. If the paper does go out for review, the subsequent review process can be very slow. There's an account of the trials and tribulations of dealing with Nature and Science which makes for depressing reading. Slow reviewing is clearly not a problem restricted to high impact journals. My experience is that lower-impact journals can be even worse. But the impression from the comments on FemaleScienceProfessor's blog, is that reviewers can be unduly picky when the stakes are high.

So what can be done? I'd like to see us return to a time when the purpose of publishing was to communicate, and the purpose of research funding was to enable a scientist to pursue interesting ideas. The current methods of evaluation have encouraged an unstoppable tide of publications and grant proposals, many of which are poor quality. Many scientists are spending time on writing doomed proposals and papers when they would be better off engaging in research and scholarship in a less frenetic and more considered manner. But they won't do that so long as the pressures are on them to bring in grants and generate publications. I'll conclude with a few thoughts on how the system might be improved.

1. My first suggestions, regarding publications, are already adopted widely in the UK, but my impression is they may be less common elsewhere. Requiring applicants for jobs or fellowships to specify their five best publications rather than providing a full list rewards those who publish significant pieces of work, and punishes piecemeal publication. Use of the H-index as an evaluation metric rather than either number of publications or journal impact factor is another way to encourage people to focus on producing substantial papers rather than a flood of trivial pieces, as papers with low citations have no impact whatever on the H-index.There are downsides: we have the lag problem, which makes the H-index pretty useless for evaluating junior people, and in its current form the index does not take into account the contribution of authors, thereby encouraging multiple authorship, since anyone who can get their name on a highly-cited paper will boost their H-index, regardless of whether they are a main investigator or freeloader.

2. Younger researchers should be made aware that a sole focus on publishing in very high impact journals may be counter-productive. Rapid publication in an Open Access journals (many of which have perfectly respectable impact factors) may be more beneficial to ones career (http://openaccess.eprints.org/) because the work is widely accessible and so more likely to be cited. A further benefit of the PLOS journals, for instance, is that they don't impose strict length limits, so research can be properly described and put in context, rather than being restricted to the soundbite format required by very high impact journals.

3. Instead of using metrics based on grant income, those doing evaluations should use those based on efficiency, i.e. an input/output function. Two problems here: the lag in output is considerable, and the best metric for measuring output is unclear. The lag means it would be necessary to rely on track record, which can be problematic for those starting out in the field. Nevertheless, a move in this direction would at least encourage applicants and funders to think more about value for money, rather than maximising the size of a grant – a trend that has been exacerbated by Full Economic Costing (don't get me started on that). And it might make grant-holders and their bosses see the value of putting less time and energy into writing multiple proposals and more into getting a project done well, so that it will generate good outputs on a reasonable time scale.

4. The most radical suggestion is that we abandon formal institutional rankings (i.e. the successor to the RAE, the REF). I've been asking colleagues who were around before the RAE, what they think it achieved. The general view was that the first ever RAE was a useful exercise that exposed weaknesses in insitutions and individuals and got everyone to sharpen up their act. But the costs of subsequent RAEs (especially in terms of time) have not been justified by any benefit. I remember a speech given by Prof Colin Blakemore at the British Association for the Advancement of Science some years ago where he made this point, arguing that rankings changed rather little after the first exercise, and certainly not enough to justify the mammoth bureaucratic task involved. When I talk to people who have not known life without an RAE, they find it hard to imagine such a thing, but nobody has put forward a good argument that has convinced me it should be retained. I'd be interested to see what others think.

The difference between p < .05 and a screening test

Here's a fairly standard science story. A researcher compares a group of 30 children with autism/dyslexia/depression/chilblains and a matched control group on something: genotype/reactions to pharamaceuticals/behaviour profile/brain activation. A statistically significant difference is found between the groups. The result is published in a high profile journal.
The media want to know the relevance of this finding for their readers. It is not clear that this will lead to immediate treatment for autism/dyslexia/depression/chilblains, but they do feel justified in saying that the research will be useful in improving screening. After all, we all know that the earlier we can identify something, the easier it is to treat.

What's wrong with this? Well, alas, everything. And before going further, it is important to note that it is not just the media that get it wrong; many scientists blithely assert that their findings will have application in screening with no understanding of what is required to make a good screening test.

The four key considerations for a screening programme are take-up, costs, accuracy and intervention. Take-up can be surprisingly high, even for embarrassing and uncomfortable procedures. In the UK, most women go along with screening for cervical and breast cancer because, hey, having a total stranger introduce your private parts to cold metallic objects is, on balance, less bad than finding too late that you have a terminal disease.

But there are costs, of course. First, to the individual being screened, who has to take time out of a busy life to go along for screening, and second to the NHS, which must pay for the staff, equipment, facilities and infrastructure to do the screening. None of this is going to be worthwhile unless the screening is very accurate. If you are screening for cancer, then you cannot afford to make errors of omission: giving the all-clear when in fact the person has disease. But equally, you will be very unpopular if you make errors of commission – if you tell healthy person they have failed the screen, causing anxiety and leading them to have further tests that then fail to confirm the diagnosis. This has been a problem for the PSA screening test for prostate cancer.

Quite apart from the accuracy of diagnosis, the screening would not be worthwhile if the disorder you are screening for was not readily treatable – or preventable. There would be little point in identifying large numbers of people with disease X, even if you could do accurately, if nothing could be done about X. Indeed, it could be counterproductive.

In this light, let us consider recent stories in the media about using automated analysis of vocalisations to screen for autism. The study, by Oller and colleages, was published in Proceedings of the National Academy of Sciences in July 2010. It is an impressive and innovative study: the researchers had children wearing body-microphones so that all their utterances could be recorded, and they developed software that was able, with a high degree of accuracy, to distinguish the child's own utterances from those of other people and background noise. Using their extensive knowledge of the acoustic characteristics of babble and early language, they derived algorithms to characterise aspects such as pitch and rhythmic features in the vocalisations. They compared results for 106 typically-developing children, 77 children with autism and 49 children with language delay, and demonstrated that on a combination of these indices the groups were well discriminated. They concluded "We interpret this as a proof of concept that automated analysis can now be included in the tool repertoire for research on vocal development and should be expected soon to yield further scientific benefits with clinical implications." Screening is mentioned twice in the paper; in the abstract it states: "automated analysis should soon be able to contribute to screening and diagnosis procedures for early disorders", and in the introduction it is noted that autism is currently diagnosed mainly in terms of negative features, such as joint attention deficits and communication deficits and "abnormal vocal characteristics would constitute a positive marker that might enhance precision of screening or diagnosis." Note that these claims are pretty cautious: the vocal analysis is seen as 'contributing to' and 'enhancing precision' of screening. Nevertheless, the potential of the method as a screening tool has been emphasised in some media reports.

So how accurate is it? The key statistics for any screening instrument are sensitivity and specificity. Sensitivity is the proportion of 'hits', i.e. cases with autism who are correctly identified as having autism. Specificity is the proportion of 'true negatives', i.e, non-autistic children who are correctly identified as non-autistic. These two indices can only be defined in relation to a specific population, and their interpretation will depend crucially on the base rate of disorder in that population.

Oller et al reported identification of autism versus typical development showed sensitivity of 0.75 and specificity of 0.98. This corresponds to the situation shown below: 104/106 (98%) of typical children are accurately diagnosed, and 58/77 (75%) of autistic children are accurately diagnosed.

		Actual diagnosis
		Typical	Autistic	% autistic
Diagnosis	Typical	104	19
from vocal	Autistic	2	58
analysis	Total	106	77	42%

But their study sample was selected to contain a high proportion of children with autism, 42%. In the general population, rates of autism are much lower than this. The most liberal estimate, which includes children with far less severe problems than those included in this study, is around 1% (Baird et al, 2006). If we apply the sensitivity and specificity figures from Oller et al to a population with a base rate of autism of 1% - just by multiplying the number of typical children while keeping specificity constant – then the numbers look like this:

		Actual diagnosis
		Typical	Autistic	% autistic
Diagnosis	Typical	7471	19
from vocal	Autistic	152	58
analysis	Total	7623	77	1%

We see that in this simulated general population sample around three quarters of those who'd be diagnosed as autistic on the basis of vocal analysis are non-autistic. In reality, accuracy is likely to be lower than this in such a sample, because it would include children with milder cases of autism, including those with Asperger syndrome, who were excluded from the study by Oller et al.

It gets worse. A population sample would also include other non-autistic children with poor communication skills, including those with specific language difficulites, or conditions associated with more general developmental delay. Oller et al did include a non-autistic sample with language impairments, and it is clear from the presented data that there was considerable overlap between the autism and language impaired groups; my reading of the plotted data suggests around 43% of children with language impairment fell below the cutoff for abnormality on the vocal measures. The notion that vocal abnormality is a 'unique signature' for autism, as reported by some of the media, is clearly false.

Might there be situations where the vocal analysis could be clinically useful? Potentially, yes. Autism diagnosis is particularly difficult in young, low-functioning children, and even our gold-standard diagnostic tools can have difficulty distinguishing autism from non-autistic mental handicap in children under 18 months of age (Lord et al, 1993). The vocal measures of Oller et al might provide useful supplementary information in such cases, but bear in mind that the data presented so far suggest the vocal abnormalities can occur in conditions other than autism. Another context where the vocal analysis could be of value is in evaluation of very young children at high risk of autism, in particular young siblings of children with autism, around 10-20% of whom will develop autism or a related neurodevelopmental condition. It is possible that vocal analysis will be useful in identifying those likely to have problems. However, as noted by Zwaigenbaum et al (2006), even if we are able to identify such children early on, we then have to confront the fact that little is known about effective early intervention for autism. A further issue in the study of infant sibs is that vocal behaviours could be abnormal in such a sample if children imitate their older autistic sibling. Overall, automated vocal analysis could prove useful in early diagnosis of high risk groups, but we cannot be certain without further research. And even if we did find it could be used in this role, we would want to demonstrate that it provided information that could not be obtained by other simpler means. For instance, a brief parental questionnaire can be pretty effective in identifying autism (Berument et al., 1999).

Despite these caveats, I want to stress that I think the Oller et al study is an excellent piece of science. The method they have developed could provide important insights into the nature of communication problems in young children, and also be used to study how typical children learn to speak. It is a scholarly piece of work built on many years of painstaking research.

I've used the example of the Oller et al study because it provides a recent example of a good study whose clinical implications have been overstated, but there are many, many other instances. Take for instance, media coverage of findings of genes associated with risk for dyslexia. I just did so by Googling 'genes dyslexia screening', and found this article in the New York Times, which is typical of how genetic associations are reported in the media. The newspaper article states " Researchers said a genetic test for dyslexia should be available within a year or less. Children in families that have a history of the disorder could then be tested, with a cheek swab, before they are exposed to reading instruction. If children carry a genetic risk, they could be placed in early intervention programs". Now, I recently reviewed the research on genetic studies of dyslexia (Bishop, 2009). Even though I thought I knew the literature, I was pretty surprised by what I found. I looked at one of the more robust results in the field, on a gene called KIAA0319, described in a paper with the title " Strong evidence that KIAA0319 is a susceptibility gene for developmental dyslexia". The authors compared children with different haplotypes (allele combinations) in relation to dyslexia status. Here's my summary:

"The most common form with alleles 1–1 was equally frequent for affected versus unaffected cases, but two other common forms, 1–2 and 2–1, showed contrasting effects. (Conventionally, 1 is the more frequent allele, and 2 is the less frequent). The 1–2 haplotype was found in 35% of those with dyslexia and 27% of those without, whereas for 2–1, the figures were 24% in those with dyslexia and 36% in unaffected controls.... Extrapolating to the general population (i.e. taking base rates into account), one would expect that most individuals with the 1–2 risk haplotype would not be dyslexic, and most dyslexic individuals would not have the 1– 2 haplotype."

Note that, even without taking base rates into account, the strength of the association with genotypes is considerably lower than the Oller et al association between vocal behaviour and autism. This result causes excitement among geneticists because it has been replicated, and the evidence for association is strong in terms of p-values in a very large sample size. This finding potentially can tell us something about the development of brain regions that lead to risk for dyslexia, and it indicates that biology plays a role in dyslexia, and it is not just the result of poor parenting or poor teaching. But there is a difference between "strong evidence" and "strong effect": in fact, the effect size for individual genotypes is very small – far too small to play a useful role in screening. Since the 1970s we have had reasonably accurate ways of predicting which preschoolers are at risk for reading difficulties, based on simple behavioural tests and family background (see e.g. Satz & Friel, 1978). They are unlikely to be surpassed by any genetic tests in the foreseeable future.

Nevertheless, the belief that genes are destiny and are somehow more accurate than simple behavioural or observational measures in predicting problems is persistent. This encourages disturbing developments, such as firms offering genetic screening to identify your child's talents. Apparently you can pay to have your child tested for genes related to a host of personal characteristics, ranging from intelligence to optimism to dancing. Just one small problem: for many of these characteristics there is no published data linking gene variants to the trait. And for those that exist, the effects are tiny.

The arguments I present here are very old; I was taught about the importance of base rates by the late A. E. Maxwell when training as a clinical psychologist in 1975. However, knowledge of this aspect of epidemiology is not widespread, even among academic psychologists who are otherwise sophisticated users of statistics. I recently asked a group of MSc students if any of them knew about base rates, and they thought I was talking about the Bank of England.

So, the bottom line: just because someone finds a statistically significant difference between two groups, don't assume that will translate into a useful screening test.

(Red underline denotes corrections made 19th August 2010).

Cited references

Baird, G., Simonoff, E., Pickles, A., Chandler, S., Loucas, T., Meldrum, D., et al. (2006). Prevalence of disorders of the autism spectrum in a population cohort of children in South Thames: the Special Needs and Autism Project (SNAP). Lancet, 368 (9531), 210-215.

Berument, S. K., Rutter, M., Lord, C., Pickles, A., & Bailey, A. (1999). Autism screening questionnaire: Diagnostic validity. British Journal of Psychiatry, 175, 444-451.

Bishop, D. V. M. (2009). Genes, cognition and communication: insights from neurodevelopmental disorders. The Year in Cognitive Neuroscience: Annals of the New York Academy of Sciences, 1156, 1-18.

Lord, C., Storoschuk, S., Rutter, M., & Pickles, A. (1993). Using the ADI-R to diagnose autism in preschool children. Infant Mental Health Journal, 14, 234-252.

Satz, P., & Friel, J. (1978). Predictive validity of an abbreviated screening battery. Journal of Learning Disabilities, 11(6), 347-351.

Zwaigenbaum, L., Thurm, A., Stone, W., Baranek, G., Bryson, S., Iverson, J., et al. (2006). Studying the emergence of autism spectrum disorders in high-risk infants: Methodological and practical Issues. Journal of Autism and Developmental Disorders, 37, 466-480.

Further reading

Loong, T. W. (2003). Understanding sensitivity and specificity with the right side of the brain. (10.1136/bmj.327.7417.716). British Medical Journal, 327(7417), 716-719.

Postscript, 12^th August 2010

An article by Ecker et al that was published yesterday in Journal of Neuroscience comprehensively illustrates the points above.  The researchers gathered MRI brain scans on 20 adults with autism, 19 adults with ADHD, and 20 control adults with no neuropsychiatric diagnosis. Previous research has reported various differences between brains of people with autism and controls, but there have been few findings that are sufficiently strong and convincing to discriminate the groups. Indeed, it has been surprisingly difficult to find any consistent abnormalities on brain scanning, even in the case of children with severe impairments. Ecker et al adopted a different approach, by arguing that a constellation of features would be more effective than individual indices in distinguishing autism from control brains.

The three groups did not differ in terms of three measures of brain capacity: intracranial volume, total brain volume, and gray matter volume.

They used five measures to derive a 'classifier': an equation that puts the measures together in a way that maximally distinguishes the autism and control groups. These measures were (as far as I understand the highly technical account):

1. The average convexity or concavity of large-scale features of the brain surface. This measures sulcal depth and gyral height
2. Mean (radial) curvature. This assesses number of smaller convolutions on the surface of the brain.
3. A measure of the degree of cortical folding in terms of metric distortion relative to an average template.
4. Cortical thickness
5. Pial area, a measure of the grey matter surface.

To avoid the bias that can arise when the same data is used to derive and test a classifier, they adopted a 'leave one out' method, whereby one person with autism and one control were omitted when deriving the estimation of the equation. The resulting equation was then applied to classify the left-out cases. This was done separately for the left- and right sides of the brain.  Only the left-sided classifier was effective, giving the following results.

		Actual diagnosis
		Autistic	Control
Diagnosis	Autistic	18	4
from MRI	Control	2	16
classifier	Total	20	20

When the same left-hemisphere classifier was appplied to the ADHD group, the picture was similar. 4 of 19 cases were classified as autistic, and the remainder as non-autistic.

In the discussion of the results, the authors note the potential for clinical application as follows:

"... the existence of an ASD biomarker such as brain anatomy might be useful to facilitate and guide the behavioral diagnosis. This would, however, require further extensive exploration in the clinical setting, particularly with regards to classifier specificity to ASD rather than neurodevelopmental conditions in general. "

In other words, the classifier needs to be validated on other samples, and it is as yet unclear whether it would be as accurate if people with autism were contrasted, say, with people with language impairments. The authors also noted that the adult sample used here was largely composed of high-functioning adults with Asperger syndrome, and it was not clear whether results would generalise to other cases of autism.  In the final sentence of the paper it is stated: "classification values and specific patterns we report must be considered as preliminary".

In sum, the paper illustrates a promising approach to understanding more about brain development in autism, but it does not demonstrate the utility of this approach for differential diagnosis of autism from other conditions, nor for assisting diagnosis in marginal and difficult cases.

Encouraged by an enthusiastic press release from the Medical Research Council, who funded this work, the media picked up the story yesterday.  It was featured prominently on BBC television and radio news, made the front page story on the Independent, and was given extensive coverage in other national newspapers.  The account of the work showed little evidence of the caution and qualification in the published account.  In this case, I don't think the media can be blamed for misrepresenting the work, as much of what was said was taken from the MRC press release.  Here are some statements from the Independent, under the headline: "Brain scan promises to identify the hidden sufferers of autism":

• Autism could in future be diagnosed in 15 minutes from a brain scan

• Scientists ... have devised a method which can distinguish the autistic brain from the normal brain with 90 per cent accuracy.

• The method could lead to the introduction of screening for the disorder in children.

Dr Ecker was quoted as saying:

"The [computer programmme] says yes this is autism or no it is not, and also gives an indication of the severity."  She also contrasted the 15 minutes it took to do an MRI scan with conventional diagnostic procedures which "takes the whole day and involves asking embarrassing questions such as how many friends they have and how they are doing at school"

Dr Ecker seems to be implying that we should stop using diagnostic interviews to diagnose autism, because a brain scan is more accurate.  Since autism is currently defined in terms of a pattern of impairments in different domains of functioning, this is a remarkably bold claim, especially since her method awarded an autism diagnosis to 20% of controls.

Much is also made of the potential of the method for diagnosing autism in children. It's possible this may be effective, but children's brains are very different from adult brains, and one would need a whole new study to evaluate this. Furthermore, although Dr Ecker says of brain scanning "It doesn't hurt and you can even go to sleep", it is not an easy procedure to apply to young children without sedation.  Lying still in a very noisy tube is not something that comes naturally to a child, even if you can explain the procedure to them clearly so they know what to expect. Most children with autism have poor language comprehension and may also become anxious when in an unfamiliar situation. It is not impossible to do brain scanning studies with them, but it certainly is not easy.

A final thought. In recent years, brain research on autism has increased exponentially. There are other MRI datasets for adults and children with autism and other clinical groups as well as controls. It should be possible for Ecker and colleagues to cross-validate their findings fairly rapidly against other datasets. I hope that others will be willing to share their data to make this possible.