My thoughts on the dyslexia debate

©CartoonStock.com

During February, there was widespread media coverage of a forthcoming book by Julian Elliott and Elena Grigorenko called The Dyslexia Debate. I've seen an advance copy of the book, whose central message is to argue that the construct of dyslexia lacks coherence. Quite simply, dyslexia does not constitute a natural category, in terms of cognitive profile, neurobiology or genetics.

The authors' arguments are backed by a large body of research: people have tried over many years to find something distinctive about dyslexia, without success. Some children are good at reading and others are not, but it's arbitrary where you place a cutoff to specify that a child has a problem. There's a popular belief that you can identify dyslexics in terms of a particular ability profile, and that affected children have a particular kind of brain organisation that makes them do things like reverse letters (b vs d) or have left-right confusion. In fact, those types of problem are common in typically-developing children at early stages of learning to read and appear to be as much a symptom as a cause of reading problems. Researchers have found neurobiological and genetic correlates of developmental reading problems, but the effects tend to be small and inconsistent from person to person: you could not currently diagnose dyslexia on the basis of brain scans or genetic analysis. It is, of course, possible that one day we may hit upon a new diagnostic test that does clearly differentiate a dyslexic subgroup from other poor readers, but many of us in the field are dubious as to whether this will ever happen.
 
The first thing to get clear is that Elliott and Grigorenko are not denying the reality of children's reading problems. Their point is a much more specific one about the way we conceptualise reading difficulties and how this affects access to services in everyday life. Their concern is that "dyslexia" implies we are dealing with a specific medical syndrome. Their view is that no such syndrome exists and it is not helpful to behave as if it does. How should we respond to this? I think we need to distinguish three questions:

1. Should we identify those in need of extra help?

Children vary in the ease with which they learn to read. Some need only the briefest exposure to books to crack the code; others struggle for years despite skilled help from expert teachers.
I think most people who have spent time with poor readers (and I would include Elliott and Grigorenko among these) would conclude that the answer to question #1 is yes. It doesn't matter whether those in the latter group have a distinct medical syndrome or not: It is up to us to ensure that all get the best teaching.

2. How should we identify those in need of extra help?

Elliott wrote a piece in the Times Higher Education where he argued that dyslexia diagnoses in Universities were skyrocketing, and that some people were unfairly exploiting the system in order to get accommodations such as a laptop computer and extra time in exams. To my mind, the problem here is less to do with the "dyslexia" label, and more to do with the haphazard way in which individuals are identified, and the lack of consistent criteria for determining who needs extra help. Like Elliott, I think it is entirely right that we should make accommodations for students who have serious difficulties in processing written information at speed. However, as he highlights, the current system is based on an unsustainable idea that "dyslexia" is a distinct disorder that can be reliably identified, and which is often diagnosed on the basis of supposed markers of dyslexia that have no scientific basis. So the current system is both invalid and unfair. Instead, it would be sensible to settle on consistent criteria for allocating extra help to students who are struggling, and to ensure that extra resources are directed to those who are most needy. As Castles and colleagues have noted, there are guidelines that can be used to identify those with severe and persistent problems, but they are not well-known or widely applied.

3. What terminology should we use to refer to those we identify?

So can we just agree that we need to find consistent ways of identifying poor readers and do away with the term "dyslexia"? While this might seem a logical response to the evidence, I think we should not underestimate the implications in practice. On the positive side, we'd get rid of the idea that we're dealing with a special condition that forms a distinct syndrome. Since few scientists would attempt to defend that notion, this would be a good thing. But we should also be aware of negative consequences.

Those commenting on the dyslexia debate so far have talked about it as if it is a particular issue relating to literacy difficulties, but in fact it's just one instance of a much more pervasive problem.  Other neurodevelopmental disorders such as autism spectrum disorder, specific language impairment, attention deficit hyperactivity disorder, developmental dyspraxia and dyscalculia are all beset by the same issues: there is no diagnostic biomarker, the condition is defined purely in terms of behaviour, different disorders overlap and there's no clear boundary between disorder and normality.
 
Similar issues have been much discussed in relation to adult psychiatric disorders, which are also diagnosed in terms of behavioural features rather than biological tests. In a fascinating paper, Kendell and Jablensky (2003) came to the conclusion that the categories of schizophrenia and depression are massively problematic in terms of validity and reliability – that is to say, just like dyslexia, they don't constitute natural categories clearly demarcated from other disorders, and furthermore, people can't even agree on who merits these diagnoses. So should we just stop using the labels? Kendell and Jablensky considered this possibility but concluded it would be impossible to abandon terms like schizophrenia and depression, on the grounds that they have utility. These labels have been used for many years by practitioners to determine the most effective intervention, and by researchers interested in discovering the underlying causes and likely outcome of a disorder. Similarly, using the construct of "dyslexia" we have discovered much about the nature of the cognitive deficits that characterise many poor readers, about underlying causes, about outcomes, and about effectiveness of intervention. For instance, we know that genes play a part in determining who is a poor reader, and that many children who have poor literacy skills also have subtle problems with oral language.
 
This argument, though, is not really watertight. We may congratulate ourselves on what we have learned, but on the other hand, it could be argued that there are also barriers to progress that arise from continued use of imprecise terms. It's clear to anyone who knows the research literature that findings can vary from study to study and from child to child within a study. This does not necessarily invalidate the research – it's rare to obtain perfect consistency of findings even within mainstream medicine – but it does make many people wonder whether we might obtain clearer results if we took a different approach. But then we have to consider what alternative approach would be better.

I suggested a few years ago that it might be helpful to treat neurodevelopmental disorders differently, as multidimensional composites, rather than regarding problems with reading, language, arithmetic, attention, motor skills and social behaviour as separate conditions. However, I did not really expect anyone to embrace this idea, as it would be too radical a change, and we are too wedded to current terminology.

Here too, comparisons with psychiatry are interesting. Last year, Tom Insel, director of the US National Institute of Mental Health, ruffled feathers by stating that his organisation would be reorienting its research away from traditional psychiatric diagnostic categories, to develop Research Domain Criteria, i.e. "new ways of classifying mental disorders based on dimensions of observable behavior and neurobiological measures." Yet the domains that are proposed seem to me just as arbitrary as the original diagnostic categories, and the associations between genetic, neurobiological and behavioural measures are mostly weak and poorly understood. So although Insel's vision might seem a rational way of trying to make sense of psychiatric disorders, it is years away from being clinically applicable – as he is the first to admit.  Even though multivariate, dimensional classification seems more logical, our current categories of autism, schizophrenia and dyslexia, though imperfect, may be as good as we can manage in terms of utility in day-to-day clinical practice.

Perhaps the strongest arguments in favour of retention of a term like "dyslexia" come not from science but from public perception. Like it or not, "dyslexia" has been around for over 100 years. In that time, a range of organisations have sprung up to help people with this diagnosis. Some of the most passionate defences of the dyslexia label come from those who have built up a sense of identity around this condition, and who feel they benefit from being part of a community that can offer information and support – see, for instance, this comment by the International Dyslexia Association to the suggestion that "dyslexia" be removed from the DSM5.

One could, of course, argue, that we shouldn't stick with a label just because it has always been there – if we were to adopt that line of argument, we'd still be talking about "maladjusted" and "educationally subnormal" children. But it's clear that many of those diagnosed with dyslexia do see this label as positive. In particular many people worry that if they were to simply switch to a more neutral, less medical term, such as "poor readers", this could trivialise reading problems, and lead people to assume that the difficulties are just caused by poor teaching. Furthermore, legal entitlement to special help under disability legislation could disappear. This, I think, is a key part of the problem, which can get overlooked when just focusing on the scientific evidence: what you call a condition determines two things: how seriously people take it, and where they place blame for the difficulties and responsibility for doing something about it.

To illustrate my point, see this recent piece in the Daily Mail by Peter Hitchens, which appeared under the headline: "Dyslexia is NOT a disease. It is an excuse for bad teachers". This displays a remarkably simplistic world view in which a poor reader either has a "disease", in which case they are blameless victims of an external force, or else it is someone's fault – in this case lacklustre teachers.

In his triumphalist piece against the "pseudoscience and quackery" of dyslexia, Peter Hitchens achieves exactly the opposite of what he intends. This is because he demonstrates one negative consequence of removing the label, which is that many people will no longer think that children who struggle to read need any kind of special help. Instead, we'll be told that "What they need, what we all need, is proper old-fashioned teaching."

A rather more sophisticated version of the same argument was given in the Green Paper that introduced the Government's proposed revision to legislation for Special Educational Needs (SEN) (see: my blogpost on this). There it was stated that too many children were being over-identified with SEN: “Previous measures of school performance created perverse incentives to over-identify children as having SEN. There is compelling evidence that these labels of SEN have perpetuated a culture of low expectations and have not led to the right support being put in place.” (point 22).

We really need to escape this polarised view of children's problems being caused either by a medical disease or by poor teaching. Yes, some children's reading may be held back because their teachers either don't know about or reject evidence-based methods of teaching, but it is seldom black and white, and some children fail despite intensive, high-quality teaching.

My concern is that those holding the purse-strings have a strong incentive to blame all problems on bad teaching or bad parenting, as it absolves them of any responsibility to do anything about them. We need to recognise that for most children, the causal influences are likely to be complex and may involve both constitutional factors and aspects of home and school environment. Unfortunately, most people don't seem able to deal with this complexity, and the language we use determines how problems are viewed. At present we are between a rock and a hard place. The rock is the term "dyslexia", which has inaccurate connotations of a distinct neurobiological syndrome. The hard place is a term like "poor readers" which leads people to think we are dealing with a trivial problem caused by bad teaching.

As Allen Frances argued in the case of psychiatry, we need to resist a growing tendency to use medical labels for what is essentially normal behaviour. However, he wisely notes that this should not blind us to the reality that there are people with problems that are severe, clearcut, and unlikely to go away on their own.  In the current debate, several commentators have made this point and have added that it doesn't really matter what we call them; the more important issue is to ensure affected individuals get appropriate help. But I'd suggest it does matter, because the label we use does much more than just identify a subset of people: it carries connotations of causation, blame and responsibility. While I can see all the disadvantages of the dyslexia label outlined by Elliott and Grigorenko, I think it will survive into the future because it provides many people with a positive view of their difficulties which also helps them get taken seriously. For that reason, I think we may find it easier to work with the label and try to ensure it is used in a consistent and meaningful way, rather than to argue for its abolition.

Reference
Kendell, R., & Jablensky, A. (2003). Distinguishing between the validity and utility of psychiatric diagnoses American Journal of Psychiatry, 160 (1) DOI: 10.1176/appi.ajp.160.1.4


This article (Figshare version) can be cited as:
Bishop, Dorothy V M (2014): My thoughts on the dyslexia debate. figshare
http://dx.doi.org/10.6084/m9.figshare.1030404
 

Parent talk and child language

© www.CartoonStock.com

There's been a lot in the media lately about the impacts of parental talk on children's language development. Some of it has been opinion, as in this piece in the Daily Telegraph, in which the headline proclaimed that children were "starting school unable to speak". This reflected the views of a head teacher, who claimed that the proportion of children with poor language skills had increased in his lifetime, and that this was the fault of parents who did not have time to talk to their children any more. There is nothing new here: versions of this story pop up every few years or so (here's one from 2003,  and a blogpost on another case from 2011): Editors know that stories about feckless parents sell newspapers: readers love the sense of complacency and moral superiority they induce.

But there is also more evidence-based stuff. Some children do have serious difficulties mastering spoken language, and there is research demonstrating links between parent talk and child language outcomes. We've known since the influential study of Hart and Risley (1995) that there is massive variation in the amount of language children are exposed to at home, and this is predicted by socio-economic status. There are many subsequent studies showing positive associations between aspects of the language that babies and toddlers hear and the rate and complexity of their language development.

When the Guardian ran a piece last week on the latest of these studies, someone tweeted "do we really need a study to demonstrate that?" – to most people it's blindingly obvious that children's language development will be determined by the language that they hear at home. This assumption is shared by many professionals in the field of language development; for instance, in a recent review, Leffel and Suskind (2013) describe poor attainment of children from disadvantaged homes and unambiguously state: "Parent linguistic input lies at the heart of the problem".

Except that it's not so simple. And the complexities become apparent when we look at the type of evidence that we have, which is mostly correlational. Students learn in Psychology #101 that correlation does not equal causation, yet when a causal interpretation seems so obvious to most people, this can get forgotten. I have lost count of the number of times I've seen a study showing that parent talk predicts child language development, where the conclusion drawn by the authors (and press offices and the media) is that limited parental language causes child language problems. No other explanation is even countenanced. Yet if we were well taught in Psychology #101, we would realise that we need to consider alternative explanations for the observed association. The figure below shows three possible causal models; these are not mutually exclusive and so all could play a role.

Different Models to account for association between parent talk and child language

Model A is the one that is typically assumed by most people: parent talk to children boosts their language development, and accordingly, if a child has poor language skills, this is likely to be caused by inadequate talk from parents.

In Model B, the association goes in the other direction. Poor language in the child leads to less talk from the parent. This could occur if, for instance, parents are discouraged from talking to a child who is unresponsive and appears not to understand. Consider too, this recent study looking at outcomes of infants in a special care baby unit . Children who were exposed to more adult language in hospital had better language outcomes; however, as the authors noted, "It could be that parents and caregivers have more opportunity to talk to infants who are less sick."

Model C explains the association without postulating a direct link from parental talk to child language. Instead it sees both of these as outcomes of some other cause. This could be an environmental factor, such as poor diet, or a genetic risk that is shared by parents and their children.

It is the job of researchers to try and find evidence to establish the relative importance of these different causal routes. In the case of child language, this is not just a theoretical exercise: it potentially makes a difference to the kinds of intervention that are likely to be effective in helping children. In particular, if model A is the main explanation for the association, then we should be able to boost poor child language by encouraging reticent parents to interact more like talkative parents. This is unlikely to be effective if model B explains the association. And if model C applies, then we would need to either modify the third factor (X) itself, or clarify how it operated in order to alter its association with poor outcomes in children.

I am concerned about the near-universal acceptance of model A as the sole explanation, because there are two lines of evidence that go against it. First, we can to some extent disentangle the impact of socioeconomic disadvantage and parental talk if we study children whose parents produce little spoken language input because they have a congenital hearing impairment. Some profoundly deaf parents have children with normal hearing. In the past there was concern about such children: how would they learn spoken language if their parents produced little intelligible speech? In fact, the studies that were done obtained unexpectedly positive results, leading to the conclusion that although young children clearly need some exposure to spoken language in order to learn to speak, they could develop normal language on the basis of exposure to other adults outside the home and language on TV (Schiff-Myers, 1988).

The second line of evidence comes from studies that disentangle genetic and environmental influences by considering language development in twins. If parental talk is an important determinant of child language, then we would expect twins growing up together in the same home to resemble each other. However, if model A is all-important, we would not expect the genetic relationship between the twins to have any effect. But it does make a difference, and on many language measures this effect is quite substantial. So we find that twins do resemble each other in general, but that resemblance is quite a bit higher if the twins are genetically identical (monozygotic) than if they are fraternal (dizygotic, and sharing around half their DNA for genes that vary between people).

I remember being struck when I first did twin studies of children's language difficulties at how different two twins growing up in the same family could be – provided they were non-identical. It was, however, unusual to find identical twin pairs where one had a significant language problem and the other was unaffected. The overall pattern of results tells us that the child's genetic makeup plays a role in determining their language development (Bishop, 2006).

So what has this to do with models A, B and C? Quite simply, the twin data support a version of model C: given that genes affect language development, we expect parents (who share around half their genes with their children) to resemble their children. We already know that parents of children with language impairments are more likely than other parents to have some kind of language or literacy problem themselves (Barry et al, 2007). This doesn't affect everyone: of course there are many literate and articulate parents whose children have language difficulties. But on balance, these kinds of difficulties run through generations, and we therefore expect to see an association between limited language ability in parents and language difficulties in their children. Note that a genetic account will also predict that language difficulties in children will predominate among those of lower social-economic status: parents who themselves are language-impaired are likely to have low levels of educational attainment and poor occupational prospects.

This kind of genetic explanation for parent-child similarities has a lot of evidential support, but people are very reluctant to accept it. If you propose that genes may play a role in children's developmental difficulties, people will tend to assume that you have a political agenda aligned with the Third Reich, with a goal of identifying a genetic underclass who should not be helped because they are just 'made that way'. This reflects a wrong-headed genetic determinism that is at odds with contemporary understanding of how genes work. Genes do not determine your fate: their impact is likely to vary according to the environment, and by modifying environments we may alter outcomes. Unlike in model A, though, model C predicts that sensitivity to specific environments may depend on one's genes. The arguments have been cogently put in a recent book by Asbury and Plomin (2013), who lament the way in which genetic influences on children's development have been ignored in favour of a political stance that blames educational and developmental difficulties on either poor parenting or poor teaching. If, as has been repeatedly shown, there is evidence that genes are important in influencing children's language development, then we may be squandering our intervention resources by ignoring this fact.

The bottom line is that we need more research. Well-conducted randomized controlled trials on the impact of modifying parent input have been thin on the ground to date, and have not generated impressive evidence of efficacy (see my earlier blogpost) . Obviously, it's early days, and I'd cheer on others who are attempting such research. Results may depend on the nature of the intervention, the aspects of language that are assessed, and the type of population the intervention is used with. My suggestion is that rather than denying the reality of genetic effects, we should be conducting research to find out what kinds of input are most effective for children who are at genetic risk. It is possible that rather than more language input, they may do best with a different kind of language input, specifically tailored to take into account their cognitive strengths and weaknesses. We are a long way from understanding how best to do this, and meanwhile, ingenious and dedicated practitioners are working hard to tackle the very real problems that some children experience. My message is simply that to lay the blame for these difficulties at the door of parents, and to anticipate that problems can be readily overcome by encouraging parents to talk more to their children may be oversimplistic.

To finish, I cannot resist adding my favourite quote from Richard Dawkins, which focuses on mathematics rather than language learning, but gets to the nub of inappropriate concerns about genetic explanations:

People seem to have little difficulty in accepting the modifiability of "environmental" effects on human development. If a child has had bad teaching in mathematics, it is accepted that the resulting deficiency can be remedied by extra good teaching the following year. But any suggestion that the child's mathematical deficiency might have a genetic origin is likely to be greeted with something approaching despair: if it is in the genes "it is written", it is "determined" and nothing can be done about it: you might as well give up attempting to teach the child mathematics. This is pernicious rubbish on an almost astrological scale ..... What did genes do to deserve their sinister juggernaut-like reputation? Why do we not make a similar bogey out of, say, nursery education or confirmation classes? Why are genes thought to be so much more fixed and inescapable in their effects than television, nuns, or books? 

Richard Dawkins (1982) The extended phenotype, Oxford University Press (p. 13) 

References 
Asbury, K., & Plomin, R. (2013). G is for genes: The impact of genetics on education and achievement. Chichester: Wiley Blackwell.
Barry, J. G., Yasin, I., & Bishop, D. V. M. (2007). Heritable risk factors associated with language impairments. Genes, Brain and Behavior, 6, 66-76.
Bishop, D. V. M. (2006). What causes specific language impairment in children? Current Directions in Psychological Science, 15, 217-221.
Caskey, M., Stephens, B., Tucker, R., & Vohr, B. (2014). Adult talk in the NICU with preterm infants and developmental outcomes Pediatrics DOI: 10.1542/peds.2013-0104
Hart, B., & Risley, T. R. (1995). Meaningful differences in the everyday experience of young American children. Baltimore, MD: Paul H. Brookes Publishing Co.
Leffel, K., & Suskind, D. (2013). Parent-directed approaches to enrich the early language environments of children living in poverty. Seminars in Speech and Language, 34(4), 267-277. doi: 10.1055/s-0033-1353443
Schiff-Myers, N. (1988). Hearing children of deaf parents. In D. Bishop & K. Mogford (Eds.), Language development in exceptional circumstances (pp. 47-61). Edinburgh: Churchill Livingstone.

This article (Figshare version) can be cited as:
Bishop, Dorothy V M (2014): Parent talk and child language. figshare.
http://dx.doi.org/10.6084/m9.figshare.1030407

What is educational neuroscience?

©CartoonStock.com

As someone who works at the interface of child development and neuroscience, I've been struck by the relentless rise of the sub-discipline of 'educational neuroscience'. New imaging technologies have led to a burgeoning of knowledge about the developing brain, and it is natural to want to apply this knowledge to improving children's learning. Centres for educational neuroscience have sprung up all over the place, with support from universities who see them as ticking two important boxes: interdisciplinarity and impact.

But at the heart of this enterprise, there seems to be a massive disconnect. Neuroscientists can tell you which brain regions are most involved in particular cognitive activities and how this changes with age or training. But these indicators of learning do not tell you how to achieve learning. Suppose I find out that the left angular gyrus becomes more active as children learn to read. What is a teacher supposed to do with that information?

As John Bruer pointed out back in 1997, the people who can be useful to teachers are psychologists. Psychological experiments can establish the cognitive underpinnings of skills such as reading, and can evaluate which are the most effective ways of teaching, and whether these differ from child to child. They can address questions such as whether there are optimal ages at which to teach different skills, how motivation and learning interact, and whether it is better to learn material in large chunks all at once or spaced out over intervals. At a trivial level, these could all be designated as aspects of 'educational neuroscience', insofar as the brain is necessarily involved in cognition and motivation. But they can all be studied without taking any measurements of brain function.

It is possible, of course, to look at the brain correlates of all of these things, but that's unlikely to influence what's done in the classroom. Suppose I want to see whether training in phonological awareness improves children's reading outcomes. I measure brain activation before and after training, and compare results with those of a control group who don't get the training. There are various possible patterns of results, as laid out in the table below:

As pointed out by Coltheart and McArthur (2012), what matters to the teacher is whether the training is effective in improving reading. It's really not going to make any difference whether detectable brain changes have happened, so either outcome A or B would give good justification for adopting the training, whereas outcomes C and D would not.

Well, you might say, children differ, and the brain measures might show up differences between those who do and don't respond to training. Indeed, but how would that be useful educationally? I've seen several studies that propose brain scans might be useful in identifying which children will and won't benefit from an intervention. That's a logical possibility, but given that brain scanning costs several hundred pounds per person, it's not realistic to suggest this has any utility in the real world, especially when there are likely to be behavioural indicators that predict outcomes just as well.

So are there actual or potential examples of how knowledge of neuroscience - as opposed to psychology - might influence educational practice? I mentioned three examples in this review: neurofeedback, neuropharmacology and brain stimulation are all methods that focus directly on changing the brain in ways that might potentially affect learning, and so could validly be designated as educational neuroscience. They are, however, as yet exploratory and experimental. The last of these, brain stimulation, was described this week in a blogpost by Roi Cohen Kadosh, who notes promising early results, but emphasizes that we need more experimental work establishing both risks and benefits before we could consider direct application of this method to improving children's learning.

I'm all in favour of cognitive neuroscience and basic research that discovers more about the neural underpinnings of typical and atypical development. By all means, let's do such studies, but let's do them because we want to find out more about the brain, and not pretend it has educational relevance.

If our goal is to develop better educational interventions, then we should be directing research funds into well-designed trials of cognitive and behavioural studies of learning, rather than fixating on neuroscience. Let me leave the last word to Hirsh-Pasek and Bruer, who described a Chilean conference in 2007 on Early Education and Human Brain Development. They noted: "The Chilean educators were looking to brain science for insights about which type of preschool would be the most effective, whether children are safe in child care, and how best to teach reading. The brain research presented at the conference that day was mute on these issues. However, cognitive and behavioral science could help."

References
Bishop, D. V. M. (2013). Neuroscientific studies of intervention for language impairment in children: interpretive and methodological problems Journal of Child Psychology and Psychiatry, 54 (3), 247-259 DOI: 10.1111/jcpp.12034

Bruer, J. T. (1997). Education and the brain: A bridge too far. Educational researcher, 26(8), 4-16. doi: 10.3102/0013189X026008004

Coltheart, M., & McArthur, G. (2012). Neuroscience, education and educational efficacy research. In M. Anderson & S. Della Sala (Eds.), Neuroscience in Education (pp. 215-221). Oxford: Oxford University Press.

This article (Figshare version) can be cited as: 
Bishop, Dorothy V M (2014): What is educational neuroscience?. figshare.
http://dx.doi.org/10.6084/m9.figshare.1030405

Why does so much research go unpublished?

As described in my last blogpost, I attended an excellent symposium on waste in research this week. A recurring theme was research that never got published. Rosalind Smyth described her experience of sitting on the funding panel of a medium-sized charity. The panel went to great pains to select the most promising projects, and would end a meeting with a sense of excitement about the great work that they were able to fund. A few years down the line, though, they'd find that many of the funds had been squandered. The work had either not been done, or had been completed but not published.

In order to tackle this problem, we need to understand the underlying causes. Sometimes, as Robert Burns noted, the best-laid schemes go wrong. Until you've tried to run a few research projects, it's hard to imagine the myriad different ways in which life can conspire to mess up your plans. The eight laws of psychological research formulated by Hodgson and Rollnick are as true today as they were 25 years ago.

But much research remains unpublished despite being completed. Reasons are multiple, and the strategies needed to overcome them are varied, but here is my list of the top three problems and potential solutions.

Inconclusive results

Probably the commonest reason for inconclusive results is lack of statistical power. A study is undertaken in the fond hope that a difference will be found between condition X and condition Y, and if the difference is found, there is great rejoicing and a rush to publish. A negative result should also be of interest, provided the study was well-designed and adequately motivated. But if the sample is small, then we can't be sure whether our failure to observe the effect is because it is absent: a real but small effect could be swamped by noise. 

I think the solution to this problem lies in the hands of funding panels and researchers: quite simply, they need to take statistical power very seriously indeed and to consider carefully whether anything will be learned from a study if the anticipated effects are not obtained. If not, then the research needs to be rethought. In the fields of genetics and clinical trials, it is now recognised that multicentre collaborations are the way forward to ensure that studies are conducted with sufficient power to obtain a conclusive result.

Rejection of completed work by journals

Even well-conducted and adequately powered studies may be rejected by journals if the results are not deemed to be exciting. To solve this problem, we must look to journals. We need recognition that - provided a study is methodologically strong and well-motivated - negative results can be as informative as positive ones. Otherwise we are doomed to waste time and money pursuing false leads.  As Paul Glasziou has emphasised, failure is part of the research process. It is important to tell people about what doesn't work if we are not to repeat our mistakes.

We do now have some journals that will publish negative results, and there is a growing move toward pre-registration of studies, with guaranteed publication if the methods meet quality criteria. But there is still a lot to be done, and we need a radical change of mindset about what kinds of research results are valuable.

Lack of time

Here, I lay the blame squarely on the incentive structures that operate in universities. To get a job, or to get promoted, you need to demonstrate that you can pull in research income. In many UK institutions this is quite explicit, and promotions criteria may give a specific figure to aim for of X thousand pounds research income per annum. There are few UK universities whose strategic plan does not include a statement about increasing research funding. This has changed the culture dramatically;  as Fergus Millar put it: "in the modern British university, it is not that funding is sought in order to carry out research, but that research projects are formulated in order to get funding".

Of course, for research to thrive, our Universities need people who can compete for funding to support their work. But the acquisition of funding has become an end in itself, rather than a means to an end. This has the pernicious effect of driving people to apply for grant after grant, without adequately budgeting for the time it takes to analyse and write up research, or indeed to carefully think about what they are doing.  As I argued previously, even junior researchers these days have an 'academic backlog' of unwritten papers.

At the Lancet meeting there were some useful suggestions for how we might change incentive structures to avoid such waste. Malcolm MacLeod argued researchers should be evaluated not by research income and high-impact publications, but by the quality of their methods, the extent to which their research was fully reported, and the reproducibility of findings. An-Wen Chan echoed this, arguing for performance metrics that recognise full dissemination of research and use of research datasets by other groups. However, we may ask whether such proposals have any chance of being adopted when University funding is directly linked to grant income, and Universities increasingly view themselves as businesses.

I suspect we would need revised incentives to be reflected at the level of those allocating central funding before vice-chancellors took them seriously.  It would, however, be feasible for behaviour to be shaped at the supply end, if funders adopted new guidelines. For a start, they could look more carefully at the time commitments of those to whom grants are given: in my experience this is never taken into consideration, and one can see successful 'fat cats' accumulating grant after grant, as success builds on success. Funders could also monitor more closely the outcomes of grants: Chan noted that NIHR withholds 10% of research funds until a paper based on the research has been submitted for publication. Moves like this could help us change the climate so that an award of a grant would confer responsibility on the recipient to carry through the work to completion, rather than acting solely to embellish the researcher's curriculum vitae.

References

Chan, A., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gotzsche, P., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014). Increasing value and reducing waste: addressing inaccessible research Lancet (8 Jan ) : 10.1016/S0140-6736(13)62296-5

Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., . . . Glasziou, P. (2014). Biomedical research: increasing value, reducing waste. Lancet, 383(9912), 101-104.

Off with the old and on with the new: the pressures against cumulative research

 

Yesterday I escaped a very soggy Oxford to make it down to London for a symposium on "Increasing value, reducing waste" in Research. The meeting marked the publication of a special issue of the Lancet containing five papers and two commentaries, which can be downloaded here.

I was excited by the symposium because, although the focus was on medicine, it raised a number of issues that have much broader relevance for science, including several that I have raised on this blog, including pre-registration of research, criteria used by high-impact journals,  ethics regulation, academic backlogs, and incentives for researchers. It was impressive to see that major players in the field of medicine are now recognizing that there is a massive problem of waste in research. Better still, they are taking seriously the need to devise ways in which this could be fixed.

I hope to blog about more of the issues that came up in the meeting, but for today I'll confine myself to one topic that I hadn't really thought about much before, but which I see as important, namely the importance of doing research that builds on previous research, and the current pressures against this.

Iain Chalmers presented one of the most disturbing slides of the day, a forest plot of effect sizes found in medical trials for a treatment to prevent bleeding during surgery.

Based on Figure 3 of Chalmers et al, 2014

Time is along the x-axis, and the horizontal line corresponds to a result where the active and control treatments do not differ. Points which are below the line and whose fins do not cross it show a beneficial effect of treatment. The graph shows that the effectiveness of the treatment was clearly established by around 2002, yet a further 20 studies including several hundred patients were reported in the literature after that date. Chalmers made the point that it is simply unethical to do a clinical trial if previous research has already established an effect. The problem is that researchers often don't check the literature to see what has already been done, and so there is wasteful repetition of studies. In the field of medicine this is particularly serious because patients may be denied the most effective treatment if they enrol in a research project.

Outside medicine, I'm not sure this is so much of an issue. In fact, as I've argued elsewhere, in psychology and neuroscience I think there's more of a problem with lack of replication. But there definitely is much neglect of prior research. I lose count of the number of papers I review where the introduction presents a biased view of the literature that supports the authors' conclusions. For instance, if you are interested in the relation between auditory deficit and children's language disorders, it is possible to write an introduction presenting this association as an established fact, or to write one arguing that it has been comprehensively debunked. I have seen both.

Is this just lazy, biased or ignorant authors? In part, I suspect it is. But I think there is a deeper problem which has to do with the insatiable demand for novelty shown by many journals, especially the high-impact ones. These journals typically have a lot of pressure on page space and often allow only 500 words or less for an introduction. Unless authors can refer to a systematic review of the topic they are working on, they are obliged to give the briefest account of prior literature. It seems we no longer value the idea that research should build on what has gone before: rather, everyone wants studies that are so exciting that they stand alone. Indeed, if a study is described as 'incremental' research, that is typically the death knell in a funding committee.

We need good syntheses of past research, yet these are not valued because they are not deemed novel. One point made by Iain Chalmers was that funders have in the past been reluctant to give grants for systematic reviews. Reviews also aren't rated highly in academia: for instance, I'm proud of a review on mismatch negativity that I published in Psychological Bulletin in 2007. It not only condensed and critiqued existing research, but also discovered patterns in data that had not previously been noted. However, for the REF, and for my publications list on a grant renewal, reviews don't count.

We need a rethink of our attitude to reviews. Medicine has led the way and specified rigorous criteria for systematic reviews, so that authors can't just cherrypick specific studies of interest. But it has also shown us that such reviews are an invaluable part of the research process. They help ensure that we do not waste resources by addressing questions that have already been answered, and they encourage us to think of research as a cumulative, developing process, rather than a series of disconnected, dramatic events.

Reference
Chalmers, Iain, Bracken, Michael B., Djulbegovic, Ben, Garattini, Silvio, Grant, Jonathan, Gülmezoglu, A. Metin, Howells, David W., Ioannidis, John P. A., & Oliver, Sandy (2014). How to increase value and reduce waste when research priorities are set Lancet : 10.1016/S0140-6736(13)62229-1

A New Year's letter to academic publishers

My relationships with journals are rather like a bad marriage: a mixture of dependency and hatred. Part of the problem is that journal editors and academics often have a rather different view of the process. Scientific journals could not survive without academics. We do the research, often spending several years of our lives to produce a piece of work that is then distilled into one short paper, which the fond author invariably regards as a fascinating contribution to the field. But when we try to place our work in a journal, we find that it's a buyer's market: most journals are overwhelmed with more submitted papers than they can cope with, and rejection rates are high. So there is a total mismatch: we set out naively dreaming of journals leaping at the opportunity to secure our best work, only to be met with coldness and rejection.  As in the best Barbara Cartland novels, for a lucky few, persistence is ultimately rewarded, and the stony-hearted editor is won over. But many potential authors fall by the wayside long before that point.

But times are changing. We are moving from a traditional "dead tree technology" model, where journals have to be expensively printed and distributed, to electronic-only media. These not only cost less to produce, but also avoid the length limits that traditionally have forced journals to be so highly selective. Alongside the technological changes, there has been rapid growth of the Open Access movement. The main motivations behind this movement were idealistic (making science available to all) and economic (escaping the stranglehold of expensive library subscriptions to closed-access journals). It's early days, but I am starting to sense that there's another consequence of the shift, which is that, as the field opens up, publishers are starting to change how they approach authors: less as supplicants, and more as customers.

In the past, the top journals had no incentive to be accommodating to authors. There were too many of us chasing scarce page space. But there are now some new boys on the open access block, and some of them have recognised that if they want to attract people to publish with them, they should listen to what authors want. And if they want academics to continue to referee papers for no reward, then they had better treat them well too.

This really is not too hard to do. I have two main gripes with journals, a big one and a little one. The big one concerns my time. The older I get, the less patient I am with organizations that behave as if I have all the time in the world to do the small bureaucratic chores that they wish to impose on me. For instance, many journals specify pointless formatting requirements for an initial submission. I really, really resent jumping through arbitrary hoops when the world is full of interesting things I could be doing. And cutting my toenails is considerably more interesting than reformatting references.

I recently encountered a journal whose website required you to enter details (name/address/email) of all authors in order to submit a pre-submission enquiry. Surely the whole point of a pre-submission enquiry is to save time, so you can get a quick decision on whether it's likely to be worth your while battling with the submission portal! There's also the horror of journals that require signatures from all authors at the point when you submit a manuscript: seems a harmless enough requirement, except that authors are often widely dispersed - on maternity leave or sailing the Atlantic - by the time the paper is submitted. The idea is to avoid fraud, of course, but like so many ethics regulations, the main effect of this requirement is to encourage honest, law-abiding people to take up forgery.

Oh, and then there are the 'invitations to review' (makes it sound so enticing, like being invited to a party), which require you to login in order to register your response – which for me invariably means selecting the option that I have forgotten my password, then looking at email to find how to update the password, meanwhile getting distracted by other email messages so I forget what I was doing, and eventually returning to the site to find it wants me now to change the password and enter mandatory contact details before it will accept my response. Well, no.  I'm usually a good citizen but I'm afraid I've just stopped responding to those.

You'd think the advent of electronic submission would make life easier, but in fact it can just open up a whole new world of tiny, fiddly things that you are required to do before your paper is submitted. Each individual thing is usually fairly trivial, but they do add up. So, for instance, if you'd like your authors to suggest referees, please allow them to paste in a list. DO NOT require them to cut and paste title, forename, initial, surname, email and institution into your horrible little boxes for each of six potential referees.  It all takes TIME. And we have more important things in life to be getting on with. Including doing the science that allows us to get the point of writing a paper.

Even worse, some of the requirements of journals are just historical artefacts with no more rationale than male nipples.  Here's a splendid post by Kate Jeffery which in fact was the impetus for this blogpost. I thought of Kate when, having carefully constructed a single manuscript document including figures, as instructed by the Instructions for Authors, I got to the submission portal to be strictly told that ON NO ACCOUNT must the figures be included in the main manuscript. Instead, they had to be separated, not only from the manuscript, but also from their captions (which had to be put as a list at the end of the manuscript). This makes sense ONCE THE PAPER IS ACCEPTED, when it needs to be typeset.  But not at the point of initial submission, when the paper's fate is undecided: it may well be rejected, and if not, it will certainly require revision. And meanwhile, you have referees tearing their hair out trying to link up the text, the Figures and their captions.

The smaller gripe is just about treating people with respect. I do have a preference for journal editors whose correspondence indicates that they are a human being and not an automaton. I've moaned about this before, in an old post describing a taxonomy of journal editors, but my feeling is that in the three years since I wrote that, things have got worse rather than better. Publishers and editors may think they make their referees happy by writing and telling them how useful their review of a paper has been – but the opposite effect is created if it is clear that this is a form letter that goes to all referees, however hopeless.It is really better to be ignored than to be sent an insincere, meaningless email - it just implies that the sender thinks you are stupid enough to be taken in by it.

So my message to publishers in 2014 is really very simple. The market is getting competitive and if you want to attract authors to send their best work to you, and referees to keep reviewing for you, you need to become more sensitive to our needs.  Two journals that appear to be trying hard are eLife and PeerJ, who avoid most of the bad practices I have outlined. I am hoping their example will cause others to up their game. We are mostly very simple souls who are not hard to please, but we hate having our time wasted, and we do like being treated like human beings.

How the government spins a crisis: the blame game

from: http://www.youtube.com/watch?v=PkHb9q-jpDU

Thousands of people in the UK had a truly miserable Christmas, with extreme weather leading to flooding and power cuts. They were shocked and cold, blundering around in the dark, sometimes for as long as three days. When David Cameron went to visit Yalding in Kent on 27th December, he got an earful from local residents, who complained they had been abandoned, and had no help from the council, who had "all decided to go on holiday."

Cameron's visit was widely seen as a PR disaster: he was criticised for using the floods as a way of getting cheap publicity, and his government's cuts in spending on flood defences were commented on.

On 30th December, we had Owen Paterson, the Energy Secretary stating that energy companies had "let customers down" in their response to the storm.

Yesterday we heard that Tim Yeo chairman of the energy select committee, planned to summon bosses of energy companies to explain their poor performance.

Now, I have no love for the energy companies, whose rapacious pricing strategies are causing real hardship to many. But I find myself wondering what exactly they were supposed to do over the Christmas period. Presumably, if a power line comes down, it requires specialised machinery and replacement parts to be sourced and brought to the site – which may well be affected by flooding – and engineers who not only have the expertise to diagnose and correct the problem, but who are also fit and brave enough to do this in horrendous weather conditions. I doubt that large numbers of such people are just sitting around waiting to be called upon, and indeed over the Christmas period, some of them may have gone away on holiday, and others may themselves be affected by the flooding.  There was much criticism concerning the lack of information given to those affected by flooding and power cuts. But it's just not realistic to expect an organization to magic up large numbers of call centre staff out of nowhere in the middle of a crisis-ridden Christmas break. It's also worth noting that much of the valiant work of helping people deal with the flooding crisis was the responsibility of the fire service, currently under pressure from cuts to funding.

I simply don't know whether the energy companies could have done better; maybe they could have done more with live updates of information through websites, Twitter or local radio. Maybe they could have issued earlier warnings, or cancelled leave for key staff. But it concerns me that we have the Environment Secretary making a very public judgement on this matter, directing blame at energy companies, just a few days after the Prime Minister has been criticised, and long before there has been a chance to evaluate what happened, and which agencies were responsible for what, in a calm and thorough manner.

Forgive me if I seem cynical, but a rapid and punitive response seems to have become a standard reaction of government to situations where they are attracting adverse publicity. Find a scapegoat and come down on them heavily, whether it be Brodie Smith, Sharon Shoesmith or David Kelly. This deflects criticism from the government and makes them look strong. All the better if the criticism can be laid instead at the door of a person or organization who is already unpopular.

By all means, let us consider the response to the crisis to see what could have been done better. But the issues are far too important to be used as propaganda to enhance a government's popularity. Let us not be distracted from a much more important priority: calling the government to account for its policy of cutting back on measures of flood prevention.