What is educational neuroscience?

©CartoonStock.com

As someone who works at the interface of child development and neuroscience, I've been struck by the relentless rise of the sub-discipline of 'educational neuroscience'. New imaging technologies have led to a burgeoning of knowledge about the developing brain, and it is natural to want to apply this knowledge to improving children's learning. Centres for educational neuroscience have sprung up all over the place, with support from universities who see them as ticking two important boxes: interdisciplinarity and impact.

But at the heart of this enterprise, there seems to be a massive disconnect. Neuroscientists can tell you which brain regions are most involved in particular cognitive activities and how this changes with age or training. But these indicators of learning do not tell you how to achieve learning. Suppose I find out that the left angular gyrus becomes more active as children learn to read. What is a teacher supposed to do with that information?

As John Bruer pointed out back in 1997, the people who can be useful to teachers are psychologists. Psychological experiments can establish the cognitive underpinnings of skills such as reading, and can evaluate which are the most effective ways of teaching, and whether these differ from child to child. They can address questions such as whether there are optimal ages at which to teach different skills, how motivation and learning interact, and whether it is better to learn material in large chunks all at once or spaced out over intervals. At a trivial level, these could all be designated as aspects of 'educational neuroscience', insofar as the brain is necessarily involved in cognition and motivation. But they can all be studied without taking any measurements of brain function.

It is possible, of course, to look at the brain correlates of all of these things, but that's unlikely to influence what's done in the classroom. Suppose I want to see whether training in phonological awareness improves children's reading outcomes. I measure brain activation before and after training, and compare results with those of a control group who don't get the training. There are various possible patterns of results, as laid out in the table below:

As pointed out by Coltheart and McArthur (2012), what matters to the teacher is whether the training is effective in improving reading. It's really not going to make any difference whether detectable brain changes have happened, so either outcome A or B would give good justification for adopting the training, whereas outcomes C and D would not.

Well, you might say, children differ, and the brain measures might show up differences between those who do and don't respond to training. Indeed, but how would that be useful educationally? I've seen several studies that propose brain scans might be useful in identifying which children will and won't benefit from an intervention. That's a logical possibility, but given that brain scanning costs several hundred pounds per person, it's not realistic to suggest this has any utility in the real world, especially when there are likely to be behavioural indicators that predict outcomes just as well.

So are there actual or potential examples of how knowledge of neuroscience - as opposed to psychology - might influence educational practice? I mentioned three examples in this review: neurofeedback, neuropharmacology and brain stimulation are all methods that focus directly on changing the brain in ways that might potentially affect learning, and so could validly be designated as educational neuroscience. They are, however, as yet exploratory and experimental. The last of these, brain stimulation, was described this week in a blogpost by Roi Cohen Kadosh, who notes promising early results, but emphasizes that we need more experimental work establishing both risks and benefits before we could consider direct application of this method to improving children's learning.

I'm all in favour of cognitive neuroscience and basic research that discovers more about the neural underpinnings of typical and atypical development. By all means, let's do such studies, but let's do them because we want to find out more about the brain, and not pretend it has educational relevance.

If our goal is to develop better educational interventions, then we should be directing research funds into well-designed trials of cognitive and behavioural studies of learning, rather than fixating on neuroscience. Let me leave the last word to Hirsh-Pasek and Bruer, who described a Chilean conference in 2007 on Early Education and Human Brain Development. They noted: "The Chilean educators were looking to brain science for insights about which type of preschool would be the most effective, whether children are safe in child care, and how best to teach reading. The brain research presented at the conference that day was mute on these issues. However, cognitive and behavioral science could help."

References
Bishop, D. V. M. (2013). Neuroscientific studies of intervention for language impairment in children: interpretive and methodological problems Journal of Child Psychology and Psychiatry, 54 (3), 247-259 DOI: 10.1111/jcpp.12034

Bruer, J. T. (1997). Education and the brain: A bridge too far. Educational researcher, 26(8), 4-16. doi: 10.3102/0013189X026008004

Coltheart, M., & McArthur, G. (2012). Neuroscience, education and educational efficacy research. In M. Anderson & S. Della Sala (Eds.), Neuroscience in Education (pp. 215-221). Oxford: Oxford University Press.

This article (Figshare version) can be cited as: 
Bishop, Dorothy V M (2014): What is educational neuroscience?. figshare.
http://dx.doi.org/10.6084/m9.figshare.1030405

Why does so much research go unpublished?

As described in my last blogpost, I attended an excellent symposium on waste in research this week. A recurring theme was research that never got published. Rosalind Smyth described her experience of sitting on the funding panel of a medium-sized charity. The panel went to great pains to select the most promising projects, and would end a meeting with a sense of excitement about the great work that they were able to fund. A few years down the line, though, they'd find that many of the funds had been squandered. The work had either not been done, or had been completed but not published.

In order to tackle this problem, we need to understand the underlying causes. Sometimes, as Robert Burns noted, the best-laid schemes go wrong. Until you've tried to run a few research projects, it's hard to imagine the myriad different ways in which life can conspire to mess up your plans. The eight laws of psychological research formulated by Hodgson and Rollnick are as true today as they were 25 years ago.

But much research remains unpublished despite being completed. Reasons are multiple, and the strategies needed to overcome them are varied, but here is my list of the top three problems and potential solutions.

Inconclusive results

Probably the commonest reason for inconclusive results is lack of statistical power. A study is undertaken in the fond hope that a difference will be found between condition X and condition Y, and if the difference is found, there is great rejoicing and a rush to publish. A negative result should also be of interest, provided the study was well-designed and adequately motivated. But if the sample is small, then we can't be sure whether our failure to observe the effect is because it is absent: a real but small effect could be swamped by noise. 

I think the solution to this problem lies in the hands of funding panels and researchers: quite simply, they need to take statistical power very seriously indeed and to consider carefully whether anything will be learned from a study if the anticipated effects are not obtained. If not, then the research needs to be rethought. In the fields of genetics and clinical trials, it is now recognised that multicentre collaborations are the way forward to ensure that studies are conducted with sufficient power to obtain a conclusive result.

Rejection of completed work by journals

Even well-conducted and adequately powered studies may be rejected by journals if the results are not deemed to be exciting. To solve this problem, we must look to journals. We need recognition that - provided a study is methodologically strong and well-motivated - negative results can be as informative as positive ones. Otherwise we are doomed to waste time and money pursuing false leads.  As Paul Glasziou has emphasised, failure is part of the research process. It is important to tell people about what doesn't work if we are not to repeat our mistakes.

We do now have some journals that will publish negative results, and there is a growing move toward pre-registration of studies, with guaranteed publication if the methods meet quality criteria. But there is still a lot to be done, and we need a radical change of mindset about what kinds of research results are valuable.

Lack of time

Here, I lay the blame squarely on the incentive structures that operate in universities. To get a job, or to get promoted, you need to demonstrate that you can pull in research income. In many UK institutions this is quite explicit, and promotions criteria may give a specific figure to aim for of X thousand pounds research income per annum. There are few UK universities whose strategic plan does not include a statement about increasing research funding. This has changed the culture dramatically;  as Fergus Millar put it: "in the modern British university, it is not that funding is sought in order to carry out research, but that research projects are formulated in order to get funding".

Of course, for research to thrive, our Universities need people who can compete for funding to support their work. But the acquisition of funding has become an end in itself, rather than a means to an end. This has the pernicious effect of driving people to apply for grant after grant, without adequately budgeting for the time it takes to analyse and write up research, or indeed to carefully think about what they are doing.  As I argued previously, even junior researchers these days have an 'academic backlog' of unwritten papers.

At the Lancet meeting there were some useful suggestions for how we might change incentive structures to avoid such waste. Malcolm MacLeod argued researchers should be evaluated not by research income and high-impact publications, but by the quality of their methods, the extent to which their research was fully reported, and the reproducibility of findings. An-Wen Chan echoed this, arguing for performance metrics that recognise full dissemination of research and use of research datasets by other groups. However, we may ask whether such proposals have any chance of being adopted when University funding is directly linked to grant income, and Universities increasingly view themselves as businesses.

I suspect we would need revised incentives to be reflected at the level of those allocating central funding before vice-chancellors took them seriously.  It would, however, be feasible for behaviour to be shaped at the supply end, if funders adopted new guidelines. For a start, they could look more carefully at the time commitments of those to whom grants are given: in my experience this is never taken into consideration, and one can see successful 'fat cats' accumulating grant after grant, as success builds on success. Funders could also monitor more closely the outcomes of grants: Chan noted that NIHR withholds 10% of research funds until a paper based on the research has been submitted for publication. Moves like this could help us change the climate so that an award of a grant would confer responsibility on the recipient to carry through the work to completion, rather than acting solely to embellish the researcher's curriculum vitae.

References

Chan, A., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gotzsche, P., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014). Increasing value and reducing waste: addressing inaccessible research Lancet (8 Jan ) : 10.1016/S0140-6736(13)62296-5

Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., . . . Glasziou, P. (2014). Biomedical research: increasing value, reducing waste. Lancet, 383(9912), 101-104.

Off with the old and on with the new: the pressures against cumulative research

 

Yesterday I escaped a very soggy Oxford to make it down to London for a symposium on "Increasing value, reducing waste" in Research. The meeting marked the publication of a special issue of the Lancet containing five papers and two commentaries, which can be downloaded here.

I was excited by the symposium because, although the focus was on medicine, it raised a number of issues that have much broader relevance for science, including several that I have raised on this blog, including pre-registration of research, criteria used by high-impact journals,  ethics regulation, academic backlogs, and incentives for researchers. It was impressive to see that major players in the field of medicine are now recognizing that there is a massive problem of waste in research. Better still, they are taking seriously the need to devise ways in which this could be fixed.

I hope to blog about more of the issues that came up in the meeting, but for today I'll confine myself to one topic that I hadn't really thought about much before, but which I see as important, namely the importance of doing research that builds on previous research, and the current pressures against this.

Iain Chalmers presented one of the most disturbing slides of the day, a forest plot of effect sizes found in medical trials for a treatment to prevent bleeding during surgery.

Based on Figure 3 of Chalmers et al, 2014

Time is along the x-axis, and the horizontal line corresponds to a result where the active and control treatments do not differ. Points which are below the line and whose fins do not cross it show a beneficial effect of treatment. The graph shows that the effectiveness of the treatment was clearly established by around 2002, yet a further 20 studies including several hundred patients were reported in the literature after that date. Chalmers made the point that it is simply unethical to do a clinical trial if previous research has already established an effect. The problem is that researchers often don't check the literature to see what has already been done, and so there is wasteful repetition of studies. In the field of medicine this is particularly serious because patients may be denied the most effective treatment if they enrol in a research project.

Outside medicine, I'm not sure this is so much of an issue. In fact, as I've argued elsewhere, in psychology and neuroscience I think there's more of a problem with lack of replication. But there definitely is much neglect of prior research. I lose count of the number of papers I review where the introduction presents a biased view of the literature that supports the authors' conclusions. For instance, if you are interested in the relation between auditory deficit and children's language disorders, it is possible to write an introduction presenting this association as an established fact, or to write one arguing that it has been comprehensively debunked. I have seen both.

Is this just lazy, biased or ignorant authors? In part, I suspect it is. But I think there is a deeper problem which has to do with the insatiable demand for novelty shown by many journals, especially the high-impact ones. These journals typically have a lot of pressure on page space and often allow only 500 words or less for an introduction. Unless authors can refer to a systematic review of the topic they are working on, they are obliged to give the briefest account of prior literature. It seems we no longer value the idea that research should build on what has gone before: rather, everyone wants studies that are so exciting that they stand alone. Indeed, if a study is described as 'incremental' research, that is typically the death knell in a funding committee.

We need good syntheses of past research, yet these are not valued because they are not deemed novel. One point made by Iain Chalmers was that funders have in the past been reluctant to give grants for systematic reviews. Reviews also aren't rated highly in academia: for instance, I'm proud of a review on mismatch negativity that I published in Psychological Bulletin in 2007. It not only condensed and critiqued existing research, but also discovered patterns in data that had not previously been noted. However, for the REF, and for my publications list on a grant renewal, reviews don't count.

We need a rethink of our attitude to reviews. Medicine has led the way and specified rigorous criteria for systematic reviews, so that authors can't just cherrypick specific studies of interest. But it has also shown us that such reviews are an invaluable part of the research process. They help ensure that we do not waste resources by addressing questions that have already been answered, and they encourage us to think of research as a cumulative, developing process, rather than a series of disconnected, dramatic events.

Reference
Chalmers, Iain, Bracken, Michael B., Djulbegovic, Ben, Garattini, Silvio, Grant, Jonathan, Gülmezoglu, A. Metin, Howells, David W., Ioannidis, John P. A., & Oliver, Sandy (2014). How to increase value and reduce waste when research priorities are set Lancet : 10.1016/S0140-6736(13)62229-1

A New Year's letter to academic publishers

My relationships with journals are rather like a bad marriage: a mixture of dependency and hatred. Part of the problem is that journal editors and academics often have a rather different view of the process. Scientific journals could not survive without academics. We do the research, often spending several years of our lives to produce a piece of work that is then distilled into one short paper, which the fond author invariably regards as a fascinating contribution to the field. But when we try to place our work in a journal, we find that it's a buyer's market: most journals are overwhelmed with more submitted papers than they can cope with, and rejection rates are high. So there is a total mismatch: we set out naively dreaming of journals leaping at the opportunity to secure our best work, only to be met with coldness and rejection.  As in the best Barbara Cartland novels, for a lucky few, persistence is ultimately rewarded, and the stony-hearted editor is won over. But many potential authors fall by the wayside long before that point.

But times are changing. We are moving from a traditional "dead tree technology" model, where journals have to be expensively printed and distributed, to electronic-only media. These not only cost less to produce, but also avoid the length limits that traditionally have forced journals to be so highly selective. Alongside the technological changes, there has been rapid growth of the Open Access movement. The main motivations behind this movement were idealistic (making science available to all) and economic (escaping the stranglehold of expensive library subscriptions to closed-access journals). It's early days, but I am starting to sense that there's another consequence of the shift, which is that, as the field opens up, publishers are starting to change how they approach authors: less as supplicants, and more as customers.

In the past, the top journals had no incentive to be accommodating to authors. There were too many of us chasing scarce page space. But there are now some new boys on the open access block, and some of them have recognised that if they want to attract people to publish with them, they should listen to what authors want. And if they want academics to continue to referee papers for no reward, then they had better treat them well too.

This really is not too hard to do. I have two main gripes with journals, a big one and a little one. The big one concerns my time. The older I get, the less patient I am with organizations that behave as if I have all the time in the world to do the small bureaucratic chores that they wish to impose on me. For instance, many journals specify pointless formatting requirements for an initial submission. I really, really resent jumping through arbitrary hoops when the world is full of interesting things I could be doing. And cutting my toenails is considerably more interesting than reformatting references.

I recently encountered a journal whose website required you to enter details (name/address/email) of all authors in order to submit a pre-submission enquiry. Surely the whole point of a pre-submission enquiry is to save time, so you can get a quick decision on whether it's likely to be worth your while battling with the submission portal! There's also the horror of journals that require signatures from all authors at the point when you submit a manuscript: seems a harmless enough requirement, except that authors are often widely dispersed - on maternity leave or sailing the Atlantic - by the time the paper is submitted. The idea is to avoid fraud, of course, but like so many ethics regulations, the main effect of this requirement is to encourage honest, law-abiding people to take up forgery.

Oh, and then there are the 'invitations to review' (makes it sound so enticing, like being invited to a party), which require you to login in order to register your response – which for me invariably means selecting the option that I have forgotten my password, then looking at email to find how to update the password, meanwhile getting distracted by other email messages so I forget what I was doing, and eventually returning to the site to find it wants me now to change the password and enter mandatory contact details before it will accept my response. Well, no.  I'm usually a good citizen but I'm afraid I've just stopped responding to those.

You'd think the advent of electronic submission would make life easier, but in fact it can just open up a whole new world of tiny, fiddly things that you are required to do before your paper is submitted. Each individual thing is usually fairly trivial, but they do add up. So, for instance, if you'd like your authors to suggest referees, please allow them to paste in a list. DO NOT require them to cut and paste title, forename, initial, surname, email and institution into your horrible little boxes for each of six potential referees.  It all takes TIME. And we have more important things in life to be getting on with. Including doing the science that allows us to get the point of writing a paper.

Even worse, some of the requirements of journals are just historical artefacts with no more rationale than male nipples.  Here's a splendid post by Kate Jeffery which in fact was the impetus for this blogpost. I thought of Kate when, having carefully constructed a single manuscript document including figures, as instructed by the Instructions for Authors, I got to the submission portal to be strictly told that ON NO ACCOUNT must the figures be included in the main manuscript. Instead, they had to be separated, not only from the manuscript, but also from their captions (which had to be put as a list at the end of the manuscript). This makes sense ONCE THE PAPER IS ACCEPTED, when it needs to be typeset.  But not at the point of initial submission, when the paper's fate is undecided: it may well be rejected, and if not, it will certainly require revision. And meanwhile, you have referees tearing their hair out trying to link up the text, the Figures and their captions.

The smaller gripe is just about treating people with respect. I do have a preference for journal editors whose correspondence indicates that they are a human being and not an automaton. I've moaned about this before, in an old post describing a taxonomy of journal editors, but my feeling is that in the three years since I wrote that, things have got worse rather than better. Publishers and editors may think they make their referees happy by writing and telling them how useful their review of a paper has been – but the opposite effect is created if it is clear that this is a form letter that goes to all referees, however hopeless.It is really better to be ignored than to be sent an insincere, meaningless email - it just implies that the sender thinks you are stupid enough to be taken in by it.

So my message to publishers in 2014 is really very simple. The market is getting competitive and if you want to attract authors to send their best work to you, and referees to keep reviewing for you, you need to become more sensitive to our needs.  Two journals that appear to be trying hard are eLife and PeerJ, who avoid most of the bad practices I have outlined. I am hoping their example will cause others to up their game. We are mostly very simple souls who are not hard to please, but we hate having our time wasted, and we do like being treated like human beings.

How the government spins a crisis: the blame game

from: http://www.youtube.com/watch?v=PkHb9q-jpDU

Thousands of people in the UK had a truly miserable Christmas, with extreme weather leading to flooding and power cuts. They were shocked and cold, blundering around in the dark, sometimes for as long as three days. When David Cameron went to visit Yalding in Kent on 27th December, he got an earful from local residents, who complained they had been abandoned, and had no help from the council, who had "all decided to go on holiday."

Cameron's visit was widely seen as a PR disaster: he was criticised for using the floods as a way of getting cheap publicity, and his government's cuts in spending on flood defences were commented on.

On 30th December, we had Owen Paterson, the Energy Secretary stating that energy companies had "let customers down" in their response to the storm.

Yesterday we heard that Tim Yeo chairman of the energy select committee, planned to summon bosses of energy companies to explain their poor performance.

Now, I have no love for the energy companies, whose rapacious pricing strategies are causing real hardship to many. But I find myself wondering what exactly they were supposed to do over the Christmas period. Presumably, if a power line comes down, it requires specialised machinery and replacement parts to be sourced and brought to the site – which may well be affected by flooding – and engineers who not only have the expertise to diagnose and correct the problem, but who are also fit and brave enough to do this in horrendous weather conditions. I doubt that large numbers of such people are just sitting around waiting to be called upon, and indeed over the Christmas period, some of them may have gone away on holiday, and others may themselves be affected by the flooding.  There was much criticism concerning the lack of information given to those affected by flooding and power cuts. But it's just not realistic to expect an organization to magic up large numbers of call centre staff out of nowhere in the middle of a crisis-ridden Christmas break. It's also worth noting that much of the valiant work of helping people deal with the flooding crisis was the responsibility of the fire service, currently under pressure from cuts to funding.

I simply don't know whether the energy companies could have done better; maybe they could have done more with live updates of information through websites, Twitter or local radio. Maybe they could have issued earlier warnings, or cancelled leave for key staff. But it concerns me that we have the Environment Secretary making a very public judgement on this matter, directing blame at energy companies, just a few days after the Prime Minister has been criticised, and long before there has been a chance to evaluate what happened, and which agencies were responsible for what, in a calm and thorough manner.

Forgive me if I seem cynical, but a rapid and punitive response seems to have become a standard reaction of government to situations where they are attracting adverse publicity. Find a scapegoat and come down on them heavily, whether it be Brodie Smith, Sharon Shoesmith or David Kelly. This deflects criticism from the government and makes them look strong. All the better if the criticism can be laid instead at the door of a person or organization who is already unpopular.

By all means, let us consider the response to the crisis to see what could have been done better. But the issues are far too important to be used as propaganda to enhance a government's popularity. Let us not be distracted from a much more important priority: calling the government to account for its policy of cutting back on measures of flood prevention.

The impact of blogging on reputation

I was alerted this morning on Twitter to this blogpost by Brian LePort on the first of 5 reasons why students shouldn't blog. Its central thesis is that "it is almost impossible to avoid writing something that will offend someone". Consequently, bloggers run the risk of doing themselves reputational harm at best, or failing to get a job or even getting fired at worst.

LePort illustrates his thesis by the extraordinary case of Christopher Rollston, who tells how he was forced to resign from a post at Emmanuel Christian Seminary because he wrote a piece for the Huffington Post on the marginalization of women in the Bible. Rollston, who describes himself as a Christian, concluded: "Gender equality may not have been the norm two or three millennia ago, but it is essential. So, the next time someone refers to 'biblical values,' it's worth mentioning to them that the Bible often marginalized women and that's not something anyone should value." Apparently, a major funder of the seminary disapproved of such incendiary sentiments and Rollston's career there was toast.

I have to say, I find LePort's reaction to this story disappointing. Yes, people who blog should think carefully about what they say and the impact it may have. Yes, it's impossible to avoid offending someone somewhere, unless what you write is so boring and anodyne that nobody would want to read it. But I despair at the idea of a future generation so cowed with fear that nobody ever says anything original or controversial.

I'm not arguing that students and junior academics should sacrifice themselves on the altar of freedom of speech, but rather that they should have confidence in the positive as well as the negative power of the internet. If what they say is worth saying, they will get support. LePort focuses on the negative consequences of Rollston's blogging, but, as this post by Robert Cargill pointed out, he attracted huge support online and ended up in a better job, whereas Emmanuel Christian Seminary suffered massive reputational damage.

LePort makes the important point that blogs are very different to more formal academic writing and often represent a point of view at a particular point in time, which may subsequently change. To my mind, this is one of the huge benefits of blogging – if you are lucky, your blog will attract comments that expose you to a wide range of reactions and help clarify and develop your thinking. This can be both fun and useful. LePort worries, though, that this may mean your incomplete and half-baked thoughts on an issue are used against you by those in positions of authority.

As a senior academic, I hope I can offer some reassurance. In general, I see blogging as an indication that the author is a bit out of the ordinary – someone who cares enough about things to write about them, and who is willing to try and move discussion forward. If in addition they change their views on the basis of feedback, that's fine. Obviously, it's possible to reveal yourself on a blog as uninformed, irrational or bigoted, and that is definitely not good. But most of the blogs I read aren't like that.

Well, I can hear you saying, that's all very well. You are someone who actually blogs and understands social media, but most academics aren't like that. My reply is that social media is an unstoppable force and even the most traditional institutions are starting to focus on developing strategies for harnessing its power.  So I'd say, yes, LePort is right in that we need to be aware that blogging is a public medium, and anything we say on a blog can be read by anyone. But it would be a shame if we allowed ourselves to become so worried about potential problems that we failed to see the advantages of blogging for fostering academic debate.That would be like staying at home with the door locked because you're scared of what may happen if you go outside.

A letter to Boris Johnson

Dear Boris,

Last week you  created a media storm with your speech at the Centre for Policy Studies in London. As a self-publicist, you make Katie Price look like a case of social phobia, and maybe you are pleased with the column inches you attracted. But perhaps you are also made nervous by the many commentators who have suggested that you have damaged your chances of greater political glory by letting the avuncular mask slip and revealing what lies behind it.

As a eulogy for Margaret Thatcher, your speech was most effective, though it came across as a transparent bid to take on her mantle.  But it also revealed a blind spot in your understanding of your fellow human beings.  This is evidence in the two themes that run through the speech: (1) people are motivated solely by competition with others; (2) those who win competitions are morally superior and more deserving than those who lose.

Let’s look at the evidence in the speech.  The first comment that gave me pause was this: “Of the 193 present members of the UN, we have conquered or at least invaded 171 – that is 90 per cent.” This is presented as if it is something to be proud of. As someone who’d like to peaceably get on with my neighbours, I find it disturbing that a man who has ambitions to lead our country sees domination of other nations as an admirable goal.

You then go on to argue that as Britain’s empire waned, we suffered from a “spiritual morosity that bordered on self-loathing”.  Not my memory of the 1960s-70s. It was all rather jolly what with flower power and the sexual revolution (all made possible through science and the advent of the pill).  According to you, part of Thatcher’s greatness was that she revived the nation and created “a buccaneering environment where there was no shame – quite the reverse – in getting rich.” So instead of invading other countries and stealing their resources, we could elbow our way ahead of others in our own country, and feel smug about it at the same time. I do remember those days, which was exactly when my own spiritual morosity set in. Encounters with various businesses – utilities, banks, large shops, car hire firms, airlines – which had previously been straightforward and uneventful became obstacle courses that you had now to negotiate with extreme caution, because they were all trying to rip you off. You needed to be on your guard, as the default assumption was that they’d try to stitch you up and lock you in to the wrong kind of deal, with unnecessary insurance to boot and nasty little charges added on at the last moment. Those of a more recent generation may find this hard to believe, but you used to be able to interact with any large-sized company on the assumption that they were honest and cared about their reputation.  The “buccaneering environment” that Thatcher introduced delivered us into the hands of the pirates. Which was a good thing if you were a pirate, but not much fun for everyone else.

The part of your speech that has attracted most comment is when you talked about IQ. Full marks for demonstrating an understanding of the normal distribution, but less than full marks for the logic of your argument:

“Whatever you may think of the value of IQ tests, it is surely relevant to a conversation about equality that as many as 16 per cent of our species have an IQ below 85, while about 2 per cent have an IQ above 130. The harder you shake the pack, the easier it will be for some cornflakes to get to the top.

And for one reason or another – boardroom greed or, as I am assured, the natural and god-given talent of boardroom inhabitants - the income gap between the top cornflakes and the bottom cornflakes is getting wider than ever. I stress: I don’t believe that economic equality is possible; indeed, some measure of inequality is essential for the spirit of envy and keeping up with the Joneses that is, like greed, a valuable spur to economic activity.”

It’s unclear how far you understand IQ – some people have suggested you think it’s a measure of innate ability. But in a sense it doesn’t matter whether you recognise it is modifiable or not; the real problem is you confuse someone’s intellectual abilities with their worth. You seem to be saying that bright people deserve to succeed (and indeed we should give them knighthoods for their wealth generation activity) – whereas at the same time you seem to accept that there are some individual differences in ability. So are you saying that the less able people deserve to be poor? The cornflake analogy you use is not entirely clear, but it seems to amount to saying that you want to stimulate further inequalities between people – and surely that means the poor getting poorer while the rich get richer?

What I find particularly chilling in your speech is the view of human beings as motivated primarily by envy, greed, and the need to get to the top, as well as the implication that if you don’t you are stupid. You seem unaware that there are large numbers of people who are motivated by things like interest in what they are doing (e.g. scientists), a desire to help others (e.g. doctors, nurses, teachers or carers), or a creative urge (e.g. writers and artists). When confronted with a banker whose annual income is fifty times as much as average earnings, for many of us the feeling is not so much envy as incomprehension. Why would anyone need all that money? Don’t they feel embarrassed at having so much more than everyone else? Weren’t they taught that you care for those less fortunate than yourself and that greed was a bad thing? Well, clearly not, because, as you tell us, Thatcher changed the culture so that there was “no shame – quite the reverse – in getting rich.”  

You don’t say what you think about the legions of people who exist happily on moderate incomes, but the implication is that they are all suckers, who’d be better off with a good dose of greed and envy. Indeed, you imply they are parasitic on the rich, who create all this wealth by paying massive amounts of tax. This is a good way to enrage a substantial part of the electorate.

At least from time to time you give a nod of recognition of the need for philanthrophy:

“But I also hope that there is no return to that spirit of Loadsamoney heartlessness – figuratively riffling banknotes under the noses of the homeless; and I hope that this time the Gordon Gekkos of London are conspicuous not just for their greed – valid motivator thought greed may be for economic progress – as for what they give and do for the rest of the population”

But, dear Boris, if you set up a system that rewards the greedy, you can hardly expect them to change their ways and start being philanthropic once they’ve made it. The main reason the very rich end up commandeering all the resources is because they are never satisfied. Like you, they are motivated by a sense of intense competition with everyone else. They won’t think about whether their income is sufficient to live a comfortable life; all they care about is having more money than everyone else.  And people like that are not going to make philanthropic donations; on the contrary they will avoid paying tax using any means that is legally available to them.  And you, by clapping them on the back for their avarice, are just encouraging them.

I agree with you that capitalism and competition are unlikely to disappear; they are an inherent part of our economic system. I also agree that we cannot prevent inequality; the key question for politicians is how best to manage it. According to you we should give greed and envy a free rein because they have such good consequences.  I disagree. I do not feel proud to be British in a country that treats its most vulnerable citizens with contempt, and values the qualities of competition, envy and greed over those of co-operation, compassion and moderation.

The Matthew effect and REF2014

For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath. Matthew 25:29

So you’ve slaved over your departmental submission for REF2014, and shortly will be handing it in. A nervous few months await before the results are announced. You’ve sweated blood over deciding whether staff publications or impact statements will be graded as 1*, 2*, 3* or 4*, but it’s not possible to predict how the committee will judge them, nor, more importantly, how these ratings will translate into funding. In the last round of evaluation, in 2008, a weighted formula was used, such that a submission earned 1 point for every 2* output, 3 points for every 3* output, and 7 points for every 4* output. Rumour has it that this year there may be no money for 2* outputs and even more for 4*. It will be more complicated than this, because funding allocations will also take into account ratings of ‘impact statements’, and the ‘environment’.

I’ve blogged previously about concerns I have with the inefficiency of the REF2014 as a method for allocating funds. Today I want to look at a different issue: the extent to which the REF increases disparities between universities over time. To examine this, I created a simulation which made a few simple assumptions. We start with a sample of 100 universities, each of which is submitting 50 staff in a Unit of Assessment. At the outset, we start with all universities equal in terms of the research quality of their staff: they are selected at random from a pool of possible staff whose research quality is normally distributed. Funding is then allocated according to the formula used in RAE2008. The key feature of the simulation is that over every assessment period there is turnover of staff (estimated at 10% in simulation shown here), and universities with higher funding levels are able to recruit replacement staff with higher scores on the research quality scale. These new staff are then the basis for computing funding allocations in the next cycle – and so on, through as many cycles as one wishes. This simulation shows that funding starts out fairly normally distributed, but as we progress through each cycle, it becomes increasingly skewed, with the top-performers moving steadily away from the rest (Figure A). In the graphs, funding is shown over time for universities grouped in deciles, i.e., bands of 10 universities after ranking by funding level.

Simulation: Mean income for universities in each of 10 deciles over 6 funding cycles

Depending on specific settings of parameters in the model, we may even see a bimodal distribution developing over time: a large pool of ‘have-nots’ vs an elite group of ‘haves’. Despite the over-simplifications of the model, I would argue that it captures an essential feature of the current funding framework: funding goes to those who are successful, allowing them to enter a positive feedback loop whereby they can recruit more high-calibre researchers and become even more successful – and hence gain even more funds in the next round. For those who are unsuccessful, it can be hard to break out of a downward spiral into research inactivity.

We could do things differently. Figure B shows how tweaking the funding model could avoid opening up such a wide gulf between the richest and poorest, and retain a solid core of middle-ranking universities.

Simulation using linear weighting of * levels. Each line is average for institutions in a given decile

Figure C, on the other hand, shows how a formula that predominantly rewards 4* outputs (weighting of 1 for 3* and 7 for 4*, which is rumoured to be a possible formula used in REF2014). This would dramatically increase the gulf between the elite and other institutions.

Simulation where 4* outputs get favoured. Each line is average for institutions in a given decile

I’m sure people will have very different views about whether or not the consequences illustrated here are desirable. One argument is that it is best to concentrate our research strength in a few elite institutions. That way the UK will be able to compete with the rest of the world in University league tables. Furthermore, by pooling the brightest brains in places where they have the best resources to do research, we have a chance of making serious breakthroughs. We could even use biblical precedent to justify such an approach: the Matthew effect refers to the biblical parable of the talents, in which servants are entrusted different sums of money by their master, and those who have most make the best use of it. There is no sympathy for those with few resources: they fail to make good use of what they do have and end up cast out into outer darkness, where there is weeping and gnashing of teeth. This robust attitude characterises those who argue that only internationally outstanding research should receive serious funding.

However, given that finances are always limited, there will be a cost to the focus on an elite; the middle-ranking universities will get less funding, and be correspondingly less able to attract high-calibre researchers. And it could be argued that we don’t just need an elite: we need a reasonable number of institutions in which there is a strong research environment, where more senior researchers feel valued and their graduate students and postdocs are encouraged to aim high. Our best strategy for retaining international competitiveness might be by fostering those who are doing well but have potential to do even better. In any case, much research funding is awarded through competition for grants, and most of this goes to people in elite institutions, so these places will not be starved of income if we were to adopt a more balanced system of awarding central funds.

What worries me most is that I haven’t been able to find any discussion of this issue – namely, whether the goal of a funding formula should be to focus on elite institutions or distribute funds more widely. The nearest thing I’ve found so far is a paper analysing a parallel issue in grant awards (Fortin & Curry, 2013) – which comes to the conclusion that broader distribution of smaller grants is more effective than narrowly distributed large grants. Very soon, somebody somewhere is going to decide on the funding formula, and if rumours are to be believed, it will widen the gap between the haves and have-nots even further. I'm concerned that if we continue to concentrate funding only in those institutions with a high proportion of research superstars, we may be creating an imbalance in our system of funding that will be bad for UK research in the long run.

Reference  

Fortin JM, & Currie DJ (2013). Big Science vs. Little Science: How Scientific Impact Scales with Funding. PloS one, 8 (6) PMID: 23840323

On the need for responsible reporting of research to the media

This was one of the first tweets I saw when I woke up this morning :

In response, a parent of two girls with autism tweeted "gutted to read this. B's statement has been final for 1 yr but no therapy has been done. we're still waiting."

I was really angry. A parent who is waiting for therapy for a child has many reasons to be upset. But the study described on the BBC Website did NOT identify a 'critical window'. It was not about autism and not about intervention.

I was aware of the study because I'd been asked by the Science Media Centre to comment on an embargoed version a couple of days ago.

These requests for commentary on embargoed papers always occur very late in the day, which makes it difficult to give a thorough appraisal. But I felt I'd got the gist: the researchers had recruited 108 children aged between 1 and 6 years and done scans to look at the development of white matter in the brain. They also gave children a well-known test of cognitive development, the Mullen scales, which assesses language, visual and fine motor skills. It's not clear where the children came from, but their scores on the Mullen scales were pretty average, and as far as I can tell, none of them had any developmental disorders.

The researchers were particularly interested in lateralisation: the tendency to have more white matter on one side of the brain than the other. Left-sided lateralisation of white matter in some brain regions is well-established in adults but there's been debate as to whether this is something that develops early in life, or whether it is present from birth. In the introduction, the authors state that this lateralisation is strongly heritable, but although that's often claimed, the evidence doesn't support it (Bishop, 2013). A preponderance of white matter in the left hemisphere is of interest because in most people, the left side of the brain is strongly involved in language processing.

The authors estimated lateralisation in numerous regions of the left and right brain using a measure termed the myelin water fraction. Myelin is a fatty sheath that develops around the axons of cells in the brain, leading to improved efficiency of neural transmission. Myelination is a well-established phenomenon in brain development.

The main findings I took away from the paper were (a) myelin is asymmetrically distributed in the brains of young children, with many regions showing greater myelin density in the left than the right; (b) although the amount of myelin increases with age, the extent of lateralisation is stable from 1 to 6 years. This is an important finding.

The authors, however, put most focus on another aspect of the study: the relationship between myelin lateralisation and language level. Overall, there was no relationship with asymmetry of a temporal-occipital region that overlapped with the arcuate fasciculus, a fibre tract important for language that previously had given rather inconsistent results (see Bishop, 2013). However, looking at a total of eight brain regions and four cognitive measures, they found two regions where leftward asymmetry was related to language or visual measures, and one where rightward asymmetry was related to expressive and receptive language.

Their primary emphasis, however, was on another finding, that there were interactions between age and lateralisation, so that, for instance, left-sided lateralisation of myelin in a region encompassing caudate/thalamus and frontal cortex only became correlated with language level in older children. I found it hard to know how much confidence to place in this result: the authors stated that they corrected for multiple comparisons using false discovery rate, but if, as seems the case, they looked at both main effects and interaction terms in 32 statistical analyses, then some of these findings could be chance.

Be that as it may, it is an odd result. Remember that this was a cross-sectional study and that on no index was there an age effect on lateralisation. So it does not show that changes in language ability - which are substantial over this age range - are driven by changes in lateralisation of myelin. So what do the authors say? Well, in the paper, they conclude "The data presented here are cross sectional, longitudinal analysis will allow us to confirm these findings; however, the changing interaction between ability and myelin may be mediated by progressive functional specialization in these connected cortical regions, which itself is partly mediated by environmental influences" (p. 16175). But this is pure speculation: they have not measured functional specialisation, and, as they appear to recognise, without longitudinal data, it is premature to interpret their results as indicating change with age.

If you've followed me so far, you may be wondering when I'm going to get on to the bit about intervention for autism and critical periods. Well, there's no data in this paper on that topic. So why did the BBC publish an account of the paper likely to cause dismay and alarm in parents of children with language and communication problems? The answer is because King's College London put out a press release about this study that contained at least as much speculation as fact. We are told that the study "reveals a particular window, from 2 years to the age of 4, during which environmental influence on language development may be greatest." It doesn't do anything of the kind. They say: "the findings help explain why, in a bilingual environment, very young typically developing children are better capable of becoming fluent in both languages; and why interventions for neurodevelopmental disorders where language is impaired, such as autism, may be much more successful if implemented at a very young age. " Poppycock.

A few months ago the same press office put out a similarly misleading press release about another study, quoting the principal researcher as stating: “Now we understand that this is how we learn new words, our concern is that children will have less vocabulary as much of their interaction is via screen, text and email rather than using their external prosthetic memory. This research reinforces the need for us to maintain the oral tradition of talking to our children.” As I noted elsewhere, the study was not about children, computers or word learning.

I can see that there is a problem for researchers doing studies of structural brain development. It can be hard to excite the general public about the results unless you talk about potential implications. It is frankly irresponsible, though, to go so far beyond your data that the headline is based on the speculation rather than the findings.

I am tired of researchers trying to make their studies relevant by dragging in potential applications to autism, schizophrenia, or dyslexia, when they haven't done any research on clinical groups. They need to remember that there are real people out there whose everyday life is affected by these conditions, and that neither they nor the media can easily discriminate what a study actually found from speculations about its implications. It is the duty of researchers and press officers to be crystal clear about that distinction to avoid causing confusion and distress.

POSTSCRIPT
11/10/13: Dr O'Muircheartaigh has commented below to absolve the KCL Press Office of any responsibility for the content of their press release. I apologise for assuming that they were involved in decisions about how to publicise this research and have reworded parts of this blogpost to remove that implication.


References 

Bishop, D. V. M. (2013). Cerebral asymmetry and language development: Cause, correlate, or consequence? Science, 340 (6138) DOI: 10.1126/science.1230531

O'Muircheartaigh, J., Dean, D. C., Dirks, H., Waskiewicz, N., Lehman, K., Jerskey, B. A., & Deoni, S. C. L. (2013). Interactions between white matter asymmetry and language during neurodevelopment. Journal of Neuroscience, 33(41), 16170-16177. doi: 10.1523/jneurosci.1463-13.2013

 

High time to revise the PhD thesis format

Before the electronic age: Henry Wellcome's dissertation from 1874

I don't know how it works in other countries, but in the UK, if you agree to examine a PhD thesis, odds are you will receive a bound document of some 250-400 pages to evaluate. You are not supposed to write on it. You may be explicitly forbidden to obtain an electronic version of the document.

There are ways of dealing with this: the most useful one, taught to me by Uta Frith when we co-examined a thesis some years ago, was to make ample use of post-it notes. However, this is still pretty tedious. What I want is a loose-leaf document that I can write on. I want, when travelling on a train to be able to take a chapter or two with me.

Please, can somebody fix this?

Good and bad news on the phonics screen

Teaching children to read is a remarkably fraught topic. Last year the UK Government introduced a screening check to assess children’s ability to use phonics – i.e., to decode letters into sounds. Judging from the reaction in some quarters they might as well have announced they were going to teach 6-year-olds calculus. The test, we were told, would confuse and upset children and not tell teachers anything they did not already know. Some people implied that there was an agenda to teach children to read solely using meaningless materials. This, of course, is not the case. Nonwords are used in assessment precisely because you need to find out if the child has the skills to attack an unfamiliar word by working out the sounds. Phonics has been ignored or rejected for many years by those who assumed that if you taught phonics the child would be doomed to an educational approach that involved boring drills in meaningless materials. This is not the case: for instance, Kevin Wheldall argues that children need to combine teaching of phonics with training in vocabulary and comprehension, and storybook reading with real texts should be a key component of reading instruction.

There is evidence for the effectiveness of phonics training from controlled trials,  and I therefore regard it as a positive move that the government has endorsed the  use of phonics in schools. However, they continue to meet resistance from many teachers, for a whole range of reasons. Some just don’t like phonics. Some don’t like testing children, especially when the outcome is a pass/fail classification. Many fear that the government will use results of a screening test to create league tables of schools, or to identify bad teachers. Others question the whole point of screening: This recent piece from the BBC website quotes Christine Blower, the head of the National Union of Teachers, as saying: "Children develop at different levels, the slow reader at five can easily be the good reader by the age of 11.” To anyone familiar with the literature on predictors of children’s reading, this shows startling levels of complacency and ignorance. We have known for years that you can predict with good accuracy which children are likely to be poor readers at 11 years from their reading ability at 6 (Butler et al, 1985).

When the results from last year's phonics screen came out I blogged about them, because they looked disturbingly dodgy, with a spike in the frequency distribution at the pass mark of 32. On Twitter, @SusanGodsland has pointed me to a report on the 2012 data where this spike was discussed. This noted that the spike in the distribution was not seen in a pilot study where the pass mark had not been known in advance. The spike was played down in this report, and attributed to “teachers accounting for potential misclassification in the check results, and using their teacher judgment to determine if children are indeed working at the expected standard.” It was further argued that the impact of the spike was small, and would lead to only around 4% misclassification.

However, a more detailed research report on the results was rather less mealy-mouthed about the spike and noted “the national distribution of scores suggests that pupils on the borderline may have been marked up to meet the expected standard.” The authors of that report did the best they could with the data and carried out two analyses to try to correct for the spike. In the first, they deleted points in the distribution where the linear pattern of increase in scores was disrupted, and instead interpolated the line. They concluded that this gave 54% rather than 58% of children passing the screen. The second approach, which they described as more statistically robust, was to take all the factors that they had measured that predicted scores on the phonics screen, ignoring cases with scores close to the spike, and then use these to predict the percentage passing the screen in the whole population. When this method was used, only 46% of children were estimated to have passed the screen when the spike was corrected for.

Well, this year’s results have just been published. The good news is that there is an impressive increase in percentage of children passing from 2012 to 2013, up from 58% to 69%. This suggests that the emphasis on phonics is encouraging teachers to teach children about how letters and sounds go together.

But any positive reaction to this news is tinged with a sense of disappointment that once again we have a most peculiar distribution with a spike at the pass mark. 

 

Proportions of children with different scores on phonics screen in 2012 and 2013. Dotted lines show interpolated values.

I applied the same correction as had been used for the 2012 data, i.e. interpolating the curve over the dodgy area. This suggested that the proportion of cases passing the screen was overestimated by about 6% for both 2012 and 2013. (The precise figure will depend on the exact way the interpolation is done). 

Of course I recognise that any pass mark is arbitrary, and children’s performance may fluctuate and not always represent their true ability. The children who scored just below the pass mark may indeed not warrant extra help with reading, and one can see how a teacher may be tempted to nudge a score upward if that is their judgement. Nevertheless, teachers who do this are making it difficult to rely on the screen data and to detect whether there are any improvements year on year. And it undermines their professional status if they cannot be trusted to administer a simple reading test objectively.

It has been announced that the pass mark for the phonics screen won’t be disclosed in advance in 2014, which should reduce the tendency to nudge scores up. However, if the pass mark differs from previous years, then the tests won’t be comparable, so it seems likely that teachers will be able to guess it will remain at 32. Perhaps one solution would be to ask the teacher to make a rating of whether or not the test result agrees with their judgement of the child’s ability. If they have an opportunity to give their professional opinion, they may be less tempted to tweak test results. I await with interest the results from 2014!

Reference

Butler, Susan R., Marsh, Herbert W., Sheppard, Marlene J., & Sheppard, John L (1985). Seven-year longitudinal study of the early prediction of reading achievement Journal of Educational Psychology, 77, 349-361 DOI: 10.1037//0022-0663.77.3.349