Good and bad news on the phonics screen

Teaching children to read is a remarkably fraught topic. Last year the UK Government introduced a screening check to assess children’s ability to use phonics – i.e., to decode letters into sounds. Judging from the reaction in some quarters they might as well have announced they were going to teach 6-year-olds calculus. The test, we were told, would confuse and upset children and not tell teachers anything they did not already know. Some people implied that there was an agenda to teach children to read solely using meaningless materials. This, of course, is not the case. Nonwords are used in assessment precisely because you need to find out if the child has the skills to attack an unfamiliar word by working out the sounds. Phonics has been ignored or rejected for many years by those who assumed that if you taught phonics the child would be doomed to an educational approach that involved boring drills in meaningless materials. This is not the case: for instance, Kevin Wheldall argues that children need to combine teaching of phonics with training in vocabulary and comprehension, and storybook reading with real texts should be a key component of reading instruction.

There is evidence for the effectiveness of phonics training from controlled trials,  and I therefore regard it as a positive move that the government has endorsed the  use of phonics in schools. However, they continue to meet resistance from many teachers, for a whole range of reasons. Some just don’t like phonics. Some don’t like testing children, especially when the outcome is a pass/fail classification. Many fear that the government will use results of a screening test to create league tables of schools, or to identify bad teachers. Others question the whole point of screening: This recent piece from the BBC website quotes Christine Blower, the head of the National Union of Teachers, as saying: "Children develop at different levels, the slow reader at five can easily be the good reader by the age of 11.” To anyone familiar with the literature on predictors of children’s reading, this shows startling levels of complacency and ignorance. We have known for years that you can predict with good accuracy which children are likely to be poor readers at 11 years from their reading ability at 6 (Butler et al, 1985).

When the results from last year's phonics screen came out I blogged about them, because they looked disturbingly dodgy, with a spike in the frequency distribution at the pass mark of 32. On Twitter, @SusanGodsland has pointed me to a report on the 2012 data where this spike was discussed. This noted that the spike in the distribution was not seen in a pilot study where the pass mark had not been known in advance. The spike was played down in this report, and attributed to “teachers accounting for potential misclassification in the check results, and using their teacher judgment to determine if children are indeed working at the expected standard.” It was further argued that the impact of the spike was small, and would lead to only around 4% misclassification.

However, a more detailed research report on the results was rather less mealy-mouthed about the spike and noted “the national distribution of scores suggests that pupils on the borderline may have been marked up to meet the expected standard.” The authors of that report did the best they could with the data and carried out two analyses to try to correct for the spike. In the first, they deleted points in the distribution where the linear pattern of increase in scores was disrupted, and instead interpolated the line. They concluded that this gave 54% rather than 58% of children passing the screen. The second approach, which they described as more statistically robust, was to take all the factors that they had measured that predicted scores on the phonics screen, ignoring cases with scores close to the spike, and then use these to predict the percentage passing the screen in the whole population. When this method was used, only 46% of children were estimated to have passed the screen when the spike was corrected for.

Well, this year’s results have just been published. The good news is that there is an impressive increase in percentage of children passing from 2012 to 2013, up from 58% to 69%. This suggests that the emphasis on phonics is encouraging teachers to teach children about how letters and sounds go together.

But any positive reaction to this news is tinged with a sense of disappointment that once again we have a most peculiar distribution with a spike at the pass mark. 

 

Proportions of children with different scores on phonics screen in 2012 and 2013. Dotted lines show interpolated values.

I applied the same correction as had been used for the 2012 data, i.e. interpolating the curve over the dodgy area. This suggested that the proportion of cases passing the screen was overestimated by about 6% for both 2012 and 2013. (The precise figure will depend on the exact way the interpolation is done). 

Of course I recognise that any pass mark is arbitrary, and children’s performance may fluctuate and not always represent their true ability. The children who scored just below the pass mark may indeed not warrant extra help with reading, and one can see how a teacher may be tempted to nudge a score upward if that is their judgement. Nevertheless, teachers who do this are making it difficult to rely on the screen data and to detect whether there are any improvements year on year. And it undermines their professional status if they cannot be trusted to administer a simple reading test objectively.

It has been announced that the pass mark for the phonics screen won’t be disclosed in advance in 2014, which should reduce the tendency to nudge scores up. However, if the pass mark differs from previous years, then the tests won’t be comparable, so it seems likely that teachers will be able to guess it will remain at 32. Perhaps one solution would be to ask the teacher to make a rating of whether or not the test result agrees with their judgement of the child’s ability. If they have an opportunity to give their professional opinion, they may be less tempted to tweak test results. I await with interest the results from 2014!

Reference

Butler, Susan R., Marsh, Herbert W., Sheppard, Marlene J., & Sheppard, John L (1985). Seven-year longitudinal study of the early prediction of reading achievement Journal of Educational Psychology, 77, 349-361 DOI: 10.1037//0022-0663.77.3.349

Raising awareness of language learning impairments

A couple of years ago I did a Google search for ‘Specific language impairment’. I was  appalled by what I found. The top hit was a video by a chiropractor who explained he’d read a paper about neurological basis of language difficulties; he proceeded to mangle its contents, concluding that cranial osteopathy would help affected children.

I’ve previously described how I got together with colleagues in 2012 to try and remedy this situation, culminating in a campaign for Raising Awareness of Language Learning Impairments (RALLI). The practicalities have sometimes been challenging but I’m pleased to say that the collection of videos on our RALLI site has now attracted over 90,000 hits, providing an accessible and evidence-based source of information about developmental language impairments. As well as research-based films we have videos with practical information for parents, children and teachers.

So here, for those of you interested in this topic, is an index of what we have so far:

Background to RALLI

• What is RALLI?
• Why we've launched RALLI
• What is SLI?
• Meet Seb

Research topics

• Identification of Specific Language Impairment Video Slides References 
• How specific is SLI? Video Slides References 
• Speech, language and communication Video Slides 
• What is Pragmatic Language Impairment? Video Slides 
• How common is specific language impairment? Video Slides References 
• When should we be concerned about late-talkers? Video Slides References 
• SLI and Autism Video Slides References 
• Auditory processing disorder (APD) and language impairment Video Slides References 
• The causes of Specific Language Impairment Video Slides References 
• Specific language impairment and the brain Video Slides References 
• An extra X or Y chromosome: a rare cause of language impairment Video Slides References 
• SLI & reading: 1. Decoding (phonics) Video Slides References 
• SLI and reading: 2. Understanding written language Video Slides References 
•  Reading intervention for children with language impairments Video  References 
• Friendships in teenagers with SLI Video Slides 
• La Amistad y el TEL (Spanish) Video Slides 

Information for teachers

• Signs of SLI
• Literacy difficulties and SLI
• Looking behind behaviour
• Teacher tips: practical strategies
• Teacher tips: adapting your language
• What is the Better Communication Research Programme?

Support for parents and children

• Finding my strengths
• A Parent's Perspective of SLI: Meet Suzanne.
• You are not alone
• It's OK to ask for help
• Getting support
• Helping your child
• Sophie's Story: SLI in adulthood
• Talking about friendships: Megan and Eleshia
• Supporting families: the role of Afasic
• Speech and language therapy: Helping Michael  
• It's not your fault
• Things I wish I'd known: a parent's perspective 

International

• O que é o DEL? (Portuguese)
• 什麼是特定型語言障礙? (Mandarin)
• ¿Qué es el TEL? (Spanish)
• Qu'est-ce que le TSL? (French)
• Was sind spezifische Sprachentwicklungsstörungen? (German)
• Wat is een taalstoornis? (Dutch)
• 甚麽是特殊語言障礙（廣東話版）(Cantonese)
• Che cos'é il DSL? (Italian)
• Beth yw NIP? (Welsh)
• Τι είναι η Ειδική Γλωσσική Διαταραχή ( Greek)  

     Spanish translations/subtitled versions

• ¿Porqué hemos lanzado la campaña RALLI?
• Introducción a Spanish RALLI
• No es tu culpa para niños

Reference  
Bishop, D. V. M., Clark, B., Conti-Ramsden, G., Norbury, C., & Snowling, M. J. (2012). RALLI: An internet campaign for raising awareness of language learning impairments Child Language Teaching & Therapy, 28 (3), 259-262 DOI: 10.1177/0265659012459467

Evaluate, evaluate, evaluate

© www.CartoonStock.com

When I was starting out on a doctorate, I’d look at the senior people in my field and wonder if I’d ever be like them. It must be great, I thought, to reach the advanced age of 40. By then you’d have learned everything you needed to know to do great science, and you could just focus on doing it. I suspect today’s crop of grad students are a bit more savvy than I was, but all the same, I wonder if they realise just how wrong that picture is – for two reasons.

First, you never stop learning. The field moves on. Instead of getting easier, it gets harder. I remember when techniques such as functional brain imaging first came along. The most competent people in that area were either those who had developed the methods, or young people who learned them as grad students. If you were of the generation above, you had three choices: ignore the methods, spend time learning them, or hire junior people who knew what they were doing. As the methods evolve, they get ever more complex, and meanwhile, your own brain starts to shrink. So if you are anticipating making it to a tenured post and then settling down in your armchair, think again.

Second, the more senior you get, the more of your time is spent, not on doing your own research, but on evaluation. You learn that an email entitled ‘invitation’ should not make your spirits rise: it’s just a desperate attempt to put a positive spin on a request for you to do more work for no reward. You get regular ‘invitations’ to review papers and grants, write job references, appraise promotion bids, sit on interview panels and examine theses. If you are involved in teaching, you’ll also be engaged in numerous other forms of appraisal.

I was prompted to think about this when someone asked on an electronic forum what was a reasonable number of doctoral theses to examine each year. The general consensus was two: though it will obviously depend on what other commitments someone has. It also varies from country to country. There are some jolly places in Europe where a PhD viva is just an excuse for a boozy party with a lot of dressing up in funny gowns and hats. In UK psychology, the whole thing is no fun at all: you have to read a document of 50,000-70,000 words reporting a body of work based on a series of experimental studies. You then write a report on it and see the candidate for a face-to-face viva, which is typically 2 to 3 hours long. Although failure is uncommon, it is not assumed that the candidate will pass (unlike in the viva-as-party countries), and weeping or catatonic candidates are not unheard of. Taking into account travel, etc., if you are going to do a proper job, you are probably talking about three days’ work. For this you get paid around the minimum wage – the fee for examining is typically somewhere between £120 and £200.

So why do we do it? The major reason is because the entire academic enterprise depends on reciprocity: we want people to examine our students and review our papers and grants. In addition, it’s important to maintain standards, and to ensure that degrees, promotions, publications and grants go to those who merit them. But the demands keep growing. In the 37 weeks of this year I’ve been asked to review 76 papers and six grants. I agreed to review 16 papers and three of the grants. This, of course, is nothing compared with being a journal editor or serving on a grants board, something that most of us will do at some point.

Clearly, if I agreed to do everything I was asked, I’d have no time for anything else. Of course, one learns to say no. But awareness of these pressures has made me look with rather a critical eye at how we use evaluation. There is, for instance, research suggesting that job interviews aren’t very useful at identifying good candidates:  we tend to be seduced by immediate impressions, which may not be a good indicator of a person’s suitability. Like most people, I’d be reluctant to take on an employee I hadn’t interviewed, but if Daniel Kahneman is to be believed, this is just because I am a victim of the Illusion of Validity.

I’m a supporter of the peer review system used by journals, and here I feel  I’m on more solid ground, because I can point to instances where my papers have been improved by input from reviewers. Nevertheless, where reviewing is used simply to reject/accept papers or grant proposals,  and where fine-grained decisions have to be made between many high-quality submissions, agreement between experts may be little better than chance (e.g. Fogelholm et al, 2012). Nevertheless, we stick with it, because it’s hard to know what to put in its place.

I’ve written a fair bit about that expensive and time-consuming evaluation process that UK academics engage in, the REF. It requires experts to make judgements of whether, for instance, papers are of 3* or 4* quality, a distinction based on whether the research is “world leading” or “internationally excellent…. but falls short of the highest standards of excellence.” The reliability of such judgements has not, to my knowledge, been evaluated, yet large amounts of funding depend on them. Those on REF committees are in the same situation as Pavlov’s poor dogs, having to make distinctions that are on the one hand impossible (discriminating circles and ellipses that become increasingly similar) and on the other hand very important (get it wrong and you get a shock).

There is one good thing about doing so much evaluation. You have the opportunity to see what others are doing – you may be the first person to read an important new paper, or examine a ground-breaking thesis. You may be forced to engage with different ways of thinking, and confronted with new topics and ideas. You may be able to provide useful input to authors. And since you yourself will be evaluated, it can be useful to see life from the other side of the table, as the person doing the evaluating. But all too often, even these advantages fail to compensate for the fact that as a senior academic you will spend more and more time on evaluation of others and less and less doing your own research.

Reference
Fogelholm, Mikael, Leppinen, Saara, Auvinen, Anssi, Raitanen, Jani, Nuutinen, Anu, & Väänänen, Kalervo (2012). Panel discussion does not improve reliability of peer review for medical research grant proposals Journal of Clinical Epidemiology, 65 (1), 47-52 DOI: 10.1016/j.jclinepi.2011.05.001

Changing children's brains

Portraits of Serafino & Francesco Falzacappa; Pier Leone Ghezzi [1674 - 1755]
Source: J. Paul Getty Museum

Our children are being exposed to an experience that alters their brains in ways we do not fully understand. There is now strong evidence that patterns of brain connectivity are different in individuals who have been exposed compared to those who have not¹. Influential figures have expressed concern that people’s memories will be restricted by this experience, which removes the need for them to memorise material, and allows them to look things up instead². And indeed, there is clear evidence of changes in cognitive processing, as predicted³. Furthermore, instead of learning in a social context, our children are increasingly being encouraged to engage in solitary activities that deprive them from the benefits of interacting with other people. And, rather than embracing traditional influences, they are exposed to alien ideas from other cultures^4,5 

Does this sound familiar? Are you thinking computer games, ipads, smart phones? If so, then consider: We are changing children’s brains, altering their memories, and influencing their ideas by exposing them to books.

References

1. Dehaene, Stanislas, Pegado, Felipe, Braga, Lucia W., Ventura, Paulo, Filho, Gilberto Nunes, Jobert, Antoinette, Dehaene-Lambertz, Ghislaine, Kolinsky, Régine, Morais, José, & Cohen, Laurent (2010). How learning to read changes the cortical networks for vision and language Science, 330, 1359-1364 DOI: 10.1126/science.1194140 
2. http://outofthejungle.blogspot.co.uk/2007/11/socrates-objections-to-writing.html

3. Ong, W. J. (1982). Orality and Literacy. London and New York: Routledge.

4. http://yalebooks.wordpress.com/2012/06/11/a-history-of-women-readers-belinda-jack-discusses-the-relationship-between-gender-and-literacy/

5. Nafisi, A. (2003). Reading Lolita in Tehran. New York: Random House.

The arcuate fasciculus and word learning: a critique

The arcuate fasciculus is a white matter tract linking areas in the temporal lobe involved in interpreting speech with areas in the frontal lobe that control motor movements. Its role in language was established years ago when it was proposed that conduction aphasia, characterised by poor repetition despite good understanding and fluent spontaneous speech, was a disconnection syndrome resulting from lesions of the arcuate fasciculus.

Compared with apes and monkeys, humans have much stronger structural connections between temporal and frontal regions of the brain, suggesting that evolution of the arcuate fasciculus played a key role in language evolution.

Study of white matter tracts in the brain has advanced rapidly since the advent of diffusion tensor imaging (DTI). DTI makes it possible to measure parameters such as fractional anisotropy and radial diffusivity, indirect measures of myelination and/or axonal density within white matter.

Use of DTI has revealed an intriguing aspect of the arcuate fasciculus: it shows wide individual variation. In most people, the left arcuate fasciculus is larger than the right, but in some a more bilateral pattern is seen, and in others, a right arcuate fasciculus may not be visible on DTI. This immediately raises the question of whether this individual variation corresponds to functional differences in language ability. Two studies considered whether the degree of lateralisation of the arcuate fasciculus related to language level, but they obtained conflicting results. Lebel and Beaulieu (2009) found that laterality of the arcuate fasciculus, measured on diffusion tensor imaging, was modestly correlated (r = 0.32) with receptive vocabulary in 68 children, with the highest scores for those with strong left lateralization. However, a study of adults found no relation between left lateralization of the arcuate fasciculus and vocabulary; instead, higher verbal memory was found to be associated with weak lateralisation.

A couple of weeks ago, López-Barroso et al published a paper in the Proceedings of the National Academy of Sciences claiming that structural and functional  measures of the left arcuate fasciculus predicted word learning ability. The authors started with 27 young adults who had brain scans that yielded measures of structural and functional connectivity between temporal and frontal language areas of the brain. Twenty of these individuals also did a learning task while in the scanner. They heard a rapid sequence of novel words, each consisting of three syllables, and were asked to concentrate on them, as they would be asked to recognise them later. After this learning phase, they were presented with the same nonwords mixed in with other nonwords made from the same syllables in a different order, and were asked to make a left or right keypress to indicate if each item was familiar or not. Their responses were transformed into a measure called d-prime, which indicates how well the person discriminates between familiar and unfamiliar items.

Figure 1A from López-Barroso et al, showing the learning task 

From previous research, one might have expected to see an association between nonword learning and lateralisation of the arcuate fasciculus. This was not found, but accuracy in the nonword learning task was significantly correlated with structural and functional measures of strength of connectivity in the left hemisphere. The authors’ conclusion is given in the title of the paper: “Word learning is mediated by the left arcuate fasciculus”.

Given what we know about the arcuate fasciculus, this is a plausible finding, but how robust is the evidence? I think there are at least three problems with this study, which lead me to be cautious about accepting its claims.

First, there is the perennial problem of multiple comparisons. The authors considered three different DTI measures (number of streamlines, fractional anisotropy and radial diffusivity) for left and right sides of four tracts (arcuate long, arcuate anterior, arcuate posterior, and inferior fronto-occipital fasciculus). They used, however, a Bonferroni correction appropriate for 8 correlations (p = .0062) rather than for 24 correlations (.002).  None of the reported correlations is significant if the appropriate correction is used.

Second, the authors emphasised that the correlation between word learning and radial diffusivity was significant only for the direct arcuate tract in the left hemisphere. This, however, confuses difference in significance levels with significance of differences: as Nieuwenhuis et al (2011)  remarked: "when making a comparison between two effects, researchers should report the statistical significance of their difference rather than the difference between their significance levels". Table 1 shows the correlations of radial diffusivity with nonword learning for different regions, with 95% confidence intervals added, and it is clear that there is overlap between these. In other words, these correlations do not differ significantly from one another. See here for further discussion of these issues.

Table 1: Correlations (r) between nonword learning and radial diffusivity in different pathways, with 95% confidence intervals

In this study, the problem is compounded by the fact that different subsets of individuals are included in the correlations for different brain regions. It is not unusual to have to exclude participants from DTI studies because of measurement difficulties, but this does mean that when comparing one brain region with another one is not comparing like with like. And since statistical significance depends on sample size, if this varies from brain region to brain region, this further complicates interpretation. This is evident from Figures 2 and 3 of the López-Barroso et al paper; in both cases the absolute value of the correlation is .42, yet for radial diffusivity of the right posterior segment, this is dismissed as nonsignificant (with N = 19), whereas for the  fMRI analysis it is heralded as significant (with N = 25).

To establish what results would look like if the same subset of participants was used in all analyses, I requested the raw data for radial diffusivity from the first author, who kindly provided it. There were just 13 participants with DTI data for all brain regions: if analysis was restricted to them, then just one of the correlations with word learning was significant by the authors' criterion, that with the right posterior arcuate fasciculus (r = .73, p = .005). This analysis does not prove that this pathway is important: rather, it emphasises that a similar pattern of associations is seen in all pathways, and the study  is underpowered to detect reliable associations, particularly if the interest is in selective associations with one pathway and not another.

Perhaps of greatest concern, though, is the measure of ‘word learning’. For a start, this was not word learning in the usual sense, as the participants were not required to associate speech sounds with meanings. Instead, they had to recognise familiar strings of meaningless sounds. There is a serious oddity about the results. Measures of d-prime usually range from zero (no ability to discriminate familiar from unfamiliar items, i.e. chance performance) to 2 or 3 (highly significant ability to discriminate familiar from unfamiliar items). But in this study, five of the twenty participants obtained negative values of d-prime. A negative value means performance is below chance: i.e., the person was more likely to treat the unfamiliar items as familiar, and vice versa. This is frankly weird, and makes one wonder whether some participants simply got confused about which key corresponded to which response. The authors give a different explanation: “Negative values indicate discrimination is achieved but individuals segmented incorrectly, classifying nonwords as words of the artificial language.” I find this unconvincing, as it would only make sense if the distractor items were made by taking sequences from the original input that crossed word boundaries: this does not seem to have been the case. But even if it were the explanation, does it make sense to treat those who discriminate the nonwords, but segment them wrongly, as doing worse on word learning than those who don’t discriminate the nonwords at all?

Does this matter? I re-ran the correlations excluding four participants with a negative d-prime value of less than -0.42 (which as far as I can work out corresponds to below chance performance). The correlations no longer reached conventional levels of statistical significance, and the largest value was now for a right-sided pathway. This is pretty meaningless, however, because the sample size, already small, becomes so tiny that one cannot do an adequately powered test of the association. The best one can say is that ‘further data are needed’.

I hope the authors will look further at this issue, as the role of the arcuate fasciculus in language  learning is fascinating and potentially important. One possibility would be to look at the associations between vocabulary level and analogous connectivity measures in the sample of 50 adults reported by Catani et al (2007), where the same DTI methods were used.

After I had drafted this critique, I Googled to see if anyone else had blogged about this study. I didn’t find blogs, but I did find extensive media coverage. I was astonished to see that, in discussing implications of this study, one of the authors, Marco Catani, a respected expert in tractography, appeared to be channeling Susan Greenfield. He was quoted as claiming that children’s vocabularies will be restricted by their use of iPads. The newspapers have picked up on these quotes, coming out with headlines such as: “Experts say too much time is spent learning via tablets and computers. Children's vocabulary could be stunted because they listen to teachers and parents less.”  For further sensationalist and misleading accounts, see here and here.

Just to be clear, this was a study looking at structural and functional brain connectivity in relation to a task that involved extracting syllabic patterns from auditory input. It did not feature children, vocabulary learning or iPads.

It really does a disservice to families of children with language learning problems to come out with scaremongering claims about modern technology on the basis of no hard evidence. And, for the record, auditory input is not the only way to learn new words: reading provides an  increasingly important route for vocabulary learning as children grow older.



Reference López-Barroso D, Catani M, Ripollés P, Dell'acqua F, Rodríguez-Fornells A, & de Diego-Balaguer R (2013). Word learning is mediated by the left arcuate fasciculus. Proceedings of the National Academy of Sciences of the United States of America, 110 (32), 13168-73 PMID: 23884655