Good and bad news on the phonics screen

Teaching children to read is a remarkably fraught topic. Last year the UK Government introduced a screening check to assess children’s ability to use phonics – i.e., to decode letters into sounds. Judging from the reaction in some quarters they might as well have announced they were going to teach 6-year-olds calculus. The test, we were told, would confuse and upset children and not tell teachers anything they did not already know. Some people implied that there was an agenda to teach children to read solely using meaningless materials. This, of course, is not the case. Nonwords are used in assessment precisely because you need to find out if the child has the skills to attack an unfamiliar word by working out the sounds. Phonics has been ignored or rejected for many years by those who assumed that if you taught phonics the child would be doomed to an educational approach that involved boring drills in meaningless materials. This is not the case: for instance, Kevin Wheldall argues that children need to combine teaching of phonics with training in vocabulary and comprehension, and storybook reading with real texts should be a key component of reading instruction.

There is evidence for the effectiveness of phonics training from controlled trials,  and I therefore regard it as a positive move that the government has endorsed the  use of phonics in schools. However, they continue to meet resistance from many teachers, for a whole range of reasons. Some just don’t like phonics. Some don’t like testing children, especially when the outcome is a pass/fail classification. Many fear that the government will use results of a screening test to create league tables of schools, or to identify bad teachers. Others question the whole point of screening: This recent piece from the BBC website quotes Christine Blower, the head of the National Union of Teachers, as saying: "Children develop at different levels, the slow reader at five can easily be the good reader by the age of 11.” To anyone familiar with the literature on predictors of children’s reading, this shows startling levels of complacency and ignorance. We have known for years that you can predict with good accuracy which children are likely to be poor readers at 11 years from their reading ability at 6 (Butler et al, 1985).

When the results from last year's phonics screen came out I blogged about them, because they looked disturbingly dodgy, with a spike in the frequency distribution at the pass mark of 32. On Twitter, @SusanGodsland has pointed me to a report on the 2012 data where this spike was discussed. This noted that the spike in the distribution was not seen in a pilot study where the pass mark had not been known in advance. The spike was played down in this report, and attributed to “teachers accounting for potential misclassification in the check results, and using their teacher judgment to determine if children are indeed working at the expected standard.” It was further argued that the impact of the spike was small, and would lead to only around 4% misclassification.

However, a more detailed research report on the results was rather less mealy-mouthed about the spike and noted “the national distribution of scores suggests that pupils on the borderline may have been marked up to meet the expected standard.” The authors of that report did the best they could with the data and carried out two analyses to try to correct for the spike. In the first, they deleted points in the distribution where the linear pattern of increase in scores was disrupted, and instead interpolated the line. They concluded that this gave 54% rather than 58% of children passing the screen. The second approach, which they described as more statistically robust, was to take all the factors that they had measured that predicted scores on the phonics screen, ignoring cases with scores close to the spike, and then use these to predict the percentage passing the screen in the whole population. When this method was used, only 46% of children were estimated to have passed the screen when the spike was corrected for.

Well, this year’s results have just been published. The good news is that there is an impressive increase in percentage of children passing from 2012 to 2013, up from 58% to 69%. This suggests that the emphasis on phonics is encouraging teachers to teach children about how letters and sounds go together.

But any positive reaction to this news is tinged with a sense of disappointment that once again we have a most peculiar distribution with a spike at the pass mark. 

 

Proportions of children with different scores on phonics screen in 2012 and 2013. Dotted lines show interpolated values.

I applied the same correction as had been used for the 2012 data, i.e. interpolating the curve over the dodgy area. This suggested that the proportion of cases passing the screen was overestimated by about 6% for both 2012 and 2013. (The precise figure will depend on the exact way the interpolation is done). 

Of course I recognise that any pass mark is arbitrary, and children’s performance may fluctuate and not always represent their true ability. The children who scored just below the pass mark may indeed not warrant extra help with reading, and one can see how a teacher may be tempted to nudge a score upward if that is their judgement. Nevertheless, teachers who do this are making it difficult to rely on the screen data and to detect whether there are any improvements year on year. And it undermines their professional status if they cannot be trusted to administer a simple reading test objectively.

It has been announced that the pass mark for the phonics screen won’t be disclosed in advance in 2014, which should reduce the tendency to nudge scores up. However, if the pass mark differs from previous years, then the tests won’t be comparable, so it seems likely that teachers will be able to guess it will remain at 32. Perhaps one solution would be to ask the teacher to make a rating of whether or not the test result agrees with their judgement of the child’s ability. If they have an opportunity to give their professional opinion, they may be less tempted to tweak test results. I await with interest the results from 2014!

Reference

Butler, Susan R., Marsh, Herbert W., Sheppard, Marlene J., & Sheppard, John L (1985). Seven-year longitudinal study of the early prediction of reading achievement Journal of Educational Psychology, 77, 349-361 DOI: 10.1037//0022-0663.77.3.349

Data from the phonics screen: a worryingly abnormal distribution

The new phonics screening test for children has been highly controversial.  I’ve been surprised at the amount of hostility engendered by the idea of testing children’s knowledge of how letters and sounds go together. There’s plenty of evidence that this is a foundational skill for reading, and poor ability to do phonics is a good predictor of later reading problems. So while I can see there are aspects of the implementation of the phonics screen that could be improved,  I don’t buy arguments that it will ‘confuse’ children, or prevent them reading for meaning.

I discovered today that some early data on the phonics screen had recently been published by the Department for Education, and my inner nerd was immediately stimulated to visit the website and download the tables.  What I found was both surprising and disturbing.

Most of the results are presented in terms of proportions of children ‘passing’ the screen, i.e. scoring 32 or more. There are tables showing how this proportion varies with gender, ethnic background, language background, and provision of free school meals. But I was more interested in raw scores: after all, a cutoff of 32 is pretty arbitrary. I wanted to see the range and distribution of scores.  I found just one table showing the relevant data, subdivided by gender, and I have plotted the results here.

Data from Table 4, Additional Tables 2, SFR21/2012
Department for Education (weblink above)

Those of you who are also statistics nerds will immediately see something very odd, but other readers may need a bit more explanation.  When you have a test like the phonics test, where each item is scored right or wrong, and the number of correct items is totalled up, you’d normally expect to get a continuous distribution of scores. That is to say, the numbers of children obtaining a given score should increase gradually up to some point corresponding to the most typical score (the mode), and then gradually decline again. If the test is pretty easy, you may get a ceiling effect, i.e. the mode may be at or close to the maximum score, so you will see a peak at the right hand side of the plot, with a long straggly tail of lower scores.  There may also be a ‘bump’ at the left hand edge of the distribution, corresponding to those children who can’t read at all – a so-called ‘floor’ effect.  That's evident in the scores for boys. But there's also something else. There’s a sudden upswing in the distribution, just at the ‘pass’ mark. Okay, you might think, that’s because the clever people at the DfE have devised the phonics test that way, so that 31 of the items are really easy, and most children can read them, but then they suddenly get much harder.  Well, that seems unlikely, and it would be a rather odd way to develop a test, but it’s not impossible. The really unbelievable bit is the distribution of scores just above and below the cutoff. What you can see is that for both boys and girls, fewer children score 31 than 30, in contrast to the general upward trend that was seen for lower scores. Then there’s a sudden leap , so that about five times as many children score 32 than 31. But then there’s another dip: fewer children score 33 than 32. Overall, there’s a kind of ‘scalloped’ pattern to the distribution of scores above 32, which is exactly the kind of distribution you’d expect if a score of 32 was giving a kind of ‘floor effect’.  But, of course, 32 is not the test floor.

This is so striking, and so abnormal, that I fear it provides clear-cut evidence that the data have been manipulated, so that children whose scores would put them just one or two points below the magic cutoff of 32 have been given the benefit of the doubt, and had their scores nudged up above cutoff.

This is most unlikely to indicate a problem inherent in the test itself. It looks like human bias that arises when people know there is a cutoff and, for whatever reason, are reluctant to have children score below that cutoff.  As one who is basically in favour of phonics testing, I’m sorry to put another cat among the educational pigeons, but on the basis of this evidence, I do query whether these data can be trusted.

Phonics screening: sense and sensibility

There’s been a lot written about the new phonics test that is being introduced in UK schools in June. Michael Rosen cogently put the arguments against it on his blog this morning. A major concern is that the test involves asking children to read a list of items, and takes no account of whether they understand them. Indeed, the list includes nonwords (i.e. pronounceable letter strings, such as "doop" or "barg") as well as meaningful words. So children will be “barking at print” - a very different skill from reading for meaning.

I can absolutely see where Rosen is coming from, but he’s missing a key point. You can’t read for meaning if you can’t decode the words. It’s possible to learn some words by rote, even if you don’t know how letters and sounds go together, but in order to have a strategy for decoding novel words, you need the phonics skills. Sure, English is an irritatingly irregular language, so phonics doesn’t always give you the right answer, but without phonics, you have no strategy for approaching an unfamiliar word.

Back in 1990, Hoover and Gough wrote an influential paper in 1990 called “The Simple View of Reading”. This is clearly explained in this series of slides by Morag Stuart from the Institute of Education. It boils down to saying that in order to be an effective reader you need two things: the ability to decode words, and the ability to understand the language in a text. Some children can say the words but don’t understand what they’ve read. These are the ones Michael Rosen is worried about. They won’t be detected by a nonword reading test. They are all-too-often missed by teachers who don’t realise they are having problems because when asked to read aloud, they do fine. There’s a fair bit of research on these so-called “poor comprehenders”, and how best to help them (some of which is reviewed here). But there are other children with the opposite pattern: good language understanding but difficulties in decoding: this corresponds to classic dyslexia. There are decades of research showing that one of the most effective ways of identifying these children is to assess their ability to read novel letter sequences that they haven’t encountered before - nonwords. Nonword reading ability has also been shown to predict which children are at risk for later reading failure.  It's useful precisely because it tests children's ability to attack unfamiliar material, rather than testing what they have already learned. It's a bit like a doctor giving someone a stress test on a treadmill. They may never encounter a treadmill in everyday life, but by observing how they cope with it, the doctor can tell whether they are at risk of cardiovascular problems.

Some children don’t need explicit teaching of phonics - they pick it up spontaneously through exposure to print. But others just don’t get it unless it is made explicit. I’m coming at this as someone who sees children who just don’t get past first base in learning to read, and who fall increasingly far behind if their difficulties aren’t identified. A nonword reading test around age 6 to 7 years will help identify those children who could benefit from extra support in the classroom.

So that’s the rationale, and it is well-grounded in a great deal of reading research. But is there a downside? Potentially, there are numerous risks. It would be catastrophic if teachers got the message from this exercise that reading instruction should involve training children to read lists of words, or worse still, nonwords. Unfortunately, testing in schools is increasingly conflated with evaluation of the school, and so teaching-to-the-test is routinely done. The language comprehension side of reading is hugely important, and shouldn't be neglected. Developing children’s oral language skills is an important component of making children literate. It is also important for children to be read to, and to learn that books are a source of pleasure.

Another concern is children being identified at an early age as failing. The cutoff that is used is crucial, and there are concerns that the bar may be set too high.  Children at real risk are those who bomb on nonword reading, not those who are just a bit below average.

The impact on children’s self-perception is also key. There is already evidence that some primary school children are unduly stressed by SATS. There’s nothing more likely to put a child off reading than being given a test that they don’t understand and being told they’ve failed it. When I was at school, we had the 11+ examination that divided children into those who went to grammar school and those who didn’t. I had friends whose parents promised them a bicycle if they passed - even though there was precious little practice that you could do for the 11+, which was designed to test skills that had not been explicitly taught. Schoolfriends who failed were left with a chip on their shoulder for years. I’d hope that this reading screen is introduced in a more sensitive manner, but the onus is on parents, teachers and the media to ensure this happens. This screening test should serve as a simple diagnostic that will allow teachers to identify those children whose weak letter-sound-knowledge means that they could benefit from extra support. It should not be used to evaluate schools, make children feel they are failures, worry their parents, or support a sterile phonics-only approach to reading.

References

Connor, M. J. (2003). Pupil stress and standard assessment tasks (SATs) An update. Emotional and Behavioural Difficulties, 8(2), 101-107. doi: 10.1080/13632750300507010

Hoover, W. A., & Gough, P. B. (1990). The simple view of reading. Reading and Writing, 2, 127-160.

Nation, K., & Angell, P. (2006). Learning to read and learning to comprehend. London Review of Education, 4(1), 77–87. doi: 10.1080/13603110600574538

Rack, J. P., Snowling, M. J., & Olson, R. K. (1992). The nonword reading deficit in developmental dyslexia. Reading Research Quarterly, 27, 29-53.

Snowling, M., & Hulme, C. (2012). Interventions for children's language and literacy difficulties International Journal of Language & Communication Disorders, 47 (1), 27-34 DOI: 10.1111/j.1460-6984.2011.00081.x