Who wants the TEF?

I'll say this for the White Paper on Higher Education "Success as a Knowledge Economy": it's not as bad as the Green Paper that preceded it. The Green Paper had me abandoning my Christmas shopping for furious tirades against the errors and illogicality that were scattered among the exhausted clichés and management speak (see here, here, here, here and here). So appalled was I at the shoddy standards evident in the Green Paper that I actually went through all the sources quoted in the first section of the White Paper to contact the authors to ask if they were happy with how their work had been reported. I'm pleased to say that out of 12 responses I got, ten were entirely satisfied, and one had just a minor quibble. But what about the twelfth, you ask. What indeed?

When justifying the need for a Teaching Excellence Framework (TEF) last November, Jo Johnson used some extremely dodgy statistical analysis of the National Student Survey to support his case that teaching in some quarters was 'lamentable'. I was pleased to see that this reference was expunged from the White Paper. But that left a motheaten hole in the fabric of the argument: if students aren't dissatisfied, then do we really need a TEF?  One could imagine the civil servants rushing around desperate to find a suitably negative statistic. And so they did, citing the 2015 HEPI-HEA Student Academic Experience Survey as showing that "Many students are dissatisfied with the provision they receive, with over 60% of students feeling that all or some elements of their course are worse than expected and a third of these attributing this to concerns with teaching quality." (p 8, para 5).  The same report is subsequently cited as showing that: ".. applicants are currently poorly-informed about the content and teaching structure of courses, as well as the job prospects they can expect. This can lead to regret: the recent Higher Education Academy (HEA)–Higher Education Policy Institute (HEPI) Student Academic Experience Survey found that over one third of undergraduates in England believe their course represents very poor or poor value for money." The trouble is, both of these quotes again use spin and dodgy statistics.

Let's take the 60% dissatisfaction statistic first. The executive summary of the report stated; "Most students are satisfied with their course, with 87% saying that they are very or fairly satisfied, and only 12% feeling that their course is worse than they expected. However, for those students who feel that their course is worse than expected, or worse in some ways and better than others, the number one reason is not the number of contact hours, the size of classes or any problems with feedback but the lack of effort they themselves put in." So how do we get to 60% dissatisfied? This number is arrived at from the finding that 12% said that their experience had been worse than expected, 49% said that it had been better in some ways and worse in others. So it is literally true that there is dissatisfaction with 'some or all elements', but the presentation of the data is clearly biased to accentuate the negative. One is reminded of Hugh in 'The Thick of It' saying "I did not knowingly not tell the truth".

But it gets worse: As pointed out on the Wonkhe blog, among 'key facts' in a briefing note accompanying the White Paper, the claim was reworded to say “over 60% of students said they feel their course is worse than expected”. The author of the blogpost referred to this as substantial misrepresentation of the survey. This is serious because it appears that in order to make a political point, the government is spreading falsehoods that could cause reputational damage to Universities.

Moving on to perceptions of 'value for money', there are two reasons for giving this low ratings  - you are paying a reasonable amount for something of poor quality, or you are paying an unreasonable amount for something of good quality. Alex Buckley, one of the authors of the report replied to my query to say that while the numeric data were presented accurately, crucial context was omitted. This made it crystal clear it was the money side of the equation that concerned students. He wrote:

"Figure 11 on page 17 of the 2015 HEPI-HEA survey report shows that students from England (paying £9k) and students from Scotland studying in Scotland (paying no fees) have very different perceptions of value for money. And Figure 12 shows that the perceptions of value for money of students from England plummeted at the time of the increase in fees. Half of 2nd year students from England in 2013 thought they were getting good or very good value for money. In 2014, when 2nd years were paying £9k, that figure was a third. (Other global perceptions of quality - satisfaction etc. - did not change). There is something troubling about the Government citing students' perceptions of value for money as a problem for the sector, when they appear to be substantially determined by Government policy, i.e. the level of fees. The survey suggests that an easy way to improve students' perceptions of the value for money of their degree would be to reduce the level of fees - presumably not the message that the Government is trying to get across."

So do students want the TEF? All the indicators say no. Chris Havergal wrote yesterday in the Times Higher about a report by David Greatbatch and Jane Holland in which students in focus groups gave decidedly lukewarm responses to questions about the usefulness of TEF. Insofar as anyone wants information about teaching quality, they want it at the level of courses rather than institutions, but, as an ONS interim review pointed out, the data is mostly too sparse to reliably differentiate among institutions at the subject level. Meanwhile, the NUS has recommended boycotting the National Student Survey, which forms a key part of the metrics to be used by TEF.

This is all rather rum, given that the government claims its reforms will put students at the heart of higher education. It seems that they have underestimated the intelligence of students, who can see through the weasel words and recognise that the main outcome of all the reforms will be further increases in fees.

It's widely anticipated that fees will rise because of the market competition that the White Paper lauds as a positive stimulus to the sector, and it was clear in the Green Paper that one goal of the reforms was to tie the TEF to a regulatory mechanism that would allow higher fees to be set by those with good TEF scores. Perhaps less widely appreciated is that the plan is for the new Office for Students to be funded largely by subscriptions paid by Higher Education Providers. They will have to find the money somewhere, and the obvious way to raise the cash will be by raising fees. So students will be in the heart of the reforms in the sense that having already endured dramatic rises in fees and loss of the maintenance grant, they will now also be picking up the bill for a new regulatory apparatus whose main function is to satisfy a need for information that they do not want.

2011 Orwellian Prize for Journalistic Misrepresentation

© www.CartoonStock.com

So the time has come round for the announcement of the 2011 Orwellian Prize. The prize is given for an article in an English-language national newspaper that achieves an unusually high level of inaccuracy. Only articles that describe a piece of published scientific research are eligible. Points are given for every statement in the article that does not match the original source, as follows:

•     Factual error in the headline: 3 points
•     Factual error in a subtitle: 2 points
•     Factual error in the body of the article: 1 point

Last year, I got two nominations, but, as described here, neither adequately met the criteria. This year, I’ve had just one nomination, from Neurobonkers, but it’s set a standard for inaccuracy that will be hard to beat. The article that I first selected in 2010 to illustrate the scoring system scored 16 points. This one achieves a startling 23 points. The source article by Kucewicz et al (2011) can be found here. Here is a screenshot of the account in the Daily Mail, with errors marked in red (3 points), orange (2 points) and blue (1 point).

There is a detailed analysis of errors in this blogpost by Neurobonkers, which I urge you to look at. Suffice it to say,  the academic paper is not about cannabis, smoking or schizophrenia. Rather it is about an artificial compound that is not present in cannabis, which was injected into rats, and which led to changes in their brain waves.

There were some complaints to the Press Complaints Commission, and presumably in response to this, the article was modified. The headline, which originally read Just ONE cannabis joint 'can bring on schizophrenia' as well as damaging memory was altered to Just ONE cannabis joint 'can cause psychiatric episodes similar to schizophrenia' as well as damaging memory. Perhaps even the Daily Mail found the notion of a schizophrenic rat implausible. But the rest of the article remains, as a scare story about cannabis. And here is what makes this article such a prime candidate for the Orwellian Award: this is not about a hyped press release by a university, or misunderstanding of complex science. It's not even about sensationalising a scientific finding to draw readers in. No, this is about using a scientific paper as a prop in the Daily Mail's anti-cannabis campaign. A ploy that the newspaper has previously used in another ideological battle, on climate change. When reporting research, no respect is given to the truth: scientists are simply used to bolster a preconceived opinion, and if they don't do that, their findings are distorted.

Twelve of this article's 23 points came from the headline. Journalists don’t write the headlines. They therefore dislike my scoring system because it penalises errors in headlines more than errors in the body of the text. My view is that it’s the headlines that count for most: far more people will read the headline than the text, and for many readers it's the only part of the article they will process. It’s important that it's accurate. Although few would defend frank lying, some editors seem to think it doesn’t matter if a headline is hyperbolic, provided it sells the paper or gets someone to read further. This very issue has been a topic of debate in the recent Leveson Inquiry into culture, practice and ethics in the UK media. I feel strongly that it's a cop-out to just wash one's hands of it and blame anonymous sub-editors for misleading headlines. I shall therefore continue to award points in proportion to the prominence of the material. But I appreciate it’s not then fair to make the award to the journalist. Indeed, given the Mail's agenda on cannabis, the journalist in this case may well have been under duress to write a scare story.   I will accordingly be making the award to Paul Dacre, Editor of the Daily Mail. I will be happy also to send the token of appreciation for the nomination to Neurobonkers if he/she is willing to email me to tell me where to send it.

I'm pleased not to have had more nominations this year: it suggests that, despite all the grumblings about science journalism, the field is in rude health. I've certainly read a lot of good science reportage in our national newspapers, and where articles have made me angry, it's often because of hype by a press office or scientist, rather than distortion by the press. There are, however, still a few topics, among them drugs policy, where the political stakes are high and scientific reporting is cynically exploited to support an otherwise weak argument.

Coming up with an award certificate and item turns out to be an excellent way of avoiding doing serious work.....

Reference
Kucewicz, M., Tricklebank, M., Bogacz, R., & Jones, M. (2011). Dysfunctional Prefrontal Cortical Network Activity and Interactions following Cannabinoid Receptor Activation Journal of Neuroscience, 31 (43), 15560-15568 DOI: 10.1523/JNEUROSCI.2970-11.2011