Saturday, 6 June 2020

Frogs or termites? Gunshot or cumulative science?

"Tell us again about Monet, Grandpa."

The tl;dr version of this post is that we're all so obsessed with doing new studies that we disregard prior literature. This is largely due to a scientific culture that gives disproportionate value to novel work. This, I argue, weakens our science.

This post has been brewing in my mind ever since I took part in a reading group about systematic reviews. We were discussing the new NIRO guidelines for systematic reviews outside the clinical trials context that are under development by Marta Topor and Jade Pickering. I'd been recommending systematic review as a useful research contribution that could be undertaken when other activities had stalled because of the pandemic. But the enthusiasm of those in the reading group seemed to wane as the session progressed. Yes, everyone agreed, the guidelines were excellent: clear and comprehensive. But it was evident that doing a proper review would not be a "quick win"; the amount of work would of course depend on the number of papers on a topic, but even for a circumscribed subject it was likely to be substantial and involve close reading of a lot of material. Was it a good use of time, people asked. I defended the importance of looking at past literature: it's concerning if we don't read scientific papers because we are all so busy writing them. To my mind, being a serious scholar means being very familiar with past work in a subject area. However, it's concerning that our reward system doesn't value that, making early-career researchers nervous about investing time in it.

The thing that prompted me to put my thoughts into words was a tweet I saw this morning by Mike Johansen (@mikejohansenmd). It seems at first to be on an unrelated topic, but I think it is another symptom of the same issue: a disregard for prior literature. Mike wrote:
Manuscripts should look like: Question: Methods: Results: Limitations: Figures/Tables: Who does these things? Things that don't matter: introduction, discussion. Who does these things?
I replied that he seemed to be recommending that we disregard the prior literature, which I think is a bad idea. I argued "One study is never enough to answer a question - important to consider how this study fits in - or if it doesn't , why."

Noah Haber (@noahhaber) jumped in at this point to say: 
I'm sympathetic (~45% convinced) to the argument that literature reviews in introductions do more harm than good. In practice, they are rarely more than cursory and uncritical, and make us beholden to ideas that have long outlived their usefulness. Space better used in methods.
But I don't think that's a good argument. I'm the first to agree that literature reviews are usually terrible: people only cite the work that confirms their position, and often do that inaccurately. You can see slides from a talk I gave on 'Why your literature review should be systematic' here. But I worry if the response to current unscholarly and biased approaches to the literature is to say that we can just disregard the literature. If you assume that the study you are doing is so important that you don't have time to read other people's studies, it is on the one hand illogical (if we all did that, who would read your studies), on the other hand disrespectful to fellow scientists, and on the most important third hand (yes, assume a mutant for now) bad for science.

Why is it bad for science? Because science seldom advances by a single study. Solid progress is made when work is cumulative. We have far more confidence in a theory that is supported by a series of experiments than by a single study, however large the effect. Indeed, we know that studies heralding a novel result often overestimate the size of effect – the "winner's curse". So to interpret your study, I want to know how far it is consistent with prior work, and if it isn't whether there might be a good reason for that.

Alas, this approach to science is discouraged by many funders and institutions: calls for research proposals are peppered with words such as "groundbreaking", "transformational", and "novel". There is a horror of doing work that is merely "cumulative". As a consequence, many researchers hop around like frogs in a lilypond, trying to land on a lilypad that is hiding buried treasure. It may sound dull, but I think we should model ourselves more on termites – we can only build an impressive edifice if we collaborate to each do our part and build on what has gone before.

Of course, the termite mound approach is a disaster if the work we try to build on is biased, poorly conducted and over-hyped. Unfortunately that is often the case, as noted by Noah. We come rather full circle here, because I think a motivation for Mike and Noah's tweets is recognition of the importance of reporting work in a way that will make it a solid foundation for a cumulative science of the future. I'm in full agreement with that. Where I disagree, though, is in how we integrate what we are doing now with what has gone before. We do need to see what we are doing as part of a cumulative, collaborative process in taking ideas forward, rather than a series of single-shot studies.

Friday, 29 May 2020

Boris Bingo: Strategies for (not) answering questions

On Wednesday 27th May, the Prime Minister, Boris Johnson, appeared before the House of Commons Liaison Committee, to answer questions about the coronavirus crisis. The Liaison Committee is made up of all the Chairs of Select Committees, which are where much of the serious business of government is done. The proceedings are available online, and contrast markedly with Hansard reports from the House of Commons, where the atmosphere is typically gladiatorial, with a lot of political point-scoring. In Select Committees, members from a mix of parties aim to work constructively together. It is customary for the Prime Minister to give evidence to the Liaison Committee three times a year, but this was Boris Johnson's first appearance.

The circumstances were extraordinary. The PM himself did not look well: perhaps not surprising when one considers that he was in intensive care with COVID-19 in April, only leaving hospital on 12th April, with a new baby born on 29th April. Since then, the UK achieved the dubious distinction of having one of the worst rates of COVID-19 infection in the world. Then, last weekend a scandal broke around Dominic Cummings, Chief Advisor to the PM, who gave a Press Conference on Monday to explain why he had been travelling around the country with his wife and son, when both he and his wife had suspected COVID-19.

I watched the Liaison Committee live on TV and was agog. There had been fears that the Chair, Bernard Jenkin, would give the PM an easy time. He did not; he chaired impeccably, ensuring committee members stuck to time and that the PM stuck to the point. Questions were polite but challenging, regardless of the political affiliation of the committee member. Did the PM rise to the challenge? This was not the sneering, combative PM that we saw in Brexit debates – he, no doubt, could see that would not go down well with this committee. Rather, the impression he gave was of a man who was winging it and relying on his famous charm in the hope that bluster and bonhomie would win the day. Alas, they did not.

Intrigued by Johnson's strategy – if it can be called that – for answering questions,  I have spent some time poring over the transcript of the proceedings, and realised in so doing that I have the material for a new Bingo game. When watching the PM answer questions, you have a point for each of the following strategies you identify. If you do the drinking game version, it may ease the angst otherwise generated by listening to the leader of our nation.


This term refers to a common strategy of politicians of appearing to answer a question, without actually doing so. It can give at least a superficial impression that the question has been answered, while deflecting to a related topic. In the following exchange, 'I have no reason to believe' is a big red flag for paltering. The Chair asked what advice the PM had sought from the Cabinet Secretary about Cummings' behaviour in relation to compliance with the code of integrity, and the PM replied:
I have no reason to believe that there is any dissent from what I said a few days ago.
Asked whether Scottish and Welsh first ministers had any influence on the approach to lockdown (Q14)
Stephen, we all work together, and I listen very carefully to what Mark says, to what Arlene and Michelle say, to what Nicola says. Of course we think about it together.
Response to Jeremy Hunt on why there were delays in implementing testing
As you know, Jeremy, we faced several difficulties with this virus. First, this was a totally new virus and it had some properties that everybody was quite slow to recognise across the world. For instance, it is possible to transmit coronavirus when you are pre-symptomatic—when you do not have symptoms—and I do not think people understood that to begin with.
When Hunt later asked the straightforward question "Why don't we get our test results back in 24 hours", (Q45) Johnson replied:
That is a very good question. Actually, we are reducing the time—the delay—on getting your test results back. I really pay tribute to Dido Harding and her team. The UK is now testing more people than any other country in Europe. She has got a staff now of 40,000 people, with 7,500 clinicians and 25,000 trackers in all, and they are rapidly trying to accelerate the turnaround time.
When asked by Caroline Nokes about the specific impact of phased school opening on women's ability to get back to work (Q73), Johnson answered a completely different question:
I think your question, Caroline, is directed at whether or not we have sufficient female representation at the top of Government helping us to inform these decisions, and I really think we have

This could take the form of bland agreement with the questioner, but without any clear commitment to action. Greg Clark asked (Q27) why we have a policy of 2 meters for social distancing when the WHO recommends 1 meter. The response was
...You are making a very important point, one that I have made myself several times—many times—in the course of the debates that we have had.
Pressed further on whether he had asked SAGE whether the 2-meter rule could be revised (Q32) he replied
I can not only make that commitment—I can tell you that I have already done just that, so I hope we will make progress.
Asked about firms who put their employees on furlough and then threatened them with redundancy (Q99), Johnson agreed this was a Very Bad Thing, but did not actually undertake to do anything about it.
...You are raising a very important point, Huw. This country is nothing without its workforce—its labour. We have to look after people properly, and I am well aware of some of the issues that are starting to arise. People should not be using furlough cynically to keep people on their books and then get rid of them. We want people back in jobs. We want this country back on its feet. That is the whole point of the furlough scheme.
Asked about how the Cabinet were consulted about the unprecedented Press Conference by Cummings (Q9), the PM was remarkably vague, replying:
...I thought that it would be a very good thing if people could understand what I had understood myself previously, I think on the previous day, about what took place—and there you go. We had a long go at it.
Asked to be specific about advice to parents who are in the same situation as Dominic Cummings re childcare (Q 21)
...The clear advice is to stay at home unless you absolutely have to go to work to do your job. If you have exceptional problems with childcare, that may cause you to vary your arrangements; that is clear.
The use of the word 'clear' in the PM responses is often a flag for vagueness.

A direct question by Greg Clark on whether contact tracing was compulsory or advisory (Q34) led to a confused answer:
We intend to make it absolutely clear to people that they must stay at home, but let me be clear—
When the questioner followed up to ask whether it was law or advice, he continued:
We will be asking people to stay at home. If they do not follow that advice, we will consider what sanctions may be necessary—financial sanctions, fines or whatever.
It is not always easy to distinguish vagueness from paltering. The PM has a tendency to agree that something is a Very Good Thing, to speak in glowing and over-general terms about initiatives, and about his desire to implement them, without any clear commitment to do more than 'looking at' them. Here he is responding to Robert Halfon on whether there will be additional resources for children whose education has been adversely affected by the shutdown (Q63)
The short answer is that I want to support any measures we can take to level up. You know what we want to do in this Government. There is no doubt that huge social injustice is taking place at the moment because some kids are going to have better access to tutoring and to schooling at home, and other kids are not going to get nearly as much, and that is not fair.
and again, when Halfon asked about apprenticeships (Q64)
All I will say to you, Rob, is that I totally agree that apprenticeships can play a huge part in getting people back on to the jobs market and into work, and we will look at anything to help people.
Halfon pressed on, asking for an apprenticeship guarantee, but the PM descended further into vagueness.
We will be doing absolutely everything we can to get people into jobs, and I will look at the idea of an apprenticeship guarantee. I suppose it is something that we would have to work with employers to deliver.
Other examples came from answers to Darren Jones, who asked about financial support to different sectors, and payments after the furlough scheme ended; e.g. the response to Q89:
We are going to do everything we can, Darren, to get everybody back into work.


This was the first strategy to appear, in response to a question by the Chair (Q2) about when the committee might expect to see him again. Johnson made it clear he wasn't going to commit to anything:
You are very kind to want to see me again more frequently, even before we have completed this session, but can I possibly get back to you on that? Obviously, there is a lot on at the moment.
Stephen Timms asked about people who were destitute because, despite having leave to remain, they had no recourse to public funds when they suddenly lost their jobs (Q68). The PM responded:
I am going to have to come back to you on that, Stephen.
It is perhaps unfair to count this one as deliberate strategy: Johnson seemed genuinely baffled as to how 100,000 children could be living in destitution in a civilised country.

When asked by Mel Stride about whether there would be significant increases in the overall tax burden, the PM replied:
I understand exactly where you are going with your question, Mel, but I think you are going to have to wait, if you can, until the Chancellor, Rishi Sunak, brings forward his various proposals.

Refusal to answer

Refusals were mostly polite. An illustration appeared early in the proceedings, when asked by the Chair about Dominic Cummings (Q6), the PM replied:
I do think that is a reasonable question to ask, but as I say, we have a huge amount of exegesis and discussion of what happened in the life of my adviser between 27 March and 14 April. Quite frankly, I am not certain, right now, that an inquiry into that matter is a very good use of official time. We are working flat out on coronavirus.
So the question is accepted as reasonable, but we are asked to understand that it is not high priority for a PM in these challenging times.

Asked by Meg Hillier whether the Cabinet Secretary should see evidence provided by Cummings, the PM responds that this is inappropriate – again arguing this would be a distraction from higher priorities:
I think, actually, I would not be doing my job if I were now to shuffle this problem into the hands of officials, who are—believe me, Meg—working flat out to deal with coronavirus, as the public would want.
At times, when paltering had been detected, and a follow-up question put him on the spot, Johnson simply dug his heels in, often claiming to have already answered the question. Asked whether the Cabinet Secretary has interviewed Cummings (Q8), Johnson replied:
I am not going to go into the discussions that have taken place, but I have no reason to depart from what I have already said.
And asked whether he'd seen evidence to prove that allegations about Cummings were false (Q17), the PM again replied:
I don’t want to go into much more than I have said—
Asked by Jeremy Hunt on when a 24-hour test turnaround time would be met (Q48), the rather remarkable reply was:
I am not going to give you a deadline right now, Jeremy, because I have been forbidden from announcing any more targets and deadlines.

Challenge questioner

This strategy where unwelcome questions were dismissed as either having false premises, and/or being politically motivated. Pete Wishart (SNP) asked if Cummings' behaviour would make people less likely to obey lockdown rules (Q10). Johnson did not engage with the question, denied any wrongdoing by Cummings and added:
Notwithstanding the various party political points that you may seek to make and your point about the message, I respectfully disagree.
Similar phrases are seen in response to Yvette Copper (Q24), who was accused of political point-scoring, and then blamed for confusing the British public (see also Churchillian gambit, below):
I think that this conversation, to my mind, has illuminated why it is so important for us to move on, and be very clear with the British public about how we want to deal with that, and how we want to make progress. And, frankly, when they hear nothing but politicians squabbling and bickering, it is no wonder that they feel confused and bewildered.
And in response to a similar point from Simon Hoare (Q25)
...what they [the people] want now is for us to focus on them and their needs, rather than on a political ding-dong about what one adviser may or may not have done

False claim

This doesn't always involve lying; it can be unclear whether or not the PM knows what is actually the case. But there was at least one instance in his evidence where what he said is widely reported as untrue. Intriguingly, this was not an answer to a direct question, but rather an additional detail when asked about testing in care homes by Jeremy Hunt (Q44)
Do not forget that, as Chris Hopson of NHS Providers has said, every discharge from the NHS into care homes was made by clinicians, and in no case was that done when people were suspected of being coronavirus victims. Actually, the number of discharges from the NHS into care homes went down by 40% from January to March, so it is just not true that there was some concerted effort to move people out of NHS beds into care homes. That is just not right.
A report by ITV news asserted that, contrary to this claim, places in hospitals were block booked for discharged NHS patients.

The Churchillian gambit

When allowed to divert from answering questions, the PM would attempt the kind of rhetoric that had been so successful in Brexit debates, referring to what 'the people' wanted, and to government attempts to 'defeat the virus'.
For instance, this extended response to Q9 re Dominic Cummings;
What we need to do really is move on and get on to how we are going to sort out coronavirus, which is really the overwhelming priority of the people of this country
After a lengthy inquisition by Yvette Cooper, culminating in a direct question about whether he put Dominic Cummings above the national interest (Q24), we again had the appeal to what the British public want.
I think my choice is the choice that the British people want us all to make, Yvette, and that is, as far as we possibly can, to lay aside party political point-scoring, and to put the national interest first, and to be very clear with the British public about what we want to do and how we want to take this country forward.
Overall, there were four mentions of 'getting the country back on its feet', including this statement, appended to a question on whether sanctions would be needed to ensure compliance with contact tracing (Q58)
Obviously, we are relying very much on the common sense of the public to recognise the extreme seriousness of this. This is our way out. This is our way of defeating the virus and getting our country back on its feet, and I think people will want to work together-
And in response to a further request for clarification about Dominic Cummings from Darren Jones (Q94)
It is my strong belief that what the country wants is for us to be focusing on how to go forward on the test and trace scheme that we are announcing today, and on how we are going to protect their jobs and livelihoods, and defeat this virus.
In all these exchanges, the 'British people' are depicted as decent, long-suffering people, who are having a bad time, and may be anxious or confused. During Brexit debates, this might have worked, but the problem is that now a large percentage of people of all political stripes are just plain angry, and telling them that they want to 'move on' just makes them angrier.

Ironic politeness

The final characterstic has less to do with content of answers than with their style. British political discourse is a goldmine for researchers in pragmatics – the study of how language is used. Attacking your opponent in obsequiously polite language has perhaps arisen in response to historical prohibitions on uncivil discourse in the House of Commons. Boris Johnson is a master of this art, which can be used to put down an opponent while getting a laugh from the audience. He had to be careful with the Liaison Committee, but his comments that they were 'kind to want to see me' and that he was 'delighted to be here today' were transparently insincere, and presumably designed to amuse the audience while establishing his dominance as someone who could choose whether to attend or not.

The final exchange between the Chair and the PM was priceless. The PM reiterated his enjoyment of his session with the committee but refused to undertake to return, because he was 'working flat out to defeat coronavirus and get our country back on its feet'. The Chair replied:
I should just point out that the questions on which you hesitated and decided to go away and think were some of the most positive answers you gave, in some respects. That is where we want to help. I hope you will come back soon.
I read that to mean, on the one hand, most answers were useless, but on the other hand, where the PM had pleaded for deferral, he would be held to account, and required to provide responses to the Committee in future. We shall see if that happens.

Wednesday, 13 May 2020

Manipulated images: hiding in plain sight?

Many years ago, I took a taxi from Manchester Airport to my home in Didsbury. It’s a 10 minute drive, but the taxi driver took me on a roundabout route that was twice as long. I started to query this as we veered off course, and was given a rambling story about road closures. I paid the fare but took a note of his details. Next day, having confirmed that there were no road closures, I wrote to complain to Manchester City Council.  I was phoned up by a man from the council who cheerfully told me that this driver had a record of this kind of thing, but not to worry, he’d be made to refund me by sending me a postal order for the difference in correct fare and what I’d paid. He sounded quite triumphant about this, because, as he explained, it would be tedious for the driver to have to go to a Post Office.

What on earth does this have to do with manipulated images? Well, it’s a parable for what happens when scientists are found to have published papers in which images with crucial data have been manipulated. It seems that typically, when this is discovered, the only consequence for the scientists is that they are required to put things right. So, just as with the taxi driver, there is no incentive for honesty. If you get caught out, you can just make excuses (oh, I got the photos mixed up), and your paper might have a little correction added. This has been documented over and over again by Elisabeth Bik: you can hear a compelling interview with her on the Everything Hertz podcast here.

There are two things about this that I just don’t get. First, why do people take the risk? I work with data in the form of numbers rather than images, so I wonder if I missing something. If someone makes up numbers, that can be really hard to detect (though there are some sleuthing methods available). But if you publish a paper with manipulated images, the evidence of the fraud is right there for everyone to see. In practice, it was only when Bik appeared on the scene, with her amazing ability to spot manipulated images, that the scale of the problem became apparent (see note below). Nevertheless, I am baffled that scientists would leave such a trail of incriminating evidence in their publications, and not worry that at some future date, they’d be found out.

But I guess the answer to this first question is contained within the second: why isn’t image manipulation taken more seriously? It’s depressing to read how time after time, Bik has contacted journals to point out irregularities in published images only to be ignored. The minority of editors who do decide to act behave like Manchester City Council: the authors have to put the error right, but it seems there are no serious consequences. And meanwhile, like many whistleblowers, far from being thanked for cleaning up science, Elisabeth has suffered repeated assaults on her credibility and integrity from those she has offended.

This week I saw the latest tale in this saga: Bik tweeted about a paper published in Nature that was being taken seriously in relation to treatment for coronavirus. Something in me snapped and I felt it was time to speak out. Image manipulation is fraud. If authors are found to have done it, the paper should be retracted and they should be banned from publishing in that journal in future. I call on the ‘high impact’ journals such as Nature to lead the way in implementing such a policy. I’d like to see some sanctions from institutions and funders as well, but I’ve learned that issues like this need a prolonged campaign to achieve small goals.

I’d be the first to argue that scientists should not be punished for honest errors (see this paper, or free preprint version). It's important to recognise that we are all fallible and prone to make mistakes. I can see how it is possible that someone might mix up two images, for instance. But in many of the cases detected by Elisabeth, part of one image is photoshopped into another, and then resized or rotated. I can’t see how this can be blamed on honest error. The only defence that seems left for the PI is to blame a single rogue member of the lab. If someone they trust is cooking the data, an innocent PI could get implicated in fraud unwittingly. But the best way to avoid that is to have a lab culture in which honesty and integrity are valued above Nature papers. And we’ll only see such a culture become widespread if malpractice has consequences.

Hiding in Plain Sight’ is a book by Sarah Kendzior that covers overt criminality in the US political scene, which the author describes as ‘a transnational crime syndicate masquerading as a government’. The culture can be likened to that seen in some areas of high-stakes science. The people who manipulate figures don’t worry about getting found out, because they achieve fame and grants, with no apparent consequences, even when the fraud is detected.

Notes (14th May 2020)
1. Coincidentally, a profile of Elisabeth Bik appeared in Nature the same day as this blogpost
2. Correction: Both Elisabeth Bik and Boris Barbour (comment below) pointed out that she was not the first to investigate image manipulation:

Friday, 8 May 2020

Attempting to communicate with the BBC: A masterclass in paltering

My original blogpost from October 2019 is immediate below - scroll to the end for latest response from BBC

A couple of weeks ago there was an outburst of public indignation after it emerged that the BBC had censured their presenter Naga Munchetty. As reported by the Independent, in July BBC Breakfast reported on comments made by President Trump to four US congresswomen, none of whom was white, whom he told to "go back and help fix the totally broken and crime-infested places from which they came." Naga commented "Every time I've been told as a woman of colour to 'go home', to 'go back to where I've come from', that was embedded in racism."

Most of the commentary at the time focused on whether or not Naga had behaved unprofessionally in making the comment, or whether she was justified in describing Trump's comment as racist. The public outcry has been heard: the Director General of the BBC, Tony Hall, has since overturned the decision to censure her.

There is, however, another concern about the BBC's action, which is why did they choose to act on this matter in the first place. All accounts of the story talk of 'a complaint'. The BBC complaints website explains that they can get as many as 200,000 complaints every year, which averages out at 547 a day. Now, I would have thought that they might have some guidelines in place about which complaints to act upon. In particular, they would be expected to take most seriously issues about which there were a large number of complaints. So it seems curious, to say the least, if they had decided to act on a single complaint, and I started wondering whether it had been made by someone with political clout.

The complaints website allows you to submit a complaint or to make a comment, but not to ask a question, but I submitted some questions anyhow through the complaints portal, and this morning I received a response, which I append in full below. Here are my questions and the answers:

Q1. Was there really just ONE complaint?
BBC: Ignored

Q2: If yes, how often does the BBC complaints department act on a SINGLE complaint?
BBC: Ignored

Q3: Who made the complaint?
BBC: We appreciate you would like specific information about the audience member who complained about Naga's comments but we can't disclose details of the complainant, but any viewer or listener can make a complaint and pursue it through the BBC's Complaints framework.

Q4: If you cannot disclose identity of the complainant, can you confirm whether it was anyone in public life?
BBC: Ignored

Q5: Can you reassure me that any action against Munchetty was not made because of any political pressure on the BBC?
BBC: Ignored

I guess the BBC are so used to politicians not answering questions that they feel it is acceptable behaviour. I don't, and I treat evasion as evidence of hiding something they don't want us to hear. I was interested to see that Ofcom is on the case, but have been fobbed off just as I was. Let's keep digging. I smell a large and ugly rat.

Here is the full text of the response:

Dear Prof Bishop
Reference CAS-5652646-TNKKXL
Thank you for contacting the BBC.
I understand you have concerns about the BBC Complaints process specifically with regard to a complaint made regarding Breakfast presenter Naga Munchetty and comments about US President Trump. 
Naturally we regret when any member of our audience is unhappy with any aspect of what we do. We treat all complaints seriously, but what matters is whether the complaint is justified and the BBC acted wrongly. If so we apologise. If we don’t agree that our standards or public service obligations were breached, we try to explain why. We appreciate you would like specific information about the audience member who complained about Naga's comments but we can't disclose details of the complainant, but any viewer or listener can make a complaint and pursue it through the BBC's Complaints framework.
Nonetheless, I understand this is something you feel strongly about and I’ve included your points on our audience feedback report that is sent to senior management each morning and ensures that your complaint has been seen by the right people quickly. 
We appreciate you taking the time to register your views on this matter as it is greatly helpful in informing future decisions at the BBC.
Thanks again for getting in touch.
Kind regards
John Hamill
BBC Complaints Team

8th May 2020
 Well, I had complained again, to say the original complaint did not address the points raised. Nothing happened until today, 7 months later, when out of the blue I received another email. The evasion continues.  The answers provide a masterclass in what is known as paltering - here's an article by the BBC explaining what that is. The story, of course, is now so old it will be buried, but I'm minded to conclude that the continuing failure to answer my questions means that this case was escalated on the basis of one complaint by a Very Important Person.

From BBC Complaints, 8th May 2020
Reference CAS-5652646-TNKKXL 

Dear Ms Bishop,

Thank you for getting back in touch with us and please accept our apologies for the long and regrettable delay in responding.

Our initial response didn’t address all of the specific concerns you raised, so we’d like to offer you a further response here addressing your four other questions.

1) Was there really just ONE complaint?
As widely reported in the media, one complaint was escalated to our Executive Complaints Unit (ECU).

2) If yes, how often does the BBC complaints department act on a SINGLE complaint?
Anyone can proceed through the BBC Complaints Framework and take their complaint to the ECU. Ultimately what matters is whether the complaint is justified and each complaint is judged on its own merit - sometimes complaints that go to the ECU are individual, sometimes more than one audience member will make a complaint about the same broadcast.

However, it is worth noting that the number of complaints are not the key factor and our main concern is whether the BBC acted wrongly. Full detail of the ECU’s findings can be found via the links below:

Recent ECU Findings:

Archived ECU reports:

3) If you cannot disclose identity of the complainant, can you confirm whether it was anyone in public life?
For reasons of confidentiality, and our responsibility to protect the identity of an individual who complained, we won't be providing any information about them.

4) Can you reassure me that any action against Munchetty was not made because of any political pressure on the BBC?
We can assure you of this. The BBC is independent, and the ECU came to their judgement based on the merits of the case before them, not as a result of any pressure or lobbying.


Saturday, 2 May 2020

Stepping back a level: stress management for academics in the pandemic

In my last blogpost, I suggested some things scientists could do if the pandemic prevented them conducting planned studies. Here I develop that theme. The bottom line is that in the current circumstances it may help to re-evaluate what you are doing by taking a step back to consider your broader goals. I'm prompted to write this by two tweets: one by a frustrated journal editor, who complained about the difficulty of finding reviewers right now, and the other by my colleague Laura Fortunato (@anthrolog) who tweeted about something I'd said recently.

We'd been in an online meeting of our steering group to discuss progress of Reproducible Research Oxford. We had had a clear planned timeline, including doing surveys to discover what activities/training people thought they needed, and what was already available at Oxford University. Laura noted that we were behind schedule. However, I thought it would be a mistake to go ahead and try to run any kind of survey at a time when most people were already overwhelmed by the adaptations to the pandemic, working from home, often with children present, and trying to adapt to deliver teaching online. In addition to the practical difficulties, many people were experiencing mood swings and periods of poor concentration. So this was most definitely not the time to ask them to fill in a survey. Instead, thanks to our splendid co-ordinator Malika Ihle, we had done things that were not planned, including setting up a range of online activities, focused on training and interaction, which had brought a large number of new people on board. So I just remarked: "We're not behind schedule: We have strategically adapted our activities to the current situation." Laura found this change in perspective helpful and tweeted about it, prompting me to explain my thoughts a bit more.

The economist John Kay in his book Obliquity drew a distinction between actions, goals and objectives, illustrated by the story of a visitor who encounters three stonemasons working on a cathedral. When asked what they are doing, the first person says "I am cutting this stone to shape"; the second says "I am building a great cathedral"; the third says "I am working for the glory of God". According to Kay, across a wide range of human activities, projects fail when they lose sight of the objectives and focus only on goals or actions. Of course, to function you need to translate your objectives, which are typically fairly abstract, into goals, and then specify actions to achieve those goals. But if you then get fixated on carrying out the actions, and fail to adapt when circumstances change, you are likely to come unstuck.

I'm seeing many people, both senior and junior, who are made miserable and frustrated by their current inability to carry on as before and do planned work. I can feel for the journal editor who sees her job as getting papers reviewed efficiently, only to find that she can't find reviewers. But the current blockage is not the fault of either the editor or the potential reviewer: we're up against external circumstances that none of us has experienced before. So we need to think in a more agile and flexible fashion.

The approach is going to vary hugely depending on one's personal circumstances. I'm one of the lucky ones. I can continue to do my job from home, my house is spacious with good wifi, I don't have to supervise home-schooling of children, and, to date, I and my loved ones have escaped the virus. Nevertheless, I find the situation strange and unsettling, my attention and emotions are up and down, and I'm much less efficient than usual. I'm aware that for many, many people it's much worse.

In this situation one needs to reconsider whether to attempt to carry on as usual, or whether to do something else. And here's where I think stepping back a level, from activities to goals, is helpful. For many postgraduates and early-career researchers it is particularly stressful not to be able to do a planned piece of work, because the clock is ticking and, in some cases, opportunities to gather data may be lost forever. It may be that you can find creative ways to do the work (e.g. in psychology, many studies are moving online, as I discussed in the previous blogpost), but before you do that, consider the question: why am I doing this study? There's usually two reasons: the more lofty one is genuine academic curiosity; the more prosaic is to take the next step in one's career. For postgraduates, gathering data is required for a higher degree. For postdocs, it is part of building a research portfolio. By considering the broader picture, you may realise that your planned actions are not the only way to move towards your goals – again, there are some suggestions in my last blogpost. Of course, you will mourn the sunk costs and missed opportunities, that's only natural. But the key is to recognise that there can be alternative ways to move forward. For some people, this may be impossible and you have to accept that the best way to hold things together is to press the pause button on your plans; this would be similar to a period of parental leave, and it's good to see at least some funders and institutions supporting this.

You can step further back and ask about broader objectives, why do I want this career? Is it to bring in an income, to create a sense of fulfilment, to prove to myself and others I can do it? Or even further back: what do I want out of life? To be happy, wealthy, famous, fulfilled? I've been watching TV interviews with those who survived COVID-19 and a common theme is that it makes them re-evaluate their life, often leading to a change in priorities. In particular, many people start to realise that family, friends and good health should be the highest priorities, and that working in a way that ignores these will ultimately lead to misery. In the UK, even before the pandemic, academic institutions have adopted different standards to managing staff, and the pandemic has exaggerated these. Some have tried to keep on with "business as usual" whereas others have been flexible and sympathetic to the varied impacts on their staff. You may come to the conclusion that if your employer ignores your circumstances and expects you to damage your physical and mental health to deliver on their agenda, you might be happier doing something else.

I don't want to come across as playing down the seriousness of the stresses that people are experiencing. There are no simple solutions, and the uncertain future makes planning difficult. But I hope it may help some people who are stuck in a dark place because they can't move forward to think about dropping immediate plans and taking a step back to consider if other options might allow them achieve the objectives that matter most to them.


Wednesday, 11 March 2020

What can scientists do in an emergency shutdown?

More and more universities are closing as a precaution against coronavirus.  Scientists who work on educational interventions are seeing whole projects go up in smoke when schools close, preventing gathering of endpoint data: and even if they don't close, school staff may be understandably reluctant to have researchers on the premises. Even where university labs remain open, research with humans is likely to be impacted by participants pulling out in order to keep themselves out of harm's way. I'm aware that a major push for data collection that I had planned over the next 12 months is now at risk.  I'm writing this post in part in the hope that people who have creative solutions for adapting to this situation can add them as comments, so we might find a shared way through this difficult time.

Of course, it is hoped that funders will be sympathetic to shutdowns and may offer extensions of funding, but that won't be much comfort to anyone doing a time-sensitive project, such as an intervention trial or longitudinal study, or for students who have to complete a project before their course ends. In my own case, I had planned a programme of work to take me up to retirement, and so extension of funding would not be a solution.  And if normality resumes only after a vaccine is developed and deployed, this will be a long time off.

So what to do? For some projects, particularly in psychology, online data-gathering may be a solution. Just over the past year, we have been increasingly developing methods for doing this using the Gorilla platform, including various types of language test for adults and children, as well as measures of brain lateralisation. Gorilla has a nice interface that helps new users to develop tasks, and also helps with issues such as compliance with regulations such as GDPR. For regular studies with adults, Gorilla combines nicely with Prolific, a platform for recruitment to online studies. We have had ethics approval for several studies using this approach.

But online testing won't be the solution for everyone, and one proposal I have is that where new data collection is not possible, we need to think hard about getting more value out of existing data. There are several ways we can do this. 
  • First, search for existing datasets that are relevant to your research question. Some big datasets are already well-known. For instance, the CHILDES database for child language samples, ALSPAC for longitudinal data on child development, UK Biobank for genetic and medical data, including brain imaging.  But there are many other sources of data that are relatively underused, but can be found on sites such as the UK Data Archive (social sciences) or Open Science Framework. These are just a few I happen to know about: I'm sure readers will know of many more – and maybe one useful task that underemployed scientists could undertake would be to create directories of existing accessible data on specific topics.
  • Dan Quintana tweeted that this could be a good time to learn about doing meta-analysis, and provided some useful references here.  Meta-analysis is often seen as a rather pedestrian scientific activity, but in my experience it has benefits that go far beyond the generation of a forest plot – it forces you to engage deeply with the literature and become aware of methodological issues that might otherwise be overlooked.
  • Another suggestion is to spend time curating your own existing data from published studies, to make them fully reproducible. A major aspect of the Open Science movement has been to emphasise the importance of being able to reproduce a full analysis, right through from raw data to analyses, tables and figures in a published paper.  A nice introduction by Stodden and colleagues was published in 2016, and there are more references in the slides from her recent webcast from Project Tier.
    I've been trying to adopt this approach in my own lab over the last few years and it's on the one hand much harder than you might think, but on the other hand it is immensely satisfying.  Making code open has a huge advantage beyond making the work reproducible – it also allows others to learn from your scripts, rather than re-invent the wheel when analysing data. But be warned: you will almost certainly find errors in your published work. The main thing is to anticipate that this will happen and to ensure that you correct them and learn from them.

    A major reason why many people don't adopt methods of open, reproducible science is that it takes time. Well, we may be in the weird situation of having time on our hands, and this could be a great way of using it to give a new lease of life to old data.

Saturday, 29 February 2020

Developmental Language Disorder (DLD) in relation to DSM-5

The tl;dr (too long, didn't read) message in this post is that for all intents and purposes, Developmental Language Disorder (DLD), as defined in the CATALISE project, can be regarded as equivalent to DSM-5 Language Disorder (American Psychiatric Association, 2013). This is a question of interest to people working in systems that require a DSM-5 diagnosis for access to services or insurance payments.

Diagnostic flowchart for CATALISE, based on Figure 1 from Bishop et al (2017).

Figure 1 shows how DLD is defined in the CATALISE project (Bishop et al, 2017). In this framework, DLD is a subset of the broader term 'Language Disorder', with the 'Developmental' prefix used to indicate that the language problems are not associated with a known biomedical condition. The list of biomedical conditions includes brain injury, acquired epileptic aphasia in childhood, certain neurodegenerative conditions, cerebral palsy, oral language limitations associated with sensori-neural hearing loss, autism, intellectual disability and genetic conditions such as Down syndrome.

In DSM-5, 'Language Disorder' is used with a meaning that closely corresponds to DLD. DSM-5 Diagnostic criteria for Language Disorder (category 315.32; F80.2) are as follows:

A. Persistent difficulties in the acquisition and use of language across modalities (I.e. spoken, written, sign language, or other) due to deficits in comprehension of production that include the following: 
  • Reduced vocabulary (word knowledge and use) 
  • Limited sentence structure (ability to put word and word endings together to form sentences based on the rules of grammar and morphology 
  • Impairments in discourse (ability to use vocabulary and connect sentences to explain or describe a topic or series of events or have a conversation) 
B. Language abilities are substantially and quantifiably below those expected for age, resulting in functional limitations in effective communication, social participation, academic achievement, or occupational performance, individually or in any combination 
C. Onset of symptoms is in the early developmental period 
D. The difficulties are not attributable to hearing or other sensory impairment, motor dysfunction, or another medical or neurological condition, and are not better explained by intellectual disability (intellectual developmental disorder) or global developmental delay. 

The relationship between the CATALISE terminology and DSM-5 are shown in Figure 2. In both CATALISE DLD and DSM-5 Language Disorder, cognitive referencing is not used (i.e. there is no IQ cutoff for inclusion in the category), the language problems are seen in early childhood and are persistent and lead to functional impairment, and children with associated biomedical conditions are excluded. 
Figure 2: Set diagram showing relationship between CATALISE and DSM-5 terminology
The CATALISE criteria are more explicit than DSM-5 in relation to bilingual/multilingual children, making it clear that one would not diagnose DLD unless the child showed poor language competence in their best language, but a similar idea is conveyed in the "Differential diagnosis" section of DSM-5, where it is noted that "Language disorder needs to be distinguished from normal developmental variations... Regional, social, or cultural/ethnic variations of language must be considered when an individual is being assessed for language impairment." Unfortunately, a recent account of DLD by a member of the CATALISE panel stated that bilingual/multilingual children were excluded from the DLD category (Rice, 2020). It is unclear how this misunderstanding arose, but it is clearly discrepant with Figure 1.

One might ask why the CATALISE panel decided against alignment with the DSM-5 terminology, given that there is such overlap between CATALISE DLD and DSM-5 Language Disorder. The answer is that "Language Disorder" as defined in DSM-5 is a problematic label because on the one hand it refers to a specific DSM-5 category 315.32 (F80.2), but on the other hand it is widely used to refer to symptom-level problems in many conditions. "Language disorder" is a hopeless term to use in a literature search because it will turn up a far broader range of conditions than those defined by the diagnostic criteria above, including many types of acquired language disorder associated with both developmental and adult-onset conditions.

This broader meaning of the term gives ample scope for confusion. Indeed, even within the DSM-5 manual, ambiguity is apparent. Under "Differential diagnosis" mention is made of Neurological disorders, with the statement "Language disorder can be acquired in association with neurological disorders, including epilepsy (e.g. acquired aphasia or Landau-Kleffner syndrome)". A literal reading of this would mean that, contrary to Diagnostic criteria point D, children with acquired aphasia or Landau-Kleffner syndrome can be regarded as having Language Disorder. I don't think that is what the authors intended: rather they imply that one should differentiate acquired language disorder associated with neurological conditions from "Language Disorder" that corresponds to DSM-5 category 315.32 (F80.2).

In short, in proposing the DLD label, the CATALISE panel were capturing a grouping that is conceptually the same as DSM-5 Language Disorder. We felt, however, that 'Language Disorder" was a problematic label that would generate confusion, and so we used the more traditional term "Developmental Language Disorder" specifically to identify a category of language disorder without associated biomedical conditions.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.
Bishop, D. V. M., Snowling, M. J., Thompson, P. A., Greenhalgh, T., & CATALISE Consortium. (2017). Phase 2 of CATALISE: a multinational and multidisciplinary Delphi consensus study of problems with language development: Terminology. Journal of Child Psychology and Psychiatry, 58(10), 1068-1080. doi:10.1111/jcpp.12721
Rice, M. (2020). Clinical lessons from studies of children with Specific Language Impairment. Perspectives of the ASHA Special Interest Groups, 5(1), 12-29. doi:

Sunday, 23 February 2020

Changing terminology for children's language disorders: reflections on special issue of Perspectives of the ASHA Special Interest Groups

In 2016-7, I joined with expert colleagues to try to tackle the thorny issue of terminology for children who had difficulties speaking or understanding in their native language. This was incredibly challenging: we had a panel of 57 experts, moderated by Maggie Snowling and me, engage in a Delphi process, whereby we iteratively rated and commented on statements to do with identification and diagnosis to converge on a set of recommendations (Bishop et al., 2016; 2017). There was substantial disagreement between experts, who came from a range of disciplines and nationalities, but the one thing we agreed upon was that the current situation – of a plethora of different terms being used with various meanings – was doing a grave disservice to families and their children affected by language problems. There was a lack of recognition of language problems among professionals and the general public, as well as a paucity of research relative to other conditions of similar severity and prevalence. Eventually we converged on a recommendation that the term 'Developmental Language Disorder' (DLD) should be used to describe children with persistent problems with language expression or reception that had no obvious cause, and that impacted on everyday life.

Publication of the 2017 recommendations had considerably greater impact than I had anticipated, thanks to the generally enthusiastic take-up of the new terminology, which has been adopted by professional organisations in the UK, Ireland and Australia. A brief summary of the salient points was written by Susan Ebbels for the Royal College of Speech and Language Therapists.

Following the publication of the CATALISE papers, big changes have happened just in a couple of years:
  • The terminology has stimulated development of grassroots advocacy, with campaigns to promote awareness of DLD adopting ingenious and innovative methods to publicise events. See e.g. here.
  • The DLD label has raised awareness that this is a life-long condition*. Adults with DLD have started to talk about their experiences on social media (e.g. here). 
  • Figures on terminology from Web of Science show that DLD is starting to overtake the term 'Specific Language Impairment' in published papers.
  • Figure 1: N publications with full term 'Specific Language Impairment' or 'Developmental Language Disorder' in the Topic field; data from Web of Science

    Nevertheless, despite these positive developments, not everyone has been convinced that adoption of DLD is a good thing. This is not surprising: a couple of years ago, I reflected on some of the reasons why this particular topic is a terminological minefield (Bishop, 2017). It is difficult to balance the pros and cons of new terminology, especially when one is dealing with a condition that is heterogeneous, changes with age, can co-occur with many other problems, and is of relevance to several professions with very different perspectives, in particular speech-language therapists/pathologists, educators and medics.

    Resistance to new terminology appears to be strongest in the USA, with the American Speech-Language-Hearing Association being particularly cautious about replacing the term Specific Language Impairment with DLD. This week they published a set of papers by individuals with different viewpoints on the topic, and reading through this I was reminded of many of the debates we had in the course of our Delphi exercises. The special issue includes a  tutorial overview by Karla McGregor and colleagues (McGregor et al., 2020), who summarise the background to the CATALISE project, and subsequent developments. They are supportive of DLD, not least because of the way the new terminology has boosted the profile of children's language problems and stimulated greater efforts towards advocacy. But we need to take seriously the concerns of those who have doubts.

    Two authors in particular, Larry Leonard and Mabel Rice, were concerned that if we adopt DLD as the preferred term, decades of research on SLI would be ignored and ultimately forgotten. Both of them have made major lifetime contributions to the field, and so their concerns are understandable. It is vital that we heed Larry's advice (Leonard, 2020) to ensure that we include the search term SLI as well as DLD when drawing together relevant literature, to ensure valuable work from previous decades is not lost.

    A related issue is that, although they overlap, SLI and DLD are not identical. In general, DLD is a more inclusive term, as it does not require that nonverbal IQ is 85 or over (although it does exclude cases of intellectual disability**), and it is compatible with the presence of other neurodevelopmental disorders, especially developmental co-ordination disorder and attention-deficit-hyperactivity disorder. Rice (2020) remains strongly opposed to the use of DLD, mainly because this would entail including children with nonverbal IQs below 85 (who were termed cases of nonspecific language impairment, or NLI, in a big epidemiological study, the Iowa study). Yet this stance is hard to reconcile with her view that nonverbal ability is independent of language. She spends several paragraphs presenting evidence that it is possible to have low nonverbal ability and good language, or high nonverbal ability and poor language, culminating in the statement: 'The common assumption that children with SLI are not very smart is unwarranted, as is the assumption that all children with strong language aptitude are smart.' It is unclear to whom she is referring as making this 'common assumption'. A major reason why the CATALISE group decided to drop 'cognitive referencing' in defining DLD was because the bivariate distribution of nonverbal IQ and language demonstrates rather a weak association – just as she claims - but this makes it clear that any division into those above and below an IQ threshold is arbitrary (see Figure 2). Just to be absolutely clear, in proposing we do away with nonverbal IQ criteria for DLD, the CATALISE consortium were not saying that children with SLI are 'not very smart'. We were saying that they key thing is what your language is like, and that nonverbal ability is largely irrelevant when making that judgement.
Figure 2: Language composite measure plotted against non‐verbal IQ for 603 eight‐year‐old children in the Iowa study. Figure from Reilly et al (2014). Blue points correspond to SLI, and green points to NLI. NB Axes resized to address criticism by Rice (2020), who found the scaling misleading.

    Rice goes on to argue that we need to study children's mastery of specific grammatical markers over time to get a precise measure of children's language in SLI, yet she then goes on to present data showing that children with NLI are also identified by low scores on those markers. Although she claimed a difference in language profiles, this was not evident from the data she presented, and in general, most studies that have looked for qualitative differences between NLI and SLI have failed to find these (see also further evidence from McGregor et al, 2020). Rice maintains that only by sticking with SLI would we find sufficiently clear findings to advance our understanding, and that if we adopt less precise categories, this will lead to muddying of the waters and less replicable results. I regard this as a hypothesis that can be tested, but for which there is currently no good evidence. In fact, it was the failure to find such evidence in my own studies 25 years ago that led me to start to question the construct of SLI (Bishop, 1994).

    Larry Leonard makes the sensible suggestion that if we study DLD we should also provide data on which children would meet the more restrictive definition of SLI:  this would, in fact, allow us to test Rice's hypothesis. The best way to do this, and to build on prior studies with SLI would be to make raw data open (see Meyer, 2018), so that results can be analysed to investigate the extent to which findings depend on levels of nonverbal ability, attention, motor skills and so on. There is already precedent for this with the EpiSLI database from the Iowa study (Tomblin, 2010) – this includes data on children who would meet broader DLD criteria as well as more selective SLI cases. This way we could investigate empirically the impact of the choice of criteria, rather than just asserting that one classification approach is better than the other.

    I turn next to the other commentary that is predominantly negative about a change of criteria to DLD, that by Murza and Ehren (2020). Their perspective is diametrically opposed to that of Rice. They explain that for speech-language pathologists (SLPs) working in the school system, there is a mismatch between what is taught in their university courses (including diagnosis of SLI) and what they experience in the classroom. There were some parallels here with the views of education experts on the CATALISE panel, some of whom felt the whole debate about labels was irrelevant, and would empathise with the view that: 'No matter the label, students should receive the services they need, and no label should drive services'. But, as Murza and Ehren recognise, labels are needed to demonstrate eligibility for services, and in the USA, this is determined by the Individuals with Disabilities Education Act (IDEA), which has its own set of labels: these don't include either SLI or DLD. I felt a sense of despair coming through from this article, as the authors documented the confusing array of possible terms they could use, each of which had an operational definition, but with that definition varying from one state to another. The impression was that SLPs in US education are up against a massive, unbending bureaucracy that they had to negotiate in order to deliver services to a subset of the children that might benefit from them. In principle, yes, it would be nice to have some consistent terminology that mapped on to the population they were serving, but their conclusion was that they lacked the time, money, energy and focus to embark on a campaign to adopt new labels. This was a considerably more sober and downbeat chapter than that of Rhea Paul, who regarded DLD as potentially compatible with IDEA criteria, and a useful focus for advocacy efforts. 

    When discussing relevance of labels for SLP practitioners, another issue concerned insurance eligibility. Karla McGregor reported that she had been told by an insurer: 'If a child just isn't talking or not talking clearly - it is not covered'. The authors also noted a common concern that the term Developmental would be interpreted by insurers, and perhaps others, as implying that the condition would improve of its own accord. But SLPs should not take these kinds of responses lying down: it is grossly inequitable if insurers are willing to pay for a condition such as ADHD or Developmental Co-ordination Disorder, but not for speech-language therapy for a child with DLD. The arguments can and should be made, and it is hoped that the growing awareness of, and advocacy for DLD will empower US SLPs to be more proactive in demanding that children with language disorders are taken seriously and given the support they need.

    * Damian Quinn, a man with DLD, has written an autobiography. The timing in relation to CATALISE is probably co-incidental, but this will help with growing awareness that children with DLD do not disappear when they become adults!

    ** The introductory article for the special issue stated that DLD is used for children 'with or without intellectual disability'. This is wrong. Where the child meets criteria for intellectual disability, the diagnosis would be Language Disorder associated with Intellectual Disability.

    P.S. (24 Feb 2020). All the materials from the CATALISE project, including anonymised ratings and comments from the 57 panel members, are available on Open Science Framework.

     Bishop, D. V. M. (1994). Is specific language impairment a valid diagnostic category? Genetic and psycholinguistic evidence. Philosophical Transactions of the Royal Society, series B, 346, 105-111. 
    Bishop, D. V. M. (2017). Why is it so hard to reach agreement on terminology? The case of developmental language disorder (DLD). International Journal of Language & Communication Disorders, 52(6), 671-680. doi:10.1111/1460-6984.12335

    Bishop, D. V. M., Snowling, M. J., Thompson, P. A., Greenhalgh, T., & CATALISE Consortium. (2016). CATALISE: a multinational and multidisciplinary Delphi consensus study. Identifying language impairments in children. . PLOS One, 11(7), e0158753. doi:doi:10.1371/journal.pone.0158753

    Bishop, D. V. M., Snowling, M. J., Thompson, P. A., Greenhalgh, T., & CATALISE Consortium. (2017). Phase 2 of CATALISE: a multinational and multidisciplinary Delphi consensus study of problems with language development: Terminology. Journal of Child Psychology and Psychiatry, 58(10), 1068-1080. doi:10.1111/jcpp.12721

    Leonard, L. B. (2020). A 200-year history of the study of childhood language disorders of unknown origin: changes in terminology. Perspectives of the ASHA Special Interest Groups, 5(1), 6-11. doi:

    McGregor, K. K., Goffman, L., Owen Van Horne, A., Hogan, T. P., & Finestack, L. H. (2020). Developmental Language Disorder: Applications for advocacy, research, and clinical service. Perspectives of the ASHA Special Interest Groups, 5(1), 38-46. doi:

    Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science. doi:

    Murza, K. A., & Ehren, B. J. (2020). Considering the language disorder label debate from a school speech-language pathology lens. Perspectives of the ASHA Special Interest Groups, 5(1), 47-54. doi:

    Paul, R. (2020). Children's language disorders: What's in a name. Perspectives of the ASHA Special Interest Groups, 5(1), 30-37. doi:
    Rice, M. (2020). Clinical lessons from studies of children with Specific Language Impairment. Perspectives of the ASHA Special Interest Groups, 5(1), 12-29. doi:

    Reilly, S., Tomblin, B., Law, J., McKean, C., Mensah, F. K., Morgan, A., . . . Wake, M. (2014). Specific language impairment: a convenient label for whom? International Journal of Language & Communication Disorders, 49(4), 416-451. doi:10.1111/1460-6984.12102

    Tomblin, J. B. (2010). The EpiSLI database: a publicly available database on speech and language. Lang Speech Hear Serv Sch, 41(1), 108-117.