A proposal for data-sharing that discourages p-hacking

Open data is a great way of helping give confidence in the reproducibility of research findings. Although we are still a long way from having adequate implementation of data-sharing in psychology journals (see, for example, this commentary by Kathy Rastle, editor of Journal of Memory and Language), things are moving in the right direction, with an increasing number of journals and funders requiring sharing of data and code. But there is a downside, and I've been thinking about it this week, as we've just published a big paper on language lateralisation, where all the code and data are available on Open Science Framework. 

One problem is p-hacking. If you put a large and complex dataset in the public domain, anyone can download it and then run multiple unconstrained analyses until they find something, which is then retrospectively fitted to a plausible-sounding hypothesis. The potential to generate non-replicable false positives by such a process is extremely high - far higher than many scientists recognise. I illustrated this with a fictitious example here. 

Another problem is self-imposed publication bias: the researcher runs a whole set of analyses to test promising theories, but forgets about them as soon as they turn up a null result. With both of these processes in operation, data sharing becomes a poisoned chalice: instead of increasing scientific progress by encouraging novel analyses of existing data, it just means more unreliable dross is deposited in the literature. So how can we prevent this? 

In this Commentary paper, I noted several solutions. One is to require anyone accessing the data to submit a protocol which specifies the hypotheses and the analyses that will be used to test them. In effect this amounts to preregistration of secondary data analysis. This is the method used for some big epidemiological and medical databases. But it is cumbersome and also costly - you need the funding to support additional infrastructure for gatekeeping and registration. For many psychology projects, this is not going to be feasible. 

A simpler solution would be to split the data into two halves - those doing secondary data analysis only have access to part A, which allows them to do exploratory analyses, after which they can then see if any findings hold up in part B. Statistical power will be reduced by this approach, but with large datasets it may be high enough to detect effects of interest.  I wonder if it would be relatively easy to incorporate this option into Open Science Framework: i.e. someone who commits a preregistration of a secondary data analysis on the basis of exploratory analysis of half a dataset then receives a code that unlocks the second half of the data (the hold-out sample). A rough outline of how this might work is shown in Figure 1.

Figure 1: A possible flowchart for secondary data analysis on a platform such as OSF

An alternative that has been discussed by MacCoun and Perlmutter is blind analysis - "temporarily and judiciously removing data labels and altering data values to fight bias and error". The idea is that you can explore a dataset and run a planned analysis on it, but it won't be possible for the results to affect your analysis, because the data have been changed, so you won't know what is correct. A variant of this approach would be multiple datasets with shuffled data in all but one of them. The shuffling would be similar to what is done in permutation analysis - so there might be ten versions of the dataset deposited, with only one having the original unshuffled data. Those downloading the data would not know whether or not they had the correct version - only after they had decided on an analysis plan, would they be told which dataset it should be run on. 

I don't know if these methods would work, but I think they have potential for keeping people honest in secondary data analysis, while minimising bureaucracy and cost. On a platform such as Open Science Framework it is already possible to create a time-stamped preregistration of an analysis plan. I assume that within OSF there is already a log that indicates who has downloaded a dataset. So someone who wanted to do things right and just download one dataset (either a random half, or one of a set of shuffled datasets) would just need to have a mechanism that allowed them to gain access to the full, correct data after they had preregistered an analysis, similar to that outlined above.

These methods are not foolproof. Two researchers could collude - or one researcher could adopt multiple personas - so that they get to see the correct data as person X and then start a new process as person B, when they can preregister an analysis where results are already known. But my sense is that there are many honest researchers who would welcome this approach - precisely because it would keep them honest. Many of us enjoy exploring datasets, but it is all too easy to fool yourself into thinking that you've turned up something exciting when it is really just a fluke arising in the course of excessive data-mining. 

Like a lot of my blogposts, this is just a brain dump of an idea that is not fully thought through. I hope by sharing it, I will encourage people to come up with criticisms that I haven't thought of, or alternatives that might work better. Comments on the blog are moderated to prevent spam, but please do not be deterred - I will post any that are on topic. 

P.S. 5th July 2022 

Florian Naudet drew my attention to this v relevant paper: 

Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R. (2022). Protecting against researcher bias in secondary data analysis: Challenges and potential solutions. European Journal of Epidemiology, 37(1), 1–10. https://doi.org/10.1007/s10654-021-00839-0

Book Review. Fiona Fox: Beyond the Hype

If you're a scientist reading this, you may well think, as I used to, that running a Science Media Centre (SMC) would be a worthy but rather dull existence. Surely, it's just a case of getting scientists to explain things clearly in non-technical language to journalists. The fact that the SMC was created in part as a response to the damaging debacle of the MMR scandal might suggest that it would be a straightforward job of providing journalists with input from experts rather than mavericks, and helping them distinguish between the two. 

I now know it's not like that, after being on the Science Media Centre's panel of experts for many years, and having also served on their advisory committee for a few of them. The reality is described in this book by SMC's Director, Fiona Fox, and it's riveting stuff.

In part this is because no science story is simple. People will disagree about the quality of the science, the meaning of the results, and the practical implications. Topics such as climate change, chronic fatigue syndrome/ME and therapeutic cloning elicit highly charged responses from those who are affected by the science. More recently, we have found that when a pandemic descends upon the world, some of the bitterest disagreements are not between scientists and the media, but between well-respected, expert scientists. The idea that scientists can hand down tablets of stone inscribed with the truth to the media is a fiction that is clearly exposed in this book.

Essentially, the SMC might be seen as acting like a therapist in the midst of a seriously dysfunctional family where everyone misunderstands everyone else, and everyone wants different things out of life. On the one hand we have the scientists. They get frustrated because they feel they should be able to make exciting new discoveries, with the media then helping communicate these to the world. Instead, they complain that the media has two responses: either they're not interested in the science, or they want to sensationalise it. If you find a mild impact of grains on sexual behaviour in rats, you'll find it translated into the headline 'Cornflakes make you impotent'.

On the other hand, we have the media. They want a good story, but find that the scientists are reluctant to talk to them, or want total control of how the story is presented. In the worst case, scientists are prima donnas who want days or weeks to prepare for a media interview and will then shower the journalist with detailed information that is incomprehensible, irrelevant, or both. When the public desperately needs a clear, simple message, the scientists will refuse to deliver it, hedging every statement.

Fox has worked over the years to challenge these stereotypes: journalists do want a good story, but the serious science journalists want a true story, and are glad of the opportunity to pose questions directly to scientists. And many scientists do a fantastic job of explaining their subject matter to a non-specialist audience. In the varied chapters of the book, Fox is an irrepressible optimist, who keeps coming back to the importance of having scientists communicating directly with the media. Her optimism is not founded in ignorance: she knows exactly how messy and complicated science can be. But she persists in believing that more good is done by communicating what we know, warts and all, rather than pretending that uncertainties and disagreements do not exist.

The role of the SMC is, however, complicated by further factions. The dramatis personae includes two other groups. First, there are science press officers, who are appointed by institutions to help scientists promote their work, and then there are government officials and civil servants, who are concerned with policy implications of science.

In her penultimate chapter, Fox bemoans the fact that the traditional press officer - passionate about science and viewing themselves as "purveyors of truth and accuracy" - is a dying breed. There remain notable exceptions, but all too often science communication has become conflated with a public relations role: pushing a corporate message, defending the institutional reputation, and even using scientific discoveries as a marketing tool. Fox notes a 2014 survey of exaggerated science reports in the media that concluded: "Exaggeration in news is strongly associated with exaggeration in press releases." I had been one of those scientists who thought the media were mostly to blame for over-hyped science reporting, but this study showed that journalists are often recycling exaggerated accounts handed to them by those speaking for the scientists.

But the problems posed by scientists, journalists and press officers are trivial compared to the obstacles created by those involved in policy. They want to use science when convenient, but also want to exert control over which aspects of science policy gets talked about. Scientists working for government-funded organisations are often muzzled, with explicit instructions not to talk to the media. One can see that this cautious approach, attempting to control the message and keep things simple, puts many civil servants and government scientists on a collision course with Fox, whose view is: "Explaining preliminary and contradictory science is messy: that should not be seen as a failure of communications".

A refreshing aspect of Fox's account is that she does not brush aside the occasions when the SMC - or she personally - may have handled a situation badly. Of course, it's easy to point the finger of blame when something does go horribly wrong, and Fox has come under fire on many occasions. Rather than being defensive, she accepts that things might have been done differently, while at the same time explaining the logic of the decisions that were taken. This is in line with my memories of meetings of the SMC advisory committee, where there were frequent post mortems - "this is how we handled it; this is how it turned out; should we have done it differently?" - with frank discussions from the committee members. When you are working in contentious areas where things are bound to blow up every now and again, this is a sensible strategy that helps the organisation learn and develop. I'm glad that after 20 years, the ethos of the SMC is still very much on the side of open, transparent communication between scientists and the media.  

Fox, Fiona (2022) Beyond the Hype: The Inside Story of Science's Biggest Media Controversies. London: Elliott and Thompson Ltd.

Some quick thoughts on academic boycotts of Russia

 

 

In response to the dramatic developments in Ukraine, several instances of academic boycotts of Russia have been proposed or implemented.  These are a few of the initiatives I have heard about on Twitter:

 

European Federation of Academies of Sciences and Humanities (ALLEA) has suspended the Russian Academy of Sciences and the National Academy of Sciencesof Belarus

 

The British Psychological Society has supported a call from Ukrainian psychologists for the Russian Psychological Society to be suspended from the European Federation of Psychologists Associations (EFPA)

 

The Australian National University suspended all institutional ties with Russian institutions.

 

Yesterday, Chris Chambers, who is a journal editor, asked people's views on whether academic journals should be considering contributions from authors based in Russia.

 

Responses were pretty polarised. I have strong views against all these kinds of academic boycott, which have not really been changed by the numerous, well-argued points made by those responding to Chris, and so I thought I'd set out my position here.

 

I should start by saying that the disagreement here is not about the rights and wrongs of the war in Ukraine. I take as a basic premise that the invasion of Ukraine is an abominable atrocity, that, as its instigator, Vladimir Putin is guilty of war crimes, and that we all should do all we can to stop the war and ensure Putin is brought to justice.

 

At a national level, sanctions are imposed by governments to put pressure on Russia that, on the one hand, serves a symbolic function of demonstrating disapproval, and on the other hand serves a practical function of weakening the Russian economy, in the hope this might help bring the invasion to an end without starting World War 3.  They are not undertaken lightly, as it is recognised that many innocent people who play no part in the invasion will be adversely affected - both Russians who protest the invasion and people in the countries who are applying the sanctions, who may have to pay higher prices, or suffer shortages of goods.

 

Other organisations, notably sports organisations, have taken the line of moving major sporting events away from Russia. I fully support such moves, because they will have a financial impact on Russia, as well as denying Putin an opportunity to act as a host, which is a role that carries considerable prestige as well as opportunities for positive publicity. I also agree with banning Russian athletes from sporting events where they are explicitly representing their country - in effect, including these athletes creates the impression that Russia is a normal member of the sporting community. So these sanctions affecting sports have a very visible symbolic role as well as a potential financial one.  That makes the cost-benefit ratio of the actions seem worthwhile, despite the unfairness to individual athletes.

 

I see academic interactions differently in terms of the cost-benefit ratio.  I don't think there'd be any financial impact to Russia of Chris or other editors refusing to consider papers by Russian authors, and the symbolic impact is likely to be pretty small. I doubt most people would notice that such a sanction had occurred, and, even if Putin was aware of it, I can't see him getting particularly upset about it.  In saying this, I'm not arguing that academics are 'special' - just that the practical impact of adopting sanctions against them would be close to zero, except for the damage to the sanctioned individuals. Add to this the fact that academics are typically among the early casualties of dictators, who dislike them for their tendency to criticise, and I think the case for this kind of academic sanction is very weak.  

 

In the responses to Chris, there are many people, on both sides, putting forward 'what about' arguments. So, for instance, what about a small Russian trader, innocent of any war crimes, whose livelihood is blighted by sanctions that affect her business? For me the question is whether such sanctions have an effect that is worthwhile - in terms of its visibility and economic impact -  given that all sanctions will trap the deserving as well as the undeserving in their net?  So this needs a response on a case-by-case basis, rather than treating it as a generic principle to be adopted in all sectors.

 

Another 'what about' issue that is raised concerns countries other than Russia. Should we start to sanction academics from any government that behaves inhumanely? It would be quite a long list, and would be difficult to draw the line.  It's worth noting that on the same principle, we could well find ourselves sanctioned by academics from other countries who regard the UK's colonial past and/or escapades in the Middle East worthy of censure.  The only reason we don't have to worry about that is because we happen to have considerable power over the machinery of academic publishing.

 

So my answer here again is that in general, academics should only consider sanctions as an extreme resort, given that they are a blunt instrument that can have serious unintended consequences. The economic sanctions currently imposed on Russia by governments are unusually effective because they have a big effect on many sectors, and we are seeing a co-ordinated activity of many nations, as well as nations prepared to take a financial hit themselves.  That's not the case for academic sanctions.

 

There are many other ways we can express our support for Ukraine. Moves are already afoot among learned societies to help Ukrainian academics who have had to leave the country, and also Russian academics who have put themselves in harm's way by speaking out against the invasion. Personally, I think the most useful thing I can do is to donate money to that cause as well as to other causes that will strengthen Ukraine's position in the conflict, rather than calling for symbolic gestures.

 

I'm prepared to listen to counter-arguments. It's clear this is one of those issues that is pretty divisive among people of good will who are usually on the same side of political debate.  To help cope with a tsunami of spam, comments on this blog are moderated, but I will monitor them and publish those that contribute (respectfully) to this discussion.

 

 

Why I am not engaging with the Reading Wars

I’m not sure whether to feel flattered or irritated by Jeff Bowers’ latest blogpost, where he expresses his disappointment in my failure to support his critique of research on phonics intervention.

The short explanation, as I already conveyed to Jeff, is that I'm busy with other stuff. I’m aware that there’s a pretty intemperate debate going on at present about the value of phonics in teaching children to read, with various people writing lengthy critiques of one another’s positions. So clearly it’s a complex and murky situation. The last thing this needs is for someone else, me, who hasn’t looked in depth at the relevant research, to weigh in with an opinion.

As background: I am a (soon-to-be retired) researcher who has never worked on reading intervention, though I do work on topics tangentially related to this. I'm currently finishing off a big project on cerebral lateralization for language. I’m also involved in a couple of other projects unrelated to reading. I am a  hands-on researcher, so am heavily involved in designing, analysing and writing up research. I currently work a 4 day week, and these three projects, plus regular reviewing, giving talks, etc, take up all of my time.

One of the fun parts of my job is that I am my own boss and, as far as time allows, I can pursue odd topics that take my fancy. I sometimes put my head above the parapet and engage in, or even initiate, academic disputes about research. These consume much time and emotional energy, so I choose carefully which battles I want to get involved in. In particular, I think hard about what can be gained - I do get passionate when something is wrong, especially where someone is behaving badly and deliberately distorting science for some nefarious ends. In such cases I will get engaged to try and put things right.

The reading wars don't look like that to me. They actually look rather like the equally heated debates going on among epidemiologists and public health experts around Covid – another topic on which I don’t’ speak out because I lack expertise. The most I’m prepared to offer is a general suggestion about perspective that might be helpful in achieving some truth and reconciliation.

In both Covid and reading wars there are arguments about research evidence and arguments about policy. Typically, there isn't really good research on which to base important decisions, but we don’t have the luxury of waiting for such studies to be done. Doing nothing is in itself a choice, for which, in the case of an epidemic, we know the consequences are likely to be bad. So it’s all about making the best decisions we can, given the patchy evidence that we have – a point nicely made by Trisha Greenhalgh in the context of COVID-19. 

The current debates have highlighted the shortage of well-designed studies to show which teaching method is most effective for which children at which age. As with medical interventions, studies also need to evaluate potential costs and harms – something which was neglected in the past, but which is relevant given that some of the disagreements are about whether particular approaches can be counterproductive.

But, as with Covid, policy decisions have to be pragmatic. This was the remit of the 2006 Rose Report, which attempted to synthesise evidence and expert opinion on reading intervention and came out recommending use of synthetic phonics. Overturning the status quo to radically change how reading is taught would only be reasonable if there was some other intervention approach for which there was better evidence of effectiveness – so much so that the costs of making a change would be justified. Maybe there is, but it hasn’t come up on my radar. 

My apologies to all of those engaged in current discussions whose contributions I have ignored. I hope this blogpost will suffice to explain why.

Boris Johnson leads the world ....in the number of false facts he can squeeze into a session of PMQs

 

The first weeks of 2022 have had quite enough drama to last us the whole year. Prime Minister’s Questions (PMQs) on Wednesdays has developed a must-watch quality that was previously reserved for the final of the Great British Bake-off. I find myself studying the oratorical approach adopted by Boris Johnson. Previously, I analysed his strategies for (usually not) answering questions in a session of a Select Committee. In PMQs the style is different. It's a gladiatorial context where the goal is to earn howls of approval for trouncing your opponent.

Last week we had the unusual sight of a remorseful Boris Johnson, trying to convince the nation that he was truly, truly sorry for allowing parties to take place on his watch, during a period when we had extremely severe lockdown rules that were preventing people seeing dying relatives or attending funerals. It was one of those apologies where you sensed he was sorry at having been found out, rather than for having done something wrong. But he clearly hoped this would be enough to settle the matter so we could put it behind us – at least until a whitewashing report is published. He had not reckoned with the Machiavellian cunning of his ex-advisor Dominic Cummings, who is drip-feeding damaging information to the media. This started a new story at the weekend, after Cummings contradicted Johnson’s denial that he knew about a party. So I wondered if we would have Remorse, the Sequel at yesterday’s PMQs. 

Johnson, however, adopted the very strategy that he had previously learned from Cummings – brazen it out. So we had the bizarre sight of a Prime Minister who has lost confidence of his own party because they no longer trust what he says, doubling down and telling yet more lies to convince everyone that this is an Absolutely Splendid Government, and he is a the best chap to lead them. (My thanks to Hansard for this transcript of the debate). 

Here is a selection of some of the more questionable claims (there is more, but I have a day job!):

 “We are building 40 new hospitals.” Cf this analysis by FullFact.  One might add that the biggest problem faced by the NHS is lack of staff, rather than buildings.

“We have also been cutting crime by 10% and putting 11,000 more police officers out on the streets” – I guess this is an improvement on previous claims of 20,000 more police officers, but it’s been debunked as misleading, given that this is reinstating police that were previously cut.  

“We now have the fastest-growing economy in the G7 and GDP is now back up above pre-pandemic levels.” Here, Google just led me to a Government report from the previous day which stated: “In Q3 2021, UK GDP grew by 1.1% compared with the previous quarter (Q2 2021), while GDP rose by 2.2% in the Eurozone and 0.6% in the US. Compared to the pre-pandemic peak, UK GDP in Q3 2021 was 1.5% lower, compared with 1.1% lower in Germany and 0.3% lower in the Eurozone. In the US GDP was 1.4% higher. 

“Thanks to the efforts of people across Whitehall, this country is now capable of producing 80% of our own PPE.” I couldn’t find recent data that might be the basis for this claim, but talking up their record on PPE as a source of pride seems a rash move by a Government whose shambolic handling of this issue was documented in a report by the Public Accounts Committee in Feb 2021:  “...its failure to be transparent about decisions, publish contracts in a timely manner or maintain proper records of key decisions left it open to accusations of poor value for money, conflicts of interest and preferential treatment of some suppliers, and undermines public trust in government procurement and the use of taxpayers’ money. Between February and July 2020, the Department of Health and Social Care spent over 12 billion on 32 billion items of PPE. We are concerned that the Department has so far identified items worth hundreds of millions of pounds which are unusable for their intended purpose, putting the efficient use of taxpayers’ money at further risk.”

As Daisy Cooper noted on Twitter, Johnson also repeated the claim that the vaccine rollout would not have been possible except for Brexit – a claim that has been debunked more than once 

Then there was this extraordinary claim: “I will tell you what this Government have been doing to look after the people of this country throughout this pandemic and beyond. We have been cutting the cost of living and helping them with the living wage. We have been cutting taxes for people on low pay. We have been increasing payments for people suffering the costs of fuel- “ People don’t need to consult Google to challenge this rosy vision of life in the UK. We can simply take use of foodbanks as an indicator of how many people are living in serious poverty. We had barely heard of foodbanks before this Government came into power – now more and more people depend on them just to exist. Not only is the cost of living going up, Jack Monroe noted that the way it is computed underestimates the impact on the poorest 

No, Prime Minister, you can’t talk your way out of this one. People are worried about what the future holds, but they are not going to reject the evidence of their eyes and ears

Response to Philip Ball's critique of scientific advisors

This week, Philip Ball wrote a piece in the New Statesman headlined: “Muted and deferential, the UK’s scientists have failed the pandemic test”. I regard Ball as a thoughtful and knowledgeable science writer, and I usually agree with him, but this piece made me uneasy. I expressed disagreement on Twitter and Ball asked for my reasons, so here is a brief response.

I should start by saying that, while I try to keep myself informed about the science of coronavirus, I have no background in infectious diseases, epidemiology or public health. I follow a lot of genuine experts on Twitter, and what has been striking is the lack of agreement between them on many policy issues. Initially there was uncertainty about the nature of the disease – it seems WHO held out for a long time against the view that the virus was airborne, a view that I think is now pretty universally accepted. This led to early mis-steps with a focus on handwashing and surface-cleaning rather than masks and indoor air quality. On other matters, such as vaccination of children, disagreements still remain. And when we move to consider potential non-pharmaceutical public health measures, there is a wide range of views – do lock-downs do more harm than good? Should we restrict foreign travel? How should a test-and-trace system be organised? Although people who know what they are talking about sometimes take strong and passionate positions on these issues, there is variation from country to country in how they are addressed, and my impression is the outcomes do not always support what we might have expected.

I’m not saying it doesn't matter what we do – on the contrary, I think it matters enormously, as the health of the world depends on getting it right. But it’s much harder to know what is right when the experts themselves are uncertain. So scientists will lay out scenarios A, B and C, and estimate a range of possible outcomes for each one, rather than agreeing that A is the only valid course of action. I suspect everyone would be happier if it was very clear what we had to do – even if it was painful – but instead both scientists and politicians have to weigh up pros and cons, knowing that they won’t be forgiven for a wrong decision. On top of everything else there is time pressure: if we wait for further research, it may be too late. 

The decisions that governments have to make are complex and high-stakes, and the fact that we have a Prime Minister and front bench who were selected solely because of their commitment to Brexit does not inspire confidence that they will select the right policies. But should scientific advisors be doing more, as Philip Ball suggests, to challenge government?

 It seems to me there are four different scenarios, each of which requires a different response.

1. Bad science/fake news

I agree that if a government minister were to state openly something that was scientific misinformation, then it is the job of senior advisors to challenge them publicly. I was among the first to criticise Deborah Birx early in 2020 when she said of Trump “He’s been so attentive to the scientific literature and the details and the data…. I think his ability to analyze and integrate data that comes out of his long history in business has really been a real benefit during these discussions about medical issues.”  See also here. There were also times when I thought Anthony Fauci should have just stood up and said ‘No, No, No!’ rather than sit in the same room while Trump spouted dangerous nonsense. However, bad though the situation in the UK is, I don’t think we have had anything like that situation. Certainly there are politicians and others who have tried to discredit scientific advisors as “scare-mongering”, including some who see a pandemic as a nice opportunity for a bit of disaster capitalism. But I haven’t seen government ministers making scientific claims about coronavirus that are so wildly wrong that no serious scientist could support them.

2. Policies that prove misjudged in hindsight 

Most people would agree that in the early days of the pandemic, the wrong policies were adopted, with an emphasis on handwashing and cleaning surfaces, and a severe lockdown imposed too late. Jeremy Farrar, who was on SAGE at the time, described these events in his book with Ajana Ahuja, Spike, and it is clear that he, and other SAGE members, regret that they did not consider more options, or argue for more rapid action. To that extent, the criticism may be valid, but my impression is that the advisors had significant input into the policies, which at the time they thought was the least bad course of action. There were other scientists who spoke out strongly in criticism of lock-downs, but it’s not clear that this helped the situation. I agree, it would be helpful for everyone who was involved in those decisions to reflect now about what they got wrong, but it would have been hard to do so at the time - not because they were "muted and deferential" but because they had significant input into the policies.

3. Ongoing policies 

We became accustomed during the past two years of seeing government advisers alongside the PM or another government minister at press briefings when a policy is announced. Ball says they should “openly criticise bad policy” – this would be fine if everyone could agree what good and bad policy were. The policy we get will typically be the least bad option, formed by trying to integrate scientific and political considerations. Nobody will be entirely happy with it – the question is not whether it has flaws, but whether there are better options that everyone could agree on. Once it is decided, it is the job of the scientific advisers to ensure it is clearly communicated and explained.

For instance, Ball says Chris Whitty was being timid in December 2021 when he suggested that we “prioritise the social interactions that really matter” at a press briefing when Johnson said “we’re not cancelling people’s parties”. This seems to me to be an unfair charge: the policy is at this point agreed. Is Whitty supposed to stand up at the podium and say “I think we should cancel parties.”?  I heard the December briefing and thought Whitty’s contribution provided useful clarification, in emphasising the importance of not dropping one’s guard, and restricting contact to a small circle. 

4. Cases where policies are not followed 

Ball also notes cases where rules or guidance were not followed: Dominic Cummings’ trip to Barnard Castle, MPs not wearing masks, or drinks at no. 10. He feels scientific advisors are “appeasing” government when they refuse to comment on such matters. I disagree. Their role is to help ensure that policies are based on the best scientific evidence. Where there is failure to adhere to policies, it’s the job of the police, politicians and the media to hold people to account. 

Government advisers and Independent SAGE

Concerns about lack of transparency around SAGE led to the formation of Independent SAGE. I regard this as a very useful development, that serves much of the function that Ball sees as missing in regular SAGE. They provide a wealth of information about different aspects of pandemic management, and can express their views robustly, and suggest alternatives to current policies. Perhaps one way forward would be to formalise the need for a body separate from scientific advisors that is quite independent from government.  But we do still need a group of scientists who work directly with government to provide scientific input into rapid decisions about what is needed now. 

I should make clear that I totally agree with Ball’s criticisms of the government – my sense is that most of his piece is aimed at highlighting how dishonest and incompetent they have been in responding to coronavirus. But it strikes me as unfair to extend the criticism to Whitty, Vallance and members of SAGE who have tried to steer us through the pandemic as best they can, even though that means working with this dysfunctional government. My view is that we would be in a far worse position without their input.

Linking responsibility for climate refugees to emissions

 

I followed the COP summit in Glasgow with interest, particularly via Channel 4 News, which is good at reporting minority voices. There were incredibly powerful speeches by people whose homes are going to be uninhabitable if we exceed 1.5 degrees warming - and indeed by some for whom extreme weather has already caused famine. For examples see speeches by Elizabeth Wathuti, youth climate activist from Kenya, Mia Mottley, Prime Minister of Barbados, and the interview with Shauna Aminath, minister of environment for the Maldives, and Seve Paeniu, Tuvalu’s minister for finance and climate change. 

The plan at COP seemed to focus on decelerating warming in order to, as far as possible, prevent this happening. But it's pretty clear that, even in the unlikely event that we hold warming to 1.5 degrees, there will be enormous numbers of people who will become climate refugees. People don't want to be forced out of their homes. It makes far more sense to adopt strategies that will avoid that than to plan for mass migrations. But sadly, human beings have a habit of not believing the worse until it is upon them. So I think we're going to have mass migrations and I'm worried nobody seems to be planning for that.

The reaction of many wealthy countries to refugees has been to keep them out at all costs and hope they go away. People fleeing their homelands are regarded like toxic waste: a problem to be exported. That pretty much sums up the attitude of the current UK government, who have at various times suggested that asylum seekers be sent to Ascension Island, confined on ship hulks  and, most recently, and to the surprise of the Albanians, sent to Albania. But this problem is not going to go away. We will see increasing numbers of people roaming around desperately trying to find somewhere to live. Quite apart from the humanitarian disaster for the refugees themselves, this will create social unrest and conflict for those in wealthy countries unless it is planned for. Germany has shown that it is possible to integrate large numbers of refugees, and that they can contribute positively to a society if given support.

So, as well as anticipating the impacts of global heating on the physical environment, we need to be planning for mass migrations of people who have lost their homeland. We will need a global strategy, where everyone plays their part. And it should be those countries who are are doing least to curb global emissions who should step up to play their part in taking in refugees. 

So I suggest that the next COP summit should aim for a more direct linkage between emissions generated and responsibility for refugees: the more you emit, the more you have to commit to giving citizenship to those who have been rendered homeless by climate change. Note I am saying citizenship, not temporary accommodation in a tent in the desert (or a ship hulk for that matter).

Heads of state who genuinely don't believe that things are that bad, and that it's okay to continue to generate emissions, will find this an easy commitment to make, as it would only kick in when a homeland has become uninhabitable; it should be possible to devise a formal criterion for this: e.g. the place is under water, or experiencing temperatures of 40 degrees or more over a sustained period.

Take, for example, Australia, whose PM, Scott Morrison, tries to blag his way out of a crisis and thinks it's fine to continue mining and exporting fossil fuels. It's ironic that Australia's solution to refugees has previously been to send them off to be imprisoned on a Pacific Island. That sort of solution may no longer be available in future, as Nauru succumbs to rising sea levels.

Given the major role that Australia has played in causing global warming, wouldn't it make sense if at the next COP it was required to step up to its responsibilities and undertake to award citizenship to Pacific Islanders whose homes will be submerged in a few decades? The greater the emissions, the higher the quota of people. And who knows, if it were put to them like that, they might even start taking the phasing out of fossil fuels more seriously?

Universities vs Elsevier: who has the upper hand?

 

The academic publisher Elsevier is currently negotiating a deal with UK universities. In Oxford, as in other universities, there have been extensive discussions about the proposed deal; the goals are to reduce costs to sustainable levels and to provide full and immediate open access to UK research. I have a nasty feeling that we could end up in the situation of those at the COP summit: with a deal that where the publishers feel they are giving away a huge amount, while the consumers are still unsatisfied. 

In the print era, publishers already had a large profit margin.  I was a journal editor when electronic publishing first came in, and I remember discussions with the publisher - who was clearly very nervous about how this might damage their bottom line. With some clever business practices, such as bundling, they managed to keep going, making more rather than less money.

Other new developments - the advance of metrics and requirements for open access publishing - also pose challenges to them, but their response, like any savvy business, is to take control of these new developments and find a way to profit from them as well.

So should we go along with this? The discussions have largely centred around money, and this is a real concern. Publishers charge massively inflated subscription charges, taking as profit money that our libraries could otherwise put to good use. But the problems go way beyond that, including blocking open access to authors' own work, and giving poor quality and even fraudulent material a veneer of respectability. The publisher currently has considerable power over what gets published but takes little responsibility when things go wrong.

But we also have power, and I think it's time we started to wield it. I have two proposals: one radical and the other less so, both of which go beyond what is currently being considered in the JISC discussions.

The radical option: move publishing in-house  

In an ideal world, we would not have any deals with for-profit publishers. Universities would take control of academic publishing. A model which I have found works well is the Wellcome Trust journal Wellcome Open Research (WOR). I served on its Editorial Advisory Board in the first years of its operation and I have published eight papers in WOR. 

Quality control is maintained by requiring that work published in WOR has been Wellcome-Trust funded. The platform is paid for by the Wellcome Trust, and all material is Open Access, with no charge to authors.

Universities could use this as a model for developing their own platforms for publishing work of their researchers. As with Wellcome Trust, researchers would be encouraged rather than required to publish there. Setting up and maintaining the platform would cost money, but this might be covered by savings from stepping back from deals with big publishers. This is not a perfect solution, of course; we would still want access to past material published by traditional publishers. But I feel this option should be given serious consideration.

The less radical option. An agreement with Elsevier that reflects what we want and need 

If we are to retain a working relationship with Elsevier, then we need to ensure that we are getting value for money, and that they are delivering what we want.

One way in which they fail to do this is through the copyright agreements that they require authors to sign. I agree with Sally Rumsey that any publisher that does not allow an author to retain rights to their own work is problematic. Furthermore, there is concern that even if Elsevier were to say they agreed to author rights retention, they would continue to adopt practices that would mislead authors into signing away their rights.

Another issue is how publishers deal with matters related to misbehaviour by editors and authors. We have a number of serious problems with the academic publishing system that have been developing for years, and have been thrown into stark relief by the pandemic. Publications in journals are a kind of academic currency, with potentially huge rewards in terms of promotion, pay and tenure of academic staff. Unfortunately, this means that some people will go to great lengths to procure publications, and this can include fraudulent means. A reputable publisher should recognise and indeed anticipate such problems and take steps to deal with them. Elsevier has been notably poor in doing so. I will give a handful of examples.

Clearly erroneous/fraudulent work remains in the literature In recent years, a massive problem has been discovered with so-called paper mills - a form of industrialised cheating that involves generation of literally thousands of fake papers.  This phenomenon was discussed at a meeting this summer, the Computational Research Integrity Conference (CRI-Conf), which brought together some of the scientists who uncovered paper mills and representatives of publishers (including Elsevier) and university integrity officers. A repeated complaint of the 'data sleuths' (scientists who uncovered the paper mills) was that when the problems were drawn to the attention of the editors or publishers, typically nothing was done.  Some of these papers are highly cited and have medical implications - see, for instance work by Jennifer Byrne and Cyril Labbé on cancer biology, or by Elisabeth Bik on image manipulations.

To its credit, Elsevier was recently cited as acting promptly when a cache of nonsense papers in special issues was uncovered, but one still has to ask how these papers, some of which were complete gibberish, originally got through the supposedly rigorous peer-review process. For another recent example, see here.

Part of the problem seems to be that there is very little oversight of editors.  I first became aware of Elsevier's lax attitude to research integrity when I discovered a ring of editors. There was overwhelming evidence that a set of editors were publishing one another's papers in their journals, with publication lags so brief as to indicate there had not been peer review. Emails to Elsevier received no reply, and so I started reporting the events on my blog. This created such publicity that Elsevier then posted a reply, but initially they denied that any papers had been published without peer review. Eventually, as the evidence mounted up, they were forced to acknowledge the problem, and claimed to be investigating it, but at no point did they communicate with me, and their investigation went on for years and has never led to any consequences. One of the offending editors, Johnny Matson, quietly disappeared without comment and is now editing another journal for another publisher. The unreviewed papers, many of which are concerned with intervention for children with developmental disorders, remain in the literature. 

More recently I did a bibliometric analysis of editors of psychology journals who publish prolifically in their own journals, and broke down the results by publisher. Elsevier had the highest number of editors-in-chief in this category,

A related case in another field is that of Didier Raoult, who was senior author on a paper advocating hydroxychloroquine as a treatment for Covid-19 that appeared in 2020 in an Elsevier journal for which he was Editor-in-Chief. It has subsequently been shown to be riddled with methodological errors, but, although Elsevier claimed to be investigating it in April 2020, it has not been retracted.

At the CRI-Conf publishers said that they were doing their best, but the problem of fake and fraudulent practices was so huge they could not be expected to tackle all of it. Yet we know publishers make enormous profits, so why aren't they putting more effort into checking the integrity of the work they publish - and dealing promptly with the problems that are drawn to their attention? Elsevier is not the only large publisher to be affected, but if it wants to be seen as a leader in the field, it should recognise that integrity of published material is of supreme importance - especially in clinical fields where people's health and wellbeing can be affected (as in the cases of Raoult and Matson noted above).

If UK Universities do strike a deal with Elsevier, then they should take the opportunity to include in any future contract conditions requirements that: 

(a) Elsevier allows authors the retention of the rights to their publications and  

(b) it provides annually an open report on the number of integrity issues that have been raised about its publications and how these have been dealt with. 

This would go some way toward ensuring that we get value for money, and are not simply adding to the profits of a company whose underlying ethos is not aligned with promoting scientific quality.

Universities have historically been extremely meek in accepting the terms and conditions imposed by big publishers, who have in turn treated academics as a cash cow. We on the one hand produce for free the product that they sell, and then pay to publish and read it. They cannot survive without the research created in Universities: we should start recognising that we need not bow down to their demands.

DISCLAIMER: I am writing in a personal capacity and not as a representative of the University of Oxford.

14/11/21

P.S. Some further examples of integrity problems with Elsevier have been pointed out to me on Twitter:

Geoff Frampton (@Geoff_Frampton) noted 'Elsevier had the largest market share among the publishers of retracted Covid-19 articles in our study' 

Patricia Murray (@PMurray_65) tweeted: Elsevier have failed to retract this 2008 Lancet paper that is known to be fraudulent and poses a risk to patients: https://thelancet.com/journals/lancet/article/PIIS0140-6736(08)61598-6/references… The paper stated the so-called 'bioengineered' airway had "a normal appearance and mechanical properties at 4 months." 

It is particularly shocking that the publisher is still making this article available (at a cost of $31) given that the author had a prison sentence for faked research, and conducted unethical experiments on seriously ill patients: see this report. 

PPS - it gets worse. There have been strenuous attempts to get the journal to retract the paper, to no avail. This has made the national news in the UK.

 

Comments on this blog are welcome, but may only appear after a delay as they are moderated to avoid spam. 

Open data: We know what's needed - now let's make it happen

©Cartoonstock.com

 

The week after then end of University term and before the Easter break is choc-a-bloc with conferences and meetings. A great thing about them all being virtual is that you can go to far more. My brain is zinging (and my to-do list sadly neglected) after 3 days of the Computational Research Integrity Conference (CRI-Conf) hosted at Syracuse University, and then a 1.5 hr webinar organised by the Center for Biomedical Research Transparency (CBMRT) on Accelerating Science in the Age of COVID-19. Synthesising all the discussion in my head gives one clear message: we need to get a grip on failure of authors to make their data, and analysis scripts, open. 

For a great overview of CRI-Conf, see this blog by one of the presenters, Debora Weber-Wulff . The meeting focused mainly on fraud, and the development of computational methods to detect it, with most emphasis on doctored images. There were presentations by those who detect manipulated data (notably Elisabeth Bik and Jennifer Byrne), by technology experts developing means of automating image analysis, publishers and research integrity officers who attempt to deal with the problem, and by those who have created less conventional means to counteract the tide of dodgy work (Boris Barbour from PubPeer, and Ivan Oransky from Retraction Watch). The shocking thing was the discovery that fabricated data in the literature is not down to a few bad apples: there are "paper mills" that generate papers for sale, which are readily snapped up by those who need them for professional advancement.

CRI-Conf brought together people who viewed the problem from very different perspectives, with a definite tension between those representing "the system" - publishers, institutions and funders - and those on the outside - the data sleuths, PubPeer, Retraction Watch. The latter are impatient at the failure of the system to act promptly to remove fraudulent papers from the literature; the former respond that they are doing a lot already, but the problem is immense and due process must be followed. There was, however, one point of agreement. Life would be easier for all of us if data were routinely published with papers. Research integrity officers in particular noted that a great deal of time in investigations is spent tracking down data.

The CBMRT webinar yesterday was particularly focused on the immense amount of research that has been generated by the COVID-19 pandemic. Ida Sim from Vivli noted that only 70 of 924 authors of COVID-19 clinical trials agreed to share their data within 6 months. John Inglis, co-founder of biorXiv and medrXiv cited Marty Lakary's summary of preprints: "a great disruptor of a clunky, slow system never designed for a pandemic". Deborah Dixon from Oxford University Press noted how open science assumed particular importance in the pandemic: open data not only make it possible to check a study's findings, but also can be used fruitfully as secondary data for new analyses. Finally, a clinician's perspective was provided by Sandra Petty. Her view is that there are too many small underpowered studies: we need large collaborative trials. My favourite quote: "Noise from smaller studies rapidly released to the public domain can create a public health issue in itself".

Everyone agreed that open data could be a game-changer, but clearly it was still the exception rather than the rule. I asked whether it should be made mandatory, not just for journal articles but also for preprints. The replies were not encouraging. Ida Sim, who understands the issues around data-sharing better than most, noted that there were numerous barriers - there may be legal hurdles to overcome, and those doing trials may not have the expertise, let alone the time, to get their data into appropriate format. John Inglis noted it would be difficult for moderators of preprint servers to implement a data-sharing requirement for pre-prints, and that many authors would find it challenging.

I am not, however, convinced. It's clear that there is a tsunami of research on COVID-19, much of it of very poor quality. This is creating problems for journals, reviewers, and for readers, who have to sift through a mountain of papers to try and extract a signal from the noise. Setting the bar for publication - or indeed pre-prints - higher, so that the literature only contains papers that can be (a) checked and (b) used for meta-analysis and secondary studies, would reduce the mountain to a hillock and allow us to focus on the serious stuff. 

The pandemic has indeed accelerated the pace of research, but things done in a hurry are more likely to contain errors, so it is more important than ever to be able to check findings, rather than just trusting authors to get it right. I'm thinking of the recent example where an apparent excess of COVID-19 in toddlers was found to be due to restricting age to a 2-digit number, so someone aged 102 would be listed as a 2-year-old.  We may be aghast, but I feel "there but for the grace of God go I". Unintentional errors are everywhere, and when the stakes are as high as they are now, we need to be able to check and double-check findings studies that are going to translated into clinical practice. That means sharing analysis code as well as data. As Philip Stark memorably said, "Science should be 'show me', not 'trust me'".

All in all, my sense is that we still have a long way to go before we shift the balance in publishing from a focus on the needs of authors (to get papers out rapidly) to an emphasis on users of research, for whom the integrity of the science is paramount. As Besançon et al put it: Open Science saves lives.

Time for publishers to consider the rights of readers as well as authors

 

© cartoonstock.com

I've just been reading this piece entitled: "Publication ethics: Barriers in resolving figure corrections" by Lataisia Jones, on the website of the American Society for Microbiology, which publishes several journals.  Microbiology is a long way from my expertise and interests, but I have been following the work of Elisabeth Bik, datasleuth extraordinaire, for some time - see here. As Bik points out, the responses (or more often lack of response) she gets when she raises concerns about papers are similar to those seen in other fields where whistleblowers try to flag up errors  (e.g. this Australian example). 

It's clear that there are barriers to correcting the scientific record when errors are identified, and so I was pleased to see a piece tackling this head-on, which attempts to explain why responses by journals and publishers often appear to be so slow and unsatisfactory. However, I felt the post, missed some key points that need to be taken seriously by publishers and editors. 

The post starts by saying that: "Most figure concerns are created out of error and may present themselves in the form of image duplication, splicing and various figure enhancements." I think we need to have that "most" clarified in the form of a percentage. Yes, of course, we all make mistakes, but many of the issues flagged up by Bik are not the kinds of error made by someone going "oops" as they prepare their figures. I felt that on the one hand it is crucial to be aware that many papers are flawed because they contain honest errors, but that fact should not lead us to conclude that most cases of problematic images are of this kind. At least, not until there is hard evidence on that point. 

The post goes on to document the stages that are gone through when an error has been flagged up, noting in particular these guidelines produced by the Committee on Publication Ethics (COPE). First, the author is contacted. "Since human error is a common reason behind published figure concerns, ASM remains mindful and vigilant while investigating to prevent unnecessarily tarnishing a researcher’s reputation. Oftentimes, the concern does not proceed past the authors, who tend to be extremely responsive." So here again, Jones emphasises human error as a "common reason" for mistakes in figures, and also describes authors as "extremely responsive". And here again, I suggest some stastistics on both points would be of considerable interest. 

Jones explains that this preliminary step may take a long time when several years have elapsed between publication and the flagging up of concerns. The authors may be hard to contact, and the data may no longer be available. Assuming the authors give a satisfactory response, what happens next depends on whether the error can be corrected without changing the basic results or conclusions. If so, then a Correction is published. Interestingly, Jones says nothing about what happens if an honest error does change the basic results or conclusions. I think many readers would agree that in that case there should be a retraction, but I sense a reluctance to accept that, perhaps because Jones appears to identify retraction with malpractice. 

She describes the procedure followed by ASM if the authors do not have a satisfactory response: the problem is passed on to the authors' institution for investigation. As Jones points out, this can be an extended process, as it may require identification of old data, and turn into an inquiry into possible malpractice. Such enquiries often move slowly because the committee members responsible for this work are doing their investigations on top of their regular job. And, as Jones notes: "Additionally, multiple figure concerns and multiple papers take longer to address and recovering the original data files could take months alone." So, the institution feeds back its conclusions (typically after months or possibly years), which may return us to the point where it is decided a Correction is appropriate. But, "If the figure concerns are determined to have been made intentionally or through knowingly manipulating the data, the funding agencies are notified." And yet another investigation starts up, adding a few more months or years to the process. 

So my reading of this is that if the decision to make a Correction is not reached, the publisher and journal at this point hand all responsibility over to other agencies - the institution and the funders. The post by Jones at no point mentions the conditions that need to be met for the paper to actually be retracted (in which case it remains in the public domain but with a retraction notice) or withdrawn (in which case it is removed). Indeed, the word 'retract' does not appear at all in her piece. 

What else is missing from all of this? Any sense of responsibility to other researchers and the general public. A peer-reviewed published article is widely regarded as a credible piece of work. It may be built on by other researchers, who assume they can trust the findings. Its results may be used to inform treatment of patients or, in other fields, public policy. Leaving an erroneous piece of work in a peer-reviewed journal without any indication that concerns have been raised is rather like leaving a plate of cookies out for public consumption, when you know they may be contaminated. 

Ethical judgements by publishers need to consider their readers, as well as their authors. I would suggest they should give particularly high priority to published articles where concerns have not been adequately addressed by authors, and which also have been cited by others. The more citations, the greater the urgency to act, as citations spawn citations, with the work achieving canonical status in some cases. In addition, if there are multiple papers by the same author with concerns, surely this should be treated as a smoking gun, rather than an excuse for why it takes years to act.

It should not be necessary to wait until institutions and funders have completed investigations into possible malpractice. Malpractice is actually a separate issue here: the key point for readers of the journal is whether the published record is accurate. If it is inaccurate - either due to honest error or malpractice - the work should be retracted, and there is plenty of precedent for retraction notices to specify the reason for retraction. This also applies to the situation where there is a realistic concern about the work (such as manipulated figures or internally inconsistent data) and the author cannot produce the raw data that would allow for the error to be identified and corrected. In short, it should be up to the author to ensure that the work is transparent and reproducible. Retaining erroneous work in a journal is not a neutral act. It pollutes the scientific literature and ignores the rights of readers not to be misled or misinformed.