Problems with ELife's new article type: Replication studies

 

I was interested to receive an email from eLife last week, telling me that "As part of our commitment to open science, scientific rigour and transparency we now accept submissions of Replication Studies". Sounds good, I thought, but on reading further I became increasingly dismayed. I think the way this is set up dooms it to failure.

Back in 2023, when eLife created a storm by altering its publishing model, I argued that they would not achieve their aims unless they changed the basis for selecting articles for peer review.  Although eLife has abandoned the traditional binary outcome where articles are accepted or rejected, there is still a decision delegated to editors, which is whether the article is selected for peer review. My suggestion was that they should adopt results-blind selection. This avoids the bias that favours articles with positive results. Publication bias leaves well-conducted studies with null results to languish unpublished. This matters because a cumulative science should include negative as well as positive findings. Nissen et al (2016) showed how publication bias leads to what they termed the "canonization of false facts". It is a universal phenomenon and I doubt that eLife editors are immune to it.

One method of avoiding publication bias is the Registered Reports format, whereby the introduction, methods and analysis plan are peer reviewed before data are collected, with the article accepted in principle by a journal, provided the researchers did the study as planned, or gave adequate reasons for deviating from protocol. In my experience, biomedical researchers are highly resistant to Registered Reports; they tell me the approach is incompatible with how they work, which involves development of ideas and methods in the course of doing a study 🙄. However, that argument does not hold for replication studies, where the idea is to reproduce the methods of an existing published study. Indeed, eLife took a pioneering stand in hosting the Reproducibility Project on Cancer Biology (Errington et al., 2021a). So it is disappointing that their current proposal is for a much more timid approach.

First of all, it sounds as if the plan is for individuals to complete a replication study before submitting  it to eLife. This means that we'll be up against publication bias all over again - the likelihood of an editor selecting a study for peer review will depend not on the strength and suitability of methods but on whether the replication was "successful".

Second the eLife instructions state: "The authors should work closely with the authors of the original study and summarize their interactions with the original authors as part of a cover letter". It seems entirely appropriate to liaise with original authors to ensure that materials and methods are suitable for replication. However, we know from the Cancer Biology Replication project that many authors were unresponsive or even obstructive when asked to give advice on a replication study (Errington et al, 2021b). Many attempts at replication failed because it was impossible to work out exactly what had been done in the original study or because authors would not share materials. In effect, then, the current eLife approach to replication studies gives original authors the ability to veto any replication attempt.

I can't think who on earth would take the risk of doing a replication study under these conditions. Many scientists already regard replication as an inferior type of research activity, for which replicators get little credit - and potential abuse. Furthermore, it is difficult to get funds for replication studies, because they are deemed insufficiently novel. Yet, previous large-scale replication studies have found that direct replications often fail to find original effects, or replicate the effect but with a smaller effect size. So you make yourself unpopular by attempting a replication, and then when you come to try to submit it to eLife you are told it won't be peer reviewed because you obtained a null effect.

I think that we won't get replication studies in biosciences unless they are explicitly incentivised - and judged on their methodological quality rather than their results. Meanwhile, because of the field's obsession with novelty, research progress stalls as people keep trying to build on results without knowing whether they provide a solid foundation.

My prediction: give it a year, and see how many Replication Studies have been accepted for peer review in eLife. I'll be surprised if it's more than zero.

References

Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021a). Investigating the replicability of preclinical cancer biology. eLife, 10, e71601. https://doi.org/10.7554/eLife.71601

Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Reproducibility in Cancer Biology: Challenges for assessing replicability in preclinical cancer biology | eLife. eLife. https://doi.org/10.7554/eLife.67995

Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016). Publication bias and the canonization of false facts. eLife, 5, e21451. https://doi.org/10.7554/eLife.21451

Book Review: Unreliable: Bias, Fraud, and the Reproducibility Crisis in Biomedical Research

by Csaba Szabo, Columbia University Press, 2025 

This is a rollicking good read, written in an informal style, and enlivened by cartoons, which works as a scholarly and accessible account of the so-called reproducibility crisis in biomedical research. 

I first became aware of this book back in February 2024 when the publisher asked me to review the draft. By happy coincidence I had just submitted an (ultimately unsuccessful) application for funding for a meeting on the closely-related topic of research fraud, and as I devoured the text, I felt guilty that I had not been aware of Szabo's work. As it turned out, there was a good reason for my ignorance: he had not previously written anything on this topic. As he explains in the Afterword, there is a personal backstory: 

The journey of a starryeyed young scientist entering the field of science, and continuing with the scientist working hard and hopefully contributing to the field over thirty years, but during all this time gradually realizing that the entire system suffers from major problems. And now that same scientist—not so young anymore, unfortunately—has written a book concluding that about 30 percent of the papers that come out every year are fake garbage and that 70 to 90 percent of the published scientific literature is not reproducible. 

That is a startling statement, but Szabo speaks with authority, as one who has always loved science, and has had a long and distinguished career in biomedical research both in the US and in Europe. It is clear that he does not want to attack science: as he points out, it is the only game in town. But he is dismayed at how the scientific method has been degraded, and he is concerned that nobody in power is taking responsibility for cleaning it up. 

What makes the book unlike any other on this topic is the detailed account of how we got into the current state, and what are the barriers to remedying the situation. Szabo spends some time explaining how hypercompetition for research grants drives the behaviour of researchers. Although it is customary to talk of "publish or perish", Szabo argues that the real crunch point for a biomedical scientist is success in obtaining external grants. In the USA, this usually means NIH funding in the form of a R01 grant, typically around $1 million over a 4-5 year period. While that sounds like a lot of money, it has to cover 50% of the salary of the principal investigator as well as other salaries and research supplies, which means it is not enough to support a research group. The success rate is around 20%, and researchers typically need to submit numerous proposals in order to survive. The institutions want their staff to obtain grants, not just so they can bathe in the reflected glory of impressive research results, but also because grants bring overheads in the form of indirect costs. So getting grants is extremely high-stakes.

It gets even more interesting when Szabo documents his experiences as a grant reviewer for NIH study sections. We might hope that the most successful proposals are the ones that are realistic, avoid hype, and carefully document the reliability of their methods. Alas, this is the opposite of what happens. Such is the pressure for novelty and impact, that anyone who proposed to replicate a prior finding would be quickly triaged out of the competition. I've noted similar tendencies in the UK context. Thus, the message researchers get from institutions is, if you want to keep your job, get a grant, and the message they get from funders is, if you want to get a grant, concentrate on making the research look exciting.

Szabo next moves on to discuss the way science is done in the lab. There are numerous factors that conspire to make findings irreproducible. Some of these are related to the inherent variability of biological systems, but some arise because of failure to adopt experimental designs that adequately control for bias. Typically, doing things meticulously takes time, and the pressure is to get results out fast. Furthermore, many studies involve a mixture of complicated methods, and the principal investigator may not understand all of them. When it comes to analysing the results, there is huge scope for adopting methods such as post-hoc outlier exclusion, p-hacking, and HARKing. All of these can be used to squeeze positive findings out of an unpromising dataset. If biomedical science is like psychology, many researchers regard such methods as normative and are unaware just how much they contribute to lack of reproducibility of published results. 

The next chapter takes a darker turn, moving to intentional fraud. A number of high-profile cases are reviewed, as well as the industrial-scale fraudulent operations run by so-called paper mills. This chapter is particularly depressing as Szabo takes us through the analogies that have been used to characterise fraud.  Initially, fraud was seen as rare, due to a few "bad apples"; later it was compared to an iceberg of fraudulent work, where we only see the tip but need to be aware that much is hidden. Szabo writes: 

In my view, even this analogy is severely misleading. If we want to stay with nature analogies, my feeling is that we are dealing with a big scientific swamp, with various swamp creatures of different sizes and shapes living in it. There are some relatively clean areas of water, too, and there are regular life forms as well. But there are an awful lot of swamp creatures who happily coexist in their natural environment, taking away food and resources from the regular life forms. In addition, a whole ecosystem built around the swamp is benefiting from it. The people who are supposed to manage the swamp, or perhaps drain it, are nowhere to be found. 

One group of people who might be expected to manage the swamp are those who publish research papers, but Szabo does not find them equal to the task, instead talking of "A broken scientific publishing system". Regular readers of this blog will be well-acquainted with the phenomenon whereby someone reports an obvious problem with a published paper, only to be ignored. Academic publishers are making efforts to screen new submissions for plagiarism and image manipulation, but there seems little appetite for cleaning up the existing body of scientific literature. Until that is done, we cannot regard it as a foundation for future work. 

Szabo is impressed by the efforts of "data sleuths", who perform post-publication peer review and report problems on the PubPeer website, but he regards this as unsustainable: and cleaning up the literature should not be a task for volunteers.  It seems that everyone wants someone to "do something" to fix the problem, but nobody takes it on. Organisations with some responsibility include universities, research institutes, publishers, editors and funders. Szabo's recommendations for change focus on funders, who have the power to deny funding to those who fail to take steps to ensure that their results are reliable. And ultimately, the money for research comes from taxpayers, and governments call the shots. 

This is a particularly difficult time to be conveying such a message. The only people who might be overjoyed to hear that a high proportion of published research is unreliable are politicians who are antagonistic to science and would like an excuse to defund it. In the USA, cuts to funding have been so fast and so deep that many are fearful that the science base may not recover. The swamp creatures may die, but so will the regular life forms. We urgently need therefore to look seriously at recommendations for changing how the system works at all levels - laboratory practice, funding, institutional integrity investigations, publishing, incentive structures - so that we can not only have confidence in scientific findings, but also defend science against attacks. 

I don't agree with all of Szabo's recommendations, but it is refreshing to have someone take a deep dive into the topic, and his ideas form a good basis for discussion. One point where I have a different approach concerns the emphasis on replications. There are many people arguing that more funding should be directed towards replicating prior studies. In the short term, that will be needed, because the way research has been done means we don't know which findings are solid. But in many areas it takes a large amount of time and money to replicate a study. The whole point of the statistical and experimental methods used in science is that they should allow us to assign a level of confidence in our findings without needing to perform an explicit replication. The problem is that we have misapplied those methods. The Registered Reports approach, where a study is evaluated by reviewers and accepted or rejected by a journal on the basis of introduction, methods and analysis plan, before any data are gathered, gets rid of the biases due to p-hacking, HARKing and publication bias, making it possible to interpret statistics sensibly. It also leads to improved methods overall, because independent reviewers offer feedback at a point when it can be helpful. As far as I know, the Registered Reports model has not been adopted by biomedicine, but could, I think, transform the field to make it more rigorous. 

Finally, I'm pleased to say that despite the initial rejection, I managed eventually to secure funding for that meeting on research fraud, which will be held in Oxford from 7th-9th April 2025, and will provide a great opportunity for discussing these issues. Registration is open for a few more days, so please consider attending (online or in person) if you'd like to take part. More details and registration form here: (turn off VPN if it does not load).

Reproducible practices are the future for early career researchers

This post was prompted by an interesting exchange on Twitter with Brent Roberts (@BrentWRoberts) yesterday. Brent had recently posted a piece about the difficulty of bringing about change to improve reproducibility in psychology, and this had led to some discussion about what could be done to move things forward. Matt Motyl (@mattmotyl) tweeted:

I had one colleague tell me that sharing data/scripts is "too high a bar" and that I am wrong for insisting all students who work w me do it

And Brent agreed:

We were recently told that teaching our students to pre-register, do power analysis, and replicate was "undermining" careers.

Now, as a co-author of a manifesto for reproducible science, this kind of thing makes me pretty cross, and so I weighed in, demanding to know who was issuing such rubbish advice. Brent patiently explained that most of his colleagues take this view and are skeptics, agnostics or just naïve about the need to tackle reproducibility. I said that was just shafting the next generation, but Brent replied:

Not as long as the incentive structure remains the same.  In these conditions they are helping their students.

So things have got to the point where I need more than 140 characters to make my case. I should stress that I recognise that Brent is one of the good guys, who is trying to make a difference. But I think he is way too pessimistic about the rate of progress, and far from 'helping' their students, the people who resist change are badly damaging them.  So here are my reasons.

1.     The incentive structure really is changing. The main drivers are funders, who are alarmed that they might be spending their precious funds on results that are not solid. In the UK, funders (Wellcome Trust and Research Councils) were behind a high profile symposium on Reproducibility, and subsequently have issued statements on the topic and started working to change policies and to ensure their panel members are aware of the issues. One council, the BBSRC, funded an Advanced Workshop on Reproducible Methods this April. In the US, NIH has been at the forefront of initiatives to improve reproducibility. In Germany, Open Science is high on the agenda.

2.     Some institutions are coming on board. They react more slowly than funders, but where funders lead, they will follow. Some nice examples of institution-wide initiatives toward open, reproducible science come from the Montreal Neurological Institute and the Cambridge MRC Cognition and Brain Sciences Unit. In my own department, Experimental Psychology at the University of Oxford, our Head of Department has encouraged me to hold a one-day workshop on reproducibility later this year, saying she wants our department to be at the forefront of improving psychological science.

3.     Some of the best arguments for working reproducibly have been made by Florian Markowetz. You can read about them on this blog, see him give a very entertaining talk on the topic here, or read the published paper here. So there is no escape. I won't repeat his arguments here, as he makes them better than I could, but his basic point is that you don't need to do reproducible research for ideological reasons: there are many selfish arguments for adopting this approach – in the long run it makes your life very much easier.

4.     One point Florian doesn't cover is pre-registration of studies. The idea of a 'registered report', where your paper is evaluated, and potentially accepted for publication, on basis of introduction and methods was introduced with the goal of improving science by removing publication bias, p-hacking and HARKing (hypothesising after results are known). You can read about it in these slides by Chris Chambers. But when I tried this with a graduate student, Hannah Hobson, I realised there were other huge benefits. Many people worry that pre-registration slows you down. It does at the planning stage, but you more than compensate for that by the time saved once you have completed the study. Plus you get reviewer comments at a point in the research process when they are actually useful – i.e. before you have embarked on data collection. See this blogpost for my personal experience of this.

5.     Another advantage of registered reports is that publication does not depend on getting a positive result. This starts to look very appealing to the hapless early career researcher who keeps running experiments that don't 'work'. Some people imagine that this means the literature will become full of boring registered reports with null findings that nobody is interested in. But because that would be a danger, journals who offer registered reports impose a high bar on papers they accept – basically, the usual requirement is that the study is powered at 90%, so that we can be reasonably confident that a negative result is really a null finding, and not just a type II error. But if you are willing to put in the work to do a well-powered study, and the protocol passes scrutiny of reviewers, you are virtually guaranteed a publication.

6.     If you don't have time or inclination to go the whole hog with a registered report, there are still advantages to pre-registering a study, i.e. depositing a detailed, time-stamped protocol in a public archive. You still get the benefits of establishing priority of an idea, as well as avoiding publication bias, p-hacking, etc. And you can even benefit financially: the Open Science Framework is running a pre-registration challenge – they are giving $1000 to the first 1000 entrants who succeed in publishing a pre-registered study in a peer-reviewed journal.

7.     The final advantage of adopting reproducible and open science practices is that it is good for science. Florian Markowetz does not dwell long on the argument that it is 'the right thing to do', because he can see that it has as much appeal as being told to give up drinking and stop eating Dunkin Donuts for the sake of your health. He wants to dispel the idea that those who embrace reproducibility are some kind of altruistic idealists who are prepared to sacrifice their careers to improve science. Given arguments 1-6, he is quite right. You don't need to be idealistic to be motivated to adopt reproducible practices. But it is nice when one's selfish ambitions can be aligned with the good of the field. Indeed, I'd go further and suggest that I've long suspected that this may relate to the growing rates of mental health problems among graduate students and postdocs: many people who go into science start out with high ideals, but are made to feel they have to choose between doing things properly vs. succeeding by cutting corners, over-hyping findings, or telling fairy tales in grant proposals. The reproducibility agenda provides a way of continuing to do science without feeling bad about yourself.

Brent and Matt are right that we have a problem with the current generation of established academic psychologists, who are either hostile to or unaware of the reproducibility agenda.  When I give talks on this topic, I get instant recognition of the issues by early career researchers in the audience, whereas older people can be less receptive. But what we are seeing here is 'survivor bias'. Those who are in jobs managed to succeed by sticking to the status quo, and so see no need for change. But the need for change is all too apparent to the early career researcher who has wasted two years of their life trying to build on a finding that turns out to be a type I error from an underpowered, p-hacked study. My advice to the latter is don't let yourself be scared by dire warnings of the perils of working reproducibly. Times really are changing and if you take heed now, you will be ahead of the curve.

Better control of the publication time-line: A further benefit of Registered Reports

I’ve blogged previously about waste in science. There are numerous studies that are completed but never see the light of day. When I wrote about this previously, I focused on issues such as reluctance of journals to publish null results, and the problem of writing up a study while applying for the next new grant. But here I want to focus on another factor: the protracted and unpredictable process of peer review that can lead to researchers to just give up on a paper.

Sample Gantt chart. Source: http://www.crp.kk.usm.my/pages/jepem.htm

The sample Gantt chart above nicely illustrates a typical scenario.  Let's suppose we have a postdoc with 30 months’ funding. Amazingly, she is not held up by patient recruitment issues, or ethics approvals, and everything goes according to plan, so 24 months in, she writes up the study and submits it to a journal. At the same time, she may be applying for further funding or positions. She may plan to start a family at the end of her fellowship. Depending on her area of study it may take anything from two weeks to six months to hear back from the journal*. The decision is likely to be revise and resubmit. If she’s lucky, she’ll be able to do the revisions and get the paper accepted to coincide with the end of her fellowship.  All too often, though, the reviewers suggest revisions. If she's very unlucky they may demand additional experiments, which she has no funding for.  If they just want changes to the text, that's usually do-able, but often they will suggest further analyses that take time, and she may only get to the point of resubmitting the manuscript when her money runs out. Then the odds are that the paper will go back to the reviewers – or even to new reviewers – who now have further ideas of how the paper can be improved. But now our researcher might have started a new job, have just given birth, or be unemployed and desperately applying for further funds.

The thing about this scenario, which will be all too familiar to seasoned researchers (see a nice example here), is that it is totally unpredictable. Your paper may be accepted quickly, or it may get endlessly delayed. The demands of the reviewers may involve another six month’s work on the paper, at a point when the researcher just doesn’t have the time. I’ve seen dedicated, hardworking, enthusiastic young researchers completely ground down by this situation, faced by the choice of either abandoning a project that has consumed a huge amount of energy and money, or somehow creating time out of thin air. It’s particularly harsh on those who are naturally careful and obsessive, who will be unhappy at the idea of doing a quick and dirty fix to just get the paper out. That paper which started out as their pride and joy, representing their best efforts over a period of years is now reduced to a millstone around the neck.

But there is an alternative. I’ve recently, with a graduate student, Hannah Hobson, put my toe in the waters of Registered Reports, with a paper submitted to Cortex looking at an electrophysiological phenomenon known as mu suppression. The key difference from the normal publication route is that the paper is reviewed before the study is conducted, on the basis of an introduction and protocol detailing the methods and analysis plan. This, of course takes time – reviewing always does. But if and when the paper is approved by reviewers, it is provisionally accepted for publication, provided the researchers do what they said they would.

One advantage of this process is that, after you have provisional acceptance of the submission, the timing is largely under your own control. Before the study is done, the introduction and methods are already written up, and so once the study is done, you just add the results and discussion. You are not prohibited from doing additional analyses that weren’t pre-registered, but they are clearly identified as such. One the study is written up the paper goes back to reviewers. They may make further suggestions for improving the paper, but what they can’t do is to require you to do a whole load of new analyses or experiments. Obviously, if a reviewer spots a fatal error in the paper, that is another matter. But reviewers can’t at this point start dictating that the authors do further analyses or experiments that may be interesting but not essential.

We found that the reviewer comments on our completed study were helpful: they advised on how to present the data and made suggestions about how to frame the discussion. One reviewer suggested additional analyses that would have been nice to include but were not critical; as Hannah was working to tight deadlines for thesis completion and starting a new job, we realised it would not be possible to do these, but because we have deposited the data for this paper (another requirement for a Registered Report), the door is left open for others to do further analysis.

I always liked the idea of Registered Reports, but this experience has made me even more enthusiastic for the approach. I can imagine how different the process would have been had we gone down the conventional publishing route. Hannah would have started her data collection much sooner, as we wouldn’t have had to wait for reviewer comments. So the paper might have been submitted many months earlier. But then we would have started along the long uncertain road to publication. No doubt reviewers would have asked why we didn’t include different control conditions, why we didn’t use current source density analysis, why we weren’t looking at a different frequency band, and whether our exclusionary criteria for participants were adequate. They may have argued that our null results arose because the study was underpowered. (In the pre-registered route, these were all issues that were raised in the reviews of our protocol, so had been incorporated in the study). We would have been at risk of an outright rejection at worst, or requirement for major revisions at best. We could then have spent many months responding to reviewer recommendations and then resubmitting, only to be asked for yet more analyses.  Instead, we had a pretty clear idea of the timeline for publication, and could be confident it would not be enormously protracted.

This is not a rant against peer reviewers. The role of the reviewer is to look at someone else’s work and see how it could be improved. My own papers have been massively helped by reviewer suggestions, and I am on record as defending the peer review system against attacks. It is more a rant against the way in which things are ordered in our current publication system. The uncertainty inherent in the peer review process generates an enormous amount of waste, as publications, and sometimes careers, are abandoned. There is another way, via Registered Reports, and I hope that more journals will start to offer this option.

*Less than two weeks suggests a problem!See here for an example.