Book Review: Unreliable: Bias, Fraud, and the Reproducibility Crisis in Biomedical Research

by Csaba Szabo, Columbia University Press, 2025 

This is a rollicking good read, written in an informal style, and enlivened by cartoons, which works as a scholarly and accessible account of the so-called reproducibility crisis in biomedical research. 

I first became aware of this book back in February 2024 when the publisher asked me to review the draft. By happy coincidence I had just submitted an (ultimately unsuccessful) application for funding for a meeting on the closely-related topic of research fraud, and as I devoured the text, I felt guilty that I had not been aware of Szabo's work. As it turned out, there was a good reason for my ignorance: he had not previously written anything on this topic. As he explains in the Afterword, there is a personal backstory: 

The journey of a starryeyed young scientist entering the field of science, and continuing with the scientist working hard and hopefully contributing to the field over thirty years, but during all this time gradually realizing that the entire system suffers from major problems. And now that same scientist—not so young anymore, unfortunately—has written a book concluding that about 30 percent of the papers that come out every year are fake garbage and that 70 to 90 percent of the published scientific literature is not reproducible. 

That is a startling statement, but Szabo speaks with authority, as one who has always loved science, and has had a long and distinguished career in biomedical research both in the US and in Europe. It is clear that he does not want to attack science: as he points out, it is the only game in town. But he is dismayed at how the scientific method has been degraded, and he is concerned that nobody in power is taking responsibility for cleaning it up. 

What makes the book unlike any other on this topic is the detailed account of how we got into the current state, and what are the barriers to remedying the situation. Szabo spends some time explaining how hypercompetition for research grants drives the behaviour of researchers. Although it is customary to talk of "publish or perish", Szabo argues that the real crunch point for a biomedical scientist is success in obtaining external grants. In the USA, this usually means NIH funding in the form of a R01 grant, typically around $1 million over a 4-5 year period. While that sounds like a lot of money, it has to cover 50% of the salary of the principal investigator as well as other salaries and research supplies, which means it is not enough to support a research group. The success rate is around 20%, and researchers typically need to submit numerous proposals in order to survive. The institutions want their staff to obtain grants, not just so they can bathe in the reflected glory of impressive research results, but also because grants bring overheads in the form of indirect costs. So getting grants is extremely high-stakes.

It gets even more interesting when Szabo documents his experiences as a grant reviewer for NIH study sections. We might hope that the most successful proposals are the ones that are realistic, avoid hype, and carefully document the reliability of their methods. Alas, this is the opposite of what happens. Such is the pressure for novelty and impact, that anyone who proposed to replicate a prior finding would be quickly triaged out of the competition. I've noted similar tendencies in the UK context. Thus, the message researchers get from institutions is, if you want to keep your job, get a grant, and the message they get from funders is, if you want to get a grant, concentrate on making the research look exciting.

Szabo next moves on to discuss the way science is done in the lab. There are numerous factors that conspire to make findings irreproducible. Some of these are related to the inherent variability of biological systems, but some arise because of failure to adopt experimental designs that adequately control for bias. Typically, doing things meticulously takes time, and the pressure is to get results out fast. Furthermore, many studies involve a mixture of complicated methods, and the principal investigator may not understand all of them. When it comes to analysing the results, there is huge scope for adopting methods such as post-hoc outlier exclusion, p-hacking, and HARKing. All of these can be used to squeeze positive findings out of an unpromising dataset. If biomedical science is like psychology, many researchers regard such methods as normative and are unaware just how much they contribute to lack of reproducibility of published results. 

The next chapter takes a darker turn, moving to intentional fraud. A number of high-profile cases are reviewed, as well as the industrial-scale fraudulent operations run by so-called paper mills. This chapter is particularly depressing as Szabo takes us through the analogies that have been used to characterise fraud.  Initially, fraud was seen as rare, due to a few "bad apples"; later it was compared to an iceberg of fraudulent work, where we only see the tip but need to be aware that much is hidden. Szabo writes: 

In my view, even this analogy is severely misleading. If we want to stay with nature analogies, my feeling is that we are dealing with a big scientific swamp, with various swamp creatures of different sizes and shapes living in it. There are some relatively clean areas of water, too, and there are regular life forms as well. But there are an awful lot of swamp creatures who happily coexist in their natural environment, taking away food and resources from the regular life forms. In addition, a whole ecosystem built around the swamp is benefiting from it. The people who are supposed to manage the swamp, or perhaps drain it, are nowhere to be found. 

One group of people who might be expected to manage the swamp are those who publish research papers, but Szabo does not find them equal to the task, instead talking of "A broken scientific publishing system". Regular readers of this blog will be well-acquainted with the phenomenon whereby someone reports an obvious problem with a published paper, only to be ignored. Academic publishers are making efforts to screen new submissions for plagiarism and image manipulation, but there seems little appetite for cleaning up the existing body of scientific literature. Until that is done, we cannot regard it as a foundation for future work. 

Szabo is impressed by the efforts of "data sleuths", who perform post-publication peer review and report problems on the PubPeer website, but he regards this as unsustainable: and cleaning up the literature should not be a task for volunteers.  It seems that everyone wants someone to "do something" to fix the problem, but nobody takes it on. Organisations with some responsibility include universities, research institutes, publishers, editors and funders. Szabo's recommendations for change focus on funders, who have the power to deny funding to those who fail to take steps to ensure that their results are reliable. And ultimately, the money for research comes from taxpayers, and governments call the shots. 

This is a particularly difficult time to be conveying such a message. The only people who might be overjoyed to hear that a high proportion of published research is unreliable are politicians who are antagonistic to science and would like an excuse to defund it. In the USA, cuts to funding have been so fast and so deep that many are fearful that the science base may not recover. The swamp creatures may die, but so will the regular life forms. We urgently need therefore to look seriously at recommendations for changing how the system works at all levels - laboratory practice, funding, institutional integrity investigations, publishing, incentive structures - so that we can not only have confidence in scientific findings, but also defend science against attacks. 

I don't agree with all of Szabo's recommendations, but it is refreshing to have someone take a deep dive into the topic, and his ideas form a good basis for discussion. One point where I have a different approach concerns the emphasis on replications. There are many people arguing that more funding should be directed towards replicating prior studies. In the short term, that will be needed, because the way research has been done means we don't know which findings are solid. But in many areas it takes a large amount of time and money to replicate a study. The whole point of the statistical and experimental methods used in science is that they should allow us to assign a level of confidence in our findings without needing to perform an explicit replication. The problem is that we have misapplied those methods. The Registered Reports approach, where a study is evaluated by reviewers and accepted or rejected by a journal on the basis of introduction, methods and analysis plan, before any data are gathered, gets rid of the biases due to p-hacking, HARKing and publication bias, making it possible to interpret statistics sensibly. It also leads to improved methods overall, because independent reviewers offer feedback at a point when it can be helpful. As far as I know, the Registered Reports model has not been adopted by biomedicine, but could, I think, transform the field to make it more rigorous. 

Finally, I'm pleased to say that despite the initial rejection, I managed eventually to secure funding for that meeting on research fraud, which will be held in Oxford from 7th-9th April 2025, and will provide a great opportunity for discussing these issues. Registration is open for a few more days, so please consider attending (online or in person) if you'd like to take part. More details and registration form here: (turn off VPN if it does not load).

A call for funders to ban institutions that use grant capture targets

I  caused unease on Twitter this week when I criticised a piece in the Times Higher Education on 'How to win a research grant'. As I explained in a series of tweets, I have no objection to experienced grant-holders sharing their pearls of wisdom with other academics: indeed, I've given my own tips in the past. My objection was to the sentiment behind the lede beneath the headline: "Even in disciplines in which research is inherently inexpensive, ‘grant capture’ is increasingly being adopted as a metric to judge academics and universities. But with success rates typically little better than one in five, rejection is the fate of most applications." I made the observation that it might have been better if the Times Higher had noted that grant capture is a stupid way to evaluate academics.

Science is in trouble when the getting of grant funding is seen as an end in itself rather than a means to the end of doing good research, with researchers rewarded in proportion to how much money they bring in. I've rehearsed the arguments for this view more than once on my blog (see, e.g. here); many of these points were anticipated by Raphael Gillett in 1991, long before 'grant capture' became widespread as an explicit management tool. Although my view is shared by some other senior figures (see, e.g., this piece by John Ioannidis), it is seldom voiced. When I suggested that the best approach to seeking funding was to wait until you had a great idea that you were itching to implement, the patience of my followers snapped. It was clear that to many people working in academia, this view is seen as naive and unrealistic. Quite simply, it's a case of get funded or get fired. When I started out, use of funding success may have been used informally to rate academics, but now it is often explicit, sometimes to the point whereby expected grant income targets are specified.

Encouraging more and more grant submissions is toxic, both for researchers and for science, but everyone feels trapped. So how could we escape from this fix?

I think the solution has to be down to funders. They should be motivated to tackle the problem for several reasons.

• First, they are inundated with far more proposals than they can fund - to the extent that many of them use methods of "demand management" to stem the tide. 
• Second, if people are pressurised into coming up with research projects in order to become or remain employed, this is not likely to lead to particularly good research. We might expect quality of proposals to improve if people are encouraged to take time to develop and hone a great idea.
• Third, although peer review of grants is generally thought to be the best among various unsatisfactory options for selecting grants, it is known to have poor reliability, and there is an element of lottery as to who gets funded. There's a real risk that, with grant capture being used as a metric, many researchers are being lost from the system because they were unlucky rather than untalented. 
• Fourth, if people are evaluated in terms of the amount of funding they acquire, they will be motivated to make their proposals as expensive as possible: this cannot be in the interests of the funders.

Funders have considerable power in their hands and they can use it to change the culture. This was neatly demonstrated when the Athena SWAN charter started up, originally focused on improving gender equality in STEMM subjects. Institutions paid lip service to it, but there was little action until the Chief Medical Officer, Sally Davies, declared that to be eligible for biomedical funding from NIHR, institutions would have to have a Silver Athena SWAN award.  This raising of the stakes concentrated the minds of Vice Chancellors to an impressive degree.

My suggestion is that major funders such as Research England,  Wellcome Trust and Cancer Research UK could at a stroke improve research culture in the UK by implementing a rule whereby any institution that used grant capture as a criterion for hiring, firing or promotion would be ineligible to host grants. 

Reference
Gillett, R. (1991). Pitfalls in assessing research performance by grant income. Scientometrics, 22(2), 253-263.