How paper mills go straight (without going clean): a shift to dual-use services

Guest post by Matt Spick

                           Photo: Parsadanov/Shutterstock.com

From relative obscurity, paper mills have recently moved into the spotlight of academic attention. These organisations - which sell manuscripts or citations to authors to enhance scholarly metrics - are growing so rapidly that many fields are being overwhelmed. The overall proportion of paper mill outputs was estimated at 1.5–2% of all scientific papers published in 2022, but a recent AI-screening cancer study estimated that around 10% of recent cancer manuscripts in some venues may be paper mill products, and in data-intensive fields paper mill outputs can now outnumber legitimate publications. The increased attention has also been driven by the integrity community highlighting unethical behaviours, notably in the FoSci Report 2026, and has resulted in initiatives such as United2Act. This is in addition to publishers and third parties setting up a growing number of integrity checking systems, whether for citation anomalies, duplicated images, or tortured phrases. But this is an adversarial process, and backward-looking checks will inevitably miss what happens next.

One option for paper mills is to stop selling unethical products, and shift to dual-use services instead. The dual-use business model has a long history. During the US prohibition era, manufacturing moonshine came with severe risks. To avoid these risks, and transfer both criminal intentions (mens rea) and criminal action (actus reus) to the customers, the California Vineyardist Association created a front organisation, Fruit Industries Ltd, to sell Vine-Glo. This innovative product consisted of a block of concentrated grape juice, easily dissolved in water to create a refreshing and entirely legal fruit-based drink. The grape juice block also included an explicit warning to purchasers that “After dissolving the brick in a gallon of water, do not place the liquid in a jug away in the cupboard for twenty days, because then it would turn into wine.” And of course there are other examples of unethical actors shifting into legitimate business, whether in laundry services or waste management businesses. Such operations are hard to police, precisely because they overlap with entirely legal commercial interests.

The paper mill equivalents of Vine-Glo might include proprietary software tools to data dredge and p-hack large open datasets, such as the Global Burden of Disease Study or NHANES. Such tools could be used legitimately, but if videos were created explaining how to mass produce p-hacked papers, this would be claimed as “beyond the developers’ control”. Training courses might emerge that offer primers on scientific writing to complete beginners, starting with an icebreaker at 9:00 am on day one and concluding with submission to a PubMed indexed journal at 5:00 pm on day two. Naturally, acceptance would not be guaranteed, and the trainers would stress that this was purely a training service, not a ‘pay for authorship’ model. Proving that these practices were linked to specific examples of problematic authorship would be impossible at the individual paper level.

The lack of action (sadly, there is no such thing as the science police) is especially frustrating in an industry that is slow to respond even in clear cases of retractions being needed, due to a conservative culture around accusations of wrongdoing, which - partly for good reasons - prefers to tolerate higher levels of unethical outputs rather than inadvertently criticising or punishing scientists. At the same time, the surge in publication volumes (measured in the tens of thousands across biomedical and life sciences research) makes it impossible to conclude that no problematic behaviour is occurring.

Mills turning to dual-use products would also compromise large parts of the integrity community's current toolkit: detection based on image duplication, tortured phrases and citation anomalies would largely fail, because the outputs would represent genuine analyses of real data with real (if trivial) results. In turn, this is likely to lead to prevention having to move upstream, to pre-registration before data release and gatekeeping of access.

Will service mills offering software and training replace the existing paper mills? In practice, both models are likely to continue to exist, if only because hiring, promotion and graduation decisions are all influenced by (or completely dependent upon) publication counts. As long as the traditional model remains profitable (with first-author slots advertised at up to USD 5,600, more than sufficient to grease the wheels through editor bribery elsewhere in the publication chain) and the underlying demand continues, there will be no incentive to leave money on the table. Conventional paper mills may be able to use LLMs and agentic platforms to improve the quality of their outputs (as a different form of adversarial adaptation), making them harder to detect and preserving their value proposition for unethical authors who simply want to purchase authorship. Many potential customers are likely to find the scientific publishing process too arcane, and will be happy to continue to buy an end-to-end service. Service mills seem likely to grow in importance, however, for authors who are more risk averse. At the same time - for expert users - agentic AI platforms for science may trivialise secondary outputs such as reviews and open data analyses, providing a third route to ‘enhancing’ an author’s scholarly record. 

This will present a challenge to the wider scientific community, which has only begun to wake up to the problem of paper mills as manuscript factories. In reality the mills are diversifying their offerings, creating the illusion of ‘going straight’ through dual-use products, and it seems inevitable that some expert users will be able to disintermediate the paper mills completely through agentic AI. We need to recognise that all these types of output can dilute the scholarly record, decreasing the signal to noise ratio and delaying the translation of literature into real impact, to the disadvantage of both direct users and also society as a whole.

Note: Comments are welcome, but are moderated, so there may be a delay before they appear. In general, comments are accepted if on-topic and non-anonymous.

 

 

That fireside chat with Jay Bhattacharya: a view from across the pond

On Friday I attended (virtually) a workshop organised by the National Academies of Science, Engineering and Medicine on Enhancing Scientific Integrity: Progress and Opportunities in the Social and Behavioral Sciences.

On social media, the workshop attracted a lot of critical commentary, mainly because it included a "fireside chat" with Jay Bhattacharya, who is currently the Director of the National Institutes of Health, and acting Director of the Centre for Disease Control and Prevention. The chat was hosted by Emily Oster. As a Brit I had not come across her before; she has had a distinguished career in health economics and is the author of popular books, but is controversial for questioning school shut-downs during COVID, and for questioning the use of COVID-19 vaccination mandates.

Numerous commentators on BlueSky questioned the decision to include the fireside chat on the programme, arguing that it gave a platform to Bhattacharya, one of the authors of the Great Barrington Declaration,  to promote his controversial views on COVID and vaccination. Other speakers at the meeting were criticized for sharing a platform with him, and NASEM came in for particularly harsh attack as the sponsor of the meeting.

A couple of weeks before the meeting, organizer Katy Milkman, who was the focus of much of the criticism, wrote on Bluesky: 

"To everyone responding with vitriol: please come! Our goal in choosing speakers was to platform both popular and unpopular figures advocating for scientific reforms to spark productive conversations across divides! If this makes you angry, come engage in dialogue - that's how we improve".

This did not go down well, and many people vowed not to attend the meeting. I had the opposite reaction: I was very curious to see the fireside chat, and if possible to ask a question.

So what's my verdict? The tl;dr version is that I think that NASEM was right to include the Bhattacharya/Oster chat in the meeting for two reasons. First, I've attended endless sessions on research integrity with great speakers, but most of us feel powerless to actually do anything, other than call for action from funders and governments. Even if you don't like what your government is doing, it can be helpful to know what solutions they will consider. Second, I'm a great believer in engaging with the opposition in any difference of views, so you can build better arguments. 

It was fascinating to see how Bhattacharya tried to control the conversation. While I don't always agree with Oster, I felt she came out of the encounter relatively well, considering that she was dealing with someone whose main strategies were to focus on issues that most people would agree with, and then to go on the attack if more challenging topics were raised. I'm sure the organizers will be criticized for the fact that only one audience question was asked in this session, but I doubt that I could have done any better than Oster, who had to be pretty assertive in moving the conversation away from replication to difficult topics that the audience were interested in. Bhattacharya left few gaps in his stream of speech for her to chip in. In his answers to difficult questions, Bhattacharya revealed himself as someone who gets very uncomfortable when confronted with inconsistencies in his behaviour or views.

I'm a nerd, so I have transcribed the full interview, which is appended below. Here's my take on the themes of the chat.

Replication. Bhattacharya is a big fan of John Ioannidis. They co-authored the infamous "Santa Clara County" study of COVID-19 prevalence, which "raised eyebrows" because the methodology had exactly the kind of problems with bias that Ioannidis himself had criticized. Ioannidis is also on the editorial board of a Journal founded by Bhattacharya, the Journal of the Academy of Public Health, another thing that has led to raised eyebrows. When I checked who was behind the Academy of Public Health, I found an organisation that was "founded for improving public health practice as a special purpose vehicle of the West African Institute of Public Health (WAIPH)", which is quite separate from the new journal.  It seems that Bhattacharya invented another organization with that name for the purpose of restricting authorships in JAPH members of the academy, i.e., those whose views were deemed acceptable. The JAPH has not been a success, but it's worth a read to gain insight into background to current US government policies on public health.

I reckon that most of what Bhattacharya said on replication came from Ioannidis. As an aside, Ioannidis also spoke at the workshop, arguing that though there were big problems with the replicability of much of science, it would be a disaster if the response was to cut funding for science. I felt that he was furiously trying damage control against weaponization of his views. I wanted to ask him whether he still had confidence in the Santa Clara County study, but my question was not selected (though another of my questions was). It would do wonders for his reputation if he would just reflect on those times and accept that, with hindsight, he did not do a good job.

But back to Bhattacharya. When talking about replicability, he was preaching to the choir, reflecting many of the points made by other speakers about the need for replications, for open data, code and materials, and for well-conducted meta-analyses.

So how was this to be achieved? One idea was to change the metrics by which scientists were evaluated, to encourage collaboration, error-checking, and to reward those whose work was replicable, and who adopted open science practices. PubMed is to be revamped to provide a button that allows readers to see related work and evidence of replicability. I had nothing to disagree with there.

He also identified journals as a big part of the problem, noting that AAAS were not willing to publish replications in their journals. As someone who has tried to persuade journal editors to adopt replication registered reports, I agree this is a big problem. But I disagree with Bhattacharya's solution, which is to have some kind of publication outlet for replications - and his history with JAPH doesn't inspire confidence. If NIH were to have a funding stream for replications, this would be a game-changer, but I think it would have to be in the form of registered reports, where the methods and analyses were reviewed and nailed down, with an in-principle acceptance by the journal when the study protocol was approved and before data were collected.

Changes at NIH

Bhattacharya's cosy demeanour fractured when he was asked about changes at NIH. Oster specifically asked whether the "leadership review of grants" would be part of the replication agenda. He replied "You should not be paying attention to fake news on this topic", but Oster didn't give up.

EO: So there's no leadership review of grants? 

JB: There's no- there's no political review of grants where like -       

     It's actually very destructive to bring that into this right now     

     If you want to talk about that, we can do this but we should talk about the replication thing first

He went on to reiterate all the stuff about replication, concluding:

"By the way, do you hear anything political in any of this?  I think- I think that the problem is, like, there's a lot of bad faith conversations and thinking about this"

Oster attempted to argue that the conversation was not in bad faith, but Bhattacharya spoke over her and then moved on to talk about the other big problem, stagnation.

Stagnation

We learned that Bhattacharya had co-authored a paper showing that in the 1980s, papers relied on newer ideas than in the period around 2017. I was a bit puzzled by the logic, as this could be taken to mean that ideas studied in the 1980s did not endure as long as those studied later. But I've not dug into the details of the paper, other than to note that all the data and code is openly available.

Bhattacharya interpreted this study as showing that NIH research was stagnating, and so in future, NIH grant scores should reduce the weighting given to methods, and increase the weighting for innovation. He explained that ARPA was founded in response to this sense that NIH was not funding research at the bleeding edge.

Now, I am aware that there's a hefty literature about optimal research funding models, but I have to say this change of focus seems odd, because it is the opposite of what you'd do if you were really concerned about replicability. Replicability is all about building a solid base for future research. Over the years I've seen numerous over-hyped fads eat up huge amounts of research funding (example here), and they typically take years before they run out of steam. 

NIH funding cuts

At this point Oster introduced the lumbering great elephant that had so far been ignored: NIH funding. She did so in the most tactful way possible, by asking "has this meant approving fewer grants?" She got another blast from Bhattacharya about fake news. The reason for the recent reduction in grants, he argued, was the fault of the two-month shutdown, going on to say:

"This is again a sort of politicised attack, instead of actually trying to engage with what the reforms actually are".

I suspect this will be strongly disputed by those who have seen funding opportunities decline sharply. It seemed particularly galling that Bhattacharya argued that a focus on innovation would inevitably mean higher funding rates for early-career researchers, when most commentators are arguing that they have been among the worst hit by changes in policies.

CDC and vaccination

Oster again tried a controversial question, about how Bhattacharya (who is acting head of CDC) could increase routine vaccination rates. His initial answer was:

"Tell them to get their kids vaccinated"

before noting that this wasn't about replication, but was rather a separate topic.

Oster came back rather magnificently with: 

"Yes, it's a separate topic. But I'm asking about it".

He argued there were two ways of handing vaccination policy - coercion versus having a conversation, "and not cancelling or trying to destroy people who have questions about whatever vaccines you're talking about". Oster agreed with him that a coercive approach destroyed trust. But she did not challenge his false binary, or his examples of Scandinavian countries with high vaccination rates but voluntary vaccination policies - which occur in the context of universal healthcare, quite unlike the US system. I'm no expert, but I could readily find a wealth of data on this topic, (see, e.g., here, and here) and one would expect two distinguished health economists to recognise that cherrypicking selected examples and not adjusting for contextual factors is unhelpful.

The final uncomfortable question by Oster concerned his blocking of a CDC report on COVID vaccination efficacy. He gave this short shrift. The method in the COVID study was "ridiculous" and:

"You agree to this, and any econometrician you show this method to will go 'are you freaking kidding me'". 

Oster agreed with the criticism of the method, but noted that CDC had recently published a report on flu that used the same method. Here again, he fulminated on "fake news" , and "bad faith reaction", arguing that he had not personally approved that report.

And then the time was up.

Conclusions 

Overall, I wondered if Bhattacharya might have set preconditions for his "fireside chat". He was happy to talk endlessly about replication and innovation - two topics where he felt relatively safe. He became ruffled when Oster put other topics on the agenda - blustering through them in a Trumpian fashion with accusations of "fake news" and "bad faith" arguments. 

It might help if he were to pause to consider why people don't accept his arguments are in good faith. His track record as author of the Great Barrington Declaration and the Santa Clara County study already raises questions about bias. I accept that his (and Oster's) positions about lockdowns and Covid vaccines may be sincere - when Covid-19 first became a big problem, I was amazed at how polarized views were between respected medical experts, all of whom seemed to have genuine concerns.  It was a horrible dilemma deciding how to act in the best interests of the whole population.  But Bhattacharya's setting up of the Journal of the Academy of Public Health was a huge misstep, creating a walled garden where only those who endorsed views like his were welcome. 

I'm glad we were able to see him explain and defend his position. But I think he's done nothing to reassure rank and file scientists who are worried that the "replication crisis" could be weaponized to dismiss science that was judged to use "ridiculous methods", and the "Unified Funding Strategy" could ensure that scarce research funds went to innovative but flaky ideas. For the sake of my colleagues across the pond, I hope not.

P.S. follow-up. 6th May 2026

Here's some more background reading for those interested in further details.

 

 David Sanders has a great piece in the Times Higher reflecting on a talk that Bhattacharya recently gave at Purdue University.

 

Andrew Gelman also wrote a blogpost about the NASEM meeting:

Like me, he's attracted criticism for not attacking Bhattacharya more strongly. 

 

And, the CDC report that Bhattacharya blocked has now been leaked, see here. 

My post led to some discussion on social media. Several commenters thought readers should be made aware of other venues where Bhattacharya had been interviewed, and pointed me to these sources:

 Can Faith Guide Public Health

 Dad Saves America

 

I haven't watched the first one yet (it's long). The second one on risks/benefits of vaccines in different groups, seen as cost-benefit analysis, was pretty much in line with UK policy - as explained here by David Spiegelhalter. This is not to say I approve of JB: as I noted above, he has badly damaged his reputation as an impartial interpreter of science by setting up a 'journal' as a vanity project for those who share his views on controversial topics, and by co-authoring a Covid study with shoddy methods (Santa Clara study). But, from the tone of some of the comments, I'd assumed he'd be on record as an anti-vaxxer, and I haven't seen convincing evidence of that. 

I think there is a fundamental divide between those who think that engaging with someone like Bhattacharya just gives him publicity to air his disingenuous views, so should be avoided, and those like me (and I think Andrew) who want to have him on the record, in this case making his best case arguments when confronted with an academic audience. We can scrutinise them and be ready to rebut those that are inconsistent or incoherent.  For instance, using "fake news" to just shut down anything he disagrees with is not going to enhance Bhattacharya's reputation as a serious person in academic circles. 

 

I'd welcome details of any other articles or talks where Bhattacharya states his position on public health interventions - please add these to the comments.     

 P.P.S.  8th May 2026

Andrew Tan (@andrewtanyongyi.bsky.social) sent me a couple of links to tweets by Jay Bhattacharya dating from 2024 - i.e. before he became Director of NIH (May 2025). The first endorses the film "Thank you, Dr Fauci", which puts the case for there being a cover-up of gain-of-function research that led to COVID-19.  The second gives positive comments on the HOPE Accord, which calls for suspension of mRNA COVID-19 vaccine. 

 

Comments

Comments are moderated so there may be a delay of a day or two before they appear. On topic comments are welcomed, but I seldom accept comments that are anonymous or rude.  

 Draft transcript of the conversation. 

Source, vimeo for 24 April 2026, on https://shorturl.at/JvlIs

N.B. I'm not sure if this is accessible if you did not register for the meeting. 

Transcript from discussion on 24 April 2026: Emily Oster and Jay Bhattacharya

 

Starts around 6.28.00

 

E: thank you Katie thanks for having us, and Jay it's great it's great to see you I

'm delighted to be doing this so thank you thanks for being here

J:  likewise Emily, I saw your name that you'd interview me and that just sealed it

E: amazing. all right, so let's start.

 I actually wanted to start by asking probably a couple of big picture questions about goals

 So you have said, I think pretty publicly, that your goal at the NIH, or one of them is transparency and you've talked about the value of scientific replication and of rigour in the scientific work that you that you fund, erm,  and I think that is an excellent goal. What I'm curious about is is metrics. So I'm curious about what are the metrics that you look to, that you want us to look to for success here, and how are you measuring those metrics

 

J; OK so the key thing is like what problem are we trying to solve.

In the context of of replication the question is like why are we in this situation we're in right .

So like you know everyone knows-  I guess John Ioannidis was was interviewed earlier today. He had that paper with the title, that most published biomedical research findings are false, 2006

and since then there's been all these like findings .

It was actually audit studies of various literatures starting with psychology, I think, but then you know neurobiology, cancer you know cancer bi-  a whole host of like basic science,

 and then now social science literatures that where independent people look at the same thing they don't find the same answer . (6.31)

Related to that there's a whole bunch of very high profile instances of of of scientific fraud , like Photoshop fraud and all that, again I think there there're folks earlier that- that have been-  Bik and others that have been central to this

 That that really document the the fact that um the published biomedical literature does not reflect-  in many ways is is is it it doesn't reflect scientific reality.  In many ways, right. Not entirely. 

Right now half of the-  half or more of a scientific literature doesn't replicate, meaning like independent teams look at the same thing they don't find the same answer. or a typical thing which is like with the the replication the effect sizes are er

E: smaller

J: smaller than the original published literature says.

Sometimes in the opposite direction, often in the opposite direction.

 You know, we have a problem, right, because I can't- that means that like when patients and go look at Pubmed, they don't do that often but when they do when they find some- some paper,  they don't know if it's right.

When a doctor goes and looks to see OK I have a patient with XY&Z, what does the best evidence say, they can't trust it, right, and that-

Developers of drugs,  what they do is they do their own private replication efforts before they- before they invest you know tens of millions of dollars in some phase three trials or phase two trial,  to say look this, is this worth it, can we trust the literature

So you have this like pockets of private entities that know what parts of the biomedical are true or not, and pockets or large chunks where we just don't know.

Nutrition science is a good example of this. So there are real world consequences of this. Like, are eggs good for you.

Like that's gone back and forth multiple times in my life.  I went through my youth in the- in the- in the eighties. My family never had eggs. We just didn't have them. When I first had eggs, I had like an egg white omelette which just was terrible.

I just was not - now I eat eggs. So, erm-

The point is like that these have real world consequences for the decisions people make.

And then, the other question is what - what's the - what's the reason why this happens.

 And I've seen -

I'll get to your question about metrics, I promise

E: I think the reason this happens is very complicated and it's going to be different in different cases, so why is the literature on eggs bad is different from the question of why do some of these trials that John is talking about do not replicate.

And so- OK go ahead but I do want to hear about -

J: I have a grand unifying theory why this happens, right

E: Okay

J:  And that is that science is hard

 E: Okay

J: That's the grand unifying theory ,right

If is very difficult for science- for anybody to like understand how physical reality works.

It's just a very difficult -

And the power of the scientific process is that it brings to bear - when it's working well, so that the whole collaborative process of of of people checking each other's work in order to decide whether some idea is true or not, right

The first scientific revolution happened because rather than having -  rather than having some ecclesiastical power gets to decide, whether the moons of Jupiter moved, people, very clever people with telescopes get to decide that.

And anyone can go and look at a telescope and go , yeah okay, the moons of Jupiter move.

Right, it's that collaborative process that together we check each other's work and that - and science , and so- sci , it's it's er hard-

Now, the process related to this is the fact that- the publication process, the peer review process, does not provide an adequate check

There's all kinds of problems people have talked about , right

You can't- it's very difficult to publish negative findings

Er - it's actually quite difficult to publish replication findings

It's difficult to f-, it's difficult to like - the peer review process itself-

Emily, you will have done thousands of peer reviews, as have I

How many times have you had a chance to look at the actual data underlying the paper?

 and done a replication yourself?

E: I agree the peer review has some concerns, I think in some ways it works well, I think in some ways it doesn't work well. But I mean, I hear all this Jay, but I think like the question is-

Maybe what you're saying is you want there to be more replication, and one of the metrics you're going to measure is there more replication

J: let me- let me go - let me get to the metrics, right

The metrics are things that provide incentives to have that collaborative process work well.

E: OK

J: That's the key thing. So, there's a whole science of science literature behind - underneath this, but I can give you some ideas that I have now.

I'm very open to hearing more metrics.

But like, so for instance,

Right now, the metrics, very broadly speaking that measure scientific productivity measure it along two dimensions .

 The volume of publications you have, the volume of work you do, in other words, and then the influence you have, the citations

Those are - I think everybody listening on this call understands, those are too narrow a set of metrics to decide whether a science is productive

And those by themselves do not encourage the kind of collaborative scientific behaviour that we want.

So, for instance, let me give you a metric, right

So, er, one might, one might measure, it's like you publish a paper, er, er- do you make it available in a readily - in a like standardized form, make those data available, so that other people can like get those data without having to like go beg for you for like a hundred  - for a hundred different emails.

Do you make it in a form where - do you describe your methods in a way such that replication could be done.(6.37)

Like a big problem in the literature is that a lot of the m- papers publish method sections that are not even interpretable, so that -

E: Okay. So let me pause you on that. So that would suggest that one approach to one metric that you could have would be what share of the papers, say, funded by the NIH,  are replicable in the sense that someone either has replicated them or maybe you have some independent process that tries to get their data and replicate them

 Is that an example of something you could use as a metric-

J: So if you look; it's actually a little more nuanced than that.

That so like you- I mean sometimes you talk about replications like- this narrow- I just do the same exact experiment you did under the same exact circumstances, do I find the same thing

There's a broader just conversation that happens around replication you know

 just does it does your result generalise

 If I think of it a different theoretical approach the same problem that it has a slightly different experiment is- does it tends to tend to find a similar answer , or , or -

or it doesn't replicate what specific step is not-  is being done differently so that it doesn't replicate

Can I step back- one one thing again for for the theoretical perspective before we get launched into more of the metrics

I got lots of metrics I could give you but let me just just theoretically, right

So I've seen like two different approaches to this problem - and I- I- I just I don't know if we've like been quite so explicit about

You can even see in the agenda for this conference

One is a policing mechanism , right

So what we do is we we just have a-  some function where we go root out scientific fraud, audit people and and that will that will solve the replication crisis, because the implication at least root out scientific fraud .

Now I'm not against rooting out scientific fraud

But my question is does that really address the key thing that I think is the central problem which is the science is hard

But I think even if you had rooted out all scientific fraud in some theoretical utopia you would still have the replication problem because the incentives to actually engage in that kind of collaborative behaviour that produces advances in scientific knowledge just is not there  in the same way that you would want it to be

But that's that's incentive certainly based on the metrics that we currently use which are too narrow, encourage encourage us to like find ways to have scientific influence

find ways for us to like get our stuff published a lot

But it doesn't encourage the kind of collaborative -

So for instance if you have a paper with a null result or if you have a paper with that's that that that just really just is a narrow replication of somebody else, danger- it's actually it's it's going to have a really hard time publishing it

If you are approached to do replica-

 If your paper is subject to replication you're going to view it as a threat to your career, right

 That's that's an indicator that scientific environment that we're in is an unhealthy one

 -because it doesn't have that sort of or or -

It doesn't represent the sort of erm -ideal that I've been just - as I think we agreed on before about like this sort of collaborative environment we're sort of together trying to seek the truth .

E: So then-

 So one approach that then is to sort of say OK, instead of allowing effectively the market to decide what people work on, which I think some of -

J: Hold on.

The market I mean I'm OK with the market but the issue is like we don't have a market

E: we have a market in the sense that like people are- there's something people are trying to maximise which is like the number of citations

 But let's put aside

Maybe you don't like that as the market

But I think one approach here is you could say OK we're going to decide what are the important questions like what is a problem that we were going to be more problem based, like heres a problem

We need to solve we wanna you know cure cancer

We want to understand whether eggs are good for you or whatever is the kind of core thing

And then we're going to like point people as a group in some way towards that and maybe maybe that's kind of a thing the NIH should be thinking about

 Is that-

J:  But that's exactly what NIH always has done

Like with fifty billion dollars that we spend that's exactly- that's that's the whole thing right

So the question is not are we-

That's why it's not a market, right

E: Well how are you going to do that better

J: Right, so that's the question

E: That approach is not good, it's not worked  

J: I just - it's not that it hasn't worked

It hasn't set the incentives correctly, right

So you don't have a choice right

This is not it's not a free market, in fact NIH I think partly exists as it solves a market failure

Or should exist as it solves a market failure

 It funds science that that the private - where the knowledge gained has some public good aspect here right

So so in principle I don't think would be much

 The question is like how should the- an entity like the NIH make decisions to make to create this sort of collaborative scientific environment that makes- not just incentivizes science at the bleeding edge, which we should do better ,but also make sure that the science produced to the bleeding edge is replicable

 like I see now that's just that now is that

E: It's not a metric though

[E and J speaking over each other]

 

J: But Emily, before you can decide on a metric as you know first you have to understand where you're going right

You understand the theoretical perspective

E: Okay

J: Come on, you know that right

So so you gotta have the conversation first

And so where are we going

So for instance like

How many papers do you do you have published with just you did somebody replication?

How many papers have you sitting in your drawer that where you just published where you -

I mean every single lab has it's sort of like secret knowledge of what papers were replicated versus don't

What ideas are likely true versus not

 That should be public and if we give incentives {unintelligible} measure how many replication papers you have as a as a as a metric that matters

Where you base promotions in part on it

 Then you get more of it, right

 If you if you measure so if you measure how many like uh uh I already mentioned the data sharing, code sharing, tissue samples sharing, things like that

 These are like inputs into replication rather

Now of course the the ultimate kind of measure is do you have important pa- important ideas that survive the test of replication.

 It it's it's kind of it what it does is fundamentally transforms what's rewarded in science

instead of can you get something published in a top journal, instead it's do you have an idea that that has been subject to replication at all.

 I mean that's that we should be measuring that

And has that idea survived replication, that would be another another kind of separate metric

All of those should be seen as positive as measures of scientific productivity

All of these I think in principle are measurable at scale given the data resources we have, we just need to like figure out how to do it right

And there's nuances about how exactly to do that, but the point is that the the the place you want to go you, you can you just derive the metrics that gets you to get you there

with the incentives you want there

In my view it's not are we rooting out scientific fraud - again I'm not against rooting out scientific fraud

 I'm saying that that is a very very a relatively small part of the fundamental problem

 that we don't have the right incentives for us to have this collaborative environment that you and  I just talked about two minutes ago

E: So it sounds like one, like the goal is a more collaborative environment with more replication and then maybe our metrics would be things like are we promoting people on the basis of replication or what share of the things that you're funding are replication based

or like you know -

J; I don't know how complicated this be but in principle it is possible:  do you write your methods up in ways that that that invite replication where people can just do it ,right

could you do you I mean I mean -

I think the inputs in the replication in many ways are even more important in some ways as far as incentives are concerned, than do you have a paper -important paper that's replicated that will determine the scientific status of course, in principle

But the the sort of the good scientific housekeeping like the kind of kind of behaviours that we want all sides to have ought to be measured and ought to be rewarded, even if it doesn't mean that your own paper-

I mean, Emily, think about this.  if you're subject to replication, your work is subject to replication now-

E: I totally completely agree with this like I think we're very much on the same page about the importance of this replication,

 I think my question is just how is that is not like-

J: It's not the importance.  It's the very very act of someone approaching you for replication is now seen as a threat.

E: I agree. And I think if you can if you can turn that off I think that would be very- I think that would be very good .

I would like that

I have a related question, ah,  which is you know you've talked, and I think this is in some ways probably the way you would achieve some of these goals, which is to have some review, an NIH leadership review of grants for something, you said scientific rigour, we've also talked about how maybe part of that could be, you know, how replicable  or how accessible is this going to be for replication.

I think that people have raised questions about how that will work.

er,  and whether it is different,  how will it be different from the review that goes into grant approval

 and like frankly I don't think I'm talking out of school here, people are worried about political bias in those kind of reviews

And so I guess I'm curious, is there anything for this audience, probably many of whom are applying to the NIH for money, is there anything more you can say about how these leadership reviews are going to work and what they're looking for that's beyond the sort of typical review cycle

 J: OK so this is a different topic than replication,  cos I view that as  like portfolio construction, which is different from replication

E: it's another, it's also a goal, it's a broader-

{E/J talking over each other}

J:  Let me just state, portfolio construction is very important, the idea that's its political is is absolute nonsense.

 Well let me just let me just finish with replication first right ,

So actually we can link into portfolio construction.

 Part of our portfolio has to have replication in it

Meaning I want to fund the John Ioannidises, not just John specifically, but like the people like him who do replication in clever - meta analysis -  in clever ways at scale. That develop new methods for replication, actually even just do do replication, right, so I want to fund a cadre of researchers who can get R level funding to do replication.

 I think it is very- because of the concern you just said- it is very important the government not pick and choose which topics are subject to replication.

 I think that that that should come from the scientific community, right,  so we'll have R level funding that is that is subject to the same kind of peer review that everyone's used to

 The key thing is the the proposals will say-   will identify sort of rate-limiting step-  and I can't replicate everything

I mean that'll that'll greatly  deep slow the rate of scientific progress I think.

 So what you have is essentially a competition among scientists for this R level funding, where they're saying, look here are the ideas that are rate-limiting step ideas in my field.

If if I if we know this is true then this field goes in one direction and if we know there's not there's more nuances around that that will go in another direction

 And so you have study sections arguing about what are the key ideas in a field that need to be need to be subject to replication, right

So that's that's so 1) you gotta start funding replication

2) you have to have a place where you can publish it .

And the key thing there is that- cos you know I talked to the AAA S folks and they're like they will not publish replications not even in their in their in their top journals-  right,  they just they just won't, like it - and I think that's generally true

These journ- the journals generally w- make it difficult to get that stuff get replications published

And it's partly because it's a very different kind of thing right so like

If you have if if somebody does a replication project, the kind of gatekeeping that a journal does should not actually happen

Right, so should happen, though is peer review.

Public expert peer review that are papers that are replication papers, right

 So you do a replication of some paper of mine you should get to publish it in someplace but then I'll write a response that will be public

 It will all be public, and then the journal or whatever the whatever you want to call it will invite other interested expert parties to weigh in and so you have the replication which is essentially with minimal gatekeeping accepted but alongside a whole bunch of peer reviews for that paper

And then so that's that's true ,right

You fund replication:

2, you have a place where you can publish and have a real conversation about it

I don't mean Twitter, I don't mean PubPeer where it's all anonymous.

Then 3) you have to link it to the search engine and in particular PubMed.

In particular, right so right now pubMed what it what it emphasise is when you go search for paper is, you know,  who's the authors, where it published,  an abstract, and you might want to click and you might you might be able to get the full text now that we have this new policy that NIH funded, but but really emphasises where it's published, right.

 It elevates it elevates the that-

 It is very difficult like the related papers section is not that great , it's very difficult to sort of understand the sort of like broader knowledge ecosystem around that paper.

 Imagine instead a rep- in addition to PubMed, you have a replication button and then you click it and you get a knowledge knowledge graph where like you put the paper in some context of other other papers that are that are related you can go click on them -

E: That's called Claude, Jay,  I think that I think

{both talking, unintelligible}

J: Claude is much worse because we will be able to write context. You'll have an AI summary of linked of like related of like actual replication work for for that paper and you'll be able to see what what - it'll do it the context of the search engine

 E: OK fine, so yeah so we're still on replication.  I want to move on-

J: You want to do that with Claude? I mean, it's just not going to do it. I mean, I've played with Claude. It just doesn't do. {unintelligible} a bunch of the time

I don-t - [It just won't do that].

E:  [I see]-

sure, I'm sure there are ways, yes.

We could argue all day about what we think is the potential of AI for this kind of thing but I think that it's fair to say that like some kind of additional context around these things would be helpful

  but I do want[ to think-[

J: [I think it's more] than helpful. I think it's transformative, right, so if we elevate where it's published, where a paper is published , top five journal as the bare  - most people won't even read the paper, right.

They're reading - they just  look and see - even even in promotion exercises people are not - the committees often don't read the papers; they read the abstract see where it's published and top five , right.

And so that- that sort of elevates authority over over sort of like the how many other clever people with telescopes see the moons of Jupiter moving ,right

 So it's a it's a epistemological- fundamental epistemological change in how fields start to think about what's what's valuable

E:  I think that that is-

 OK, and so what do you see so the NIH role in this the sort of -how like, how you get this done, you know

I think that the levers you have to pull are you know how you're going to choose to fund- maybe you're going to start this journal, maybe you're going to build this AI tool box, like I'm curious how how you with the platform you have now are going to make this happen and whether that's you know this kind of leadership review of grants is a part of that or it's a-

J:  Leadership review of grants it's not- is not- it's again a portfolio construction thing is not any different-

 You should not be paying attention to fake news on this topic

OK just bluntly

E: So there's no leadership review of grants-

J: There's no- there's no political review of grants where like -

It's actually very destructive to bring that into this right now

I you want to talk about that, we can do this but we should talk about the replication thing first

 So first let me just say on the replication thing, all the three things I said we could just do, right

You're going to start seeing highlighted topics

E: Right

J: Asking people to like put in R level grant applications on replication

You're going to start- we're going to have a journal

We're gonna have our platform or whatever we can publish

 doing {natural science or} medicine, your replication work

 Everyone's replication work

 And then third on the on the pubMed it's going to look different

It'll have it'll have that replication button it will have that knowledge graph

all that in the works, we'll have that this year

E: That's awesome

J: The metrics we're talking about, there's there's a there's folks who do who do science of science that -

I don't know, I mean I work with them, I published with them outside, we're going to bring in and create a team of to create metrics along the lines of the sort I just talked about , right

We have a much broader set of metrics just you know the h-index or whatever

 all of that is in the works already at the NIH

It is going to look very - the scientific landscape will look very different when those tools are in place

 And I think you're underestimating the power of it because I think

 It's very easy to think that what the structure we currently have is the structure we will always have

 And it's-  it can be different ,right, doesn't it it's-  it's just a matter of incentives and the NIH is the single biggest funder of biomedical research

iI creates incentives

 Along with along these lines

By the way do you hear anything political any of this?

 I think I think that the problem is like there's a lot of bad faith conversations and thinking about this

E: Jay, let me just -  it's I think I'm not sure the conversation is as bad faith as you think

 I think one of the things is there's a lot of fear and -

J: Led by bad faith conversation.

{E and J talking over each other)

J: OK so let me let me let me just say that there's a there's a related problem to replication to the replication crisis which is stagnation

 Right so a few years back I did a paper with Mikko Packalin where we estimated how old were the ideas in NIH published research

And so like you know if you go back to the way we measure the age of ideas you can argue about this but like basically we just took all the words that were published in biomedicine in 1940 word and Word phrases, did in 1941 the fact that often the 1940 word you get the idea were produced in 1941.

 you keep doing that you get a history of biomedicine

you go back to every paper you can figure out how old are the ideas are

Turns out that papers by NIH funded researchers in in the in the 1980s were relying on ideas or like 0 one or two years old.

 and whereas like papers in the 2017 like they were relying on ideas that were like 7 or 8 years old

 This sort of like sort of like- partly ARPA-H exists because of this problem

ARPA-H exists because there was a fe- a sense that the NIH wasn't po- funding research at the bleeding edge

But it wasn't possible to like do the kind of like very very innovative things through the NIH process

 The the the unified funding strategy, which is you're you're calling this leadership review or whatever,

E: leadership review, yeah

J: yeah it's not called that. It's called the unified funding strategy.

E yeah, okay good.

J:  it's not the same. It's the unified funding strategy.

E; I know. it's a good name, I'm saying, unified funding strategy, I like it

J: So the unified funding strategy what it says is -

Just everyone in call knows this, but I want to like set the stage

What you do, is you have er, you have peer review like standard scientific review, you've served on peer review panels at NIH , I've served on peer review panels at NIH,

You ah, those- those panels produce an overall score based on sort of a series of subcomponents

Among the sub components are the methods for the paper-, for the proposal and the innovation of the proposal, {unintelligible} proposal.

And the overall score is some kind of gestalt, it's not some like average , it's like some gestalt draw- sort of pulled from those those sub components

It's  absolutely the case that the overall score correlates very strongly with methods but does not correlate quite so strongly with innovation.

 Just a  fact .

The old way of funding proposals in many many of the institutes that we have, they would take all of these the proposal they got. They will get 100,000 proposals a year , you can make the- overall each gets at least three reviews by the by peer reviewers, scientific review.

 You rank them and then what you do is you have a pay line that says we're going to fund everything that's in the top N percentile, depending on how much money Congress gave for it ,right

 That that basically, you can probably see, guarantees that there will be -  it'll corr- that portfolio will be filled with things that the peer reviwers are certain will work because the methods are strong.

E; okay

J:  But will leave out a lot of very very innovative scientific opportunities that will fall below the pay line 'cause the correlation with the overall score and the innovation scores is less than the in the correlation with the method score.

 Instead now I have empowered the institutes - and this is your leadership review - to make portfolios of scientific projects that match the science - the strategic plans that the institutes have already published, filled with like strategic opportunities for you in each field

And so like if there's 15 paper- applications for the same idea, and a whole bunch of the parts of the strategic plan can't get filled, because because for whatever

they can pick proposals that match the, sort of like, range of strategic plan

They can pick the innovative things and they can take account of the full range of nuanced conversation comes out of scientific review, rather than just the overall score when they make their portfolio (around 7.00)

And the goal of the portfolio then is to advance the field to advance health in the field that are coming out of the field, rather than just a whole bunch of public-  proposals that are guaranteed to produce top five papers but won't necessarily advance the field

 E; so basically this is more of a-  it's not a leadership review in the way that people are [imagining ]

J: [because there's a lot of bad faith ] conversations about this

E: No, no,  but I think this is -

This is why we're having this conversation but instead this is effectively what we call like a reweighting

You wanted them to reweight innovativeness above sort of established methods

J: yeah

And also it's a revaluation of how we {unintelligible} successes, right

So it's the idea is that the portfolio as a whole is more

It's a little more  Silicon Valley ish right

 So I don't care if any particular individual proposal or or project fails

What I care is that the portfolio as a whole actually advances the health of the population

Actually fundamentally transforms the biological fields that there's they're in

Um, there are other aspects that -

E: Can I just ask, has this meant approving fewer grants, because I know when you're going to tell me this is fake news and that's fine that's we're here to clear up fake news But like earlier this week the Times had an article saying that NIH had approved fewer grants than in [previous years]

J:  [it's just, it's just, first of all-]

E: is that not true?

J: it's  ridiculous, right

We had a 2 month shutdown, right, and so if you compare the cycle where we were last year at this time, we are roughly where we were last year this time, which was a year with a lot of disruption, and we still got all the grants out

 This is again a sort of politicised attack, instead of actually trying to engage with what the reforms actually are

So anyway, so the point is that you have these portfolios that you also-  have other goals of the of the of the- again set by the institutes themselves, the careers ,right ,

So including for instance elevating elevating the ideas of early career scientists

We have a problem, we have a major problem, right

 For for the last 30 years it's gotten worse and worse

Where where you have the time to first- like the age at which you get your first large grant, the age has gone up from the mid-80s was about 35 years old to now it's  like in your mid 40s.

We have a biological research ecosystem where you need 1,2,3 postdocs

Early career scientists are the ones who have the newer ideas

and so, like, when you when you elevate the new ideas you allow- essentially empower the institutes to have newer ideas funded, you're automatically organically going to fund early career scientists. You just are.

E:  Have you thought?

Can I ask a question about - back to the replication and this is an audience question by the way

If the audience has questions put them in the- put them in the chat

But one thing that that has been raised , and this came in earlier today, was this question of whether there should be positions that are specifically for replication so I think like an alternative approach to saying, you know, people like, you know, my students faculty should be incentivized to run replications

 You could say like inside the NIH we have like a replication group and that is, you know, people who are who are researchers

But where the incentives are very closely aligned to replication and where that's part of the goal

Have you thought about that solution or you think this kind of needs to live inside the Academy in the way you've talked about it

J:  I mean I'm not against that

It's hard to do that without like playing favourites I think

 E: Yeah

J: I would much rather the government not get involved in that

I rather end - the great thing about scientific peer review at the NIH is  that it really is walled off from that, right

So it's it's the scientific research community itself that decides what things are worthy of replication, and so they send that signal by deciding which of the grant applications received again from scientific community or high score or whatever

it's sort of offloads that outside of the political process because the NIH, just in your favour Emily, sorry to be so vociferous about this, you can probably work out,

But the key thing- the thing is NIH is a political institution right

It has to be because it solves a market value

So like you have to set guardrails, and  I take those guardrails very very seriously

so that you don't get that kind of political interference

 I think it's very very easy to envision a an NIH  that does play favourites you know with with leaders of the end of the night that devastating takedowns of other people that don't agree with for instance right I'm not going to do that

E: Fair enough

So you've also been the head of the CIA-  the CDC until

J: Not the CIA

E: Well that's probably next. Cos you have enough jobs.

J: I saw a Marco Rubio meme with my head on top. That is not good.

E; Not gonna happen!

So you're still the head of the CDC for the moment.

And I wanted to ask you about what you think the CDC needs to do

to achieve a goal that you have stated is your goal, which is to increase routine childhood vaccination rates

And I think this relates to some of the trust issues in general

 I think it's you know I got the platform and this is a this is a place that I think the people who I talked to really care about young people are really really worried their kids are gonna get measles and measles vaccination rates have been declining a lot

J: tell them to get their kids vaccinated and that's a very simple

E: I tell them every year

{E and J talking over each other}

[I wanna know how the CDC is going to]-

J: [OK so let me just say ]

So let me just say, I think, I think that-  this isn't about replication but {incomprehensible}. So this is a separate topic, just so-

E: Yes, it's a separate topic. But I'm asking about it.

J: Er, OK. So let me just say,  there's two sort of philosophies I've seen to try to do this

in order to -  before I get to the philosophies let me just say like the how how we got this moment right

 Actually let me do the philosophies first

Philosophy one is is is is a you essentially coerce people into doing it, right

You see the Covid vaccine is an example of this

You you essentially mandate it , say you can't you know- in some countries you can't go - you can't like leave outside your house , you can't get a job, you can't you know you can't you can't just live your basic life unless unless you get the vaccine and you will for sure with that approach get a higher rate of vaccination {unclear},  but you also demolish public trust

And that's that's that's one way

The second way, and this is like lot of Scandinavian countries do this, with with vaccination for instance is they they essentially trust the public to do the right thing

They give the public the right the sort of the the information the good and the bad and they make a recommendation based on that and they trust the public do the right thing

And you have this sort of like symbiotic trust between the public and public health, and you end up where there is no coercion

It's not seen as coercion, it's seen as a partnership

There's no mandate for instance in in Sweden but there's no mandates in Norway and there's no mandates in Denmark and they have 95, 98 plus percent vaccination rate for MMR.

Right so those are the two approaches I've seen proposed

We've tried the first approach in this country and it has failed, and especially post COVID

Post COVID where essentially, you know, the let's just be blunt like the recommendations regarding the COVID vaccination were- were coercive and frankly had nothing to do with what scientific evidence actually was saying regarding regarding

E; I don't disagree with that.

I think that was an enormous loss of public trust

I think my concern now is how we get that trust back on some of these things like the the vaccines which you know, where kids are going to die if they don't get the measles vaccine and I think we both care a lot about that

J: I do absolutely care about that I would love to see the MMR vaccination rates go higher

Now, the specifics are important here, right

So there are particular communities of people that have low really low MMR vaccination rates in the range of 70, 70%

How you reach those communities is is maybe different

Particular religious communities or or whatever , particular immigrant communities or whatever

That may be very different than how you reach the general public

In both cases though the key thing is to to establish trust and you don't do that by coercion

And let me be very specific

If you essentially censor the public from the kinds of concerns they have which is what the Biden administration did, right ,

They went to Facebook said you must stop these vaccine injury groups from talking to each other in privately or else you're going to face threat as a company, right,

That guarantees that you're going to maintain distrust in the particular communities that don't don't trust

You have to be open to allowing that conversation and not cancelling or trying to destroy people who have questions about about whatever vaccines you're talking about

E:  I could not agree with that more

Actually I think it it leads into the last- to the last question that I wanted to ask, which is how are you thinking about your dual role as a sort of political appointee who's head of these different things and your kind of like researcher hat on

 An example that came to mind in part because I think this would be so hard for me, is this question about this COVID vaccine efficacy report where they're

J: they just used a bad method

 Like they're using this test negative design

E: I think this method is ridiculous , 

J: yeah

E: I also don't like this method

I am not a fan of this method

We grew up in the same - totally agree

But!  this is also the method that you clearly just published a paper with the flu vaccine

so I'm so like - it must be such a wrestle. Of course you're losing public trust . When you don't publish it you're losing public trust, 

J: So Emily-

E: Even though I understand the method problem

J:  OK so OK so first, we agree the method is ridiculous like it's not going to produce the truth.  

E: It is ridiculous, it's not a great method.

J: Let me correct a bit of fake news right

So like the flu vaccine version of this paper that was published I think a week or so after- I was a week or two after I was made at CDC director,  that was cleared before I got in right

 I didn't see the thing cross my desk

E; I hear, OK

J: I know for a fact this method's crap

unuv Right right, so I can't as CDC director sign off on -

Now MNWR, the publication you're talking about, is not a peer-reviewed publication

It is literally the voice of the CDC

That that's- go back in history that is the that's ini- so the CDC is conveying its sense of what good science is and what it thinks public health things sort of research and public health concerns should be, when it publishes something in there

  When that paper reached my desk-

E: Do you feel here that like there is like a little piece of , you know, you're affecting public trust in vaccines or in the COVID vaccine in particular by pulling this

Like that it's your job is different than your job as referee 2.  

J: No I don't think so. I think that's -

E: Or do you think that scientific rigour should trump all?

J: I think that scientific rigour should trump -

I mean I'm not going to put out false information for the public to make decisions on

 If I do that, that's what that's what destroys public trust

Now there's been a lot of bad faith reaction to this

In particular the Washington Post and the Times because they are harming public health by not telling the truth about this -  the the poor sort of reliability of this method

like that I think that should be the story

E; I think that should be a story,  

I think there are many stories

Alright Jay thank you for putting up with me

 I love to talk  to  you

J: It was so good talking to you, my friend

E: It was really great to talk to you, and thank you everybody.