Guest post: Stealth corrections are still a threat to scientific integrity

Authors

René Aquarius, Floris Schoeters, Alex Glynn, Guillaume Cabanac

 

An update on stealth corrections

Last year, we published an article describing stealth corrections, a phenomenon in which a publisher makes at least one post-publication change to a scientific article, without providing a correction note or any other indicator that the publication was temporarily or permanently altered.

 

Now, we have expanded our database with newly identified stealth corrections. We also wrote a freely accessible COSIG guide describing how to report stealth corrections in a transparent fashion.

 

Difficult to pinpoint

Stealth corrections are, by nature, extremely difficult to track down and most stealth corrections are identified by science sleuths who might notice a mismatch between different versions of an article. It is impossible to provide a comprehensive overview and one must assume that we have only identified a small minority of these issues.

 

For this update we applied the same pragmatic approach in documenting stealth retractions as previously: registering stealth corrections on PubPeer ourselves, asking around within the science sleuthing community and searching the PubPeer database for terms as “no erratum”, “no corrigendum”, or “stealth” (repeat the search yourself).

 

Stealth corrections were further categorized into the following types:

• Changes in author information (addition or removal of authors, changes in author affiliation, etc.);
• Changes in content (figures, data or text, etc.);
• Changes in the record of editorial process (editor name, date of submission, acceptance or publication, etc.);
• Changes in additional information (ethics statements, conflicts of interest statements, funding information, etc.).

New cases

We found 32 published articles that were affected by stealth corrections in addition to the 131 we had identified last year. An overview of all stealth corrections (#1-163) can be found in the online database, which also contains the links to all accompanying PubPeer posts for additional detail. Table 1 shows the type of correction per publisher for the 32 new cases.

 

Table 1. Type of correction per publisher for newly identified stealth corrections. 

	Changes in additional information	Changes in author information	Changes in content	Changes in the record of editorial process
ACM	3	0	0	0
Am Phytopathological Soc	0	0	1	0
CV Literasi Indonesia	0	0	1	0
Elsevier	0	0	3	0
Impact Journals	0	0	1	0
Int Soc Computational Biology	1	0	0	0
MDPI	0	0	0	19
Oxford University Press	0	1	0	0
Springer Nature	0	0	1	0
Taylor & Francis	0	0	1	0

 

 

Why are stealth corrections still a thing?

Last year we wrote “post-publication amendments that are made silently, without a visible correction note, will give rise to questions regarding the ethics and integrity of the specific journal, editors and publisher, and might undermine the validity of the published literature as a whole”. Again, we have identified stealth corrections that might be used as a shortcut to ‘repair’ more serious issues. The three publishers with the most stealth corrections in this update were: MDPI, Elsevier, and the Association of Computing Machinery (ACM).

MDPI was involved in 19 new stealth corrections. Sixteen cases (#141, #144-158) were registered in August of 2024, too late for our initial pre-print and subsequent article on stealth corrections. All of these involved moving articles out of a ‘special issue’ and into a ‘section’. What stands out for all these 16 articles, is that the special issue editor was also an author on all of these papers. The Directory of Open Access Journals (DOAJ) has dictated that the number of articles co-authored by a special issue editor needs to be below lower than the 25% for each special issue. When it is higher than 25%, the DOAJ can delist the journal for not adhering to best practice, as detailed on their change log. Thus, by moving these articles silently out of special issues, MDPI is retroactively lowering this percentage to adhere to the rules of the DOAJ and therefore preventing potential delisting of their journals. In September 2024 -after publication of our pre-print- MDPI refuted “that removing a Special Issue article from the digital SI website can be considered a ‘stealth correction’”. Possibly, the updated correction process (which now includes ‘minor corrections’) facilitated a complete stop of this practice by MDPI. We have not identified any recent cases, which is an encouraging sign.

In the remaining three cases (#135-137), the name of a peer reviewer was suddenly set to anonymous, while the contents of the peer review reports did not change. According to MDPI, this was done to adhere to GDPR requirements. However, this only happened after the peer reviewer was identified as being part of a review mill. The reviewer claims on PubPeer that they were not involved in writing the peer review report. These cases prove that a request for anonymity might hamper the desire for transparency and strengthening research integrity.

Elsevier was involved in 3 new stealth corrections. All of them involved changes in content. In 3 cases an image was silently replaced (#133, #139-141) according to PubPeer reports that were posted between December 2024 and May 2025. In response to our pre-print, Elsevier stated that they “do not correct articles without a formal notice”. However, in this update we -again- present clear evidence of major changes to the scientific record that went through without any formal acknowledgement in the form of a correction notice. This directly contradicts earlier statements from Elsevier. Eventually, all of these articles have been retracted, but only 4-12 months after the stealth correction was noticed, meaning there was a substantial window of time that allowed for interaction with these flawed articles, without any proper indication that there might have been a problem.

ACM silently made multiple changes to the introduction from three conference proceedings written by the conference chair (#159-161). References were removed and in one case the text was heavily altered. In all three cases, a notice of concern was also published to indicate that the peer review process had been compromised and the publisher strongly urged people not to cite the conference papers. It seems as if the ACM retroactively tried to erase the citations to the conference papers, but they did it by secretly making all kinds of alterations to the documents, which is far from ideal.

This update shows that some scientific publishers continue to use stealth corrections as a way to change the scientific record. Stealth corrections can undermine the entire enterprise of science; at the level of the individual article, the lack of a transparent correction minimizes the likelihood of those who read or cited the original version being informed of the change; on the macro level, the integrity of the published literature as a whole is compromised as readers never know for certain whether an article has been silently corrected or not. Meanwhile, there is still no consensus on issuing corrections.

 

Conclusion and recommendations

Stealth corrections are still problematic as they are sometimes used as a shortcut to ‘repair’ other integrity issues. Again, we stress that stealth corrections are notoriously difficult to find and that this update likely only shows chance findings by science sleuths. Correct documentation and transparency are of the utmost importance to uphold scientific integrity and the trustworthiness of science.

We still recommend:

• Tracking of all changes to the published record by all publishers in an open, uniform and transparent manner, preferably by online submission systems that log every change publicly, making stealth corrections impossible.
• Clear definitions and guidelines on all types of corrections.
• Sustained vigilance of the scientific community to publicly register stealth corrections. Now made easier by using our COSIG guide.

 

Acknowledgements

We thank Dorothy Bishop for hosting this update on her blog and we thank all (anonymous) science sleuths who have found and reported stealth corrections: your work is much appreciated.

Note from DVMB: Comments are moderated on this blog.  They are usually approved if they are on topic and non-anonymous. 

My experience as a reviewer for MDPI

 

Guest post by 

René Aquarius, PhD

Department of Neurosurgery

Radboud University Medical Center, Nijmegen, The Netherlands

 

After a recent zoom-call where Dorothy and I discussed several research-related topics, she invited me to write a guest blogpost about the experience I had as a peer-reviewer for MDPI. As I think transparency in research is important, I was happy to accept this invitation.  

 

Mid November 2023 I received a request to peer-review a manuscript for a special issue on subarachnoid hemorrhage for the Journal of Clinical Medicine, published by MDPI. This blog post summarizes that process. I hope it will give some insight on the nitty-gritty of the peer-review process for MDPI.

 

I decided to review the manuscript two days after receiving the invitation and what I found was a study like many others in the field: a single-center, retrospective analysis of a clinical case series. I ended up recommending rejection of the paper two days after accepting to review. My biggest gripes were that the authors claimed that data were collected prospectively, but their protocol was registered at the very end of the period in which they included patients. In addition, I discovered some important discrepancies between protocol and the final study. Target sample size according to the protocol was 50% bigger than what was used in their study. The minimum age for patients also differed between the protocol and the manuscript. I also had problems with their statistical analysis as they used more than 20 t-tests to test variables, which creates a high probability of Type I errors. The biggest problem was the lack of a control group, which made it impossible to establish whether changes in a physiological parameter could really predict intolerance for a certain drug in a small subset of patients.

 

When filling out the reviewer form for MDPI, certain aspects struck me as peculiar. There are four options for Overall Recommendation:

• Accept in present form
• Accept after minor revision (correction to minor methodological errors and text editing)
• Reconsider after major revision (control missing in some experiments)
• Reject (article has serious flaws, additional experiments needed, research not conducted correctly)

 

Regardless of which of the last two options you select, the response is: "If we ask the authors to revise the manuscript, the revised manuscript will be sent to you for further evaluation". 

 

Although reviewer number 2 is often jokingly referred to as "the difficult one" it couldn’t be further from the truth in this case. The reviewer liked the paper and recommended accept after minor revision. So with a total of two reviews, the paper got the editorial decision of rejected, with a possibility of resubmission after extensive revisions only one day after I handed in my peer review report.

 

Revisions were quite extensive, as you will discover below, and arrived only two days after the initial rejection. I agreed to review the revised manuscript. But before I could start my review of the revision, just four days after receiving the invitation, I received a response from the editorial office that my review was no longer needed because they already had enough peer-reviewers for the manuscript. I politely ignored this request, because I wanted to know if the manuscript had improved. What happened next was quite a bit of a surprise, but not in a good way. 

 

The manuscript had indeed undergone extensive revisions. The biggest change, however, was also the biggest red flag. Without any explanation the study had lost almost 20% of its participants. An additional problem was that all the issues I had raised in my previous review report remained unaddressed. I sent my newly written feedback report the same day, exactly one week after my initial rejection.

 

When I handed in my second review report, I understood why I initially got an email that my review was not needed anymore. One peer reviewer had also rejected the manuscript and had concerns similar to mine. Two other reviewers, however, accepted the manuscript. One with minor revisions (English needed some improvement) and one in present form, so without any suggested revisions. This means that if I had followed the advice of the editorial office of MDPI, the paper would probably have been accepted in its current form. But because my vote was now also cast and the paper received two rejections, the editor couldn’t do much more than to reject the manuscript, which happened three days after I handed in my review report.  

 

Fifteen days after receiving my first invitation to review, the manuscript had already seen two full rounds of peer-review by at least four different peer-reviewers.

 

This is not where the story ends.  

 

In December, about a month later, I received an invitation to review a manuscript for the MDPI journal Geriatrics. You’ve guessed it by now: it was the same manuscript. It's reasonable to assume this was shifted internally through MDPI's transfer service, summarised in this figure.  I can only speculate as to why I was still attached to the manuscript as a peer-reviewer, but I guess somebody forgot to remove my name from it.

from: https://www.mdpi.com/authors/transfer-service

The manuscript had, again, transformed. It was now very similar to the very first version I reviewed. Almost word-for-word similar. That also meant that the number of included patients was restored to the initial number. However, the registered protocol that was previously mentioned in the methods section (which had led to some of the most difficult to refute critiques) was now completely left out. The icing on the cake was that, for a reason that was not explained, another author was added to the manuscript. There was no mention in this invitation of the previous reviews and rejections of the same manuscript.   Although one might wonder whether MDPI editors were aware of this, it would be strange if they were not, since they pride themselves on their Susy manuscript submission system where "editors can easily track concurrent and previous submissions from the same authors".

 

Because the same issues were still present in the manuscript, I rejected it for a third time on the same day I agreed to review it. In an accompanying message to the editor, I clearly articulated my problems with the manuscript and the review process.

 

The week after, I received a message that the editor had decided to withdraw the manuscript in consultation with the authors.

 

Late January 2024, the manuscript was published in the MDPI journal Medicina. I was not attached to the manuscript any more as a reviewer. There was no indication on the website of the name of the acting editor who accepted it. 

Note from Dorothy Bishop

Comments on this blog are moderated so there may be some delay before they appear, but legitimate, on-topic contributions are welcomed. We would be particularly interested to hear from anyone else who has experiences, good or bad, as a reviewer for MDPI journals.

 

Postscript by Dorothy Bishop: 19 Aug 2024 

Here's an example of a paper that was published with the reviews visible. Two were damning and one was agreeable.  https://www.mdpi.com/2079-6382/9/12/868.  Thanks to @LymeScience for drawing our attention to this, and noting the important clinical consequences when those promoting an alternative, non-evidenced treatment have a "peer-reviewed" study to refer to. 

Some thoughts on eLife's New Model: One year on

 

I've just been sent an email from eLife, pointing me to links to a report called "eLife's New Model: One year on" and a report by the editors "Scientific Publishing: The first year of a new era". To remind readers who may have missed it, the big change introduced by eLife in 2023 was to drop the step where an editor decides on reject or accept of a manuscript after reviewer comments are received. Instead, the author submits a preprint, and the editors then decide whether it should be reviewed. If the answer is yes, then the paper will be published, with reviewer comments. 

Given the controversy surrounding this new publishing model, it seems timely to have a retrospective look at how it's gone, and these pieces by the journal are broadly encouraging in showing that the publishing world has not fallen apart as a consequence of the changes. We are told that the proportion of submissions published has gone down slightly from 31.4% to 27.7% and the demographic characteristics of authors and reviewers are largely unchanged. The ratings of quality of submissions are similar to those from the legacy model. The most striking change has been in processing time: median time from submission to publication of the first version with reviews is 91 days, which is much faster than previously. 

As someone who has been pushing for changes to the model of scientific publishing for years (see blogsposts below), I'm generally in favour of any attempt to disrupt the conventional model. I particularly like the fact that the peer reviews are available with the published articles in eLife - I hope that will become standard for other journals in future. However, there are two things that rather rankled about the latest communication from the journal. 

First, the report describes an 'author survey' which received 325 responses, but very little detail is given as to who was surveyed, what the response rate was, and what the overall outcome was. This reads more like a marketing report than a serious scientific apprasal. Two glowing endorsements were reported from authors who had good experiences. I wondered though about authors whose work had not been selected to go forward to peer review - were they just as enthusiastic? Quite a few tables of facts and figures about the impact of the new policy were presented with the report, but if eLife really does want to present itself as embracing open and transparent policies, I think they should bite the bullet and provide more information - including fuller details of their survey methods and results, and negative as well as positive appraisals. 

Second, I continue to think there is a fatal flaw in the new model, which is that it still relies on editors to decide which papers go forward to review, using a method that will do nothing to reduce the tendency to hype and the consequent publication bias that ensues. I blogged about this a year ago, and suggested a simple solution, which is for the editors to adopt 'results-blind' review when triaging papers. This is an idea that has been around at least since 1976 (Mahoney, 1976) which has had a resurgence in popularity in recent years, with growing awareness of the dangers of publication bias (Locasio, 2017). The idea is that editorial decisions should be made based on whether the authors had identified an interesting question and whether their methods were adequate to give a definitive answer to that question. The problem with the current system is that people get swayed by exciting results, and will typically overlook weak methods when there is a dramatic finding. If you don't know the results, then you are forced to focus on the methods. The eLife report states:

 "It is important to note that we don’t ascribe value to the decision to review. Our aim is to produce high-quality reviews that will be of significant value but we are not able to review everything that is submitted." 

That is hard to believe: if you really were just ignoring quality considerations, then you should decide on which papers to review by lottery. I think this claim is not only disingenuous but also wrong-headed. If you have a limited resource - reviewer capacity - then you should be focusing it on the highest quality work. But that judgement should be made on the basis of research question and design, and not on results. 

Bibliography 

Locascio, J. J. (2017). Results blind science publishing. Basic and Applied Social Psychology, 39(5), 239–246. https://doi.org/10.1080/01973533.2017.1336093 

Mahoney, M. J. (1976). Scientist as Subject: The Psychological Imperative. Ballinger Publishing Company. 

Previous blogposts

Academic publishing: why isn't psychology like physics? 

Time for academics to withdraw free labour.

High impact journals: where newsworthiness trumps methodology

Will traditional science journals disappear?

Publishing replication failures

The impact of blogging on reputation

I was alerted this morning on Twitter to this blogpost by Brian LePort on the first of 5 reasons why students shouldn't blog. Its central thesis is that "it is almost impossible to avoid writing something that will offend someone". Consequently, bloggers run the risk of doing themselves reputational harm at best, or failing to get a job or even getting fired at worst.

LePort illustrates his thesis by the extraordinary case of Christopher Rollston, who tells how he was forced to resign from a post at Emmanuel Christian Seminary because he wrote a piece for the Huffington Post on the marginalization of women in the Bible. Rollston, who describes himself as a Christian, concluded: "Gender equality may not have been the norm two or three millennia ago, but it is essential. So, the next time someone refers to 'biblical values,' it's worth mentioning to them that the Bible often marginalized women and that's not something anyone should value." Apparently, a major funder of the seminary disapproved of such incendiary sentiments and Rollston's career there was toast.

I have to say, I find LePort's reaction to this story disappointing. Yes, people who blog should think carefully about what they say and the impact it may have. Yes, it's impossible to avoid offending someone somewhere, unless what you write is so boring and anodyne that nobody would want to read it. But I despair at the idea of a future generation so cowed with fear that nobody ever says anything original or controversial.

I'm not arguing that students and junior academics should sacrifice themselves on the altar of freedom of speech, but rather that they should have confidence in the positive as well as the negative power of the internet. If what they say is worth saying, they will get support. LePort focuses on the negative consequences of Rollston's blogging, but, as this post by Robert Cargill pointed out, he attracted huge support online and ended up in a better job, whereas Emmanuel Christian Seminary suffered massive reputational damage.

LePort makes the important point that blogs are very different to more formal academic writing and often represent a point of view at a particular point in time, which may subsequently change. To my mind, this is one of the huge benefits of blogging – if you are lucky, your blog will attract comments that expose you to a wide range of reactions and help clarify and develop your thinking. This can be both fun and useful. LePort worries, though, that this may mean your incomplete and half-baked thoughts on an issue are used against you by those in positions of authority.

As a senior academic, I hope I can offer some reassurance. In general, I see blogging as an indication that the author is a bit out of the ordinary – someone who cares enough about things to write about them, and who is willing to try and move discussion forward. If in addition they change their views on the basis of feedback, that's fine. Obviously, it's possible to reveal yourself on a blog as uninformed, irrational or bigoted, and that is definitely not good. But most of the blogs I read aren't like that.

Well, I can hear you saying, that's all very well. You are someone who actually blogs and understands social media, but most academics aren't like that. My reply is that social media is an unstoppable force and even the most traditional institutions are starting to focus on developing strategies for harnessing its power.  So I'd say, yes, LePort is right in that we need to be aware that blogging is a public medium, and anything we say on a blog can be read by anyone. But it would be a shame if we allowed ourselves to become so worried about potential problems that we failed to see the advantages of blogging for fostering academic debate.That would be like staying at home with the door locked because you're scared of what may happen if you go outside.

C’mon sisters! Speak out!

When I give a talk, I like to allow time for questions. It’s not just a matter of politeness to the audience, though that is a factor. I find it helps me gauge how the talk has gone down: what points have people picked up on, are there things they didn’t get, and are there things I didn’t get? Quite often a question coming from left field gives me good ideas. Sometimes I’m challenged and that’s good too, as it helps me either improve my arguments or revise them. But here’s the thing. After virtually every talk I give there’s a small queue of people who want to ask me a private question. Typically they’ll say, “I didn’t like to ask you this in the question period, but…”, or “This probably isn’t a very sensible thing to ask, but…”. And the thing I’ve noticed is that they are almost always women. And very often I find myself saying, “I wish you’d asked that question in public, because I think there are lots of people in the audience who’d have been interested in what you have to say.”

I’m not an expert in gender studies or feminism, and most of my information about research on gender differences comes from Virginia Valian’s scholarly review, Why So Slow. Valian reviews studies confirming that women are less likely than men to speak out in question sessions in seminars. I have to say my experience in the field of psychology is rather different, and I'm pleased to work in a department where women’s voices are as likely to be heard as men’s. But there’s no doubt that this is not the norm for many disciplines, and I've attended conferences, and given talks, where 90% of questions come from men, even when they are a minority of the audience.

So what’s the explanation? Valian recounts personal experiences as well as research evidence that women are at risk of being ignored if they attempt to speak out, and so they learn to keep quiet. But, while I'm sure there is truth in that, I find myself irritated by what I see as a kind of passivity in my fellow women. It seems too easy to lay the blame at the feet of nasty men who treat you as if you are invisible. A deeper problem seems to be that women have been socially conditioned to be nervous of putting their heads above the parapet. It is really much easier to sit quietly in an audience and think your private thoughts than to share those thoughts with the world, because the world may judge you and find you lacking. If you ask women why they didn’t speak up in a seminar, they’ll often say that they didn’t think their question was important enough, or that it might have been wrong-headed. They want to live life safely and not draw attention to themselves. This affects participation in discussion and debate at all stages of academic life - see this description of anxiety about participating in student classes. Of course, this doesn’t only affect women, nor does it affect all women. But it affects enough women to create an imbalance in who gets heard.

We do need to change this. Verbal exchanges after lectures and seminars are an important part of academic life, and women need to participate fully. There’s no point in encouraging men to listen to women’s voices if the women never speak up. If you are one of those silent women, I urge you to make an effort to overcome your bashfulness. You’ll find it less terrifying than you imagine, and it gets easier with practice. Don’t ask questions just for the sake of it, but when a speaker sparks off an interesting thought, a challenging question, or just a need for clarification, speak out. We need to change the culture here so that the next generation of women feel at ease in engaging in verbal academic debate.

Novelty, interest and replicability

So at last, your paper is written. It represents the culmination of many years’ work. You think is an important advance for the field. You write it up. You carefully format it for your favoured journal. You grapple with the journal’s portal, tracking down details of recommended reviewers and then sit back. You anticipate a delay of a few weeks before you get reviewer comments. But, no. What’s this? A decision letter within a week: “Unfortunately we receive many more papers than we can publish or indeed review and must make difficult decisions on the basis of novelty and general interest as well as technical correctness.” It’s the publishing equivalent of the grim reaper: a reject without review.

It happens increasingly often, especially if you send work to journals with high impact factors. I’ve been an editor and I know there are difficult decisions to make. It can be kinder to an author to reject immediately if you sense that the paper isn’t going to make it through the review process. One thing you learn as an author is that there’s no point protesting or moaning. You just try again with another journal. I’m confident our paper is important and will get published, and there’s no reason for me to single this journal out for complaint. But this experience has made me reflect more generally on factors affecting publication, and I do think there are things about the system that are problematic.

So, using this blog as my soapbox, there are two points I’d like to make: A little one and a big one. Let’s get the little one out of the way first. It’s simply this: if a journal commonly rejects papers without review, then it shouldn’t be fussy about the format in which a paper is submitted. It’s just silly for busy people to spend time getting the references correctly punctuated, or converting their figures to a specific format, if there’s a strong probability that their paper will be bounced. Let the formatting issues be addressed after the first round of review.

The second point concerns the criteria of “novelty and general interest”. My guess is that our paper was triaged on the novelty criterion because it involved replication. We reported a study that involved measuring electrical brain responses to sounds. We compared these responses in children with developmental language impairments and typically-developing children. The rationale is explained in a blogpost I wrote for the Wellcome Trust.

We’re not the first people to do this kind of research. There have been a few previous studies, but it’s a fair summary to say the literature is messy. I reviewed part of it a few years back and I was shocked at how bad things were. It was virtually impossible to draw any general conclusions from 26 studies. Now these studies are really hard to do. Just recruiting people is difficult and it can take months if not years to get an adequate sample. Then there is the data analysis which is not for the innumerate or faint-hearted. So a huge amount of time and money had gone into these studies, but we didn’t seem to be progressing very far. The reason was simple: you couldn’t generalise because nobody ever attempted to replicate previous research. The studies were focussed on the same big questions, but they differed in important ways. So if they got different results, you couldn’t tell why.

In response to this, part of my research strategy has been to take those studies that look the strongest and attempt to replicate them. So when we found strikingly similar results to a study by Shafer et al (2010) I was excited. The fact that two independent labs on different sides of the world had obtained virtually the same result gave me confidence in the findings. I was able to build on this result to do some novel analyses that helped establish direction of causal influences, and felt we at last we were getting somewhere. But my excitement was clearly not shared by the journal editor, who no doubt felt our findings were not sufficiently novel. I wasn’t particularly surprised by this decision, as this is the way things work. But is the focus on novelty good for science?

The problem is that unless novel findings are replicated, we don’t know which results are solid and reliable. We ought to know: we apply statistical methods with the sole goal of establishing this. But in practice, statistics are seldom used appropriately. People generate complex datasets and then explore different ways of analysing data to find statistically significant results. In electrophysiological studies, there are numerous alternative ways in which data can be analysed, by examining different peaks in a waveform, different methods of identifying peaks, different electrodes, different time windows, and so on. If you do this, it is all too easy for “false positives” to be mistaken as genuine effects (Simmons, Nelson, & Simonsohn, 2011). And the problem is compounded by the “file drawer problem” whereby people don’t publish null results. Such considerations led Ioannidis (2005) to conclude that most published research findings are false.

This is well-recognised in the field of genetics, where it became apparent that most early studies linking genetic variants to phenotypes were spurious (see Flint et al). The reaction, reflected in a recent editorial in Behavior Genetics has been to insist that authors replicate findings of associations between genes and behaviour. So if you want to say something novel, you have to demonstrate the effect in two independent samples.

This is all well and good, but requiring that authors replicate their results is unrealistic in a field where a study takes several years to complete, or involves a rare disorder. You can, however, create an expectation that researchers include a replication of prior work when designing a study, and/or use existing research to generate a priori predictions about expected effects.

It wouldn’t be good for science if journals only published boring replications of things we already knew. Once a finding is established as reliable, then there’s no point in repeating the study. But something that has been demonstrated at least twice in independent samples (replicable) is far more important to science than something that has never been shown before (novel), because the latter is likely to be spurious. I see this as a massive challenge for psychology and neuroscience.

In short, my view is that top journals should reverse their priorities and treat replicability as more important than novelty.

Unfortunately, most scientists don’t bother to attempt replications because they know the work will be hard to publish. We will only reverse that perception if journal editors begin to put emphasis on replicability.

A few individuals are speaking out on this topic. I recommend a blogpost by Brian Knutson who argued, “Replication should be celebrated rather than denigrated.” He suggested that we need a replicability index to complement the H-index. If scientists were rewarded for doing studies that others can replicate, we might see a very different rank ordering of research stars.

I leave the last word to Kent Anderson: “Perhaps we’re measuring the wrong things … Perhaps we should measure how many results have been replicated. Without that, we are pursuing a cacophony of claims, not cultivating a world of harmonious truths.”

Simmons, J., Nelson, L., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant Psychological Science, 22 (11), 1359-1366 DOI: 10.1177/0956797611417632