Better control of the publication time-line: A further benefit of Registered Reports

I’ve blogged previously about waste in science. There are numerous studies that are completed but never see the light of day. When I wrote about this previously, I focused on issues such as reluctance of journals to publish null results, and the problem of writing up a study while applying for the next new grant. But here I want to focus on another factor: the protracted and unpredictable process of peer review that can lead to researchers to just give up on a paper.

Sample Gantt chart. Source: http://www.crp.kk.usm.my/pages/jepem.htm

The sample Gantt chart above nicely illustrates a typical scenario.  Let's suppose we have a postdoc with 30 months’ funding. Amazingly, she is not held up by patient recruitment issues, or ethics approvals, and everything goes according to plan, so 24 months in, she writes up the study and submits it to a journal. At the same time, she may be applying for further funding or positions. She may plan to start a family at the end of her fellowship. Depending on her area of study it may take anything from two weeks to six months to hear back from the journal*. The decision is likely to be revise and resubmit. If she’s lucky, she’ll be able to do the revisions and get the paper accepted to coincide with the end of her fellowship.  All too often, though, the reviewers suggest revisions. If she's very unlucky they may demand additional experiments, which she has no funding for.  If they just want changes to the text, that's usually do-able, but often they will suggest further analyses that take time, and she may only get to the point of resubmitting the manuscript when her money runs out. Then the odds are that the paper will go back to the reviewers – or even to new reviewers – who now have further ideas of how the paper can be improved. But now our researcher might have started a new job, have just given birth, or be unemployed and desperately applying for further funds.

The thing about this scenario, which will be all too familiar to seasoned researchers (see a nice example here), is that it is totally unpredictable. Your paper may be accepted quickly, or it may get endlessly delayed. The demands of the reviewers may involve another six month’s work on the paper, at a point when the researcher just doesn’t have the time. I’ve seen dedicated, hardworking, enthusiastic young researchers completely ground down by this situation, faced by the choice of either abandoning a project that has consumed a huge amount of energy and money, or somehow creating time out of thin air. It’s particularly harsh on those who are naturally careful and obsessive, who will be unhappy at the idea of doing a quick and dirty fix to just get the paper out. That paper which started out as their pride and joy, representing their best efforts over a period of years is now reduced to a millstone around the neck.

But there is an alternative. I’ve recently, with a graduate student, Hannah Hobson, put my toe in the waters of Registered Reports, with a paper submitted to Cortex looking at an electrophysiological phenomenon known as mu suppression. The key difference from the normal publication route is that the paper is reviewed before the study is conducted, on the basis of an introduction and protocol detailing the methods and analysis plan. This, of course takes time – reviewing always does. But if and when the paper is approved by reviewers, it is provisionally accepted for publication, provided the researchers do what they said they would.

One advantage of this process is that, after you have provisional acceptance of the submission, the timing is largely under your own control. Before the study is done, the introduction and methods are already written up, and so once the study is done, you just add the results and discussion. You are not prohibited from doing additional analyses that weren’t pre-registered, but they are clearly identified as such. One the study is written up the paper goes back to reviewers. They may make further suggestions for improving the paper, but what they can’t do is to require you to do a whole load of new analyses or experiments. Obviously, if a reviewer spots a fatal error in the paper, that is another matter. But reviewers can’t at this point start dictating that the authors do further analyses or experiments that may be interesting but not essential.

We found that the reviewer comments on our completed study were helpful: they advised on how to present the data and made suggestions about how to frame the discussion. One reviewer suggested additional analyses that would have been nice to include but were not critical; as Hannah was working to tight deadlines for thesis completion and starting a new job, we realised it would not be possible to do these, but because we have deposited the data for this paper (another requirement for a Registered Report), the door is left open for others to do further analysis.

I always liked the idea of Registered Reports, but this experience has made me even more enthusiastic for the approach. I can imagine how different the process would have been had we gone down the conventional publishing route. Hannah would have started her data collection much sooner, as we wouldn’t have had to wait for reviewer comments. So the paper might have been submitted many months earlier. But then we would have started along the long uncertain road to publication. No doubt reviewers would have asked why we didn’t include different control conditions, why we didn’t use current source density analysis, why we weren’t looking at a different frequency band, and whether our exclusionary criteria for participants were adequate. They may have argued that our null results arose because the study was underpowered. (In the pre-registered route, these were all issues that were raised in the reviews of our protocol, so had been incorporated in the study). We would have been at risk of an outright rejection at worst, or requirement for major revisions at best. We could then have spent many months responding to reviewer recommendations and then resubmitting, only to be asked for yet more analyses.  Instead, we had a pretty clear idea of the timeline for publication, and could be confident it would not be enormously protracted.

This is not a rant against peer reviewers. The role of the reviewer is to look at someone else’s work and see how it could be improved. My own papers have been massively helped by reviewer suggestions, and I am on record as defending the peer review system against attacks. It is more a rant against the way in which things are ordered in our current publication system. The uncertainty inherent in the peer review process generates an enormous amount of waste, as publications, and sometimes careers, are abandoned. There is another way, via Registered Reports, and I hope that more journals will start to offer this option.

*Less than two weeks suggests a problem!See here for an example.