Monday 20 April 2015

How long does a scientific paper need to be?





©CartoonStock.com

There was an interesting exchange last week on PubMedCommons between Maurice Smith, senior author of a paper on motor learning, and Bjorn Brembs, a neurobiologist at the University of Regensburg. The main thrust of Brembs' critique was that the paper, which was presented as surprising, novel and original, failed adequately to cite the prior literature. I was impressed that Smith engaged seriously with the criticism, writing a reasoned defence of the choice of material in the literature review, and noting that claims of over-hyped statements were based on selective citation.  What really caught my attention was the following statement in his rebuttal: "We can reassure the reader that it was very painful to cut down the discussion, introduction, and citations to conform to Nature Neuroscience’s strict and rather arbitrary limits. We would personally be in favor of expanding these limits, or doing away with them entirely, but this is not our choice to make."
As it happens, this comment really struck home with me, as I had been internally grumbling about this very issue after a weekend of serious reading of background papers for a grant proposal I am preparing. I repeatedly found evidence that length limits were having a detrimental effect on scientific reporting. I think there are three problems here.
1. The first is exemplified by the debate around the motor learning paper. I don't know this area well enough to evaluate whether omissions in the literature review were serious, but I am all too familiar with papers in my own area where a brief introduction skates over the surface of past work. One feels that length limits play a big part in this but there is also another dimension: To some editors and reviewers, a paper that starts documenting how the research builds on prior work is at risk of being seen as merely 'incremental', rather than 'groundbreaking'. I was once explicitly told by an editor that too high a proportion of my references were more than five years old. This obsession with novelty is in danger of encouraging scientists to devalue serious scholarship as they zoom off to find the latest hot topic.  
2. In many journals, key details of methods are relegated to a supplement, or worse still, omitted altogether. I know that many people rejoiced when the Journal of Neuroscience declared it would no longer publish supplementary material: I thought it was a terrible decision. In most of the papers I read, the methodological detail is key to evaluating the science, and if we only get the cover story of the research, we can be seriously misled. Yes, it can be tedious to wade through supplementary material, but if it is not available, how do we know the work is sound?
3. The final issue concerns readability. One justification for strict length limits is that it is supposed to benefit readers if the authors write succinctly, without rambling on for pages and pages.  And we know that the longer the paper, the fewer people will even begin to read it, let alone get to the end. So, in principle, length limits should help. But in practice they often achieve the opposite effect, especially if we have papers reporting several experiments and using complex methods. For instance, I recently read a paper that reported, all within the space of a single Results section about 2000 words long, (a) a genetic association analysis; (b) replications of the association analysis on five independent samples (c) a study of methylation patterns; (d) a gene expression study in mice; and (e) a gene expression study in human brains. The authors had done their best to squeeze in all essential detail, though some was relegated to supplemental material, but the net result was that I came away feeling as if I had been hit around the head by a baseball bat. My sense was that the appropriate format for reporting such a study would have been a monograph, where each component of the study could be given a chapter, but of course, that would not have the kudos of a publication in a high impact journal, and arguably fewer people would read it.
Now that journals are becoming online-only, a major reason for imposing length limits – cost of physical production and distribution of a paper journal – is far less relevant. Yes, we should encourage authors to be succinct, but not so succinct that scientific communication is compromised.


14 comments:

  1. I also have experienced this problem. Here is an extract from the instruction to authors from a well known review journal. "Concentrate on the seminal references of the past 2–4 years (most references should be no more than 5 years old)." Nevertheless I managed to quote Helmholtz (1876) and even Spinoza (1677).

    ReplyDelete
  2. I tend to agree, but in the last example, wouldn't it have been better to split the results into two (or more) separate papers?

    ReplyDelete
  3. We don't have a word or manuscript limit on our new review journal, Psychopathology Review http://pr.textrum.com, which seems to suit authors well (other similar clinical review journals seem to have very tight manuscript page limits) - but we still ask authors to be succinct!

    ReplyDelete
  4. This reminds me of a quip by philosopher Richard Smith who remarked how peculiar it was that all knowledge fits neatly into chunks of 5000-7000 words. Surely, there must be some epistemological universal at play.

    I'd suggest that things are even worse. The word limits come from the days of print when paper was the limiting factor (it still must be to a certain extent) and the amount of text peer reviewers are willing to read. The former has been obsoleted by e-publications but the latter may still be a factor. But surely, it wouldn't be too hard to split a paper into the argument section (concise), the background section (infinitely long), methodology section (as long as need be for replication) and data section (complete dump - with possible limits due to anonymity, etc.) The 'journals' could then 'publish' the argument with links to all the other sections which the reviewers and readers could refer to as needed.

    My favourite example (from the humanities) is Ian Morris who published Why the west rules for now (a substantial book) and the published all his background research online in a PDF as substantial as the officially published volume. Since I was intrigued enough by some of the arguments in the original book, I looked up the details which (for me) tempered the argument (no doubt influenced by the editors just like the preposterous title) proposed in the published book.

    ReplyDelete
  5. Indeed, in today's day and age, word limits and even more perniciously reference limits, are detrimental to communicating your results. I may not agree with Smith's other arguments, but there can be no doubt that he is very correct that these limits must go.
    Obviously, we are not going to say "pretty please" and NPG will do our bidding. The only way we can influence publishers is by refusing to give them what they need: subscriptions. I'm urging my university library to cancel subscriptions, so should everybody who wants to increase our influence over how we communicate science.

    ReplyDelete
    Replies
    1. We can also put pressure on publishers by sending our work elsewhere and on colleagues by not citing incomplete work (i.e., missing methodological details).

      Delete
    2. Absolutely. I agree that this of course would, in principle, work. However, this would require collective action of a lot more people: there are only about 10k institutions in the world, but hundreds of thousands of scientists.

      Delete
  6. Were there really 'good old times' which we have lost 'in today's day and age'?

    When it comes to article length, the answer could be yes. I did an analysis of all reviews in Nature Reviews Neuroscience.
    There is actually a trend towards less and less references in articles, from around 150 at its inception to around 100 these days.

    What has changed? The number of articles scientists read has increased between the 70's and today from around 150 per year to around 300 per year. However, the time people invest in reading has not changed or even declined.

    So, journal editors have adapted to a change in reading patterns towards quick looks at short papers instead of lengthy reads.

    Please do not trust my numbers without seeing where they come from:
    https://brainsidea.wordpress.com/2012/12/18/how-long-should-a-scientific-publication-be/

    ReplyDelete
  7. Thank you, Dorothy, for so eloquently expressing this very important issue. It's just one of the many reasons why I will never submit a paper to one of those "high-impact" journals. What a tragedy that Science is the opposite of science.

    ReplyDelete
    Replies
    1. I agree with you Mike. But, do you know what is the greater travesty? Admissions will still focus us to publish in high impact factors like Science as it will be related to better funding and more grant proposals to be accepted. This in turn, will delve and destroy science's underlying beauty and rigorousness to find the truth, leading to younger researchers to perform worse in finding the truth, causing the literature to be fraught with more type I errors. What I am saying is, I think we need to remove the word limit to allow everyone to state what is important for replication and to remind people 5 years is not old and the past can enlighten our future. We need someone who publishes new and interesting ideas that is based off papers from the 1920s-1990s. This will give them insight that old papers alongside new papers are fundamental to move the field forward.

      Delete
  8. Entirely agree. Furthermore, some people have documented that online supplements have a tendency to become inaccessible:

    FASEB J. 2005 Dec;19(14):1943-4.
    Unavailability of online supplementary scientific information from articles published in major journals.
    Evangelou E1, Trikalinos TA, Ioannidis JP.
    Abstract
    Printed articles increasingly rely on online supplements to store critical scientific information, but such data may eventually become unavailable. We checked the current availability of online supplementary scientific information published in six top-cited scientific journals (Science, Nature, Cell, New England Journal of Medicine, Lancet, Proceedings of the National Academy of Sciences USA). Here we show that in 4.7% and 9.6% of articles with online supplementary material, some of the supplements became unavailable within 2 and 5 years of their publication, respectively.

    ReplyDelete
    Replies
    1. I suspect that things have become both worse and better since the 2005 paper you cite. Worse in that supplementary materials to papers published between 1995-2005 or so are now probably mostly gone. And better in that there are now many more ways to store them more sustainably from the generic like GitHub to the specialised like Data Verse. But the journals should be storing any supplementary material with the article. There should also be be a role for the government to have some sort of a national data library for all publicly funded research.

      Delete
  9. Beyond length limitations, another perverse invention of some so-called top journals is to relegate Methods to the end of the paper, making it impossible to understand and critically evaluate the results if you read the article in the proposed order.

    ReplyDelete
    Replies
    1. Totally agree. As if methods aren't important.

      Delete