The Discussion section: kill it or reform it?

 

I’m impelled to write a short piece about Discussion sections, after a bit of to and fro on Twitter, which started with @SchoeneggerPhil tweeting about a new paper by Schoenegger and Pils: 

 @SchoeneggerPhil: New paper out with @PilsRaimund! We propose a new solution to the crises facing the social sciences: removing the discussion sections. We argue that they harm honest scientific reporting and would provide epistemic benefits if outsourced from the standard article.

Michael C Frank then remarked: 

@mcxfrank: Hot take: no one reads discussions so removing them will improve efficiency but not solve interpretive crises (which are driven by titles and abstracts).

Well, as someone who enjoys reading and writing discussion sections, I found this all very depressing. I argued that we already have a solution to the problems Schoenegger and Pils were trying to fix – Registered Reports. In addition, I said:

@deevybee: It seems you are so worried about people HARKing and otherwise misrepresenting data, that you end up preventing exploratory research and speculation. I see them as crucial for science; it's just a case of clarifying what is hypothesis-testing and what is not.

Russ Poldrack started further discussion by weighing in on the side of Schoenegger/Pils: 

@russpoldrack: I see this as a separate issue from the utility of the discussion section, which is what I liked about the original post. I agree that we need exploration - but I'd rather read speculation in the intro (as motivation for new work) rather than in the discussion (as ad hockery)

@deevybee: not sure i get this. Are you saying introduction should anticipate results before they’ve been reported; or that nobody should ever report a new idea that was stimulated by having seen the results?

@russpoldrack: what I mean is that if you have a new idea stimulated by your results, you should go do some additional work to test the idea, and then write a paper about that. if you want to speculate in your discussion that's fine, I just don't usually feel like it's worth my time to read it

For those who don’t know me, I should start by explaining that I am fully convinced of the problems with how the research literature currently works. I have various talks and slide-decks online on this topic (see here) as well as a long article on cognitive biases. So I agree with the problem that Schoenegger and Pils want to fix: the Discussion section of a paper is often the provenance of over-hyped findings that are damaging to science as a cumulative process, because they encourage people to waste time following seductive but ultimately false leads. But it’s a bit of a leap from saying that many people misuse the Discussion section to concluding it should be banned, particularly when we do have other solutions to the problems.

What troubles me about the views expressed by Phil and Russ is that it sounds as if they are opposed to anyone reporting a new idea that emerges from consideration of the data. I spend a lot of my time arguing for solutions to the reproducibility crisis, and am familiar with the push-back from those that say “You’ll kill creativity”, and “You are forbidding exploratory research”. My response has always been to say I thoroughly approve of people reporting creative insights that come from observing data. The only thing you should not do is to formulate and test a hypothesis from the same data - The Registered Reports model, of which I am a fan, deals very nicely with what I’ve called the four horsemen of the reproducibility apocalypse  – p-hacking, HARKing, low power and publication bias – but it does not preclude the researcher coming up with novel ideas in the Discussion: instead, it draws a very clear boundary between what is hypothesis-testing and what is exploratory, and does not allow someone to include hypothesis-testing analyses that were not preregistered.

I found Russ’s comments depressing, because it implies you shouldn’t report a new idea without first doing further work to test it. If you're making a bold new claim, then of course you need to do that. But I see scientific progress as incremental, and some insights could be valuable for others working on the topic to take into consideration. If we could only report on ideas that had been shored up by more experimentation, it could slow down discovery, because people just wouldn't bother. Also, it would make research a sadly solitary activity, where instead of exchanging ideas, we all plod on in our own narrow furrow. 

I should come clean and explain that I am currently finalising the write-up of an analysis of a dataset focused on language lateralisation, where I am comparing methods of deriving a laterality index in a purely exploratory fashion. This has generated new ideas about factors that may drive individual differences in laterality – something that has intrigued me for 50 years. I think the appropriate way to handle this in the Discussion is to end with specific predictions that follow from my ideas. Maybe others would be interested in reanalysing existing data, or doing new studies that build on this work, maybe not. I am aiming to do more myself with existing datasets, but not in a position to gather new data. I am explicit in the write-up that my current study is exploratory and I do not present statistical tests. But it seems to me it would be a bit perverse if I didn’t mention how my analyses had changed my thinking to generate new predictions. If it’s true that nobody reads discussion sections, then I’ll have cast my seed on barren ground, but that still feels better than doing nothing with it - and it's useful for me to have a clear account that can be the basis for subsequent preregistered analyses. 

If we adopted the Schoenegger and Pils model, I’d just have to hope that (a) someone else would be interested enough to write a Discussion paper based on my results, and (b) they would have useful insights into what it all means. I have all the cognitive biases of a typical human, so of course I think my own ideas are likely to be better than those of others (despite years of negative feedback). But the underlying nature of laterality is a topic that has intrigued me since I started in neuropsychology, so I don’t think it’s unreasonable for me to think that my insights not be obvious to others and may also have some value. As Schoenegger and Pils noted: 

“… one central cost with our proposal is that we may lose the epistemic advantages of authors discussing their own data in some instances. Particularly, the authors of a study often have unique insights into their data that may not be immediately apparent to third-party researchers. This is especially true for studies that involve complex datasets. By outsourcing the discussion section to third-party authors, we may miss out on important nuances and insights that only the original authors can provide.”

In sum, my view is that, when used properly, the Discussion section serves two purposes. It communicates succinctly the import of the reported results in relation to a priori hypotheses, and it provides an opportunity to consider new ideas stimulated by the results. In practice, the Discussion is often misused. People play down results they don’t like, over-interpret those that accord with their preferred theory, and engage in HARKing. But to say you should get rid of the Discussion because it is misused is like saying we should all give up cars because some people drive too fast and cause accidents.

 

Note re comments. To avoid spam, comments are moderated. In general they will be approved if on-topic and non-abusive, but approval may not be immediate.