It’s been gratifying to see how this post has sparked off discussion. To me it just emphasises the value of tweeting and blogging in academic life: you can have a real debate with others all over the world. Unlike the conventional method of publishing in journals, it’s immediate. But it’s better than face-to-face debate, because people can think about what they write, and everyone can have their say.
There are three rather different issues that people have picked up on.
1. The first one concerns methods in functional brain imaging; the debate is developing nicely on Daniel Bor’s blog and I’ll not focus on it here.
2. The second issue concerns the unusual routes by which people get published in PNAS. Fellows of the National Academy of Science are able to publish material in the journal with only “light touch” review. In this article, Rand and Pfeiffer argue that this may be justified because papers that are published via this route include some with very high citation counts. My view is that the Temple et al article illustrates that this is a terrible argument. Temple et al have had 270 citations, so would be categorised by Rand and Pfeiffer as a “truly exceptional” paper. Yet, it contains basic methodological errors that compromise its conclusions. I know some people would use this as an argument against peer review, but I’d rather say this is an illustration of what happens if you ignore the need for rigorous review. Of course, peer review can go wrong, and often does. But in general, a journal’s reputation rests on it not publishing flawed work, and that’s why I think there’s still a role for journals in academic communications. I would urge the editors of PNAS, however, to rethink their publication policy sp that all papers, regardless of the authors, get properly reviewed by experts in the field. Meanwhile, people might like to add their own examples of highly cited yet flawed PNAS “contributions” to the comments on this blogpost.
3. The third issue is an interesting one raised by Neurocritic, who asked “How much of the neuroimaging literature should we discard?” Jon Simons (@js_simons) then tweeted “It’s not about discarding, but learning”. And, on further questioning, he added “No study is useless. Equally, no study means anything in isolation. Indep replication is key.” and then “Isn't it the overinterpretation of the findings that's the problem rather than paper itself?” Now, I’m afraid this was a bit too much for me. My view of the Temple et al study was that it was not so much useless as positively misleading. It was making claims about treatment efficacy that were used to promote a particular commercial treatment in which the authors had a financial interest. Because it lacked a control group, it was not possible to conclude anything about the intervention effect. So to my mind the problem was “the paper itself”, in that the study was not properly designed. Yet it had been massively influential and almost no-one had commented on its limitations.
At this point, Ben Goldacre (@bengoldacre) got involved. His concerns were rather different to mine, namely “retraction / non-publication of bad papers would leave the data inaccessible.” Now, this strikes me as a rather odd argument. Publishing a study is NOT the same as making the data available. Indeed, in many cases, as in this one, the one thing you don’t get in the publication is the data. For instance, there’s lots of stuff in Temple et al that was not reported. We’re told very little about the pattern of activations in the typical-reader group, for instance, and there’s a huge matrix of correlations that was computed with only a handful actually reported. So I think Ben’s argument about needing access to the data is beside the point. I love data as much as he does, and I’d agree with him that it would be great if people deposited data from their studies in some publicly available archive so nerdy people could pick over them. But the issue here is not about access to data. It’s about what do you do with a paper that's already published in a top journal and is actually polluting the scientific process because its misleading conclusions are getting propagated through the literature.
My own view is that it would be good for the field if this paper was removed from the journal, but I’m a realist and I know that won’t happen. Neurocritic has an excellent discussion of retraction and alternatives to retraction in a recent post, which has stimulated some great comments. As he notes, retraction is really reserved for cases of fraud or factual error, not for poor methodology. But, depressing though this is, I’m encouraged by the way that social media is changing the game here. The Arsenic Life story was a great example of how misleading, high-profile work can get put in perspective by bloggers, even if peer reviewers haven’t done their job properly. If that paper had been published five years ago, I am guessing it would have been taken far more seriously, because of the inevitable delays in challenging it through official publication routes. Bloggers allowed us to see not only what the flaws were, but also rapidly indicated a consensus of concern among experts in the field. The openness of the blogosphere means that opinions of one or two jealous or spiteful reviewers will not be allowed to hold back good work, but equally, cronyism just won’t be possible.
We already have quite a few ace neuroscientist bloggers: I hope that more will be encouraged to enter the fray and help offer an alternative, informal commentary on influential papers as they appear.
it'd be interesting if we established some utopian alternative system for disseminating the results of experiments that sidesteps academic journal publication, but right now, if you retract and delete academic papers of poor quality, the methods and results are inaccessible. i think there is often some useful information in poor quality studies, and making that inaccessible is not the answer.ReplyDelete
I'm terribly new to blogging, and still very much finding my feet, but have been delighted and inspired by the opportunity for debate it brings. I agree that in some ways this medium is a lot better for certain discussions than the standard academic routes, such as peer-reviewed publication, or face to face chats in conferences.ReplyDelete
I do discuss point 3 quite a bit in my own recent blog (and that forms a portion of the comments discussion too), but briefly, I strongly agree with you, Dorothy, that there are some papers that have negative value, because they mislead. They might only waste the time of a bunch of scientists in failures to replicate, but they could also lead to treatments that do more harm than good.
I also totally agree (and wrote about this on Twitter too!) that there is a big distinction in neuroimaging between data and what's reported in papers, which is usually very many steps away from the raw brain-scanning data. It's possible that there is some useful design or method in a bad paper, but the whole point we've been blogging about is that usually this is not the case, and that the design or analysis is deeply flawed, and such poor publications can lead to the adoption of bad experimental habits.
There are good arguments (though with ethical issues) for making much imaging data public in collective repositories, and there have been some attempts to do this. For instance, a pool of anatomical MRI data anonymised but linked to details such as age, sex, is a tremendously rich resource that can turn into many new studies. Things get more complicated with functional imaging data, but there are still many uses for this in collective repositories. But here we're not talking about papers at all, just a collaborative raw resource.
I can see the argument, but I think that we already put too much faith in closed peer review instead of our own critical faculties as readers. The paper you've drawn attention to is just one of many, many, many flawed or problematic studies which are published each year. Most of them go through the full peer review process unscathed while we're all aware of apparently unproblematic studies which are delayed and published in obscure journals because of awkward and unreasonable reviewers.ReplyDelete
I'd like to think I've never taken a study as read, just because it's been reviewed, and I always like to form my own opinion after reading it. But I can only do this if the paper is out there to read, criticise and understand.
In the case of the paper you mention the lack of the control group is really problematic, but the reporting of uncorrected statistics is a less clear cut issue in my view (each of the alternatives have some problems some of which are outlined by Jon Simons and Neuropocalypse in comments on Daniel Bor's blog). In addition I think uncritical overreliance on statistical thresholds as the basis of inference is potentially as problematic as uncritical acceptance of uncorrected stats. At present its rather easy for neuroimagers to claim "activation was restricted to region X" and to draw conclusions based erroneously on the "absence" of activation elsewhere (just as psychologists are prone to overinterpret the absence of effects in e.g., ANOVA). A bad experiment produces activation all over the place, but it is meaningless. Conservative thresholding of the data prevents the reader evaluating such claims. Again, I am arguing in favour of transparency and clarity about the data - thresholds and reviewers just get in between me and the data.
It seems to me that deliberately fraudulent, careless or misleading papers are pretty rare. It is important that methods and results are reported fully and accurately, but I am afraid that in analysis and interpretation people (usually other people!) are sometimes just wrong - to err is human. I think the nice thing about science is the systematic way it deals with human error (eventually).
I think robust, public criticism of the kind you made in your blog post is a better solution than removing flawed work from the literature.
Dorothy, I think we agree on 99% of this. Part of our apparent disagreement is down to the word Neurocritic used, "discard". I took this to mean "retract", but in your comment on Daniel Bor's excellent post, you used the word "disregard". I would agree with that. I strongly feel that retraction is too strong a requirement for "flawed" papers (fraud should be retracted, of course). Part of the reason for that is that it is so subjective as to what might constitute serious, and therefore perhaps retractable, flaws.ReplyDelete
But I completely agree with you that the field should be made aware of flawed papers, and blogging (such as your fantastic last post) and comments on journal websites are a great way of making that happen. We can all then assess the flaws, and learn about how to avoid them in the future and do better science as a result (my point in my earlier tweet). We can also then decide how large a pinch of salt we're going to take when we're considering the findings and perhaps choose to disregard them when we're compiling reviews of the area or otherwise seeking to generalise across studies.
This comment has been removed by the author.Delete
Just seen that Neurocritic has already made the very same point:ReplyDelete
"By "discarding" I meant disregarding the results from flawed articles, not retracting them from the literature entirely."
As a co-author on a very cool paper that should be coming out in PNAS soon, I'll want to note that some of your suggestions are actually PNAS policy. For example, while NAS members can still edit their own submissions, this form of submission is no longer allowed if they have any financial interests in the work.ReplyDelete
In addition, published papers submit their data to an open database. The corresponding author for the paper I'm on, just put some of our fMRI data in a publicly available database per journal requirements. It's a bit fuzzy on what level of processing one needs to submit (i.e. raw of the scanner with every script to create the final figures vs the final contrast maps). Given the amount of data in our paper, we did something in-between that should also be in interesting contribution.
You can read about both of these rules at: http://www.pnas.org/site/misc/iforc.shtml
For what it's worth, the paper I'm on was a direct submission with an editor we didn't know until acceptance and the oversight & comments regarding methodology were rigorous.
Related to Dan's point - this is the problem with PNAS. Most direct submissions are fantastic papers, as Dan's likely is. But I have read some fatally flawed articles that were 'contributed' by members, and these can bring down the reputation of PNAS within the community at large, and lead to people feeling a need to defend the journal. I think it would be in the journal's best interests to restrict publication solely to the more traditional review model.ReplyDelete
Some choice PNAS contributions I recall are:
One paper using pain stimulation to the hand as a way of studying interoception.
A paper comparing task deactivation between patient and control groups where the patients performed the task exceedingly poorly while the controls were excellent, and not addressing this issue.
I agree, to an extent, and feel no need to defend the journal... just make clear what they do & don't do. If someone says they should have authors release data, it's worth pointing out they already do this. (Though one annoyance is that we're required to release our data for free, but PNAS will only release the article open access if we pay them extra)
I also think that NAS contributed articles make us notice the problems more, but this is a glamour journal issue in general. The glamour journals are attracted to unexpected results to results that contradict established assumptions. Worrying if methods are under-reported or weak is a secondary concern. Of course, if a result contradicts established ideas, it's more likely to be wrong. There's a slew of Science & Nature papers that are thought provoking & wrong. Sometimes an author had lucky false positive, other times, the methodological flaws are rapidly obvious once the expert community starts to dig into a result.
This also goes beyond glamour journals. One could probably count on one hand the number of Alzheimers or MCI studies with fMRI before 2005 that accounted for task performance or partial voluming in atrophied tissue.
Following up the disregard/flag ideas, I've left two comments about the half-life of bad/flawed science on Neurocritic's blog piece that Dorothy mentionsReplyDelete
Here are a couple of other posts raising concerns about papers published in PNAS through one or other of the less transparent routes:ReplyDelete
Peer review and the "ole boys network"
Exercise may be good for you, but it doesn't boost your memory
Interestingly, in the case of the second one, PNAS subsequently published a letter that strongly criticised flaws in the original paper:
Update on exercise and memory story
Perhaps this is a route somebody might like to go down in relation to the Temple paper, to ensure that its flaws are communicated to the journal's audience?
very naive... by your criteria about 1/4 of past articles should be retracted .... (this paper is not, by far, the worst of this sort of thing)... more generally, the more you learn about a specific topic, the more you realize how deeply flawed are many many papers.... tReplyDelete
Catherine: Well, my preference would be for intervention studies without control groups not to get published at all in the first place. Increasingly, medical journals won't take them. So this is really a plea for journals to adopt methodologically rigorous standards when evaluating intervention papers, as this is an area where a misleading study can lead to people spending a lot of time and money on something that is ineffective.ReplyDelete
My comment is somewhat off-topic, as it doesn't have to do with neuroscience but blogging in the service of science.ReplyDelete
There have been a number of papers published that advance the anti-vaccine agenda. Not all of these papers have been published in scholarly or science journals. Almost the only criticism of the methods and techniques of the papers occur in the science blogosphere, rarely in the journals publishing them.
Some of the bloggers have used Researchblogging.org but others have not.
Sorry for arriving late on this post.ReplyDelete
I think PNAS' "contributed papers" are a real problem.
In our area, we all know that Mike Merzenich has a terrible record of letting papers in with a one-sided view on language disorders, some OK but others really poor.
If we had time for that (and were prepared for a fierce debate!) we could draw an entire list of those papers and review them collectively...