-->
This blogpost doesn't say anything new – it just uses a new
analogy (at least new to me) to make a point about the value of null results from well-designed studies. I was
thinking about this after reading this blogpost by Anne Scheel.
Think of science like prospecting for kryptonite in an
enormous desert. There's a huge amount of territory out there, and very little kryptonite.
Suppose also that the fate of the human race depends crucially on finding kryptonite
deposits.
Most prospectors don't find kryptonite. Not finding kryptonite
is disappointing: it feels like a lot of time and energy has been wasted, and
the prospector leaves empty-handed. But the failure is nonetheless useful. It
means that new prospectors won't waste their time looking for kryptonite in
places where it doesn't exist. If, however, someone finds kryptonite, everyone gets very excited and there is a stampede to
rush to the spot where it was discovered.
Contemporary science works a bit like this, except that the
whole process is messed up by reporting bias and poor methods which lead to
false information.
To take reporting bias first: suppose the prospector who
finds nothing doesn't bother to tell anyone. Then others may come back to the
same spot and waste time also finding nothing. Of course, some scientists are
like prospectors in that they are competitive and would like to prevent other
people from getting useful information. Having a competitor bogged down in a
blind alley may be just what they want for their rivals. But where there is an
urgent need for new discovery, there needs to be a collaborative rather than
competitive approach, to speed up discovery and avoid waste of scarce funds. In
this context, null results are very useful.
False information can come from the prospector who declares
there is no kryptonite on the basis of a superficial drive through a region.
This is like the researcher who does an underpowered study that gets an inconclusive
null result. It doesn't allow us to map out the region with kryptonite-rich and
kryptonite-empty areas – it just leaves us having to go back and look again
more thoroughly. Null results from poorly designed studies are not much use to
anyone.
But the worst kind of false information is fool's kryptonite:
someone declares they have found kryptonite, but they haven't. So everyone
rushes off to that spot to try and find their own kryptonite, only to find they
have been deceived. So there are a lot of wasted resources and broken hearts.
For a prospector who has been misled in this way, this situation is worse than
just not finding any kryptonite, because their hopes have been raised and they
may have put a disproportionate amount of effort and energy into pursuing the
false information.
Pre-registering a study is the equivalent of a prospectors
declaring publicly that they are doing a comprehensive survey of a specific
region, and will declare what they have found, so that the map can gradually be
filled in, with no duplication of effort.
Some will say, what about exploratory research? Of course
the prospector may hit lucky and find some other useful mineral that nobody had
anticipated. If so, that's great, and it may even turn out more important than
kryptonite. But the point I want to stress is that the norm for most prospectors
is that they won't find kryptonite or anything else. Really exciting findings
occur rarely, yet our current incentive structures create the impression that
you have to find something amazing to be valued as a scientist. It would make more sense to reward those who do
a good job of prospecting, producing results that add to our knowledge and can
be built upon.
I'll leave the last word to Ottoline Leyser, who in an interview for The Life
Scientific said: "There's an
awful lot of talk about ground-breaking research…. Ground-breaking is what you
do when you start a building. You go into a field and you dig a hole in the
ground. If you're only rewarded for ground-breaking research, there's going to
be a lot of fields with a small hole in, and no buildings."
Posts
