Saturday, 7 May 2016

Would paying by results improve reproducibility?

Twitter and Facebook were up in arms last week. "Merck wants its money back if University research is wrong" was the headline to the article that set off the outrage.

Commentators various described the idea as dangerous, preposterous and outrageous, and a 'worrying development', while at the same time accusing Merck of hypocrisy for its history of misleading claims about its vaccines and drugs.

In the comments beneath the article, similar points were made: 'No way any academic institutions will agree to this'; 'If you want absolute truth take up religion'; 'Merck wants risk-free profit'.

But if you follow the link to the article that the story was based on, it's clear that the headline in the Technology Review piece was misleading.

For a start, the author of the piece, Michael Rosenblatt, was clear that he was not representing an official position. Under 'competing interests' we are told: "M.R. is an employee of and owns stock and stock options in Merck & Co. The opinions expressed in this article are those of the author and do not necessarily correspond to the views of the author’s employer."

Rosenblatt's focus is irreproducible scientific research: a topic that is receiving increasing attention in biomedicine among other disciplines. He notes the enormous waste that occurs when a pharma company like Merck attempts to use basic biomedical research to develop new drug targets. Whether or not you approve of Merck's track record, there is no question but they have legitimate concerns. He explains that the costs of building a translational project on shaky biomedical foundations are so great that pharma companies will now routinely try to replicate original findings before taking them forward. Except that, all too often, they find they cannot do so. Since it can take between 2-6 scientists one to two years to conduct a replication, the costs of irreproducible science are substantial.

Rosenblatt notes that there have been numerous suggestions for improving reproducibility of biomedical science; mostly these concern aspects of training, and altering promotion practices so that academic scientists will have an incentive to do reproducible research. His proposal is that pharmaceutical companies could also address incentives by making their financial deals with universities contingent on the reproducibility of the results.

In effect, Rosenblatt wants those who do the research in universities to ensure it is reproducible before they bring it forward for pharmaceutical companies to develop. His proposal does indeed include a 'money back' guarantee, so that if a result did not hold up, the pharmaceutical company would be compensated. But, Rosenblatt argues, this would be compensated for by additional funds made available to universities to enable them to adopt more reproducible practices, including, where necessary, replicating findings before taking them forward.

The Technology Review headline misses all this nuance and just implies that Merck is being naïve and unrealistic in assuming that scientists have some kind of pre-cognition about the outcomes of experiments. That is far from being the case: the emphasis is rather on the importance of ensuring that findings are as solid as possible before publishing them, and suggesting that pharma companies could play a role in incentivising reproducible practies by tweaking their funding arrangements with Universities.

Obviously, the devil would be in the detail, and Rosenblatt is careful to avoid being too specific, suggesting instead that this is not intended as a panacea, but rather as an idea that would be worth piloting. I'm as cynical as the next person about the motives of pharmaceutical companies, but I have to say I think this is an interesting suggestion that could be in the interests of scientific progress, as well as benefiting both universities and pharmaceutical companies.

Tuesday, 19 April 2016

Talking about tax avoidance: weasel words

With recent revelations about rich people legally avoiding tax, it seems a good time to tell this story.

In late 2013, I received an email telling me that I could potentially save huge amounts in tax on my pension. I thought this must be a particularly sophisticated spam: poverty in the UK was rising fast, and I have high earnings, so why would pension rules be changed to benefit me? But no, I checked it out and it was all legal and above board.

All is made clear in this piece*. The phraseology has some gems: I particularly enjoyed 'the appropriate crystallisation event, such as the point of death', but behind all the talk of accrued benefits and recovery tax charges, the message seems to be that the Government had introduced a new measure that would increase the amount of tax paid by rich people, but HMRC had immediately provided a legal means for avoiding this, 'Fixed Protection'.

I'm fascinated by the use of language: the word 'protect' has entirely positive connotations: my online dictionary defines it as 'keep safe from harm or injury'. We hear of tax 'shelters', 'tax-advantaged savings vehicles' and suchlike. But what is achieved here is tax avoidance by rich people – all aided and abetted by HMRC.

If you have money and you talk to solicitors or financial advisers, you'll find that the default assumption is that you want to pay as little tax as possible. I first came across this when making a will: the solicitor concerned started describing how I could set up a complex trust that would mean less tax would be paid on my estate. I don't have dependents so this struck me as particularly pointless. I'd be dead and wouldn't care. The solicitor looked at me aghast and clearly thought I was barking mad.

I'm not a saint and I'm not an idiot. I have some savings. But I live a comfortable life and don't need to squirrel away vast amounts of dosh. And I grew up in the 1960s when, although things weren't perfect we had a reliable National Health Service; schools had adequate resources; there were grants to support people through university. The current generation may find it remarkable that until I was about 30, I never saw a beggar on the streets of any British city. Now we have food banks. An equable society that looks after the most vulnerable costs money, and that money comes from taxes.

I'll leave the last word to J.K.Rowling, who is one of the few rich people who seems to get it:

I chose to remain a domiciled taxpayer for a couple of reasons. The main one was that I wanted my children to grow up where I grew up, to have proper roots in a culture as old and magnificent as Britain’s; to be citizens, with everything that implies, of a real country, not free-floating ex-pats, living in the limbo of some tax haven and associating only with the children of similarly greedy tax exiles. 

A second reason, however, was that I am indebted to the British welfare state; the very one that Mr Cameron would like to replace with charity handouts. When my life hit rock bottom, that safety net, threadbare though it had become under John Major’s Government, was there to break the fall. I cannot help feeling, therefore, that it would have been contemptible to scarper for the West Indies at the first sniff of a seven-figure royalty cheque. This, if you like, is my notion of patriotism. On the available evidence, I suspect that it is Lord Ashcroft’s idea of being a mug.

*I found this on Google search: I don't have any dealings with this firm of solicitors, but as far as I can tell, what they say matches the advice I had earlier, and is pretty standard

Tuesday, 12 April 2016

To: the World
From: Deevybee
Re: We have a problem

Everyone I know is exhausted by email. You can spend a day battling back the incoming tide of messages, but next morning when you wake up, there it is again. Much of it is spam and can be deleted without reading, but it still absorbs attention and energy. But there's plenty of other stuff that sits there in your inbox eyeing you balefully until you respond. People I know divide into two classes: those who have given in and just live with an oppressive burden of 500 unread messages, and those who destroy themselves trying to keep on top of it. The best advice I've read on how to manage the situation is this by Tim Harford, but even obeying his rules only reduces the pain, but does not eliminate it.

Is there a solution? I've thought of one. It probably is impossible but I'm going to put it out there and see what you all say.

The idea is that there should be a cost to sending email. This seems weird for two reasons:  when email first came out we all loved it precisely because it was free: adding a cost seems perverse. Second, how on earth could it be made to work?

Well, suppose big organisations, such as Universities, could set up their systems so as to filter incoming emails and check whether the sender was registered. I assume that since filtering already occurs at some level for spam, this should not be impossible. If the sender is not registered, they get a bounce inviting them to open an account. Once they have an account, then a v small charge is made for each email that is delivered. The charge should be adequate to cover the cost of running the filtering and billing operation, but no more.

This could be arranged so that communications within a domain would be free, so it would not save us from mass communications from admin – but given that in my department most of these have recently been about the serious matter of closure of toilets, and even gender reassignment of toilets, perhaps that is as well. I can also envisage institutions have reciprocal arrangements so that all university domains, for instance, would agree not to charge each other. We would also need to be able to set up 'whitelists' of addresses that would be exempt from a charge: in my line of work we use email to communicate with volunteers and organisations who help our research, so we'd have to find a way to indicate that if we initiate the email exchange, the recipient is not charged for replying.

I've been trying to decide whether such a set-up would be good on balance, or whether it would create more problems than it would solve. There is no doubt that it would lead to an initial period of havoc, but it would wipe out spam at a stroke. The fear is that it could also prevent genuine and important messages getting through. Would I miss the opportunity of a lifetime to collaborate with a colleague, to go to a marvellous conference, or to take on an outstanding overseas student? The answer is probably yes, though, of course, if people were really unmotivated or unable to register, there is always snail mail.

Well, this is just an early morning thought, prompted by the daily routine of deleting the mass invitations to meet a sexy lady, deliver keynote at a conference on sludge disposal in China or be the recipient of a huge donation from a distressed oligarch. There must be a better way, but what is it?

Tuesday, 22 March 2016

Better control of the publication time-line: A further benefit of Registered Reports

I’ve blogged previously about waste in science. There are numerous studies that are completed but never see the light of day. When I wrote about this previously, I focused on issues such as reluctance of journals to publish null results, and the problem of writing up a study while applying for the next new grant. But here I want to focus on another factor: the protracted and unpredictable process of peer review that can lead to researchers to just give up on a paper.

Sample Gantt chart. Source:
The sample Gantt chart above nicely illustrates a typical scenario.  Let's suppose we have a postdoc with 30 months’ funding. Amazingly, she is not held up by patient recruitment issues, or ethics approvals, and everything goes according to plan, so 24 months in, she writes up the study and submits it to a journal. At the same time, she may be applying for further funding or positions. She may plan to start a family at the end of her fellowship. Depending on her area of study it may take anything from two weeks to six months to hear back from the journal*. The decision is likely to be revise and resubmit. If she’s lucky, she’ll be able to do the revisions and get the paper accepted to coincide with the end of her fellowship.  All too often, though, the reviewers suggest revisions. If she's very unlucky they may demand additional experiments, which she has no funding for.  If they just want changes to the text, that's usually do-able, but often they will suggest further analyses that take time, and she may only get to the point of resubmitting the manuscript when her money runs out. Then the odds are that the paper will go back to the reviewers – or even to new reviewers – who now have further ideas of how the paper can be improved. But now our researcher might have started a new job, have just given birth, or be unemployed and desperately applying for further funds.

The thing about this scenario, which will be all too familiar to seasoned researchers (see a nice example here), is that it is totally unpredictable. Your paper may be accepted quickly, or it may get endlessly delayed. The demands of the reviewers may involve another six month’s work on the paper, at a point when the researcher just doesn’t have the time. I’ve seen dedicated, hardworking, enthusiastic young researchers completely ground down by this situation, faced by the choice of either abandoning a project that has consumed a huge amount of energy and money, or somehow creating time out of thin air. It’s particularly harsh on those who are naturally careful and obsessive, who will be unhappy at the idea of doing a quick and dirty fix to just get the paper out. That paper which started out as their pride and joy, representing their best efforts over a period of years is now reduced to a millstone around the neck.

But there is an alternative. I’ve recently, with a graduate student, Hannah Hobson, put my toe in the waters of Registered Reports, with a paper submitted to Cortex looking at an electrophysiological phenomenon known as mu suppression. The key difference from the normal publication route is that the paper is reviewed before the study is conducted, on the basis of an introduction and protocol detailing the methods and analysis plan. This, of course takes time – reviewing always does. But if and when the paper is approved by reviewers, it is provisionally accepted for publication, provided the researchers do what they said they would.

One advantage of this process is that, after you have provisional acceptance of the submission, the timing is largely under your own control. Before the study is done, the introduction and methods are already written up, and so once the study is done, you just add the results and discussion. You are not prohibited from doing additional analyses that weren’t pre-registered, but they are clearly identified as such. One the study is written up the paper goes back to reviewers. They may make further suggestions for improving the paper, but what they can’t do is to require you to do a whole load of new analyses or experiments. Obviously, if a reviewer spots a fatal error in the paper, that is another matter. But reviewers can’t at this point start dictating that the authors do further analyses or experiments that may be interesting but not essential.

We found that the reviewer comments on our completed study were helpful: they advised on how to present the data and made suggestions about how to frame the discussion. One reviewer suggested additional analyses that would have been nice to include but were not critical; as Hannah was working to tight deadlines for thesis completion and starting a new job, we realised it would not be possible to do these, but because we have deposited the data for this paper (another requirement for a Registered Report), the door is left open for others to do further analysis.

I always liked the idea of Registered Reports, but this experience has made me even more enthusiastic for the approach. I can imagine how different the process would have been had we gone down the conventional publishing route. Hannah would have started her data collection much sooner, as we wouldn’t have had to wait for reviewer comments. So the paper might have been submitted many months earlier. But then we would have started along the long uncertain road to publication. No doubt reviewers would have asked why we didn’t include different control conditions, why we didn’t use current source density analysis, why we weren’t looking at a different frequency band, and whether our exclusionary criteria for participants were adequate. They may have argued that our null results arose because the study was underpowered. (In the pre-registered route, these were all issues that were raised in the reviews of our protocol, so had been incorporated in the study). We would have been at risk of an outright rejection at worst, or requirement for major revisions at best. We could then have spent many months responding to reviewer recommendations and then resubmitting, only to be asked for yet more analyses.  Instead, we had a pretty clear idea of the timeline for publication, and could be confident it would not be enormously protracted.

This is not a rant against peer reviewers. The role of the reviewer is to look at someone else’s work and see how it could be improved. My own papers have been massively helped by reviewer suggestions, and I am on record as defending the peer review system against attacks. It is more a rant against the way in which things are ordered in our current publication system. The uncertainty inherent in the peer review process generates an enormous amount of waste, as publications, and sometimes careers, are abandoned. There is another way, via Registered Reports, and I hope that more journals will start to offer this option.

*Less than two weeks suggests a problem!See here for an example.