Whistleblowing, research misconduct, and mental health

 

Book Review

Carl Elliott. The Occasional Human Sacrifice: Medical Experimentation and the Price of Saying No.  W. W. Norton, 2024. ISBN: 978-1-324-06550-0

 

This book should be required reading for three groups of people. First, every member of university or hospital administration, and those who sit on ethics committees (Institutional Review Boards) in such institutions. Second, people interested in the psychology and sociology of moral behaviour. And finally, anyone who has experience of being a whistleblower regarding academic malpractice, and anyone who is thinking of acting in that role.

 There are many books on bioethics, and readers may be familiar with at least some of the seven cases described in The Occasional Human Sacrifice:

• The Markingson case, where a severely disturbed man was persuaded to take part in a drug trial against the wishes of his mother, leading, as she had predicted, to his death by suicide.
•  The Tuskegee study where black men were coerced to take part in a study of the consequences of untreated syphilis.
• Willowbrook School, a residential institution where children with intellectual disability were deliberately infected with hepatitis.
•  Protocol 126 at the Fred Hutchinson Cancer Center, which involved patients being given an experimental bone marrow transplant treatment that proved to be ineffective, without adequate informed consent.
• The Radiation Effects study at Cincinnati Medical Center, where cancer patients were exposed to whole body irradiation, not as treatment, but in order to study its effects. 
•  "The Unfortunate Experiment" in Dunedin, New Zealand, an observational study that involved monitoring rather than treating precancerous abnormalities of the cervix, without sufficient explanation to the woman concerned.
• The case of Paolo Macchiarini, who performed experimental stem cell transplants on the trachea in humans without having done adequate animal studies to justify the approach, with dire outcomes for most patients.

The details of each of these cases makes for sobering reading: patients who trusted their doctors to look after them were experimented on without proper consent procedures, in some cases over a period of years. Most accounts of these events focus on the doctors who allowed this to happen: what motivated them, why did they ignore conventional medical ethics, and why didn't they listen to those who expressed misgivings? Elliott's primary focus, however, is on a different aspect of these cases - the whistleblowers who drew attention to the problems, their motivations, and the impact of their whistleblowing on their lives.

Elliott speaks from personal knowledge - he was the whistleblower in the Markingson case, and his account makes for uncomfortable reading. When he drew attention to the unethical practices in his department of psychiatry he was first ignored and then ostracised. He became obsessed with the case, with a major toll on his mental health. In the concluding chapter he notes:

I am the last person to advise anyone to ignore the demands of conscience, but to encourage potential whistleblowers to speak out without noting the long odds of success and the grim personal costs would be like advising them to leap blindfolded off a cliff. (p 290).

All but one of the whistleblowers featured in this book had a particularly difficult time of it because they were speaking out about ethical abuses within their own institutions. As Elliott noted, the line between a traitor and a whistleblower is a fine one, relating to the perceived motivation of the person involved. A whistleblower may feel they are acting on the purest of motives, to do the right thing, yet others will interpret it as disloyalty to the institution, or attribute their behaviour to a sense of self-importance and moral superiority. Even when, as in several of these cases, the whistleblower is eventually vindicated, they are not thanked, there is seldom restitution for victims, and the mental scars of their battles with authority stay with them.

My own forays into challenging dodgy research are nothing like as serious as the cases covered by Elliott: no lives were at stake in the instances of fraud and malpractice that I've focused on, all of which have been outside my own department. Nevertheless, I resonated strongly with what Elliott had to say about the transformation that occurs in whistleblowers when they first try to report a problem. 

Unlike most people, who are used to the darker side of human behavior, whistleblowers are genuinely shocked when they discover dishonesty, greed, or corruption. They naively assume that if they blow the whistle, others will be just as outraged as they are. They might even imagine they'll be thanked." (p. 158).

 And

 "...if becoming a whistleblower requires a certain naivete, then surviving the experience demands a hard shell of cynicism." (p 159). 

I've definitely been on that journey, after years of observing the delays and obfuscations that routinely occur when a sleuth reports a fraudulent paper to a publisher, or when an institution manages to exonerate a researcher despite ample evidence of misconduct.

I've also been thinking a lot lately about parallels with political life. Here in the UK we've had a number of scandals where citizens were harmed, and sometimes killed, by people in power who were at best incompetent and at worst corrupt. Politicians handed lucrative contracts for personal protective equipment to their cronies who failed to deliver, leaving health professionals without protection; NHS patients were given contaminated blood products long after a problem had been identified; A horrendous fire in a tower block in West London arose after contractors ignored regulations and installed inappropriate cladding on the building - in each of these cases the stories came out, there was a public outcry, and a sense that "something must be done" - but the process gets mired in red tape and nothing actually is done, nobody gets punished, and those affected aren't compensated. This is terrible not just because of the impact on the individuals who were directly harmed, but also because it destroys people's trust in the institutions that we assume will look after us. I see the failure of institutions to deal adequately with academic misconduct as just one part of this broader picture.

So is there anything that can be done to challenge wrongdoing without crashing out over the cliff? A major point made by Elliott is that whistleblowers should not go it alone: that way madness lies. One option that he raised in a recent talk on this topic was to consult a lawyer with expertise in whistleblowing cases: they will be able to advise on the likelihood of success, and may be able to bring a complaint directly to an organisation, so the whistleblower can fly under the radar. There is also the option of going to the press, but, though Elliott describes some cases where this was effective, he notes it can be a two-edged sword, because journalists will want a good story, and it may not coincide with the whistleblower's viewpoint. In my experience, it's worth forming links with responsible journalists: institutions who have been inert for years may spring into action when confronted with the possibility of adverse publicity. Perhaps the most important point is that you need to have allies. They can restore confidence that there are other people who share one's values, who appreciate what one is doing and provide moral support. Although Elliott doesn't say this, I think the internet has made it easier to find one's tribe in this regard.

Where, perhaps, I take issue with Elliott is in the implication that many people aren't troubled by the issues that worry whistleblowers. When I give talks about reproducible research practices, I reliably get a question along the lines of "What advice can you give an early-career researcher whose supervisor is encouraging them to adopt questionable research practices?" The questioner clearly finds it troublesome but is being told that their career is at risk if they don't cut corners, and that "everyone is doing it". I reply that this is an issue that has serious consequences for mental health: doing what you know to be wrong causes internal conflict that can eat away at you, leading to a clash between your model of a good person and your current behaviour.

For this reason I advise those starting out in research to read this piece by Sarah Kendzior. Her advice is intended to help people survive under the threat of authoritarianism, but many of the points apply to academic survival in an age of cynicism:

Write down what you value; what standards you hold for yourself and for others.... Write a list of things you would never do. .... 

But most of all, never lose sight of who you are and what you value. If you find yourself doing something that feels questionable or wrong a few months or years from now, find that essay you wrote on who you are and read it. Ask if that version of yourself would have done the same thing.

 And if the answer is no? Don’t do it.

 

 

PEPIOPs – prolific editors who publish in their own publications

I recently reported on a method for identifying authors who contribute an unusually high proportion of papers to a specific journal. As I noted in my previous blogpost, one cannot assume any wrongdoing just on the grounds of a high publication rate, but when there is a close link between the author in question and the journal's editor, then this raises concerns about preferential treatment and integrity of the peer review process.

In running my analyses, I also found some cases where there wasn't just a close link between the most prolific author in a journal and the editor: they were one and the same person! At around the time I was unearthing these results, Elisabeth Bik tweeted to ask if anyone had examples of editors publishing in their own journals. I realised the analysis scripts I had developed for the 'percent by most prolific' analysis could be readily adapted to look at this question, and so I analysed journals in the broad domain of psychology and behavioural science from six publishers: Springer Nature, Wiley, Taylor and Francis, Sage, Elsevier and American Psychological Association (APA). I focused on those responsible for editorial decisions – typically termed Editor-in-Chief or Associate Editor. Sometimes this was hard to judge: in general, I included 'Deputy editors' with 'Editors-in-Chief' if they were very few in number. I ignored members of editorial boards.

Before reporting the findings, I thought I should consult more widely about whether people think it is appropriate for an editor to publish in their own journal.

As a starting point, I looked at guidelines that the Committee on Publication Ethics (COPE) has provided for editors  

Can editors publish in their own journal? 
While you should not be denied the ability to publish in your own journal, you must take extra precautions not to exploit your position or to create an impression of impropriety. Your journal must have a procedure for handling submissions from editors or members of the editorial board that will ensure that the peer review is handled independently of the author/editor. We also recommend that you describe the process in a commentary or similar note once the paper is published.

They link to a case report that notes how this issue can be particularly problematic if the journal is in a narrow field with few publication outlets, and the author is likely to be identifiable even if blinding is adopted.

I thought that it would be interesting to see what the broader academic community thought about this. In a Twitter poll I asked specifically about Editors-in-Chief:

The modal response was that it was OK for an Editor-in-Chief to publish in their own journal at a modest rate (once a year or less). There was general disapproval of editors publishing prolifically in their own journals – even though I emphasised that I was referring to situations where the editorial process was kept independent of the author, as recommended by COPE.

My poll question was not perfectly phrased - I did not specify the type of article: it is usual, for instance, for an editor to write editorials! But I later clarified that I meant regular articles and reviews, and I think most people interpreted it in that way.

The poll provoked some useful debate among those who approved and disapproved of editors publishing in their own journals.

Let's start with some of the reasons against this practice. I should lay my cards on the table and state that personally I am in agreement with Antonia Hamilton, who tweeted to say that as Editor in Chief of the Quarterly Journal of Experimental Psychology she would not be submitting papers from her lab to that journal during her term of office, citing concerns about Conflict of Interest. When I was Editor-in-Chief of Journal of Child Psychology and Psychiatry, I felt the same: I was concerned that my editorial colleagues would be put in a difficult position if they had to handle one of my papers, and if my papers were accepted, then it might be seen as involving a biased decision, even if it that was not the case. Chris Chambers, who was among the 34% who thought an Editor-in-Chief should never publish in their own journal, expressed concerns about breaches of confidentiality, given that the Editor-in-Chief would be able to access identities of reviewers. Others, though, questioned whether that was the case at all journals.

Turning to those who thought it was acceptable for an Editor-in-Chief to publish in their own journal, several people argued that it would be unfair, not just on the editor themselves, but also on members of their research group, if presence of an editorial co-author meant they could not submit to the journal. How much of an issue this is will depend, of course, on how big your research group is, and what other publication outlets are available.  Several people felt there was a big difference between cases where the editor was lead author, and those where they played a more minor role. The extreme case is when an editor is a member of a large research consortium led by someone else. It would seem unduly restrictive to debar a paper by a consortium of 100 researchers just because one of them was the Editor in Chief of the journal. It is worth remembering too that, while being a journal editor is a prestigious role, it is hard work, and publishers may be concerned that it becomes a seriously unattractive option if a major outlet for papers is suddenly off-limits.

Be this as it may, the impression from the Twitter poll was that three papers or more per annum starts to look excessive. In my analysis, shown here I found several journals where the Editor-in-Chief had coauthored 15 or more articles (excluding editorials/commentaries) in their own journal between 2015 and 2019.  I refer to these as PEPIOPs (prolific editors who publish in their own publications). For transparency, I've made my methods and results available, so that others can extend the analysis if they wish to do so: see https://github.com/oscci/Bibliometric_analyses. More legible versions of the tables are also available.

Table 1 shows the number of journals with a PEPIOP, according to publisher. 

Table 1: Number of journals with prolific authors. Columns AE and EIC indicate cases where Associate Editor or Editor-in-Chief published 15+ papers in the journal between 2015-2019

The table makes it clear that it is relatively rare for an Editor-in-Chief to be a PEPIOP: there were no cases for APA journals, and the highest number was 5.6% for Elsevier journals. Note that there are big differences between publishers in terms of how common it is for any author to publish 15+ papers in the journal over 5 years: this was true for around 25% of the Elsevier and Springer journals, and much less common for Sage, APA and Taylor & Francis. This probably  reflects the subject matter of the journals - prolific publication, often on multiauthored papers, is more common in biosciences than social sciences.

Individual PEPIOPs are shown in Table 2. An asterisk denotes and Editor-in-Chief who authored more articles in the journal than any other person between 2015-2019.

These numbers should be treated with caution – I did not check whether any of these editors had only recently adopted the editorial role - largely because this information is not easy to find from journal websites*. I looked at a small sample of papers from individual PEPIOPs to see if there was anything I had overlooked, but I haven't checked every article - as there a great many of them.  There was one case where this revealed misclassification: the editor of Journal of Paediatrics and Child Health wrote regular short commentaries of around 800 words summarising other papers: this seems an entirely unremarkable thing for an editor to do, but they were classified by Web of Science as 'articles' which led to him being categorised as a PEPIOP.  This illustrates how bibliometrics can be misleading.

Table 2: Editors-in-chief who published 15+ papers in own journal 2015-2019

In general I did not find any ready explanation for highly prolific outputs.  And I where I did spot checks I found only one case where there was an explanatory note of the kind COPE recommended (in the journal Maturitas) - on the contrary, in many cases there was a statement confirming no conflict of interest for any of the authors. Sometimes there were lists of conflicts relating to commercial aspects, but it was clear that authors did not regard their editorial role as posing a conflict. It was also worth mentioning there were cases where an Editor-in-Chief was senior author.

The more I have pondered this, the more I realise that the reason why I am concerned particularly by Editor-in-Chief PEPIOPs is because this is the person who is ultimately responsible for integrity of the publication process. Although publishers increasingly are alert to issues of research integrity, journal websites typically advise readers to contact the Editor-in-Chief if there is a problem. Complaints about editorial decisions, demands for retractions, or concerns about potential fraud or malpractice all come to the Editor-in-Chief. That's fine provided the Editor-in-Chief is a pillar of respectability. The problem is that not all editors are the paragons that we'd like them to be: one can adopt rather a jaundiced view after encountering editors who don't even bother to reply to expressions of concern, or adopt a secretive or dismissive attitude if plausible problems in their journal are flagged up. And there are also cases on record where an editor has abused the power of their position to enhance their own publication record. For this reason, I would strongly advise any Editor-in-Chief not to be a PEPIOP; it just looks bad, even if a robust, independent editorial process has been followed. We need to have confidence and trust in those who act as gatekeepers to journals.

My principal suggestion is that publishers could improve their record for integrity if they went beyond just endorsing COPE guidelines and started to actively check whether those guidelines are adhered to. 

*Note: 21st Aug 2020: A commenter on this blogpost has noted that Helai Huang fits this category - this editor has only been in post since 1 Jan 2020, so the period of prolific publication predated being an editor-in-chief.

Why eugenics is wrong

I really didn't think I would need to write a blogpost on this topic, but it seems eugenics is having a resurgence in the UK, so here we go.

The idea of eugenics deceptively simple. Given that some traits are heritable, we should identify those with beneficial heritable traits and encourage them to have more children, while discouraging (or even preventing) those with less desirable traits to breed. This way we will improve the human race.

Those promoting eugenics have decided that high intelligence is a desirable trait, and indeed it does correlate with things like good educational outcomes, earnings and health. It is also heritable. So wouldn't it be great to improve the human race by genetic selection for intelligence?

Much of the debate on this question has focussed on whether it could be done, rather than whether it should be. Many who would blench at enforced sterilisation have warmed to the suggestion that Polygenic Risk Scores can be used to predict educational attainment, and so could be used for embryo selection. However, those promoting this idea have exaggerated the predictive power of polygenic scores (see this preprint by Tim Morris for a recent analysis, and this review of Kevin Mitchell's book Innate for other links). But let us suppose for a moment that in the future we could predict an individual's intelligence from a genetic score: would it be acceptable then to use this information to improve the human race?

The flaw in the argument is exposed when you consider the people who are making it. Typically, they are people who do well on intelligence tests. In effect, they are saying "The world needs more people like us". Looking at those advocating eugenic policies, many of us would beg to differ.

The bad state of the world we live in is not caused by unintelligent people. It is caused by intelligent people who have used their abilities to amass disproportionate wealth, manipulate others or elbow them out of the way. Eugenicists should be especially aware that their advantages are due to luck rather than merit, yet they behave as if they deserve them, and fiercely protect them from "people not like us".

If we really wanted to use our knowledge of genetics to make the world a better place, we would select for the traits of kindness and tolerance. Rather than sterilising the unintelligent, we would minimise breeding by those who are characterised by greed and a sense of superiority over other human beings. But there's the catch: it's only those who think they're superior to others who actually want to implement eugenic policies.

How wishful thinking is damaging PETA's cause

I was appalled a couple of weeks ago to see this advert by PETA - People for the Ethical Treatment of Animals.

Readers of this blog will be aware that spurious claims of links with autism is one subject that gets me extremely cross - parents have quite enough to contend with, without the host of irresponsible people who come up with a new autism cause every day.

Needless to say, there is no scientific evidence for a link between milk consumption and autism, and so I wondered why on earth PETA should be promoting such an idea.

I didn't know much about them until this advert cropped up. I first came across PETA when I was in Australia and they had a campaign protesting about "sheep ships" - live export of animals to the Middle East. I was impressed at the way they highlighted the treatment of the animals, who were taken on a long sea voyage in vile conditions just so that they could be slaughtered in a particular way on arrival.

Now that I've read more background, I realise that the autism advert is, alas, a classic case of the "wishful thinking" fallacy, which tends to crop up in debate when there are complex ethical issues involved. Human rights and animal rights often come into conflict, and many of us find it difficult to steer a course between the competing demands. For instance, should I eat meat? I like the taste and it's a good source of protein and iron, but does that justify farming and killing animals? Should we experiment on animals? It may lead to important breakthroughs for conditions such as Parkinson's disease and Alzheimer's disease, but large numbers of animals will be subjected to unpleasant procedures as a result. Many people when confronted with such choices will give priority to the needs of humans, while at the same time trying to ensure the treatment of animals is as humane as possible.

PETA campaigners, however, take an absolute stance, arguing that "animals are not ours to eat, wear, experiment on, use for entertainment or abuse in any way." Their agenda, then, is not to improve conditions for animals used by humans, but to prevent such use altogether.

This poses a problem. Most people aren't going to be persuaded to become vegan on ethical grounds. It seems that PETA therefore decided we need to be presented with other motivations, namely the idea that milk is bad for you. It would be convenient for PETA if this were true, because it would mean that sensible people would stop drinking milk, regardless of their attitude to animals. In this regard, it's wishful thinking. The claim was challenged last week by Sense About Science, whose conversation with a representative from PETA is described here. Rather than accepting that the evidence for an autism link was not supported by science, Ben Williamson of PETA responded with further outlandish claims, maintaining that consumption of milk contributes to “asthma, constipation, recurrent ear infections, iron deficiency, anaemia and even some cancers”.

The same wishful thinking style of argument is used by those who want to ban all animal experimentation and who consequently argue that all such work is pointless, has never achieved anything, and is only done to promote the careers of those doing the experiments. A moment's thought reveals the fallacy of this viewpoint: it would have to mean that all of those doing animal experiments are either so stupid they can't see the pointlessness of their work, or are sadists who enjoy being unpleasant to animals. Neither proposition is credible. But if it were true, the argument would be easy to win.

The wishful thinkers try to bypass ethically difficult decisions by arguing that there are good practical reasons for adopting their preferred solution. We should become vegan to avoid autism, they say. We should stop animal experimentation because it achieves nothing, they say. Would that life were so simple.

Not only is it logically indefensible to take this line, it is actually counterproductive. PETA's response to Sense About Science confirms that this is an organisation that cannot be trusted to get things right. They will say whatever is convenient to promote their views and will distort the evidence if it helps. Their latest campaign has destroyed any credibility they might have had with the scientifically literate public.

Off with the old and on with the new: the pressures against cumulative research

 

Yesterday I escaped a very soggy Oxford to make it down to London for a symposium on "Increasing value, reducing waste" in Research. The meeting marked the publication of a special issue of the Lancet containing five papers and two commentaries, which can be downloaded here.

I was excited by the symposium because, although the focus was on medicine, it raised a number of issues that have much broader relevance for science, including several that I have raised on this blog, including pre-registration of research, criteria used by high-impact journals,  ethics regulation, academic backlogs, and incentives for researchers. It was impressive to see that major players in the field of medicine are now recognizing that there is a massive problem of waste in research. Better still, they are taking seriously the need to devise ways in which this could be fixed.

I hope to blog about more of the issues that came up in the meeting, but for today I'll confine myself to one topic that I hadn't really thought about much before, but which I see as important, namely the importance of doing research that builds on previous research, and the current pressures against this.

Iain Chalmers presented one of the most disturbing slides of the day, a forest plot of effect sizes found in medical trials for a treatment to prevent bleeding during surgery.

Based on Figure 3 of Chalmers et al, 2014

Time is along the x-axis, and the horizontal line corresponds to a result where the active and control treatments do not differ. Points which are below the line and whose fins do not cross it show a beneficial effect of treatment. The graph shows that the effectiveness of the treatment was clearly established by around 2002, yet a further 20 studies including several hundred patients were reported in the literature after that date. Chalmers made the point that it is simply unethical to do a clinical trial if previous research has already established an effect. The problem is that researchers often don't check the literature to see what has already been done, and so there is wasteful repetition of studies. In the field of medicine this is particularly serious because patients may be denied the most effective treatment if they enrol in a research project.

Outside medicine, I'm not sure this is so much of an issue. In fact, as I've argued elsewhere, in psychology and neuroscience I think there's more of a problem with lack of replication. But there definitely is much neglect of prior research. I lose count of the number of papers I review where the introduction presents a biased view of the literature that supports the authors' conclusions. For instance, if you are interested in the relation between auditory deficit and children's language disorders, it is possible to write an introduction presenting this association as an established fact, or to write one arguing that it has been comprehensively debunked. I have seen both.

Is this just lazy, biased or ignorant authors? In part, I suspect it is. But I think there is a deeper problem which has to do with the insatiable demand for novelty shown by many journals, especially the high-impact ones. These journals typically have a lot of pressure on page space and often allow only 500 words or less for an introduction. Unless authors can refer to a systematic review of the topic they are working on, they are obliged to give the briefest account of prior literature. It seems we no longer value the idea that research should build on what has gone before: rather, everyone wants studies that are so exciting that they stand alone. Indeed, if a study is described as 'incremental' research, that is typically the death knell in a funding committee.

We need good syntheses of past research, yet these are not valued because they are not deemed novel. One point made by Iain Chalmers was that funders have in the past been reluctant to give grants for systematic reviews. Reviews also aren't rated highly in academia: for instance, I'm proud of a review on mismatch negativity that I published in Psychological Bulletin in 2007. It not only condensed and critiqued existing research, but also discovered patterns in data that had not previously been noted. However, for the REF, and for my publications list on a grant renewal, reviews don't count.

We need a rethink of our attitude to reviews. Medicine has led the way and specified rigorous criteria for systematic reviews, so that authors can't just cherrypick specific studies of interest. But it has also shown us that such reviews are an invaluable part of the research process. They help ensure that we do not waste resources by addressing questions that have already been answered, and they encourage us to think of research as a cumulative, developing process, rather than a series of disconnected, dramatic events.

Reference
Chalmers, Iain, Bracken, Michael B., Djulbegovic, Ben, Garattini, Silvio, Grant, Jonathan, Gülmezoglu, A. Metin, Howells, David W., Ioannidis, John P. A., & Oliver, Sandy (2014). How to increase value and reduce waste when research priorities are set Lancet : 10.1016/S0140-6736(13)62229-1

Research fraud: More scrutiny by administrators is not the answer

I read this piece in the Independent this morning and an icy chill gripped me. Fraudulent researchers have been damaging Britain's scientific reputation and we need to do something. But what? Sadly, it sounds like the plan is to do what is usually done when a moral panic occurs: increase the amount of regulation.

So here is my, very quick, response – I really have lots of other things I should be doing, but this seemed urgent, so apologies for typos etc.

According to the account in the Independent, Universities will not be eligible for research funding unless they sign up to a Concordat for Research Integrity which entails, among other things, that they "will have to demonstrate annually that each team member’s graphs and spreadsheets are precisely correct."

We already have massive regulation around the ethics of research on human participants that works on the assumption that nobody can be trusted, so we all have to do mountains of paperwork to prove we aren't doing anything deceptive or harmful. 

So, you will ask, am I in favour of fraud and sloppiness in research? Of course not. Indeed, I devote a fair part of my blog to criticisms of what I see as dodgy science: typically, not outright fraud, but rather over-hyped or methodologically weak work, which is, to my mind, a far greater problem. I agree we need to think about how to fix science, and that many of our current practices lead to non-replicable findings. I just don't think more scrutiny by administrators is the solution. To start scrutinising datasets is just silly: this is not where the problem lies.

So what would I do? The answers fall into three main categories: incentives, publication practices, and research methods.

Incentives is the big one. I've been arguing for years that our current reward system distorts and damages science. I won't rehearse the arguments again: you can read them here.  The current Research Excellence Framework is, to my mind, an unnecessary exercise that further incentivizes researchers against doing slow and careful work. My first recommendation is therefore that we ditch the REF and use simpler metrics to allocate research funding to University, freeing up a great deal of time and money, and improving the security of research staff. Currently, we have a situation where research stardom, assessed by REF criteria, is all-important. Instead of valuing papers in top journals, we should be valuing research replicability. 

Publication practices are problematic, mainly because the top journals prioritize exciting results over methodological rigour. There is therefore a strong temptation to do post hoc analyses of data until an exciting result emerges. Pre-registration of research projects has been recommended as a way of dealing with this - see this letter to the Guardian on which I am a signatory.  It might be even more effective if research funders adopted the practice of requiring researchers to specify the details of their methods and analyses in advance on a publicly-available database. And once the research was done, the publication should contain a link to a site where data are openly available for scrutiny – with appropriate safeguards about conditions for re-use.

As regards research methods, we need better training of scientists to become more aware of the limitations of the methods that they use. Too often statistical training is a dry and inaccessible discipline. All scientists should be taught how to generate random datasets: nothing is quite as good at instilling a proper understanding of p-values as seeing the apparent patterns in data that will inevitably arise if you look hard enough at some random numbers. In addition, not enough researchers receive training in best practices for ensuring quality of data entry, or in exploratory data analysis to check the numbers are coherent and meet assumptions of the analytic approach.

In my original post on expansion of regulators, I suggested that before a new regulation is introduced, there should be a cold-blooded cost-benefit analysis that considers, among other things, the cost of the regulation both in terms of the salaries of people who implement it, and the time and other costs to those affected by it. My concern is that among the 'other costs' is something rather nebulous that could easily get missed. Quite simply, doing good research takes time and mental space of the researchers. Most researchers are geeks who like nothing better than staring at data and thinking about complicated problems. If you require them to spend time satisfying bureaucratic requirements, this saps the spirit and reduces creativity.

I think we can learn much from the way ethics regulations have panned out. When a new system was first introduced in response to the Alder Hey scandal, I'm sure many thought it was a good idea. It has taken several years for the full impact to be appreciated. The problems are documented in a report by the Academy of Medical Sciences, which noted "Urgent changes are required to the regulation and governance of health research in the UK because unnecessary delays, bureaucracy and complexity are stifling medical advances, without additional benefits to patient safety"

If the account in the Independent is to be believed, then the Concordat for Research Integrity could lead to a similar outcome. I'm glad I will retire before the it is fully implemented.

Schoolgirls' health put at risk by Catholic view on vaccination

Today's Time Newsfeed carries a remarkable story: parents of children attending Catholic schools in Calgary were sent a special letter to accompany details of a vaccination programme against human papillomavirus (HPV), which protects against cervical cancer. In it, local bishops wrote: "Although school-based immunization delivery systems generally result in high numbers of students completing immunization, a school-based approach to vaccination sends a message that early sexual intercourse is allowed.” I find this amazing for several reasons:

• There's a complete failure to understand what affects teenagers' behaviour. Do the bishops seriously think that teenaged girls who are thinking of having sex say to themselves "Oh, wait a minute. I might get HPV. Let's not do it." Potential consequences of sex include a host of sexually transmitted diseases, as well as pregnancy. If these don't put girls off, then why should a risk of HPV? 
• HPV is a sexually transmitted disease. You can get it if you are a virgin who marries someone with HPV. You can get it if you are raped (something which has been known to occur in Catholic schools). 
• The recommendation seems theologically dubious. I'm an atheist, but my understanding of Catholicism is that whether or not something is a sin is largely to do with motivation rather than action. So if you are tempted to sex but desist because it would upset God, then that's good. If you are tempted to sex but desist only because of a fear of disease, that's still a sin. The church should be teaching girls to love God so much that they won't do things that offend him, not to conform to standards of sexual behaviour out of fear. No doubt religious readers will put me right if I've misunderstood this distinction. 
• These girls are attending a Catholic school where I assume morality is drummed into them day and night. The bishops assume that their grasp of that morality is so weak that having a HPV vaccination will be sufficient to overturn everything they have been told about sexual ethics. Doesn't say much for the religious teaching in the schools, or for the intelligence of the pupils. 
• One thing Jesus really understood is that humans aren't perfect and frequently fall short of the moral standards they try to adhere to. There's a huge emphasis on forgiveness of sin in his teachings. The Bishops are in effect saying that God won't forgive you if you stray from the straight and narrow: he'll commit you to a life with an unpleasant disease, and increase your risk of dying from cancer. That's not the Christian God I was taught about.

The weird world of US ethics regulation

There has been a lot of interest over the past week in the Burzynski Clinic, a US organisation that offers unorthodox treatment to those with cancer. To get up to speed on the backstory see this blogpost by Josephine Jones.

As someone who spends more of my time than I’d like grappling with research ethics committees, there was one aspect of this story that surprised me. According to this blogpost, the clinic is not allowed to offer medical treatment, but is allowed to recruit patients to take part in clinical trials. But this is expensive for participants. The Observer piece that started all the uproar this week described how a family needed to raise £200,000 so that their very sick little girl could undergo Burzynski’s treatment.

I had assumed that this trial hadn’t undergone ethical scrutiny, because I could not see how any committee could agree that it was ethical to charge someone enormous sums of money to take part in a research project in which there was no guarantee of benefit. I suspect that many people would pay up if they felt they’d exhausted all other options. But this doesn’t mean it’s right.

I was surprised, then, to discover that the Burzynski trial had undergone review by an Institutional Review Board (IRB - the US term for an ethics committee). A letter describing the FDA’s review of the relevant IRB is available on the web. It concludes that “the IRB did not adhere to the applicable statutory requirements and FDA regulations governing the protection of human subjects.”  There’s a detailed exposition of the failings of the Burzynski Institute IRB, but no mention of fees charged to patients. So I followed a few more links and came to a US government site that described regulatory guidelines for ethics committees, which had a specific section on Charging for Investigational Products. It seems the practice of passing on research costs to research participants is allowed in the US system.

There has been considerable debate in academic circles about the opposite situation, where participants are paid to take part in a study. I know of cases where such payments have been prohibited by an ethics committee on the grounds that they provide ‘inducement’, which is generally regarded as a Bad Thing, though there are convincing counterarguments. But I am having difficulty in tracking down any literature at all on the ethics of requiring participants to pay a fee to take part in research. Presumably this is a much rarer circumstance than cases where participants are paid, because in general people need persuading to take part in research. The only people who are likely to pay large sums to be a research participant are those who are in a vulnerable state, feeling they have nothing to lose. But these are the very people who need protection by ethics committees because it’s all too easy for unscrupulous operators to exploit their desperation. Anyone who doesn’t have approval to charge for a medical treatment could just redescribe their activities as a clinical trial and bypass regulatory controls. Surely this cannot be right.