Sunday 26 May 2024

Are commitments to open data policies worth the paper they are written on?

 

As Betteridge's law of headlines states: "Any headline that ends in a question mark can be answered by the word no."  So you know where I am going with this.  

 

I'm a longstanding fan of open data - in fact, I first blogged about this back in 2015. So I've been gratified to see the needle shift on this, in the sense that over the past decade, in a rush to present themselves as good guys, various institutions and publishers have published policies supporting open data. The problem is that when you actually ask them to implement those policies, they back down.   

 

I discussed arguments for and against data-sharing in a Commentary article in 2016. I divided the issues according to whether they focused on the impact of data-sharing on researchers or on research participants. Table 1 from that article, entitled "Conflict between interests of researchers and advancement of science" is reproduced here:

 

Argument

Counter-argument

1. Lack of time to curate data.

Unless adequately curated, data will over time become unusable, including by the original researcher.

2. Personal investment—reluctance to give data to freeloaders.

Reuse of data increases its value and the researcher benefits from additional citations. There is also an ethical case for maximizing use of data obtained via public funding.

3. Concerns about being scooped before the analysis is complete.

This is a common concern though there are few attested cases. A time-limited period of privileged use by the study team can be specified to avoid scooping.

4. Fear of errors being found in the data.

Culture change is needed to recognize errors are inevitable in any large dataset and should not be a reason for reputational damage. Data-sharing allows errors to be found and corrected.

 

I then went on to discuss two other concerns which focused on implications of data-sharing for human participants, viz:

5.  Ethical concerns about confidentiality of personal data, especially in the context of clinical research

6.  Possibility that others with a different agenda may misuse the data, e.g. perform selective analyses that misrepresent the findings.

 

These last two issues raise complex concerns and there's plenty to discuss on how address them, but I'll put that to one side for now, as the case I want to comment on concerns a simple dataset where there is limited scope for secondary analyses and where no human participants are involved.

 

My interest was piqued by comments on PubPeer about a paper entitled "Magnetic field screening in hydrogen-rich high-temperature superconductors ".  The thread on PubPeer starts with this extraordinary comment by J. E. Hirsch:

 

I requested the underlying data for Figs. 3a, 3e, 3b, 3f of this paper on Jan 11, 2023. This is because the published data for Figs. 3a and 3e, as well as for Figs. 3b and 3f, are nominally the same but incompatible with each other, and I would like to understand why that is. I asked the authors to explain, but they did not provide an explanation. Neither did they supply the data. The journal told me that it had received the data from the authors but will not share them with me because they are "confidential". I requested that the journal posts an Editor Note informing readers that data are unavailable to readers. The journal responded that because data were share with editors they "cannot write an editorial note on the published article stating the data is unavailable as this would be factually incorrect".

 

Pseudonymous commenter Orchestes quercus drew attention to the Data Availability statement in the article: "The data that support the findings of this study are available from the corresponding authors upon reasonable request".

 

J. E. Hirsch then added a further comment: 

 

The underlying data are still not available, the editor says the author deems the request "unreasonable" but it cannot divulge the reasoning behind it, nor can the journal publish an editor note that there are restrictions on data availability because the data were provided to the journal.  Springer Nature's Research Integrity Director wrote to me in September 2023 that "we recognize the right of the authors to not share the data with you, in line with the authors’ chosen data availability statement", and that "As Springer Nature considers the correspondence with the authors confidential, we cannot share with you any further details.

 

Now, I know nothing whatsoever about superconductors or J. E. Hirsch, but I think the editors, publisher and the authors are making themselves look very silly, and indeed suspicious, by refusing to share the data.  They can't plead patient confidentiality or ethical restrictions - it seems they are just refusing to comply because they don't want to.  

 

To up the ante, Orchestes quercus extracted data from the figures and did further analyses, which confirmed that J. E. Hirsch had a point - the data did not appear to be internally consistent.

 

Meanwhile, I had joined the PubPeer thread, pointing out

 

The authors and editor appear to be in breach of the policy of Nature Portfolio journals, stated here:
https://www.nature.com/nature-portfolio/editorial-policies/reporting-standards, viz:

An inherent principle of publication is that others should be able to replicate and build upon the authors' published claims. A condition of publication in a Nature Portfolio journal is that authors are required to make materials, data, code, and associated protocols promptly available to readers without undue qualifications. Any restrictions on the availability of materials or information must be disclosed to the editors at the time of submission. Any restrictions must also be disclosed in the submitted manuscript.

After publication, readers who encounter refusal by the authors to comply with these policies should contact the chief editor of the journal. In cases where editors are unable to resolve a complaint, the journal may refer the matter to the authors' funding institution and/or publish a formal statement of correction, attached online to the publication, stating that readers have been unable to obtain necessary materials to replicate the findings.

I also noted that two of the authors are based at a Max Planck Institute. The Max Planck Gesellschaft is a signatory to the BerlinDeclaration on Open Access to Knowledge in the Sciences and Humanities.  On the website it states:

the Max Planck Society (MPG) is committed to the goal of providing free and open access to all publications and data from scholarly research (my emphasis).

 

Well, the redoubtable J. E. Hirsch had already thought of that, and in a subsequent PubPeer comment made public various exchanges he had had with luminaries from the Max Planck Institutes.

 

All I can say to the Max Planck Gesellschaft is that this is not a good look. Hirsch has noted an inconsistency in the published figures.  This has been confirmed by another reader and needs to be explained. The longer people dig in defensively, attacking the person making the request rather than just showing the raw data, the more it looks as if something fishy is going on here.

 

Why am I so hung up on data-sharing? The reason is simple. The more I share my own data, or use data shared by others, the more I appreciate the value of doing so. Errors are ubiquitous, even when researchers are careful, but we'll never know about them if data are locked away.

 

Furthermore, it is a sad reality that fraudulent papers are on the rise, and open data is one way of defending against them. It's not a perfect defence: people can invent raw data as well as summary data, but realistic data are not so easy to fake, and requiring open data would slow down the fraudsters and make them easier to catch.

 

Having said that, asking for data is not tantamount to accusing researchers of fraud: it should be accepted as normal scientific practice to make data available in order that others can check the reproducibility of findings. If someone treats such a request as an accusation, or deems it "unreasonable", then I'm afraid it just makes me suspicious.  

 

And if organisations like Springer Nature and Max Planck Gesellschaft won't back up their policies with action, then I think they should delete them from their websites. They are presenting themselves as champions of open, reproducible science, while acting as defenders of non-transparent, secret practices. As we say in the UK, fine words butter no parsnips.   

 

 P.S. 27th May:  A comprehensive account of the superconductivity affair has just appeared on the website For Better Science.  This suggests things are even worse than I thought.   

 

In addition, you can see Jorge Hirsch explain his arduous journey in attempting to access the data here.    

 

NOTE ON COMMENTS: Many thanks to those who have commented. Comments are moderated to prevent spam, so there is a delay before they appear, but I will accept on-topic comments in due course.




Wednesday 27 March 2024

Some thoughts on eLife's New Model: One year on

 

I've just been sent an email from eLife, pointing me to links to a report called "eLife's New Model: One year on" and a report by the editors "Scientific Publishing: The first year of a new era". To remind readers who may have missed it, the big change introduced by eLife in 2023 was to drop the step where an editor decides on reject or accept of a manuscript after reviewer comments are received. Instead, the author submits a preprint, and the editors then decide whether it should be reviewed. If the answer is yes, then the paper will be published, with reviewer comments. 

Given the controversy surrounding this new publishing model, it seems timely to have a retrospective look at how it's gone, and these pieces by the journal are broadly encouraging in showing that the publishing world has not fallen apart as a consequence of the changes. We are told that the proportion of submissions published has gone down slightly from 31.4% to 27.7% and the demographic characteristics of authors and reviewers are largely unchanged. The ratings of quality of submissions are similar to those from the legacy model. The most striking change has been in processing time: median time from submission to publication of the first version with reviews is 91 days, which is much faster than previously. 

As someone who has been pushing for changes to the model of scientific publishing for years (see blogsposts below), I'm generally in favour of any attempt to disrupt the conventional model. I particularly like the fact that the peer reviews are available with the published articles in eLife - I hope that will become standard for other journals in future. However, there are two things that rather rankled about the latest communication from the journal. 

First, the report describes an 'author survey' which received 325 responses, but very little detail is given as to who was surveyed, what the response rate was, and what the overall outcome was. This reads more like a marketing report than a serious scientific apprasal. Two glowing endorsements were reported from authors who had good experiences. I wondered though about authors whose work had not been selected to go forward to peer review - were they just as enthusiastic? Quite a few tables of facts and figures about the impact of the new policy were presented with the report, but if eLife really does want to present itself as embracing open and transparent policies, I think they should bite the bullet and provide more information - including fuller details of their survey methods and results, and negative as well as positive appraisals. 

Second, I continue to think there is a fatal flaw in the new model, which is that it still relies on editors to decide which papers go forward to review, using a method that will do nothing to reduce the tendency to hype and the consequent publication bias that ensues. I blogged about this a year ago, and suggested a simple solution, which is for the editors to adopt 'results-blind' review when triaging papers. This is an idea that has been around at least since 1976 (Mahoney, 1976) which has had a resurgence in popularity in recent years, with growing awareness of the dangers of publication bias (Locasio, 2017). The idea is that editorial decisions should be made based on whether the authors had identified an interesting question and whether their methods were adequate to give a definitive answer to that question. The problem with the current system is that people get swayed by exciting results, and will typically overlook weak methods when there is a dramatic finding. If you don't know the results, then you are forced to focus on the methods. The eLife report states:

 "It is important to note that we don’t ascribe value to the decision to review. Our aim is to produce high-quality reviews that will be of significant value but we are not able to review everything that is submitted." 

That is hard to believe: if you really were just ignoring quality considerations, then you should decide on which papers to review by lottery. I think this claim is not only disingenuous but also wrong-headed. If you have a limited resource - reviewer capacity - then you should be focusing it on the highest quality work. But that judgement should be made on the basis of research question and design, and not on results. 

Bibliography 

Locascio, J. J. (2017). Results blind science publishing. Basic and Applied Social Psychology, 39(5), 239–246. https://doi.org/10.1080/01973533.2017.1336093 

Mahoney, M. J. (1976). Scientist as Subject: The Psychological Imperative. Ballinger Publishing Company. 

Previous blogposts

Academic publishing: why isn't psychology like physics? 

Time for academics to withdraw free labour.

High impact journals: where newsworthiness trumps methodology

Will traditional science journals disappear?

Publishing replication failures


Sunday 24 March 2024

Just make it stop! When will we say that further research isn't needed?

 

I have a lifelong interest in laterality, which is a passion that few people share. Accordingly, I am grateful to René Westerhausen who runs the Oslo Virtual Laterality Colloquium, with monthly presentations on topics as diverse as chiral variation in snails and laterality of gesture production. 

On Friday we had a great presentation from Lottie Anstee who told us about her Masters project on handedness and musicality. There have been various studies on this topic over the years, some claiming that left-handers have superior musical skills, but samples have been small and results have been mixed. Lottie described a study with an impressive sample size (nearly 3000 children aged 10-18 years) whose musical abilities were evaluated on a detailed music assessment battery that included self-report and perceptual evaluations. The result was convincingly null, with no handedness effect on musicality. 

What happened next was what always happens in my experience when someone reports a null result. The audience made helpful suggestions for reasons why the result had not been positive and suggested modifications of the sampling, measures or analysis that might be worth trying. The measure of handedness was, as Lottie was the first to admit, very simple - perhaps a more nuanced measure would reveal an association? Should the focus be on skilled musicians rather than schoolchildren? Maybe it would be worth looking at nonlinear rather than linear associations? And even though the music assessment was pretty comprehensive, maybe it missed some key factor - amount of music instruction, or experience of specific instruments. 

After a bit of to and fro, I asked the question that always bothers me. What evidence would we need to convince us that there is really no association between musicality and handedness? The earliest study that Lottie reviewed was from 1922, so we've had over 100 years to study this topic. Shouldn't there be some kind of stop rule? This led to an interesting discussion about the impossibility of proving a negative and whether we should be using Bayes Factors, and what would be the smallest effect size of interest.  

My own view is that further investigation of this association would prove fruitless. In part, this is because I think the old literature (and to some extent the current literature!) on factors associated with handedness is at particular risk of bias, so even the messy results from a meta-analysis are likely to be over-optimistic. More than 30 years ago, I pointed out that laterality research is particularly susceptible to what we now call p-hacking - post hoc selection of cut-offs and criteria for forming subgroups, which dramatically increase the chances of finding something significant. In addition, I noted that measurement of handedness by questionnaire is simple enough to be included in a study as a "bonus factor", just in case something interesting emerges. This increases the likelihood that the literature will be affected by publication bias - the handedness data will be reported if a significant result is obtained, but otherwise can be disregarded at little cost. So I suspect that most of the exciting ideas about associations between handedness and cognitive or personality traits are built on shaky foundations, and would not replicate if tested in well-powered, preregistered studies.  But somehow, the idea that there is some kind of association remains alive, even if we have a well-designed study that gives a null result.  

Laterality is not the only area where there is no apparent stop rule. I've complained of similar trends in studies of association between genetic variants and psychological traits, for instance, where instead of abandoning an idea after a null study, researchers slightly change the methods and try again. In 2019, Lisa Feldman Barrett wrote amusingly about zombie ideas in psychology, noting that some theories are so attractive that they seem impossible to kill. I hope that as preregistration becomes more normative, we may see more null results getting published, and learn to appreciate their value. But I wonder just what it takes to get people to conclude that a research seam has been mined to the point of exhaustion. 


Friday 9 February 2024

The world of Poor Things at MDPI journals


At the weekend, the Observer ran a piece by Robin McKie entitled "‘The situation has become appalling’: fake scientific papers push research credibility to crisis point". I was one of those interviewed for the article, describing my concerns about a flood of dodgy papers that was polluting the scientific literature.

Two days later I received an email from the editorial office of MDPI publishers with the header "[Children] (IF: 2.4, ISSN 2227-9067): Good Paper Sharing on the Topic of" (sic) that began:

Greetings from the Children Editorial Office!

We recently collected 10 highly cited papers in our journal related to Childhood Autism. And we sincerely invite you to visit and read these papers, because you are an excellent expert in this field of study.

Who could resist such a flattering invitation? MDPI is one of those publishers that appears to be encouraging publication of low quality work, with a massive growth in special issues where papers are published with remarkably rapid turnaround times. Only last week it was revealed that the journal is affected by fake peer review that appears to be generated by AI. So I was curious to take a look.

The first article, by Frolli et al (2022a) was weird. It reported a comparison of two types of intervention designed to improve emotion recognition in children with autism, one of which used virtual reality. The first red flag was the sample size: two groups each of 30 children, all originally from the city of Caserta. I checked Wikipedia, which told me the population of Caserta was around 76,000 in 2017. Recruiting participants for intervention studies is typically slow and laborious and this is a remarkable sample size to recruit from such a small region. But credibility is then stretched to breaking point on hearing that the selection criteria required that the children were all aged between 9 and 10 years and had IQs of 97 or above. No researcher in their right mind would impose unnecessary constraints on recruitment, and both the age and IQ criteria are far tighter than would usually be adopted. I wondered whether there might be a typo in this account, but we then hear that the IQ range of the sample is indeed remarkably narrow: 

"The first experimental group (Gr1) was composed of 30 individuals with a mean age of 9.3 (SD 0.63) and a mean IQ of 103.00 (SD 1.70). ...... The second experimental group (Gr2) was composed of 30 individuals with a mean age of 9.4 (SD 0.49) and mean IQ of 103.13 (SD 2.04)...."

Most samples for studies using Wechsler IQ scales have SD of at least 8, even if cutoffs are applied as selection criteria, so this is unbelievably low.

This dubious paper prompted me to look at others by the first author. It was rather like pulling a thread on a hole in a sweater - things started to unravel fast. A paper published by Frolli et al (2023a) in the MDPI journal Behavioral Sciences claimed to have studied eighty 18-year-olds recruited from four different high schools. The selection criteria were again unbelievably stringent: IQ assessed on the WAIS-IV fell between 95-105 "to ensure that participants fell within the average range of intellectual functioning, minimizing the impact of extreme cognitive variations on our analyses". The lower IQ range selected here corresponds to z-score of -0.33 or 37th percentile. If the population of students covered the full range of IQ, then only around 25% would meet the criterion (between 37th and 63rd centile), so to obtain a sample of 80 it would be necessary to test over 300 potential participants. Furthermore, there are IQ screening tests that can be used in this circumstance that are relatively quick to administer, but the WAIS-IV is not one of them. We are told all participants were given the full test, which requires individual administration by a qualified psychologist and takes around one hour to complete. So who did all this testing, and where? The article states: "The data were collected and analyzed at the FINDS Neuropsychiatry Outpatient Clinic by licensed psychologists in collaboration with the University of International Studies of Rome (UNINT)." So we are supposed to believe that hundreds of 18-year-olds trekked to a neuropsychiatry outpatient clinic for a full IQ screening which most of them would not have passed. I cannot imagine a less efficient way of conducting such a study. I could not find any mention of compensation for participants, which is perhaps unsurprising as the research received no external funding. All of this is described as happening remarkably fast, with ethics approval in January 2023, and submission of the article in October 2023.

Another paper in Children in 2023 focused on ADHD, and again reported recruiting two groups of 30 children for an intervention that lasted 5 months (Frolli et al., 2023b). The narrow IQ selection criteria were again used, with WISC-IV IQs in the range 95-105, and the mean IQs were 96.48 (SD =1.09) and 98.44 (SD = 1.12) for groups 1 and 2 respectively. Again, the research received no external funding. The report of ethics approval is scanty "The study was conducted in accordance with the Declaration of Helsinki. The study was approved by the Ethics Committee and the Academic Senate of the University of International Studies of Rome."

The same first author published a paper on the impact of COVID-19 on cognitive development and executive functioning in adolescents in 2021 (Frolli et al, 2021). I have not gone over it in detail, but a quick scan revealed some very odd statistical reporting. There were numerous F-ratios, but they were all negative, which is impossible, as F is a ratio between two positive numbers. Furthermore, the reported p-values and degrees of freedom didn't always correspond to the F-ratio, even if the sign was ignored.

At this point I was running out of steam, but a quick look at Frolli et al (2022a) on Executive Functions and Foreign Language Learning suggested yet more problems, with the sentence "Significance at the level of 5% (α < 0.001) has been accepted" featuring at least twice. It is hard to believe that a human being wrote this sentence, or that any human author, editor or reviewer read it without comment.

If anyone is interested in pulling at other related threads, I suspect it would be of interest to look at articles accepted for a Special Issue of the MDPI journal Disabilities co-edited by Frolli.

In his brilliant film Poor Things, Yorgos Lanthimos distorts familiar objects and places just enough to be disturbing. Lisbon looks like what I imagine Lisbon would be in the Victorian age, except that the colours are unusually vivid, there are strange flying cars in the sky, and nobody seems concerned at the central character wandering around only partially clothed (see, e.g., this review).  The combined impression is that MDPI publishes papers from that universe, where everything looks superficially like genuine science but with jarring features that tell you something is amiss. The difference is that Poor Things has a happy ending.

References 

Frolli, A.; Ricci, M.C.; Di Carmine, F.; Lombardi, A.; Bosco, A.; Saviano, E.; Franzese, L. The Impact of COVID-19 on Cognitive Development and Executive Functioning in Adolescents: A First Exploratory Investigation. Brain Sci. 2021, 11, 1222. https://doi.org/10.3390/brainsci11091222

Frolli, A.; Savarese, G.; Di Carmine, F.; Bosco, A.; Saviano, E.; Rega, A.; Carotenuto, M.; Ricci, M.C. Children on the Autism Spectrum and the Use of Virtual Reality for Supporting Social Skills. Children 2022a, 9, 181. https://doi.org/10.3390/children9020181

Frolli, A.; Cerciello, F.; Esposito, C.; Ciotola, S.; De Candia, G.; Ricci, M.C.; Russo, M.G. Executive Functions and Foreign Language Learning. Pediatr. Rep. 2022b, 14, 450-456. https://doi.org/10.3390/pediatric14040053

Frolli, A.; Cerciello, F.; Ciotola, S.; Ricci, M.C.; Esposito, C.; Sica, L.S. Narrative Approach and Mentalization. Behav. Sci. 2023a, 13, 994. https://doi.org/10.3390/bs13120994

Frolli, A.; Cerciello, F.; Esposito, C.; Ricci, M.C.; Laccone, R.P.; Bisogni, F. Universal Design for Learning for Children with ADHD. Children 2023b, 10, 1350. https://doi.org/10.3390/children10081350

Friday 2 February 2024

An (intellectually?) enriching opportunity for affiliation

Guest Post by Nick Wise 


 

A couple of months ago a professor received the following email, which they forwarded to me.

 

"Dear esteemed colleagues,

We are delighted to extend an invitation to apply for our prestigious remote research fellowships at the University of Religions and Denominations (URD). These fellowships offer substantial financial support to researchers with papers currently in press, accepted or under review by Scopus-indexed journals. We welcome scholars from diverse academic disciplines to seize this intellectually enriching opportunity.

Fellowship Details:
Fellowship Type: Remote Short-term Research Fellowship.
Research Focus: Diverse fields, spanning humanities, social sciences, interdisciplinary studies, and more.
Research Output: Publication of research articles in Scopus-indexed journals.
Affiliation: Encouragement for researchers to acknowledge URD as their additional affiliation in published articles.
Remuneration: Project-based compensation for each research article.
Payment Range: Up to $1000 USD per article (based on SJR journal ranking).
Eligibility: Papers in press, accepted, or under review by Scopus-indexed journals.

Preference: Priority for indexing before December 30, 2023.

Application Process:   

To express your interest in securing a fellowship, kindly submit your curriculum vitae to  Ahmad Moghri at moghri.urd@gmail.com. When emailing your application, please use the subject line: "Research Fellowship, FULL NAME."

Upon Selection:
Successful applicants will receive formal invitations to join our esteemed fellowship program. Invitation letters and collaboration contracts will be dispatched within a maximum of 5 days.

We firmly believe that this fellowship program provides an invaluable platform for scholars to make substantial contributions to their fields while collaborating with the distinguished University of Religions and Denominations. We encourage all eligible individuals to seize this exceptional opportunity.

For inquiries or further information, please do not hesitate to contact moghri.urd@gmail.com.

Warmest Regards,”

Why would the institution pay researchers to say that they are affiliated with them? It could be that funding for the university is related to the number of papers published in indexed journals. More articles associated with the university can also improve their placing in national or international university rankings, which could lead directly to more funding, or to more students wanting to attend and bringing in more money.

The University of Religions and Denominations is a private Iranian university specialising, as the name suggests, in the study of different religions and movements. Until recently the institution had very few published papers associated with it according to Dimensions and their subject matter was all related to religion. However, last year there was a substantial increase to 103 published papers, and so far this year there are already 35. This suggests that some academics have taken them up on the offer in the advert to include URD as an affiliation.

Surbhi Bhatia Khan is a lecturer in data science at the University of Salford in the UK since March 2023 and a top 2% scientist in the world according to Stanford University’s rankings. She published 29 research articles last year according to Dimensions, an impressive output, in which she was primarily affiliated to the University of Salford. In addition though, 5 of those submitted in the 2nd half of last year had an additional affiliation at the Department of Engineering and Environment at URD, which is not listed as one of the departments on the university website. Additionally, 19 of the 29 state that she’s affiliated to the Lebanese American University in Beirut, which she was not affiliated with before 2023. She is yet to mention her role at either of these additional affiliations on her LinkedIn profile.

Looking at the Lebanese American University, another private university, its publication numbers have shot up from 201 in 2015 to 503 in 2021 and 2,842 in 2023, according to Dimensions. So far in 2024 they have published 525, on track for over 6,000 publications for the year. By contrast, according to the university website, the faculty consisted of 547 full-time staff members in 2021 but had shrunk to 423 in 2023.  It is hard to imagine how such growth in publication numbers could occur without a similar growth in the faculty, let alone with a reduction.

How many other institutions are seeing incredible increases in publication numbers? Last year we saw gaming of the system on a grand scale by various Saudi Arabian universities, but how many offers like the one above are going around, whether by email or sent through Whatsapp groups or similar?

The Committee On Publication Ethics held a forum on claiming institutional affiliations in December 2023, in recognition of the fact that guidance for what merits affiliation to an institution is lacking and there are no accepted standards for how many affiliations an author should give. It looks like such guidance can’t come soon enough.

Nick Wise is a researcher at the University of Cambridge, UK.

Note: Comments are moderated to prevent spam and abuse, so please be patient if you post a comment and it does not appear immediately

P.S. 3rd Feb 2024

Someone on social media queried the "top 2% rating" for Khan. Nick tells me this is based on an Elsevier ranking for 2022: https://elsevier.digitalcommonsdata.com/datasets/btchxktzyw/6

Tuesday 5 December 2023

Low-level lasers. Part 2. Erchonia and the universal panacea

 

 


In my last blogpost, I looked at a study that claimed continuing improvements of symptoms of autism after eight 5-minute sessions where a low-level laser was pointed at the head.  The data were so extreme that I became interested in the company, Erchonia, who sponsored the study and in Regulatory Insight, Inc, whose statistician failed to notice anything odd.  In exploring Erchonia's research corpus, I found that they have investigated the use of their low-laser products for a remarkable range of conditions. A search of clinicaltrials.com with the keyword Erchonia produced 47 records, describing studies of pain (chronic back pain, post-surgical pain, and foot pain), body contouring (circumference reduction, cellulite treatment), sensorineural hearing loss, Alzheimer's disease, hair loss, acne and toenail fungus. After excluding the trials on autism described in my previous post, fourteen of the records described randomised controlled trials in which an active laser was compared with a placebo device that looked the same, with both patient and researcher being kept in the dark about which device was which until the data were analysed. As with the autism study, the research designs for these RCTs specified on clinicaltrials.com looked strong, with statistician Elvira Cawthon from Regulatory Insight involved in data analysis.

As shown in Figure 1, where results are reported for RCTs, they have been spectacular in virtually all cases. The raw data are mostly not available, and in general the plotted data look less extreme than in the autism trial covered in last week's post, but nonetheless, the pattern is a consistent one, where over half the active group meet the cutoff for improvement, whereas less than half (typically 25% or less) of the placebo group do so. 

FIGURE 1: Proportions in active treated group vs placebo group meeting preregistered criterion for improvement (Error bars show SE)*

I looked for results from mainstream science against which to benchmark the Erchonia findings.  I found a big review of behavioural and pharmaceutical interventions for obesity by the US Agency for Healthcare Research and Quality (LeBlanc et al, 2018). Figures 7 and 13 show results for binary outcomes - relative risk of losing 5% or more of body weight over a 12 month period; i.e. the proportion of treated individuals who met this criterion divided by the proportion of controls. In 38 trials of behavioural interventions, the mean RR was 1.94 [95% CI, 1.70 to 2.22]. For 31 pharmaeutical interventions, the effect varied with the specific medication, with RR ranging from 1.18 to 3.86. Only two pharmaceutical comparisons had RR in excess of 3.0. By contrast, for five trials of body contouring or cellulite reduction from Erchonia, the RRs ranged from 3.6 to 18.0.  Now, it is important to note that this is not comparing like with like: the people in the Erchonia trials were typically not clinically obese: they were mostly women seeking cosmetic improvements to their appearance.  So you could, and I am sure many would, argue it's an unfair comparison. If anyone knows of another literature that might provide a better benchmark, please let me know. The point is that the effect sizes reported by Erchonia are enormous relative to the kinds of effects typically seen with other treatments focused on weight reduction.

If we look more generally at the other results obtained with low-level lasers, we can compare them to an overview of effectiveness of common medications (Leucht et al, 2015). These authors presented results from a huge review of different therapies, with effect sizes represented as standardized mean differences (SMD - familiar to psychologists as Cohen's d). I converted Erchonia results into this metric*, and found that across all the studies of pain relief shown in Figure 1, the average SMD was 1.30, with a range from 0.87 to 1.77. This contrasts with Leucht et al's estimated effect size of 1.06 for oxycodone plus paracetamol, and 0.83 for Sumatriptan for migraine.  So if we are to believe the results, they indicate that the effect of Erchonia low-level lasers is as good or better than the most effective pharmaceutical medications that we have for pain relief or weight loss. I'm afraid I remain highly sceptical.

I would not have dreamed of looking at Erchonia's track record if it were not for their impossibly good results in the Leisman et al autism trial that I discussed in the previous blogpost.  When I looked in more detail, I was reminded of the kinds of claims made for alternative treatments for children's learning difficulties, where parents are drawn in with slick websites promising scientifically proven interventions, and glowing testimonials from satisfied customers. Back in 2012 I blogged about how to evaluate "neuroscientific" interventions for dyslexia.  Most of the points I made there apply to the world of "photomodulation" therapies, including the need to be wary when a provider claims that a single method is effective for a whole host of different conditions.  

Erchonia products are sold worldwide and seem popular with alternative health practitioners. For instance, in Stockport, Manchester, you can attend a chiropractic clinic where Zerona laser treatment will remove "stubborn body fat". In London there is a podiatry centre that reassures you: "There are numerous papers which show that cold laser affects the activity of cells and chemicals within the cell. It has been shown that cold laser can encourage the formation of stem cells which are key building blocks in tissue reparation. It also affects chemicals such as cytochrome c and causes a cascade of reactions which stimulates the healing. There is much research to show that cold laser affects healing and there are now several very good class 1 studies to show that laser can be effective." But when I looked for details of these "very good class 1 studies" they were nowhere to be found. In particular, it was hard to find research by scientists without vested interests in the technology.  

Of all the RCTs that I found, there were just two that were conducted at reputable universities. One of them, on hearing loss (NCT01820416) was conducted at the University of Iowa, but terminated prematurely because intermediate analysis showed no clinically or statistically significant effects (Goodman et al., 2013).  This contrasts sharply with NCT00787189, which had the dramatic results reported in Figure 1 (not, as far as I know, published outside of clinicaltrials.gov). The other university-based study was the autism study based in Boston described in my previous post: again, with unpublished, unimpressive results posted on clinicaltrials.gov.

This suggests it is important when evaluating novel therapies to have results from studies that are independent of those promoting the therapy. But, sadly, this is easier to recommend than to achieve. Running a trial takes a lot of time and effort: why would anyone do this if they thought it likely that the intervention would not work and the postulated mechanism of action was unproven? There would be a strong risk that you'd end up putting in effort that would end in a null result, which would be hard to publish. And you'd be unlikely to convince those who believed in the therapy - they would no doubt say you had the wrong wavelength of light, or insufficient duration of therapy, and so on.  

I suspect the response by those who believe in the power of low-level lasers will be that I am demonstrating prejudice, in my reluctance to accept the evidence that they provide of dramatic benefits. But, quite simply, if low-level laser treatment was so remarkably effective in melting fat and decreasing pain, surely it would have quickly been publicised through word of mouth from satisfied customers. Many of us are willing to subject our bodies to all kinds of punishments in a quest to be thin and/or pain-free. If this could be done simply and efficiently without the need for drugs, wouldn't this method have taken over the world?

*Summary files (Erchonia_proportions4.csv) and script (Erchonia_proportions_for_blog.R) are on Github, here.

Saturday 25 November 2023

Low-level lasers. Part 1. Shining a light on an unconventional treatment for autism


 

'Light enters, then a miracle happens, and good things come out!' (Quirk & Whelan, 2011*)



I'm occasionally asked to investigate weird interventions for children's neurodevelopmental conditions, and recently I've found myself immersed in the world of low-level laser treatments. The material I've dug up is not new - it's been around for some years, but has not been on my radar until now. 

A starting point is this 2018 press statement by Erchonia, a firm that makes low-level laser devices for quasi-medical interventions. 

They had tested a device that was supposed to reduce irritability in autistic children by applying low-level laser light to the temporal and posterior regions of the head (see Figure 1) for 5 minute sessions twice a week for 4 weeks.

Figure 1: sites of stimulation by low-level laser

 The study, which was reported here, was carefully designed as a randomized controlled trial. Half the children received a placebo intervention. Placebo and active laser devices were designed to look identical and both emitted light, and neither the child nor the person administering the treatment knew whether the active or placebo light was being used.

According to Erchonia “The results are so strong, nobody can argue them.” (sic). Alas, their confidence turned out to be misplaced.

The rationale given by Leisman et al (with my annotations in yellow in square brackets) is as follows: "LLLT promotes cell and neuronal repair (Dawood and Salman 2013) [This article is about wound healing, not neurons] and brain network rearrangement (Erlicher et al. 2002) [This is a study of rat cells in a dish] in many neurologic disorders identified with lesions in the hubs of default mode networks (Buckner et al. 2008)[This paper does not mention lasers]. LLLT facilitates a fast-track wound-healing (Dawood and Salman 2013) as mitochondria respond to light in the red and near-infrared spectrum (Quirk and Whelan 2011*)[review of near-infrared irradiation photobiomodulation that notes inadequate knowledge of mechanisma - see cartoon]. On the other hand, Erlicher et al. (2002) have demonstrated that weak light directs the leading edge of growth cones of a nerve [cells in a dish]. Therefore, when a light beam is positioned in front of a nerve’s leading edge, the neuron will move in the direction of the light and grow in length (Black et al. 2013 [rat cells in a dish]; Quirk and Whelan 2011). Nerve cells appear to thrive and grow in the presence of low-energy light, and we think that the effect seen here is associated with the rearrangement of connectivity."

I started out looking at the registration of the trial on ClinicalTrials.gov. This included a very thorough document that detailed a protocol and analysis plan, but there were some puzzling inconsistencies; I documented them here on PubPeer,  and subsequently a much more detailed critique was posted there by Florian Naudet and André Gillibert. Among other things, there was confusion about where the study was done. The registration document said it was done in Nazareth, Israel, which is where the first author, Gerry Leisman was based. But it also said that the PI was Calixto Machado, who is based in Havana, Cuba.

Elvira Cawthon, from Regulatory Insight, Inc, Tennessee was mentioned on the protocol as clinical consultant and study monitor. The role of the study monitor is specified as follows: 

"The study Monitor will assure that the investigator is executing the protocol as outlined and intended. This includes insuring that a signed informed consent form has been attained from each subject’s caregiver prior to commencing the protocol, that the study procedure protocol is administered as specified, and that all study evaluations and measurements are taken using the specified methods and correctly and fully recorded on the appropriate clinical case report forms."

This does not seem ideal, given that the study monitor was in Tennessee, and the study was conducted in either Nazareth or Havana. Accordingly, I contacted Ms Cawthon, who replied: 

"I can confirm that I performed statistical analysis on data from the clinical study you reference that was received from paper CRFs from Dr. Machado following completion of the trial. I was not directly involved in the recruitment, treatment, or outcomes assessment of the subjects whose data was recorded on those CRFs. I have not reviewed any of the articles you referenced below so I cannot attest to whether the data included was based on the analyses that I performed or not or comment on any of the discrepancies without further evaluation at this time."

I had copied Drs Leisman and Machado into my query, and Dr Leisman also replied. He stated:

"I am the senior author of the paper pertaining to a trial of low-level laser therapy in autism spectrum disorder.... I take full responsibility for the publication indicated above and vouch for having personally supervised the implementation of the project whose results were published under the following citation:

Leisman, G. Machado, C., Machado, Y, Chinchilla-Acosta, M. Effects of Low-Level Laser Therapy in Autism Spectrum Disorder. Advances in Experimental Medicine and Biology 2018:1116:111-130. DOI:10.1007/5584_2018_234. The publication is referenced in PubMed as: PMID: 29956199.

I hold a dual appointment at the University of Haifa and at the University of the Medical Sciences of Havana with the latter being "Professor Invitado" by the Ministry of Health of the Republic of Cuba. Ms. Elvira Walls served as the statistical consultant on this project."

However, Dr Leisman denied any knowledge of subsequent publications of follow-up data by Dr Machado. I asked if I could see the data from the Leisman et al study, and he provided a link to a data file on ResearchGate, the details of which I have put on PubPeer.

Alas, the data were amazing, but not in a good way. The main data came from five subscales of the Aberrant Behavior Checklist (ABC)**, which can be combined into a Global score. (There were a handful of typos in the dataset for the Global score, which I have corrected in the following analysis). For the placebo group, 15 of 19 children obtained exactly the same global score on all 4 sessions. Note that there is no restriction of range for this scale: reported scores range from 9 to 154. This pattern was also seen in the five individual subscales. You might think that is to be expected if the placebo intervention is ineffective, but that's not the case. Questionnaire measures such as that used here are never totally stable. In part this is because children's behaviour fluctuates. But even if the behaviour is constant, you expect to see some variability in responses, depending on how the rater interprets the scale of measurement. Furthermore, when study participants are selected because they have extreme scores on a measure, the tendency is for scores to improve on later testing - a phenomenon known as regression to the mean, Such unchanging scores are out of line with anything I have ever come across in the intervention literature. If we turn to the treated group, we see that 20 of 21 children showed a progressive decline in global scores (i.e. improvement), with each measurement improving from the previous one over 4 sessions. This again is just not credible because we'd expect some fluctuation in children's behaviour as well as variable ratings due to error of measurement. These results were judged to be abnormal in a further commentary by Gillibert and Naudet on PubPeer. They also noted that the statistical distribution of scores was highly improbable, with far more even than odd numbers.

Although Dr Machado has been copied into my correspondence, he has not responded to queries. Remember, he was PI for the study in Cuba, and he is first author on a follow-up study from which Dr Leisman dissociated himself. Indeed, I subsequently found that there were no fewer than three follow-up reports, all appearing in a strange journal whose DOIs did not appear to be genuine: 

Machado, C., Machado, Y., Chinchilla, M., & Machado, Yazmina. (2019a). Follow-up assessment of autistic children 6 months after finishing low lever (sic) laser therapy. Internet Journal of Neurology, 21(1). https://doi.org/10.5580/IJN.54101 (available from https://ispub.com/IJN/21/1/54101).

Machado, C., Machado, Y., Chinchilla, M., & Machado, Yazmina. (2019b). Twelve months follow-up comparison between autistic children vs. Initial placebo (treated) groups. Internet Journal of Neurology, 21(2). https://doi.org/10.5580/IJN.54812 (available from https://ispub.com/IJN/21/2/54812).

Machado, C., Machado, Y., Chinchilla, M., & Machado, Yazmina. (2020). Follow-up assessment of autistic children 12 months after finishing low lever (sic) laser therapy. Internet Journal of Neurology, 21(2). https://doi.org/10.5580/IJN.54809 (available from available from https://ispub.com/IJN/21/2/54809)

The 2019a paper starts by talking of a study of anatomic and functional brain connectivity in 21 children, but then segues to an extended follow-up (6 months) of the 21 treated and 19 placebo children from the Leisman et al study. The Leisman et al study is mentioned but not adequately referenced. Remarkably, all the original participants participated in the follow-up. The same trend as before continued: the placebo group stagnated, whereas the treated group continue to improve up to 6 months later, even though they received no further active treatment after the initial 4 week period. The 2020 Abstract reported a further follow-up to 12 months. The huge group difference was sustained (see Figure 2). Three of the treated group were now reported as scoring in the normal range on a measure of clinical impairment. 

Figure 2. Chart 1 from Machado et al 2020
 

In the 2019b paper, it is reported that, after the stunning success of the initial phase of the study, the placebo group were offered the intervention, and all took part, whereupon they proceeded to make an almost identical amount of remarkable progress on all five subscales, as well as the global scale (see Figure 3). We might expect the 'baseline' scores of the cross-over group to correspond to the scores reported at the final follow-up (as placebo group prior to cross-over) but they don't. 

Figure 3: Chart 2 of Machado et al 2019b

I checked for other Erchonia studies on clinicaltrials.gov. Another study, virtually identical except for the age range, was registered in 2020 with Dr Leon Morales-Quezada of Spaulding Rehabilitation Hospital, Boston as Principal Investigator.  Comments in the documents suggest this was conducted after Erchonia failed to get the desired FDA approval. Although I have not found a published report of this second trial, I found a recruitment advertisement, which confusingly cites the NCT registration number of the 2013 study. Some summary results are included on clinicaltrials.gov, and they are strikingly different from the Leisman et al trial, with no indication of any meaningful difference between active and placebo groups in the final outcome measure, and both groups showing some improvement. I have requested fuller data from Elvira Cawthon (listed as results point of contact) with cc. to Dr Morales-Quezada and will update this post if I hear back.

It would appear that at one level this is a positive story, because it shows the regulatory system working. We do not know why FDA rejected Erchonia's request for 510k Market Clearance, but the fact that they did so might indicate that they were unimpressed by the data provided by Leisman and Machado. The fact that Machado et al reported their three follow-up studies in what appears to be an unregistered journal suggests they had difficulty persuading regular journals that the findings were legitimate. If eight 5-minute sessions with a low-level laser pointed at the head really could dramatically improve the function of children with autism 12 months later, one would imagine that Nature, Cell and Science would be scrambling to publish the articles. On the other hand, any device that has the potential to stimulate neuronal growth might also ring alarm bells in terms of potential for harm.

Use of low-level lasers to treat autism is only part of the story. Questions remain about the role of Regulatory Insight, Inc., whose statistician apparently failed to notice anything strange about the data from the first autism study. In another post, I plan to look at cases where the same organisation was involved in monitoring and analysing trials of Erchonia laser devices for other conditions such as cellulite, pain, and hearing loss.

Notes

* Quirk, B. J., & Whelan, H. T. (2011). Near-infrared irradiation photobiomodulation: The need for basic science. Photomedicine and Laser Surgery, 29(3), 143–144. https://doi.org/10.1089/pho.2011.3014. This article states "clinical uses of NIR-PBM have been studied in such diverse areas as wound healing, oral mucositis, and retinal toxicity. In addition, NIR-PBM is being considered for study in connection with areas such as aging and neural degenerative diseases (Parkinson's disease in particular). One thing that is missing in all of these pre-clinical and clinical studies is a proper investigation into the basic science of the NIR-PBM phenomenon. Although there is much discussion of the uses of NIR, there is very little on how it actually works. As far as explaining what really happens, we are basically left to resort to saying 'light enters, then a miracle happens, and good things come out!' Clearly, this is insufficient, if for no other reason than our own intellectual curiosity." 

**Aman, M. G., Singh, N. N., Stewart, A. W., & Field, C. J. (1985). The aberrant behavior checklist: A behavior rating scale for the assessment of treatment effects. American Journal of Mental Deficiency, 89(5), 485–491. N. B. this is different from the Autism Behavior Checklist which is a commonly used autism assessment. 

Sunday 19 November 2023

Defence against the dark arts: a proposal for a new MSc course

 


Since I retired, an increasing amount of my time has been taken up with investigating scientific fraud. In recent months, I've become convinced of two things: first, fraud is a far more serious problem than most scientists recognise, and second, we cannot continue to leave the task of tackling it to volunteer sleuths. 

If you ask a typical scientist about fraud, they will usually tell you it is extremely rare, and that it would be a mistake to damage confidence in science because of the activities of a few unprincipled individuals. Asked to name fraudsters they may, depending on their age and discipline, mention Paolo Macchiarini, John Darsee, Elizabeth Holmes or Diederik Stapel, all high profile, successful individuals, who were brought down when unambiguous evidence of fraud was uncovered. Fraud has been around for years, as documented in an excellent book by Horace Judson (2004), and yet, we are reassured, science is self-correcting, and has prospered despite the activities of the occasional "bad apple". The problem with this argument is that, on the one hand, we only know about the fraudsters who get caught, and on the other hand, science is not prospering particularly well - numerous published papers produce results that fail to replicate and major discoveries are few and far between (Harris, 2017). We are swamped with scientific publications, but it is increasingly hard to distinguish the signal from the noise. In my view, it is getting to the point where in many fields it is impossible to build a cumulative science, because we lack a solid foundation of trustworthy findings. And it's getting worse and worse.

My gloomy prognosis is partly engendered by a consideration of a very different kind of fraud: the academic paper mill. In contrast to the lone fraudulent scientist who fakes data to achieve career advancement, the paper mill is an industrial-scale operation, where vast numbers of fraudulent papers are generated, and placed in peer-reviewed journals with authorship slots being sold to willing customers. This process is facilitated in some cases by publishers who encourage special issues, which are then taken over by "guest editors" who work for a paper mill. Some paper mill products are very hard to detect: they may be created from a convincing template with just a few details altered to make the article original. Others are incoherent nonsense, with spectacularly strange prose emerging when "tortured phrases" are inserted to evade plagiarism detectors.

You may wonder whether it matters if a proportion of the published literature is nonsense: surely any credible scientist will just ignore such material? Unfortunately, it's not so simple. First, it is likely that the paper mill products that are detected are just the tip of the iceberg - a clever fraudster will modify their methods to evade detection. Second, many fields of science attempt to synthesise findings using big data approaches, automatically combing the literature for studies with specific keywords and then creating databases, e.g. of genotypes and phenotypes. If these contain a large proportion of fictional findings, then attempts to use these databases to generate new knowledge will be frustrated. Similarly, in clinical areas, there is growing concern that systematic reviews that are supposed to synthesise evidence to get at the truth instead lead to confusion because a high proportion of studies are fraudulent. A third and more indirect negative consequence of the explosion in published fraud is that those who have committed fraud can rise to positions of influence and eminence on the back of their misdeeds. They may become editors, with the power to publish further fraudulent papers in return for money, and if promoted to professorships they will train a whole new generation of fraudsters, while being careful to sideline any honest young scientists who want to do things properly. I fear in some institutions this has already happened.

To date, the response of the scientific establishment has been wholly inadequate. There is little attempt to proactively check for fraud: science is still regarded as a gentlemanly pursuit where we should assume everyone has honourable intentions. Even when evidence of misconduct is strong, it can take months or years for a paper to be retracted. As whistleblower Raphaël Levy asked on his blog: Is it somebody else's problem to correct the scientific literature? There is dawning awareness that our methods for hiring and promotion might encourage misconduct, but getting institutions to change is a very slow business, not least because those in positions of power succeeded in the current system, and so think it must be optimal.

The task of unmasking fraud is largely left to hobbyists and volunteers, a self-styled army of "data sleuths", who are mostly motivated by anger at seeing science corrupted and the bad guys getting away with it. They have developed expertise in spotting certain kinds of fraud, such as image manipulation and improbable patterns in data, and they have also uncovered webs of bad actors who have infiltrated many corners of science. One might imagine that the scientific establishment would be grateful that someone is doing this work, but the usual response to a sleuth who finds evidence of malpractice is to ignore them, brush the evidence under the carpet, or accuse them of vexatious behaviour. Publishers and academic institutions are both at fault in this regard.

If I'm right, this relaxed attitude to the fraud epidemic is a disaster-in-waiting. There are a number of things that need to be done urgently. One is to change research culture so that rewards go to those whose work is characterised by openness and integrity, rather than those who get large grants and flashy publications. Another is for publishers to act far more promptly to investigate complaints of malpractice and issue retractions where appropriate. Both of these things are beginning to happen, slowly. But there is a third measure that I think should be taken as soon as possible, and that is to train a generation of researchers in fraud busting. We owe a huge debt of gratitude to the data sleuths, but the scale of the problem is such that we need the equivalent of a police force rather than a volunteer band. Here are some of the topics that an MSc course could cover:

  • How to spot dodgy datasets
  • How to spot manipulated figures
  • Textual characteristics of fraudulent articles
  • Checking scientific credentials
  • Checking publisher credentials/identifying predatory publishers
  • How to raise a complaint when fraud is suspected
  • How to protect yourself from legal attacks
  • Cognitive processes that lead individuals to commit fraud
  • Institutional practices that create perverse incentives
  • The other side of the coin: "Merchants of doubt" whose goal is to discredit science

I'm sure there's much more that could be added and would be glad of suggestions. 

Now, of course, the question is what could you do with such a qualification. If my predictions are right, then individuals with such expertise will increasingly be in demand in academic institutions and publishing houses, to help ensure the integrity of work they produce and publish. I also hope that there will be growing recognition of the need for more formal structures to be set up to investigate scientific fraud and take action when it is discovered: graduates of such a course would be exactly the kind of employees needed in such an organisation.

It might be argued that this is a hopeless endeavour. In Harry Potter and the Half-Blood Prince (Rowling, 2005) Professor Snape tells his pupils:

 "The Dark Arts, are many, varied, ever-changing, and eternal. Fighting them is like fighting a many-headed monster, which, each time a neck is severed, sprouts a head even fiercer and cleverer than before. You are fighting that which is unfixed, mutating, indestructible."

This is a pretty accurate description of what is involved in tackling scientific fraud. But Snape does not therefore conclude that action is pointless. On the contrary, he says: 

"Your defences must therefore be as flexible and inventive as the arts you seek to undo."

I would argue that any university that wants to be ahead of the field in this enterprise could should flexibility and inventiveness in starting up a postgraduate course to train the next generation of fraud-busting wizards. 

Bibliography

Bishop, D. V. M. (2023). Red flags for papermills need to go beyond the level of individual articles: A case study of Hindawi special issues. https://osf.io/preprints/psyarxiv/6mbgv
Boughton, S. L., Wilkinson, J., & Bero, L. (2021). When beauty is but skin deep: Dealing with problematic studies in systematic reviews | Cochrane Library. Cochrane Database of Systematic Reviews, 5. Retrieved 4 June 2021, from https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.ED000152/full
 Byrne, J. A., & Christopher, J. (2020). Digital magic, or the dark arts of the 21st century—How can journals and peer reviewers detect manuscripts and publications from paper mills? FEBS Letters, 594(4), 583–589. https://doi.org/10.1002/1873-3468.13747
Cabanac, G., Labbé, C., & Magazinov, A. (2021). Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals (arXiv:2107.06751). arXiv. https://doi.org/10.48550/arXiv.2107.06751
Carreyrou, J. (2019). Bad Blood: Secrets and Lies in a Silicon Valley Startup. Pan Macmillan.
COPE & STM. (2022). Paper mills: Research report from COPE & STM. Committee on Publication Ethics and STM. https://doi.org/10.24318/jtbG8IHL 
Culliton, B. J. (1983). Coping with fraud: The Darsee Case. Science (New York, N.Y.), 220(4592), 31–35. https://doi.org/10.1126/science.6828878 
Grey, S., & Bolland, M. (2022, August 18). Guest Post—Who Cares About Publication Integrity? The Scholarly Kitchen. https://scholarlykitchen.sspnet.org/2022/08/18/guest-post-who-cares-about-publication-integrity/ 
Hanson, M., Gómez Barreiro, P., Crosetto, P., & Brockington, D. (2023). The strain on scientific publishing (2309; p. 33343265 Bytes). arXiv. https://arxiv.org/ftp/arxiv/papers/2309/2309.15884.pdf 
Harris, R. (2017). Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions (1st edition). Basic Books.

Judson, H. F. (2004). The Great Betrayal. Orlando.

Lévy, R. (2022, December 15). Is it somebody else’s problem to correct the scientific literature? Rapha-z-Lab. https://raphazlab.wordpress.com/2022/12/15/is-it-somebody-elses-problem-to-correct-the-scientific-literature/
 Moher, D., Bouter, L., Kleinert, S., Glasziou, P., Sham, M. H., Barbour, V., Coriat, A.-M., Foeger, N., & Dirnagl, U. (2020). The Hong Kong Principles for assessing researchers: Fostering research integrity. PLOS Biology, 18(7), e3000737. https://doi.org/10.1371/journal.pbio.3000737
 Oreskes, N., & Conway, E. M. (2010). Merchants of Doubt: How a handful of scientists obscured the truth on issues from tobacco smoke to global warming. Bloomsbury Press.
 Paterlini, M. (2023). Paolo Macchiarini: Disgraced surgeon is sentenced to 30 months in prison. BMJ, 381, p1442. https://doi.org/10.1136/bmj.p1442  
Rowling, J. K. (2005) Harry Potter and the Half-Blood Prince. Bloomsbury, London. ‎ ISBN: 9780747581086
Smith, R. (2021, July 5). Time to assume that health research is fraudulent until proven otherwise? The BMJ. https://blogs.bmj.com/bmj/2021/07/05/time-to-assume-that-health-research-is-fraudulent-until-proved-otherwise/
Stapel, D. (2016). Faking science: A true story of academic fraud.  Translated by Nicholas J. Brown. http:// nick.brown.free.fr/stapel.
Stroebe, W., Postmes, T., & Spears, R. (2012). Scientific misconduct and the myth of self-correction in science. Perspectives on Psychological Science, 7(6), 670–688. https://doi.org/10.1177/1745691612460687
 

Note: On-topic comments are welcome but are moderated to avoid spam, so there may be a delay before they appear.