Monday 1 July 2024

Whistleblowing, research misconduct, and mental health


Book Review

Carl Elliott. The Occasional Human Sacrifice: Medical Experimentation and the Price of Saying No.  W. W. Norton, 2024. ISBN: 978-1-324-06550-0


This book should be required reading for three groups of people. First, every member of university or hospital administration, and those who sit on ethics committees (Institutional Review Boards) in such institutions. Second, people interested in the psychology and sociology of moral behaviour. And finally, anyone who has experience of being a whistleblower regarding academic malpractice, and anyone who is thinking of acting in that role.

 There are many books on bioethics, and readers may be familiar with at least some of the seven cases described in The Occasional Human Sacrifice:

  • The Markingson case, where a severely disturbed man was persuaded to take part in a drug trial against the wishes of his mother, leading, as she had predicted, to his death by suicide.
  •  The Tuskegee study where black men were coerced to take part in a study of the consequences of untreated syphilis.
  • Willowbrook School, a residential institution where children with intellectual disability were deliberately infected with hepatitis.
  •  Protocol 126 at the Fred Hutchinson Cancer Center, which involved patients being given an experimental bone marrow transplant treatment that proved to be ineffective, without adequate informed consent.
  • The Radiation Effects study at Cincinnati Medical Center, where cancer patients were exposed to whole body irradiation, not as treatment, but in order to study its effects. 
  •  "The Unfortunate Experiment" in Dunedin, New Zealand, an observational study that involved monitoring rather than treating precancerous abnormalities of the cervix, without sufficient explanation to the woman concerned.
  • The case of Paolo Macchiarini, who performed experimental stem cell transplants on the trachea in humans without having done adequate animal studies to justify the approach, with dire outcomes for most patients.

The details of each of these cases makes for sobering reading: patients who trusted their doctors to look after them were experimented on without proper consent procedures, in some cases over a period of years. Most accounts of these events focus on the doctors who allowed this to happen: what motivated them, why did they ignore conventional medical ethics, and why didn't they listen to those who expressed misgivings? Elliott's primary focus, however, is on a different aspect of these cases - the whistleblowers who drew attention to the problems, their motivations, and the impact of their whistleblowing on their lives.

Elliott speaks from personal knowledge - he was the whistleblower in the Markingson case, and his account makes for uncomfortable reading. When he drew attention to the unethical practices in his department of psychiatry he was first ignored and then ostracised. He became obsessed with the case, with a major toll on his mental health. In the concluding chapter he notes:

I am the last person to advise anyone to ignore the demands of conscience, but to encourage potential whistleblowers to speak out without noting the long odds of success and the grim personal costs would be like advising them to leap blindfolded off a cliff. (p 290).

All but one of the whistleblowers featured in this book had a particularly difficult time of it because they were speaking out about ethical abuses within their own institutions. As Elliott noted, the line between a traitor and a whistleblower is a fine one, relating to the perceived motivation of the person involved. A whistleblower may feel they are acting on the purest of motives, to do the right thing, yet others will interpret it as disloyalty to the institution, or attribute their behaviour to a sense of self-importance and moral superiority. Even when, as in several of these cases, the whistleblower is eventually vindicated, they are not thanked, there is seldom restitution for victims, and the mental scars of their battles with authority stay with them.

My own forays into challenging dodgy research are nothing like as serious as the cases covered by Elliott: no lives were at stake in the instances of fraud and malpractice that I've focused on, all of which have been outside my own department. Nevertheless, I resonated strongly with what Elliott had to say about the transformation that occurs in whistleblowers when they first try to report a problem. 

Unlike most people, who are used to the darker side of human behavior, whistleblowers are genuinely shocked when they discover dishonesty, greed, or corruption. They naively assume that if they blow the whistle, others will be just as outraged as they are. They might even imagine they'll be thanked." (p. 158).


 "...if becoming a whistleblower requires a certain naivete, then surviving the experience demands a hard shell of cynicism." (p 159). 

I've definitely been on that journey, after years of observing the delays and obfuscations that routinely occur when a sleuth reports a fraudulent paper to a publisher, or when an institution manages to exonerate a researcher despite ample evidence of misconduct.

I've also been thinking a lot lately about parallels with political life. Here in the UK we've had a number of scandals where citizens were harmed, and sometimes killed, by people in power who were at best incompetent and at worst corrupt. Politicians handed lucrative contracts for personal protective equipment to their cronies who failed to deliver, leaving health professionals without protection; NHS patients were given contaminated blood products long after a problem had been identified; A horrendous fire in a tower block in West London arose after contractors ignored regulations and installed inappropriate cladding on the building - in each of these cases the stories came out, there was a public outcry, and a sense that "something must be done" - but the process gets mired in red tape and nothing actually is done, nobody gets punished, and those affected aren't compensated. This is terrible not just because of the impact on the individuals who were directly harmed, but also because it destroys people's trust in the institutions that we assume will look after us. I see the failure of institutions to deal adequately with academic misconduct as just one part of this broader picture.

So is there anything that can be done to challenge wrongdoing without crashing out over the cliff? A major point made by Elliott is that whistleblowers should not go it alone: that way madness lies. One option that he raised in a recent talk on this topic was to consult a lawyer with expertise in whistleblowing cases: they will be able to advise on the likelihood of success, and may be able to bring a complaint directly to an organisation, so the whistleblower can fly under the radar. There is also the option of going to the press, but, though Elliott describes some cases where this was effective, he notes it can be a two-edged sword, because journalists will want a good story, and it may not coincide with the whistleblower's viewpoint. In my experience, it's worth forming links with responsible journalists: institutions who have been inert for years may spring into action when confronted with the possibility of adverse publicity. Perhaps the most important point is that you need to have allies. They can restore confidence that there are other people who share one's values, who appreciate what one is doing and provide moral support. Although Elliott doesn't say this, I think the internet has made it easier to find one's tribe in this regard.

Where, perhaps, I take issue with Elliott is in the implication that many people aren't troubled by the issues that worry whistleblowers. When I give talks about reproducible research practices, I reliably get a question along the lines of "What advice can you give an early-career researcher whose supervisor is encouraging them to adopt questionable research practices?" The questioner clearly finds it troublesome but is being told that their career is at risk if they don't cut corners, and that "everyone is doing it". I reply that this is an issue that has serious consequences for mental health: doing what you know to be wrong causes internal conflict that can eat away at you, leading to a clash between your model of a good person and your current behaviour.

For this reason I advise those starting out in research to read this piece by Sarah Kendzior. Her advice is intended to help people survive under the threat of authoritarianism, but many of the points apply to academic survival in an age of cynicism:

Write down what you value; what standards you hold for yourself and for others.... Write a list of things you would never do. .... 

But most of all, never lose sight of who you are and what you value. If you find yourself doing something that feels questionable or wrong a few months or years from now, find that essay you wrote on who you are and read it. Ask if that version of yourself would have done the same thing.

 And if the answer is no? Don’t do it.



Thursday 20 June 2024

Does Elsevier's negligence pose a risk to public health?


Yesterday, Retraction Watch published a piece about a notorious 2020 article by Gautret et al that had promoted the idea that hydroxychloroquine could treat COVID-19. Despite numerous concerns, the article has not been retracted from the International Journal for Antimicrobial Agents, a journal that is co-owned by the publisher, Elsevier, and The International Society of Antimicrobial Chemotherapy (ISAC).

On June 3rd, a group of concerned scientists wrote to Elsevier asking for the paper to be retracted in light of the evidence that the paper was flawed and had serious consequences for public health. The fact that the article pops up in the Web of Science database with a trophy symbol denoting it as a 'highly cited paper' just adds to concerns that it is still being taken seriously, giving credence to those who continue to promote an ineffective drug, hydroxychloroquine, to treat COVID-19.

Elsevier's response? They are "reopening the investigation" into the paper. This is the investigation that previously decided nothing needed to be done, despite a damning report by a group of respected experts. In their reply to the letter they stated:

"We cannot currently provide further detail as to the particulars of this investigation as it remains ongoing, and is necessarily confidential while we allow time to discuss our concerns with the authors as part of due process and in line with COPE recommendations. I will however be more then glad to keep you appraised of any final decisions in due course."

As one experienced in these communications, I can confirm that this is the kind of reply you get by a publisher who is kicking the issue into the long grass in the hope that you will forget about it and go away. The wonderful phrase "it remains ongoing" typically refers to something that goes on, and on, and on, and "due course" can take years.

A pertinent question is what COPE recommendations are being followed: there are a whole range of flowcharts provided by COPE for different circumstances. The most relevant in this case would appear to be this one on "Concerns about risk in published data".

Early on in the flowchart, the decision tree asks "Based on the initial assessment, should the dataset be removed or restricted during the investigation to mitigate potential risk?" If the answer is YES, then the appropriate action is "Remove public access to the dataset while following up on the concerns."

Given that the article is still freely available on the web, we have to ask ourselves, is Elsevier being negligent here? Have they decided that there is no risk, despite the evidence that we and others have provided that public health is endangered by publicising misleading data on an ineffective drug? Or do they agree that there is risk, but deem it inconvenient for them to take action?

Sunday 26 May 2024

Are commitments to open data policies worth the paper they are written on?


As Betteridge's law of headlines states: "Any headline that ends in a question mark can be answered by the word no."  So you know where I am going with this.  


I'm a longstanding fan of open data - in fact, I first blogged about this back in 2015. So I've been gratified to see the needle shift on this, in the sense that over the past decade, in a rush to present themselves as good guys, various institutions and publishers have published policies supporting open data. The problem is that when you actually ask them to implement those policies, they back down.   


I discussed arguments for and against data-sharing in a Commentary article in 2016. I divided the issues according to whether they focused on the impact of data-sharing on researchers or on research participants. Table 1 from that article, entitled "Conflict between interests of researchers and advancement of science" is reproduced here:




1. Lack of time to curate data.

Unless adequately curated, data will over time become unusable, including by the original researcher.

2. Personal investment—reluctance to give data to freeloaders.

Reuse of data increases its value and the researcher benefits from additional citations. There is also an ethical case for maximizing use of data obtained via public funding.

3. Concerns about being scooped before the analysis is complete.

This is a common concern though there are few attested cases. A time-limited period of privileged use by the study team can be specified to avoid scooping.

4. Fear of errors being found in the data.

Culture change is needed to recognize errors are inevitable in any large dataset and should not be a reason for reputational damage. Data-sharing allows errors to be found and corrected.


I then went on to discuss two other concerns which focused on implications of data-sharing for human participants, viz:

5.  Ethical concerns about confidentiality of personal data, especially in the context of clinical research

6.  Possibility that others with a different agenda may misuse the data, e.g. perform selective analyses that misrepresent the findings.


These last two issues raise complex concerns and there's plenty to discuss on how address them, but I'll put that to one side for now, as the case I want to comment on concerns a simple dataset where there is limited scope for secondary analyses and where no human participants are involved.


My interest was piqued by comments on PubPeer about a paper entitled "Magnetic field screening in hydrogen-rich high-temperature superconductors ".  The thread on PubPeer starts with this extraordinary comment by J. E. Hirsch:


I requested the underlying data for Figs. 3a, 3e, 3b, 3f of this paper on Jan 11, 2023. This is because the published data for Figs. 3a and 3e, as well as for Figs. 3b and 3f, are nominally the same but incompatible with each other, and I would like to understand why that is. I asked the authors to explain, but they did not provide an explanation. Neither did they supply the data. The journal told me that it had received the data from the authors but will not share them with me because they are "confidential". I requested that the journal posts an Editor Note informing readers that data are unavailable to readers. The journal responded that because data were share with editors they "cannot write an editorial note on the published article stating the data is unavailable as this would be factually incorrect".


Pseudonymous commenter Orchestes quercus drew attention to the Data Availability statement in the article: "The data that support the findings of this study are available from the corresponding authors upon reasonable request".


J. E. Hirsch then added a further comment: 


The underlying data are still not available, the editor says the author deems the request "unreasonable" but it cannot divulge the reasoning behind it, nor can the journal publish an editor note that there are restrictions on data availability because the data were provided to the journal.  Springer Nature's Research Integrity Director wrote to me in September 2023 that "we recognize the right of the authors to not share the data with you, in line with the authors’ chosen data availability statement", and that "As Springer Nature considers the correspondence with the authors confidential, we cannot share with you any further details.


Now, I know nothing whatsoever about superconductors or J. E. Hirsch, but I think the editors, publisher and the authors are making themselves look very silly, and indeed suspicious, by refusing to share the data.  They can't plead patient confidentiality or ethical restrictions - it seems they are just refusing to comply because they don't want to.  


To up the ante, Orchestes quercus extracted data from the figures and did further analyses, which confirmed that J. E. Hirsch had a point - the data did not appear to be internally consistent.


Meanwhile, I had joined the PubPeer thread, pointing out


The authors and editor appear to be in breach of the policy of Nature Portfolio journals, stated here:, viz:

An inherent principle of publication is that others should be able to replicate and build upon the authors' published claims. A condition of publication in a Nature Portfolio journal is that authors are required to make materials, data, code, and associated protocols promptly available to readers without undue qualifications. Any restrictions on the availability of materials or information must be disclosed to the editors at the time of submission. Any restrictions must also be disclosed in the submitted manuscript.

After publication, readers who encounter refusal by the authors to comply with these policies should contact the chief editor of the journal. In cases where editors are unable to resolve a complaint, the journal may refer the matter to the authors' funding institution and/or publish a formal statement of correction, attached online to the publication, stating that readers have been unable to obtain necessary materials to replicate the findings.

I also noted that two of the authors are based at a Max Planck Institute. The Max Planck Gesellschaft is a signatory to the BerlinDeclaration on Open Access to Knowledge in the Sciences and Humanities.  On the website it states:

the Max Planck Society (MPG) is committed to the goal of providing free and open access to all publications and data from scholarly research (my emphasis).


Well, the redoubtable J. E. Hirsch had already thought of that, and in a subsequent PubPeer comment made public various exchanges he had had with luminaries from the Max Planck Institutes.


All I can say to the Max Planck Gesellschaft is that this is not a good look. Hirsch has noted an inconsistency in the published figures.  This has been confirmed by another reader and needs to be explained. The longer people dig in defensively, attacking the person making the request rather than just showing the raw data, the more it looks as if something fishy is going on here.


Why am I so hung up on data-sharing? The reason is simple. The more I share my own data, or use data shared by others, the more I appreciate the value of doing so. Errors are ubiquitous, even when researchers are careful, but we'll never know about them if data are locked away.


Furthermore, it is a sad reality that fraudulent papers are on the rise, and open data is one way of defending against them. It's not a perfect defence: people can invent raw data as well as summary data, but realistic data are not so easy to fake, and requiring open data would slow down the fraudsters and make them easier to catch.


Having said that, asking for data is not tantamount to accusing researchers of fraud: it should be accepted as normal scientific practice to make data available in order that others can check the reproducibility of findings. If someone treats such a request as an accusation, or deems it "unreasonable", then I'm afraid it just makes me suspicious.  


And if organisations like Springer Nature and Max Planck Gesellschaft won't back up their policies with action, then I think they should delete them from their websites. They are presenting themselves as champions of open, reproducible science, while acting as defenders of non-transparent, secret practices. As we say in the UK, fine words butter no parsnips.   


 P.S. 27th May:  A comprehensive account of the superconductivity affair has just appeared on the website For Better Science.  This suggests things are even worse than I thought.   


In addition, you can see Jorge Hirsch explain his arduous journey in attempting to access the data here.    


NOTE ON COMMENTS: Many thanks to those who have commented. Comments are moderated to prevent spam, so there is a delay before they appear, but I will accept on-topic comments in due course.

Wednesday 27 March 2024

Some thoughts on eLife's New Model: One year on


I've just been sent an email from eLife, pointing me to links to a report called "eLife's New Model: One year on" and a report by the editors "Scientific Publishing: The first year of a new era". To remind readers who may have missed it, the big change introduced by eLife in 2023 was to drop the step where an editor decides on reject or accept of a manuscript after reviewer comments are received. Instead, the author submits a preprint, and the editors then decide whether it should be reviewed. If the answer is yes, then the paper will be published, with reviewer comments. 

Given the controversy surrounding this new publishing model, it seems timely to have a retrospective look at how it's gone, and these pieces by the journal are broadly encouraging in showing that the publishing world has not fallen apart as a consequence of the changes. We are told that the proportion of submissions published has gone down slightly from 31.4% to 27.7% and the demographic characteristics of authors and reviewers are largely unchanged. The ratings of quality of submissions are similar to those from the legacy model. The most striking change has been in processing time: median time from submission to publication of the first version with reviews is 91 days, which is much faster than previously. 

As someone who has been pushing for changes to the model of scientific publishing for years (see blogsposts below), I'm generally in favour of any attempt to disrupt the conventional model. I particularly like the fact that the peer reviews are available with the published articles in eLife - I hope that will become standard for other journals in future. However, there are two things that rather rankled about the latest communication from the journal. 

First, the report describes an 'author survey' which received 325 responses, but very little detail is given as to who was surveyed, what the response rate was, and what the overall outcome was. This reads more like a marketing report than a serious scientific apprasal. Two glowing endorsements were reported from authors who had good experiences. I wondered though about authors whose work had not been selected to go forward to peer review - were they just as enthusiastic? Quite a few tables of facts and figures about the impact of the new policy were presented with the report, but if eLife really does want to present itself as embracing open and transparent policies, I think they should bite the bullet and provide more information - including fuller details of their survey methods and results, and negative as well as positive appraisals. 

Second, I continue to think there is a fatal flaw in the new model, which is that it still relies on editors to decide which papers go forward to review, using a method that will do nothing to reduce the tendency to hype and the consequent publication bias that ensues. I blogged about this a year ago, and suggested a simple solution, which is for the editors to adopt 'results-blind' review when triaging papers. This is an idea that has been around at least since 1976 (Mahoney, 1976) which has had a resurgence in popularity in recent years, with growing awareness of the dangers of publication bias (Locasio, 2017). The idea is that editorial decisions should be made based on whether the authors had identified an interesting question and whether their methods were adequate to give a definitive answer to that question. The problem with the current system is that people get swayed by exciting results, and will typically overlook weak methods when there is a dramatic finding. If you don't know the results, then you are forced to focus on the methods. The eLife report states:

 "It is important to note that we don’t ascribe value to the decision to review. Our aim is to produce high-quality reviews that will be of significant value but we are not able to review everything that is submitted." 

That is hard to believe: if you really were just ignoring quality considerations, then you should decide on which papers to review by lottery. I think this claim is not only disingenuous but also wrong-headed. If you have a limited resource - reviewer capacity - then you should be focusing it on the highest quality work. But that judgement should be made on the basis of research question and design, and not on results. 


Locascio, J. J. (2017). Results blind science publishing. Basic and Applied Social Psychology, 39(5), 239–246. 

Mahoney, M. J. (1976). Scientist as Subject: The Psychological Imperative. Ballinger Publishing Company. 

Previous blogposts

Academic publishing: why isn't psychology like physics? 

Time for academics to withdraw free labour.

High impact journals: where newsworthiness trumps methodology

Will traditional science journals disappear?

Publishing replication failures

Sunday 24 March 2024

Just make it stop! When will we say that further research isn't needed?


I have a lifelong interest in laterality, which is a passion that few people share. Accordingly, I am grateful to René Westerhausen who runs the Oslo Virtual Laterality Colloquium, with monthly presentations on topics as diverse as chiral variation in snails and laterality of gesture production. 

On Friday we had a great presentation from Lottie Anstee who told us about her Masters project on handedness and musicality. There have been various studies on this topic over the years, some claiming that left-handers have superior musical skills, but samples have been small and results have been mixed. Lottie described a study with an impressive sample size (nearly 3000 children aged 10-18 years) whose musical abilities were evaluated on a detailed music assessment battery that included self-report and perceptual evaluations. The result was convincingly null, with no handedness effect on musicality. 

What happened next was what always happens in my experience when someone reports a null result. The audience made helpful suggestions for reasons why the result had not been positive and suggested modifications of the sampling, measures or analysis that might be worth trying. The measure of handedness was, as Lottie was the first to admit, very simple - perhaps a more nuanced measure would reveal an association? Should the focus be on skilled musicians rather than schoolchildren? Maybe it would be worth looking at nonlinear rather than linear associations? And even though the music assessment was pretty comprehensive, maybe it missed some key factor - amount of music instruction, or experience of specific instruments. 

After a bit of to and fro, I asked the question that always bothers me. What evidence would we need to convince us that there is really no association between musicality and handedness? The earliest study that Lottie reviewed was from 1922, so we've had over 100 years to study this topic. Shouldn't there be some kind of stop rule? This led to an interesting discussion about the impossibility of proving a negative and whether we should be using Bayes Factors, and what would be the smallest effect size of interest.  

My own view is that further investigation of this association would prove fruitless. In part, this is because I think the old literature (and to some extent the current literature!) on factors associated with handedness is at particular risk of bias, so even the messy results from a meta-analysis are likely to be over-optimistic. More than 30 years ago, I pointed out that laterality research is particularly susceptible to what we now call p-hacking - post hoc selection of cut-offs and criteria for forming subgroups, which dramatically increase the chances of finding something significant. In addition, I noted that measurement of handedness by questionnaire is simple enough to be included in a study as a "bonus factor", just in case something interesting emerges. This increases the likelihood that the literature will be affected by publication bias - the handedness data will be reported if a significant result is obtained, but otherwise can be disregarded at little cost. So I suspect that most of the exciting ideas about associations between handedness and cognitive or personality traits are built on shaky foundations, and would not replicate if tested in well-powered, preregistered studies.  But somehow, the idea that there is some kind of association remains alive, even if we have a well-designed study that gives a null result.  

Laterality is not the only area where there is no apparent stop rule. I've complained of similar trends in studies of association between genetic variants and psychological traits, for instance, where instead of abandoning an idea after a null study, researchers slightly change the methods and try again. In 2019, Lisa Feldman Barrett wrote amusingly about zombie ideas in psychology, noting that some theories are so attractive that they seem impossible to kill. I hope that as preregistration becomes more normative, we may see more null results getting published, and learn to appreciate their value. But I wonder just what it takes to get people to conclude that a research seam has been mined to the point of exhaustion. 

Friday 9 February 2024

The world of Poor Things at MDPI journals

At the weekend, the Observer ran a piece by Robin McKie entitled "‘The situation has become appalling’: fake scientific papers push research credibility to crisis point". I was one of those interviewed for the article, describing my concerns about a flood of dodgy papers that was polluting the scientific literature.

Two days later I received an email from the editorial office of MDPI publishers with the header "[Children] (IF: 2.4, ISSN 2227-9067): Good Paper Sharing on the Topic of" (sic) that began:

Greetings from the Children Editorial Office!

We recently collected 10 highly cited papers in our journal related to Childhood Autism. And we sincerely invite you to visit and read these papers, because you are an excellent expert in this field of study.

Who could resist such a flattering invitation? MDPI is one of those publishers that appears to be encouraging publication of low quality work, with a massive growth in special issues where papers are published with remarkably rapid turnaround times. Only last week it was revealed that the journal is affected by fake peer review that appears to be generated by AI. So I was curious to take a look.

The first article, by Frolli et al (2022a) was weird. It reported a comparison of two types of intervention designed to improve emotion recognition in children with autism, one of which used virtual reality. The first red flag was the sample size: two groups each of 30 children, all originally from the city of Caserta. I checked Wikipedia, which told me the population of Caserta was around 76,000 in 2017. Recruiting participants for intervention studies is typically slow and laborious and this is a remarkable sample size to recruit from such a small region. But credibility is then stretched to breaking point on hearing that the selection criteria required that the children were all aged between 9 and 10 years and had IQs of 97 or above. No researcher in their right mind would impose unnecessary constraints on recruitment, and both the age and IQ criteria are far tighter than would usually be adopted. I wondered whether there might be a typo in this account, but we then hear that the IQ range of the sample is indeed remarkably narrow: 

"The first experimental group (Gr1) was composed of 30 individuals with a mean age of 9.3 (SD 0.63) and a mean IQ of 103.00 (SD 1.70). ...... The second experimental group (Gr2) was composed of 30 individuals with a mean age of 9.4 (SD 0.49) and mean IQ of 103.13 (SD 2.04)...."

Most samples for studies using Wechsler IQ scales have SD of at least 8, even if cutoffs are applied as selection criteria, so this is unbelievably low.

This dubious paper prompted me to look at others by the first author. It was rather like pulling a thread on a hole in a sweater - things started to unravel fast. A paper published by Frolli et al (2023a) in the MDPI journal Behavioral Sciences claimed to have studied eighty 18-year-olds recruited from four different high schools. The selection criteria were again unbelievably stringent: IQ assessed on the WAIS-IV fell between 95-105 "to ensure that participants fell within the average range of intellectual functioning, minimizing the impact of extreme cognitive variations on our analyses". The lower IQ range selected here corresponds to z-score of -0.33 or 37th percentile. If the population of students covered the full range of IQ, then only around 25% would meet the criterion (between 37th and 63rd centile), so to obtain a sample of 80 it would be necessary to test over 300 potential participants. Furthermore, there are IQ screening tests that can be used in this circumstance that are relatively quick to administer, but the WAIS-IV is not one of them. We are told all participants were given the full test, which requires individual administration by a qualified psychologist and takes around one hour to complete. So who did all this testing, and where? The article states: "The data were collected and analyzed at the FINDS Neuropsychiatry Outpatient Clinic by licensed psychologists in collaboration with the University of International Studies of Rome (UNINT)." So we are supposed to believe that hundreds of 18-year-olds trekked to a neuropsychiatry outpatient clinic for a full IQ screening which most of them would not have passed. I cannot imagine a less efficient way of conducting such a study. I could not find any mention of compensation for participants, which is perhaps unsurprising as the research received no external funding. All of this is described as happening remarkably fast, with ethics approval in January 2023, and submission of the article in October 2023.

Another paper in Children in 2023 focused on ADHD, and again reported recruiting two groups of 30 children for an intervention that lasted 5 months (Frolli et al., 2023b). The narrow IQ selection criteria were again used, with WISC-IV IQs in the range 95-105, and the mean IQs were 96.48 (SD =1.09) and 98.44 (SD = 1.12) for groups 1 and 2 respectively. Again, the research received no external funding. The report of ethics approval is scanty "The study was conducted in accordance with the Declaration of Helsinki. The study was approved by the Ethics Committee and the Academic Senate of the University of International Studies of Rome."

The same first author published a paper on the impact of COVID-19 on cognitive development and executive functioning in adolescents in 2021 (Frolli et al, 2021). I have not gone over it in detail, but a quick scan revealed some very odd statistical reporting. There were numerous F-ratios, but they were all negative, which is impossible, as F is a ratio between two positive numbers. Furthermore, the reported p-values and degrees of freedom didn't always correspond to the F-ratio, even if the sign was ignored.

At this point I was running out of steam, but a quick look at Frolli et al (2022a) on Executive Functions and Foreign Language Learning suggested yet more problems, with the sentence "Significance at the level of 5% (α < 0.001) has been accepted" featuring at least twice. It is hard to believe that a human being wrote this sentence, or that any human author, editor or reviewer read it without comment.

If anyone is interested in pulling at other related threads, I suspect it would be of interest to look at articles accepted for a Special Issue of the MDPI journal Disabilities co-edited by Frolli.

In his brilliant film Poor Things, Yorgos Lanthimos distorts familiar objects and places just enough to be disturbing. Lisbon looks like what I imagine Lisbon would be in the Victorian age, except that the colours are unusually vivid, there are strange flying cars in the sky, and nobody seems concerned at the central character wandering around only partially clothed (see, e.g., this review).  The combined impression is that MDPI publishes papers from that universe, where everything looks superficially like genuine science but with jarring features that tell you something is amiss. The difference is that Poor Things has a happy ending.


Frolli, A.; Ricci, M.C.; Di Carmine, F.; Lombardi, A.; Bosco, A.; Saviano, E.; Franzese, L. The Impact of COVID-19 on Cognitive Development and Executive Functioning in Adolescents: A First Exploratory Investigation. Brain Sci. 2021, 11, 1222.

Frolli, A.; Savarese, G.; Di Carmine, F.; Bosco, A.; Saviano, E.; Rega, A.; Carotenuto, M.; Ricci, M.C. Children on the Autism Spectrum and the Use of Virtual Reality for Supporting Social Skills. Children 2022a, 9, 181.

Frolli, A.; Cerciello, F.; Esposito, C.; Ciotola, S.; De Candia, G.; Ricci, M.C.; Russo, M.G. Executive Functions and Foreign Language Learning. Pediatr. Rep. 2022b, 14, 450-456.

Frolli, A.; Cerciello, F.; Ciotola, S.; Ricci, M.C.; Esposito, C.; Sica, L.S. Narrative Approach and Mentalization. Behav. Sci. 2023a, 13, 994.

Frolli, A.; Cerciello, F.; Esposito, C.; Ricci, M.C.; Laccone, R.P.; Bisogni, F. Universal Design for Learning for Children with ADHD. Children 2023b, 10, 1350.

Friday 2 February 2024

An (intellectually?) enriching opportunity for affiliation

Guest Post by Nick Wise 


A couple of months ago a professor received the following email, which they forwarded to me.


"Dear esteemed colleagues,

We are delighted to extend an invitation to apply for our prestigious remote research fellowships at the University of Religions and Denominations (URD). These fellowships offer substantial financial support to researchers with papers currently in press, accepted or under review by Scopus-indexed journals. We welcome scholars from diverse academic disciplines to seize this intellectually enriching opportunity.

Fellowship Details:
Fellowship Type: Remote Short-term Research Fellowship.
Research Focus: Diverse fields, spanning humanities, social sciences, interdisciplinary studies, and more.
Research Output: Publication of research articles in Scopus-indexed journals.
Affiliation: Encouragement for researchers to acknowledge URD as their additional affiliation in published articles.
Remuneration: Project-based compensation for each research article.
Payment Range: Up to $1000 USD per article (based on SJR journal ranking).
Eligibility: Papers in press, accepted, or under review by Scopus-indexed journals.

Preference: Priority for indexing before December 30, 2023.

Application Process:   

To express your interest in securing a fellowship, kindly submit your curriculum vitae to  Ahmad Moghri at When emailing your application, please use the subject line: "Research Fellowship, FULL NAME."

Upon Selection:
Successful applicants will receive formal invitations to join our esteemed fellowship program. Invitation letters and collaboration contracts will be dispatched within a maximum of 5 days.

We firmly believe that this fellowship program provides an invaluable platform for scholars to make substantial contributions to their fields while collaborating with the distinguished University of Religions and Denominations. We encourage all eligible individuals to seize this exceptional opportunity.

For inquiries or further information, please do not hesitate to contact

Warmest Regards,”

Why would the institution pay researchers to say that they are affiliated with them? It could be that funding for the university is related to the number of papers published in indexed journals. More articles associated with the university can also improve their placing in national or international university rankings, which could lead directly to more funding, or to more students wanting to attend and bringing in more money.

The University of Religions and Denominations is a private Iranian university specialising, as the name suggests, in the study of different religions and movements. Until recently the institution had very few published papers associated with it according to Dimensions and their subject matter was all related to religion. However, last year there was a substantial increase to 103 published papers, and so far this year there are already 35. This suggests that some academics have taken them up on the offer in the advert to include URD as an affiliation.

Surbhi Bhatia Khan is a lecturer in data science at the University of Salford in the UK since March 2023 and a top 2% scientist in the world according to Stanford University’s rankings. She published 29 research articles last year according to Dimensions, an impressive output, in which she was primarily affiliated to the University of Salford. In addition though, 5 of those submitted in the 2nd half of last year had an additional affiliation at the Department of Engineering and Environment at URD, which is not listed as one of the departments on the university website. Additionally, 19 of the 29 state that she’s affiliated to the Lebanese American University in Beirut, which she was not affiliated with before 2023. She is yet to mention her role at either of these additional affiliations on her LinkedIn profile.

Looking at the Lebanese American University, another private university, its publication numbers have shot up from 201 in 2015 to 503 in 2021 and 2,842 in 2023, according to Dimensions. So far in 2024 they have published 525, on track for over 6,000 publications for the year. By contrast, according to the university website, the faculty consisted of 547 full-time staff members in 2021 but had shrunk to 423 in 2023.  It is hard to imagine how such growth in publication numbers could occur without a similar growth in the faculty, let alone with a reduction.

How many other institutions are seeing incredible increases in publication numbers? Last year we saw gaming of the system on a grand scale by various Saudi Arabian universities, but how many offers like the one above are going around, whether by email or sent through Whatsapp groups or similar?

The Committee On Publication Ethics held a forum on claiming institutional affiliations in December 2023, in recognition of the fact that guidance for what merits affiliation to an institution is lacking and there are no accepted standards for how many affiliations an author should give. It looks like such guidance can’t come soon enough.

Nick Wise is a researcher at the University of Cambridge, UK.

Note: Comments are moderated to prevent spam and abuse, so please be patient if you post a comment and it does not appear immediately

P.S. 3rd Feb 2024

Someone on social media queried the "top 2% rating" for Khan. Nick tells me this is based on an Elsevier ranking for 2022:

Tuesday 5 December 2023

Low-level lasers. Part 2. Erchonia and the universal panacea



In my last blogpost, I looked at a study that claimed continuing improvements of symptoms of autism after eight 5-minute sessions where a low-level laser was pointed at the head.  The data were so extreme that I became interested in the company, Erchonia, who sponsored the study and in Regulatory Insight, Inc, whose statistician failed to notice anything odd.  In exploring Erchonia's research corpus, I found that they have investigated the use of their low-laser products for a remarkable range of conditions. A search of with the keyword Erchonia produced 47 records, describing studies of pain (chronic back pain, post-surgical pain, and foot pain), body contouring (circumference reduction, cellulite treatment), sensorineural hearing loss, Alzheimer's disease, hair loss, acne and toenail fungus. After excluding the trials on autism described in my previous post, fourteen of the records described randomised controlled trials in which an active laser was compared with a placebo device that looked the same, with both patient and researcher being kept in the dark about which device was which until the data were analysed. As with the autism study, the research designs for these RCTs specified on looked strong, with statistician Elvira Cawthon from Regulatory Insight involved in data analysis.

As shown in Figure 1, where results are reported for RCTs, they have been spectacular in virtually all cases. The raw data are mostly not available, and in general the plotted data look less extreme than in the autism trial covered in last week's post, but nonetheless, the pattern is a consistent one, where over half the active group meet the cutoff for improvement, whereas less than half (typically 25% or less) of the placebo group do so. 

FIGURE 1: Proportions in active treated group vs placebo group meeting preregistered criterion for improvement (Error bars show SE)*

I looked for results from mainstream science against which to benchmark the Erchonia findings.  I found a big review of behavioural and pharmaceutical interventions for obesity by the US Agency for Healthcare Research and Quality (LeBlanc et al, 2018). Figures 7 and 13 show results for binary outcomes - relative risk of losing 5% or more of body weight over a 12 month period; i.e. the proportion of treated individuals who met this criterion divided by the proportion of controls. In 38 trials of behavioural interventions, the mean RR was 1.94 [95% CI, 1.70 to 2.22]. For 31 pharmaeutical interventions, the effect varied with the specific medication, with RR ranging from 1.18 to 3.86. Only two pharmaceutical comparisons had RR in excess of 3.0. By contrast, for five trials of body contouring or cellulite reduction from Erchonia, the RRs ranged from 3.6 to 18.0.  Now, it is important to note that this is not comparing like with like: the people in the Erchonia trials were typically not clinically obese: they were mostly women seeking cosmetic improvements to their appearance.  So you could, and I am sure many would, argue it's an unfair comparison. If anyone knows of another literature that might provide a better benchmark, please let me know. The point is that the effect sizes reported by Erchonia are enormous relative to the kinds of effects typically seen with other treatments focused on weight reduction.

If we look more generally at the other results obtained with low-level lasers, we can compare them to an overview of effectiveness of common medications (Leucht et al, 2015). These authors presented results from a huge review of different therapies, with effect sizes represented as standardized mean differences (SMD - familiar to psychologists as Cohen's d). I converted Erchonia results into this metric*, and found that across all the studies of pain relief shown in Figure 1, the average SMD was 1.30, with a range from 0.87 to 1.77. This contrasts with Leucht et al's estimated effect size of 1.06 for oxycodone plus paracetamol, and 0.83 for Sumatriptan for migraine.  So if we are to believe the results, they indicate that the effect of Erchonia low-level lasers is as good or better than the most effective pharmaceutical medications that we have for pain relief or weight loss. I'm afraid I remain highly sceptical.

I would not have dreamed of looking at Erchonia's track record if it were not for their impossibly good results in the Leisman et al autism trial that I discussed in the previous blogpost.  When I looked in more detail, I was reminded of the kinds of claims made for alternative treatments for children's learning difficulties, where parents are drawn in with slick websites promising scientifically proven interventions, and glowing testimonials from satisfied customers. Back in 2012 I blogged about how to evaluate "neuroscientific" interventions for dyslexia.  Most of the points I made there apply to the world of "photomodulation" therapies, including the need to be wary when a provider claims that a single method is effective for a whole host of different conditions.  

Erchonia products are sold worldwide and seem popular with alternative health practitioners. For instance, in Stockport, Manchester, you can attend a chiropractic clinic where Zerona laser treatment will remove "stubborn body fat". In London there is a podiatry centre that reassures you: "There are numerous papers which show that cold laser affects the activity of cells and chemicals within the cell. It has been shown that cold laser can encourage the formation of stem cells which are key building blocks in tissue reparation. It also affects chemicals such as cytochrome c and causes a cascade of reactions which stimulates the healing. There is much research to show that cold laser affects healing and there are now several very good class 1 studies to show that laser can be effective." But when I looked for details of these "very good class 1 studies" they were nowhere to be found. In particular, it was hard to find research by scientists without vested interests in the technology.  

Of all the RCTs that I found, there were just two that were conducted at reputable universities. One of them, on hearing loss (NCT01820416) was conducted at the University of Iowa, but terminated prematurely because intermediate analysis showed no clinically or statistically significant effects (Goodman et al., 2013).  This contrasts sharply with NCT00787189, which had the dramatic results reported in Figure 1 (not, as far as I know, published outside of The other university-based study was the autism study based in Boston described in my previous post: again, with unpublished, unimpressive results posted on

This suggests it is important when evaluating novel therapies to have results from studies that are independent of those promoting the therapy. But, sadly, this is easier to recommend than to achieve. Running a trial takes a lot of time and effort: why would anyone do this if they thought it likely that the intervention would not work and the postulated mechanism of action was unproven? There would be a strong risk that you'd end up putting in effort that would end in a null result, which would be hard to publish. And you'd be unlikely to convince those who believed in the therapy - they would no doubt say you had the wrong wavelength of light, or insufficient duration of therapy, and so on.  

I suspect the response by those who believe in the power of low-level lasers will be that I am demonstrating prejudice, in my reluctance to accept the evidence that they provide of dramatic benefits. But, quite simply, if low-level laser treatment was so remarkably effective in melting fat and decreasing pain, surely it would have quickly been publicised through word of mouth from satisfied customers. Many of us are willing to subject our bodies to all kinds of punishments in a quest to be thin and/or pain-free. If this could be done simply and efficiently without the need for drugs, wouldn't this method have taken over the world?

*Summary files (Erchonia_proportions4.csv) and script (Erchonia_proportions_for_blog.R) are on Github, here.

Saturday 25 November 2023

Low-level lasers. Part 1. Shining a light on an unconventional treatment for autism


'Light enters, then a miracle happens, and good things come out!' (Quirk & Whelan, 2011*)

I'm occasionally asked to investigate weird interventions for children's neurodevelopmental conditions, and recently I've found myself immersed in the world of low-level laser treatments. The material I've dug up is not new - it's been around for some years, but has not been on my radar until now. 

A starting point is this 2018 press statement by Erchonia, a firm that makes low-level laser devices for quasi-medical interventions. 

They had tested a device that was supposed to reduce irritability in autistic children by applying low-level laser light to the temporal and posterior regions of the head (see Figure 1) for 5 minute sessions twice a week for 4 weeks.

Figure 1: sites of stimulation by low-level laser

 The study, which was reported here, was carefully designed as a randomized controlled trial. Half the children received a placebo intervention. Placebo and active laser devices were designed to look identical and both emitted light, and neither the child nor the person administering the treatment knew whether the active or placebo light was being used.

According to Erchonia “The results are so strong, nobody can argue them.” (sic). Alas, their confidence turned out to be misplaced.

The rationale given by Leisman et al (with my annotations in yellow in square brackets) is as follows: "LLLT promotes cell and neuronal repair (Dawood and Salman 2013) [This article is about wound healing, not neurons] and brain network rearrangement (Erlicher et al. 2002) [This is a study of rat cells in a dish] in many neurologic disorders identified with lesions in the hubs of default mode networks (Buckner et al. 2008)[This paper does not mention lasers]. LLLT facilitates a fast-track wound-healing (Dawood and Salman 2013) as mitochondria respond to light in the red and near-infrared spectrum (Quirk and Whelan 2011*)[review of near-infrared irradiation photobiomodulation that notes inadequate knowledge of mechanisma - see cartoon]. On the other hand, Erlicher et al. (2002) have demonstrated that weak light directs the leading edge of growth cones of a nerve [cells in a dish]. Therefore, when a light beam is positioned in front of a nerve’s leading edge, the neuron will move in the direction of the light and grow in length (Black et al. 2013 [rat cells in a dish]; Quirk and Whelan 2011). Nerve cells appear to thrive and grow in the presence of low-energy light, and we think that the effect seen here is associated with the rearrangement of connectivity."

I started out looking at the registration of the trial on This included a very thorough document that detailed a protocol and analysis plan, but there were some puzzling inconsistencies; I documented them here on PubPeer,  and subsequently a much more detailed critique was posted there by Florian Naudet and André Gillibert. Among other things, there was confusion about where the study was done. The registration document said it was done in Nazareth, Israel, which is where the first author, Gerry Leisman was based. But it also said that the PI was Calixto Machado, who is based in Havana, Cuba.

Elvira Cawthon, from Regulatory Insight, Inc, Tennessee was mentioned on the protocol as clinical consultant and study monitor. The role of the study monitor is specified as follows: 

"The study Monitor will assure that the investigator is executing the protocol as outlined and intended. This includes insuring that a signed informed consent form has been attained from each subject’s caregiver prior to commencing the protocol, that the study procedure protocol is administered as specified, and that all study evaluations and measurements are taken using the specified methods and correctly and fully recorded on the appropriate clinical case report forms."

This does not seem ideal, given that the study monitor was in Tennessee, and the study was conducted in either Nazareth or Havana. Accordingly, I contacted Ms Cawthon, who replied: 

"I can confirm that I performed statistical analysis on data from the clinical study you reference that was received from paper CRFs from Dr. Machado following completion of the trial. I was not directly involved in the recruitment, treatment, or outcomes assessment of the subjects whose data was recorded on those CRFs. I have not reviewed any of the articles you referenced below so I cannot attest to whether the data included was based on the analyses that I performed or not or comment on any of the discrepancies without further evaluation at this time."

I had copied Drs Leisman and Machado into my query, and Dr Leisman also replied. He stated:

"I am the senior author of the paper pertaining to a trial of low-level laser therapy in autism spectrum disorder.... I take full responsibility for the publication indicated above and vouch for having personally supervised the implementation of the project whose results were published under the following citation:

Leisman, G. Machado, C., Machado, Y, Chinchilla-Acosta, M. Effects of Low-Level Laser Therapy in Autism Spectrum Disorder. Advances in Experimental Medicine and Biology 2018:1116:111-130. DOI:10.1007/5584_2018_234. The publication is referenced in PubMed as: PMID: 29956199.

I hold a dual appointment at the University of Haifa and at the University of the Medical Sciences of Havana with the latter being "Professor Invitado" by the Ministry of Health of the Republic of Cuba. Ms. Elvira Walls served as the statistical consultant on this project."

However, Dr Leisman denied any knowledge of subsequent publications of follow-up data by Dr Machado. I asked if I could see the data from the Leisman et al study, and he provided a link to a data file on ResearchGate, the details of which I have put on PubPeer.

Alas, the data were amazing, but not in a good way. The main data came from five subscales of the Aberrant Behavior Checklist (ABC)**, which can be combined into a Global score. (There were a handful of typos in the dataset for the Global score, which I have corrected in the following analysis). For the placebo group, 15 of 19 children obtained exactly the same global score on all 4 sessions. Note that there is no restriction of range for this scale: reported scores range from 9 to 154. This pattern was also seen in the five individual subscales. You might think that is to be expected if the placebo intervention is ineffective, but that's not the case. Questionnaire measures such as that used here are never totally stable. In part this is because children's behaviour fluctuates. But even if the behaviour is constant, you expect to see some variability in responses, depending on how the rater interprets the scale of measurement. Furthermore, when study participants are selected because they have extreme scores on a measure, the tendency is for scores to improve on later testing - a phenomenon known as regression to the mean, Such unchanging scores are out of line with anything I have ever come across in the intervention literature. If we turn to the treated group, we see that 20 of 21 children showed a progressive decline in global scores (i.e. improvement), with each measurement improving from the previous one over 4 sessions. This again is just not credible because we'd expect some fluctuation in children's behaviour as well as variable ratings due to error of measurement. These results were judged to be abnormal in a further commentary by Gillibert and Naudet on PubPeer. They also noted that the statistical distribution of scores was highly improbable, with far more even than odd numbers.

Although Dr Machado has been copied into my correspondence, he has not responded to queries. Remember, he was PI for the study in Cuba, and he is first author on a follow-up study from which Dr Leisman dissociated himself. Indeed, I subsequently found that there were no fewer than three follow-up reports, all appearing in a strange journal whose DOIs did not appear to be genuine: 

Machado, C., Machado, Y., Chinchilla, M., & Machado, Yazmina. (2019a). Follow-up assessment of autistic children 6 months after finishing low lever (sic) laser therapy. Internet Journal of Neurology, 21(1). (available from

Machado, C., Machado, Y., Chinchilla, M., & Machado, Yazmina. (2019b). Twelve months follow-up comparison between autistic children vs. Initial placebo (treated) groups. Internet Journal of Neurology, 21(2). (available from

Machado, C., Machado, Y., Chinchilla, M., & Machado, Yazmina. (2020). Follow-up assessment of autistic children 12 months after finishing low lever (sic) laser therapy. Internet Journal of Neurology, 21(2). (available from available from

The 2019a paper starts by talking of a study of anatomic and functional brain connectivity in 21 children, but then segues to an extended follow-up (6 months) of the 21 treated and 19 placebo children from the Leisman et al study. The Leisman et al study is mentioned but not adequately referenced. Remarkably, all the original participants participated in the follow-up. The same trend as before continued: the placebo group stagnated, whereas the treated group continue to improve up to 6 months later, even though they received no further active treatment after the initial 4 week period. The 2020 Abstract reported a further follow-up to 12 months. The huge group difference was sustained (see Figure 2). Three of the treated group were now reported as scoring in the normal range on a measure of clinical impairment. 

Figure 2. Chart 1 from Machado et al 2020

In the 2019b paper, it is reported that, after the stunning success of the initial phase of the study, the placebo group were offered the intervention, and all took part, whereupon they proceeded to make an almost identical amount of remarkable progress on all five subscales, as well as the global scale (see Figure 3). We might expect the 'baseline' scores of the cross-over group to correspond to the scores reported at the final follow-up (as placebo group prior to cross-over) but they don't. 

Figure 3: Chart 2 of Machado et al 2019b

I checked for other Erchonia studies on Another study, virtually identical except for the age range, was registered in 2020 with Dr Leon Morales-Quezada of Spaulding Rehabilitation Hospital, Boston as Principal Investigator.  Comments in the documents suggest this was conducted after Erchonia failed to get the desired FDA approval. Although I have not found a published report of this second trial, I found a recruitment advertisement, which confusingly cites the NCT registration number of the 2013 study. Some summary results are included on, and they are strikingly different from the Leisman et al trial, with no indication of any meaningful difference between active and placebo groups in the final outcome measure, and both groups showing some improvement. I have requested fuller data from Elvira Cawthon (listed as results point of contact) with cc. to Dr Morales-Quezada and will update this post if I hear back.

It would appear that at one level this is a positive story, because it shows the regulatory system working. We do not know why FDA rejected Erchonia's request for 510k Market Clearance, but the fact that they did so might indicate that they were unimpressed by the data provided by Leisman and Machado. The fact that Machado et al reported their three follow-up studies in what appears to be an unregistered journal suggests they had difficulty persuading regular journals that the findings were legitimate. If eight 5-minute sessions with a low-level laser pointed at the head really could dramatically improve the function of children with autism 12 months later, one would imagine that Nature, Cell and Science would be scrambling to publish the articles. On the other hand, any device that has the potential to stimulate neuronal growth might also ring alarm bells in terms of potential for harm.

Use of low-level lasers to treat autism is only part of the story. Questions remain about the role of Regulatory Insight, Inc., whose statistician apparently failed to notice anything strange about the data from the first autism study. In another post, I plan to look at cases where the same organisation was involved in monitoring and analysing trials of Erchonia laser devices for other conditions such as cellulite, pain, and hearing loss.


* Quirk, B. J., & Whelan, H. T. (2011). Near-infrared irradiation photobiomodulation: The need for basic science. Photomedicine and Laser Surgery, 29(3), 143–144. This article states "clinical uses of NIR-PBM have been studied in such diverse areas as wound healing, oral mucositis, and retinal toxicity. In addition, NIR-PBM is being considered for study in connection with areas such as aging and neural degenerative diseases (Parkinson's disease in particular). One thing that is missing in all of these pre-clinical and clinical studies is a proper investigation into the basic science of the NIR-PBM phenomenon. Although there is much discussion of the uses of NIR, there is very little on how it actually works. As far as explaining what really happens, we are basically left to resort to saying 'light enters, then a miracle happens, and good things come out!' Clearly, this is insufficient, if for no other reason than our own intellectual curiosity." 

**Aman, M. G., Singh, N. N., Stewart, A. W., & Field, C. J. (1985). The aberrant behavior checklist: A behavior rating scale for the assessment of treatment effects. American Journal of Mental Deficiency, 89(5), 485–491. N. B. this is different from the Autism Behavior Checklist which is a commonly used autism assessment.