Collapse of scientific standards at MDPI journals: a case study

 

"Here's one for you", said my husband, as he browsed the online Daily Telegraph*: "Severe autism can be reversed, groundbreaking study suggests".  My heart did not lift up at this news, which was also covered in the Daily Mail; it is a near certainty that any study making such claims is flawed. But I found it hard to believe just how flawed it turned out to be.  

 

The article, by D'Adamo et al.,  was published in a special issue of an MDPI journal, the Journal of Personalized Medicine.  The special issue "A Personalized Medicine Approach to the Diagnosis and Management of Autism Spectrum Disorder: Beyond Genetic Syndromes" appears largely to be a vehicle for papers by the guest editor Richard E. Frye, who co-authored 3/4 editorials, 3/9 articles and 1/1 review in this collection.  He was editor but not an author on the paper by D'Adamo et al, which is categorised as a "case report".

 

Essentially, this is a description of development of a pair of non-identical twin girls who were diagnosed with autism at 20 months of age, along with various gastric and motor conditions, and were subsequently subjected to a remarkable list of interventions, viz:

·      Parents worked with an autism parent coach who informed them about the link between "total allostatic load" and developmental disorders, a concept promoted by the Documenting Hope Project that one of the authors is affiliated with

·      Parents accessed resources, including free webinars, through Epidemic Answers

·      Parents took part in a parent forum called "Healing Together"

·      Applied Behavior Analysis (ABA) from 22 mo to 33 mo

·      Speech Therapy starting at 24 mo

·      Rigorous diet and nutrition intervention eliminating sources of glutamate, following Reduced Excitatory Inflammatory Diet

·      A strict gluten-free, casein-free diet that was low in sugar and additives

·      Dietary supplements, including omega-3 fatty acids, multivitamins, vitamin D, carnitine, 5-methyltetrahydrofolate and "bio-individualized homeopathic remedies"

·      Family consulted a naturopathic doctor who used IntellxxDNA genomics tool to recommend diet rich in tryptophan, vitamins B12, B6 and folate, betaine and choline, lion's mane mushroom and resolvins, as well as some dietary variants specific to each twin.

·      Neuro-sensory motor reflex integration given by an occupational therapist

·      Environmental evaluation of the home for air quality, mould and moisture, culminating in a visit by a Building Biology Environmental Consultant who identified possible water damage.

·      Cranial osteopathy, as recommended by a developmental optometrist.

 

When first diagnosed, the girls were given an assessment called the Autism Treatment Evaluation Checklist, which was repeated after 18 months. Scores declined (i.e. improved) for both girls, with one twin improving substantially, and the other less so.

 

I haven't been able to find any norms for the ATEC, but there's a bit of data on trajectories in a study by Mahapatra et al (2018) and this shows that scores normally improve between the age of 2 and 3 years.  We may also note that the ATEC is completed by a parent, and so may be influenced by parental expectations of improvement.

 

The authors conclude the study demonstrates "the clear environmental and lifestyle influences on ASD" and that there is "comparatively greater impact of these types of factors than genetics".  Neither conclusion is warranted. We can't know whether these changes would have occurred in the absence of the numerous interventions that the children were given.  If the interventions did have an effect, it would be impossible to tell which were the key ingredients, because all were given together.

 

The authors grudgingly state "...while effective at reversing ASD diagnoses, the comprehensive approach that was employed in this case may not yet be widely generalizable".  But they didn't show their approach was effective - at least not by any conventional standards of evidence-based intervention. 

 

Another limitation they noted was that this was an expensive regimen that might be out of reach for parents with limited resources.

 

Should the journalists who gave this study coverage be criticised? I think the answer is yes: science journalists can't be expected to be experts in all areas of science, but they should be aware of some basic principles, such as the need for adequate sample sizes and control groups in intervention studies, and the need to be careful in assuming causality. They can, if uncertain, ask the Science Media Centre to put them in touch with knowledgeable experts who could advise. This is particularly important for a sensitive subject matter such as autism, where publicity for unfounded interventions that portray autism as a "disease" requiring "healing" has potential to do harm.  At minimum, they might have noted the ethics statement that with only two children involved, "this is not considered a systematic investigation designed to contribute to generalizable knowledge."  So how on earth did this come to be emblazoned across national newspapers as a "groundbreaking study"?

 

Having said that, though, my strongest criticism is for the MDPI publishers, who have encouraged an explosion in "special Issues" of their journals, with scant scrutiny of the quality of published articles - each of which brings in an Article Processing Charge of CHF 2600 (around £2,200, or US $2,900). The proliferation of anecdotal reports in the literature gives ammunition to those who wish to promote all kinds of unevidenced treatments: they can point to these "peer reviewed" papers as evidence of scientific respectability.

 

In the longer term, the lax editorial standards that admit pseudoscience into the research literature will further damage the already tarnished MDPI brand, and we can only hope that papers published in the Journal of Personalized Medicine will be discounted by serious journalists. But meanwhile, every practitioner with a dubious product to sell will see this journal as a perfect outlet for their promotional material. 

 

*The headline has been modified and now reads "Autism can be reversed, scientists discover", but it still describes the study as a 'trial', which is isn't.

 

 

References

D’Adamo, C. R., Nelson, J. L., Miller, S. N., Rickert Hong, M., Lambert, E., & Tallman Ruhm, H. (2024). Reversal of autism symptoms among dizygotic twins through a personalized lifestyle and environmental modification approach: A case report and review of the literature. Journal of Personalized Medicine, 14(6), Article 6. https://doi.org/10.3390/jpm14060641

 

Mahapatra, S., Vyshedsky, D., Martinez, S., Kannel, B., Braverman, J., Edelson, S. M., & Vyshedskiy, A. (2018). Autism Treatment Evaluation Checklist (ATEC) Norms: A “Growth Chart” for ATEC score changes as a function of age. Children, 5(2), Article 2. https://doi.org/10.3390/children5020025

Whistleblowing, research misconduct, and mental health

 

Book Review

Carl Elliott. The Occasional Human Sacrifice: Medical Experimentation and the Price of Saying No.  W. W. Norton, 2024. ISBN: 978-1-324-06550-0

 

This book should be required reading for three groups of people. First, every member of university or hospital administration, and those who sit on ethics committees (Institutional Review Boards) in such institutions. Second, people interested in the psychology and sociology of moral behaviour. And finally, anyone who has experience of being a whistleblower regarding academic malpractice, and anyone who is thinking of acting in that role.

 There are many books on bioethics, and readers may be familiar with at least some of the seven cases described in The Occasional Human Sacrifice:

• The Markingson case, where a severely disturbed man was persuaded to take part in a drug trial against the wishes of his mother, leading, as she had predicted, to his death by suicide.
•  The Tuskegee study where black men were coerced to take part in a study of the consequences of untreated syphilis.
• Willowbrook School, a residential institution where children with intellectual disability were deliberately infected with hepatitis.
•  Protocol 126 at the Fred Hutchinson Cancer Center, which involved patients being given an experimental bone marrow transplant treatment that proved to be ineffective, without adequate informed consent.
• The Radiation Effects study at Cincinnati Medical Center, where cancer patients were exposed to whole body irradiation, not as treatment, but in order to study its effects. 
•  "The Unfortunate Experiment" in Dunedin, New Zealand, an observational study that involved monitoring rather than treating precancerous abnormalities of the cervix, without sufficient explanation to the woman concerned.
• The case of Paolo Macchiarini, who performed experimental stem cell transplants on the trachea in humans without having done adequate animal studies to justify the approach, with dire outcomes for most patients.

The details of each of these cases makes for sobering reading: patients who trusted their doctors to look after them were experimented on without proper consent procedures, in some cases over a period of years. Most accounts of these events focus on the doctors who allowed this to happen: what motivated them, why did they ignore conventional medical ethics, and why didn't they listen to those who expressed misgivings? Elliott's primary focus, however, is on a different aspect of these cases - the whistleblowers who drew attention to the problems, their motivations, and the impact of their whistleblowing on their lives.

Elliott speaks from personal knowledge - he was the whistleblower in the Markingson case, and his account makes for uncomfortable reading. When he drew attention to the unethical practices in his department of psychiatry he was first ignored and then ostracised. He became obsessed with the case, with a major toll on his mental health. In the concluding chapter he notes:

I am the last person to advise anyone to ignore the demands of conscience, but to encourage potential whistleblowers to speak out without noting the long odds of success and the grim personal costs would be like advising them to leap blindfolded off a cliff. (p 290).

All but one of the whistleblowers featured in this book had a particularly difficult time of it because they were speaking out about ethical abuses within their own institutions. As Elliott noted, the line between a traitor and a whistleblower is a fine one, relating to the perceived motivation of the person involved. A whistleblower may feel they are acting on the purest of motives, to do the right thing, yet others will interpret it as disloyalty to the institution, or attribute their behaviour to a sense of self-importance and moral superiority. Even when, as in several of these cases, the whistleblower is eventually vindicated, they are not thanked, there is seldom restitution for victims, and the mental scars of their battles with authority stay with them.

My own forays into challenging dodgy research are nothing like as serious as the cases covered by Elliott: no lives were at stake in the instances of fraud and malpractice that I've focused on, all of which have been outside my own department. Nevertheless, I resonated strongly with what Elliott had to say about the transformation that occurs in whistleblowers when they first try to report a problem. 

Unlike most people, who are used to the darker side of human behavior, whistleblowers are genuinely shocked when they discover dishonesty, greed, or corruption. They naively assume that if they blow the whistle, others will be just as outraged as they are. They might even imagine they'll be thanked." (p. 158).

 And

 "...if becoming a whistleblower requires a certain naivete, then surviving the experience demands a hard shell of cynicism." (p 159). 

I've definitely been on that journey, after years of observing the delays and obfuscations that routinely occur when a sleuth reports a fraudulent paper to a publisher, or when an institution manages to exonerate a researcher despite ample evidence of misconduct.

I've also been thinking a lot lately about parallels with political life. Here in the UK we've had a number of scandals where citizens were harmed, and sometimes killed, by people in power who were at best incompetent and at worst corrupt. Politicians handed lucrative contracts for personal protective equipment to their cronies who failed to deliver, leaving health professionals without protection; NHS patients were given contaminated blood products long after a problem had been identified; A horrendous fire in a tower block in West London arose after contractors ignored regulations and installed inappropriate cladding on the building - in each of these cases the stories came out, there was a public outcry, and a sense that "something must be done" - but the process gets mired in red tape and nothing actually is done, nobody gets punished, and those affected aren't compensated. This is terrible not just because of the impact on the individuals who were directly harmed, but also because it destroys people's trust in the institutions that we assume will look after us. I see the failure of institutions to deal adequately with academic misconduct as just one part of this broader picture.

So is there anything that can be done to challenge wrongdoing without crashing out over the cliff? A major point made by Elliott is that whistleblowers should not go it alone: that way madness lies. One option that he raised in a recent talk on this topic was to consult a lawyer with expertise in whistleblowing cases: they will be able to advise on the likelihood of success, and may be able to bring a complaint directly to an organisation, so the whistleblower can fly under the radar. There is also the option of going to the press, but, though Elliott describes some cases where this was effective, he notes it can be a two-edged sword, because journalists will want a good story, and it may not coincide with the whistleblower's viewpoint. In my experience, it's worth forming links with responsible journalists: institutions who have been inert for years may spring into action when confronted with the possibility of adverse publicity. Perhaps the most important point is that you need to have allies. They can restore confidence that there are other people who share one's values, who appreciate what one is doing and provide moral support. Although Elliott doesn't say this, I think the internet has made it easier to find one's tribe in this regard.

Where, perhaps, I take issue with Elliott is in the implication that many people aren't troubled by the issues that worry whistleblowers. When I give talks about reproducible research practices, I reliably get a question along the lines of "What advice can you give an early-career researcher whose supervisor is encouraging them to adopt questionable research practices?" The questioner clearly finds it troublesome but is being told that their career is at risk if they don't cut corners, and that "everyone is doing it". I reply that this is an issue that has serious consequences for mental health: doing what you know to be wrong causes internal conflict that can eat away at you, leading to a clash between your model of a good person and your current behaviour.

For this reason I advise those starting out in research to read this piece by Sarah Kendzior. Her advice is intended to help people survive under the threat of authoritarianism, but many of the points apply to academic survival in an age of cynicism:

Write down what you value; what standards you hold for yourself and for others.... Write a list of things you would never do. .... 

But most of all, never lose sight of who you are and what you value. If you find yourself doing something that feels questionable or wrong a few months or years from now, find that essay you wrote on who you are and read it. Ask if that version of yourself would have done the same thing.

 And if the answer is no? Don’t do it.

 

 

Does Elsevier's negligence pose a risk to public health?

 

Yesterday, Retraction Watch published a piece about a notorious 2020 article by Gautret et al that had promoted the idea that hydroxychloroquine could treat COVID-19. Despite numerous concerns, the article has not been retracted from the International Journal for Antimicrobial Agents, a journal that is co-owned by the publisher, Elsevier, and The International Society of Antimicrobial Chemotherapy (ISAC).

On June 3rd, a group of concerned scientists wrote to Elsevier asking for the paper to be retracted in light of the evidence that the paper was flawed and had serious consequences for public health. The fact that the article pops up in the Web of Science database with a trophy symbol denoting it as a 'highly cited paper' just adds to concerns that it is still being taken seriously, giving credence to those who continue to promote an ineffective drug, hydroxychloroquine, to treat COVID-19.

Elsevier's response? They are "reopening the investigation" into the paper. This is the investigation that previously decided nothing needed to be done, despite a damning report by a group of respected experts. In their reply to the letter they stated:

"We cannot currently provide further detail as to the particulars of this investigation as it remains ongoing, and is necessarily confidential while we allow time to discuss our concerns with the authors as part of due process and in line with COPE recommendations. I will however be more then glad to keep you appraised of any final decisions in due course."

As one experienced in these communications, I can confirm that this is the kind of reply you get by a publisher who is kicking the issue into the long grass in the hope that you will forget about it and go away. The wonderful phrase "it remains ongoing" typically refers to something that goes on, and on, and on, and "due course" can take years.

A pertinent question is what COPE recommendations are being followed: there are a whole range of flowcharts provided by COPE for different circumstances. The most relevant in this case would appear to be this one on "Concerns about risk in published data".

Early on in the flowchart, the decision tree asks "Based on the initial assessment, should the dataset be removed or restricted during the investigation to mitigate potential risk?" If the answer is YES, then the appropriate action is "Remove public access to the dataset while following up on the concerns."

Given that the article is still freely available on the web, we have to ask ourselves, is Elsevier being negligent here? Have they decided that there is no risk, despite the evidence that we and others have provided that public health is endangered by publicising misleading data on an ineffective drug? Or do they agree that there is risk, but deem it inconvenient for them to take action?

Are commitments to open data policies worth the paper they are written on?

 

As Betteridge's law of headlines states: "Any headline that ends in a question mark can be answered by the word no."  So you know where I am going with this.  

 

I'm a longstanding fan of open data - in fact, I first blogged about this back in 2015. So I've been gratified to see the needle shift on this, in the sense that over the past decade, in a rush to present themselves as good guys, various institutions and publishers have published policies supporting open data. The problem is that when you actually ask them to implement those policies, they back down.   

 

I discussed arguments for and against data-sharing in a Commentary article in 2016. I divided the issues according to whether they focused on the impact of data-sharing on researchers or on research participants. Table 1 from that article, entitled "Conflict between interests of researchers and advancement of science" is reproduced here:

 

Argument	Counter-argument
1. Lack of time to curate data.	Unless adequately curated, data will over time become unusable, including by the original researcher.
2. Personal investment—reluctance to give data to freeloaders.	Reuse of data increases its value and the researcher benefits from additional citations. There is also an ethical case for maximizing use of data obtained via public funding.
3. Concerns about being scooped before the analysis is complete.	This is a common concern though there are few attested cases. A time-limited period of privileged use by the study team can be specified to avoid scooping.
4. Fear of errors being found in the data.	Culture change is needed to recognize errors are inevitable in any large dataset and should not be a reason for reputational damage. Data-sharing allows errors to be found and corrected.

 

I then went on to discuss two other concerns which focused on implications of data-sharing for human participants, viz:

5.  Ethical concerns about confidentiality of personal data, especially in the context of clinical research

6.  Possibility that others with a different agenda may misuse the data, e.g. perform selective analyses that misrepresent the findings.

 

These last two issues raise complex concerns and there's plenty to discuss on how address them, but I'll put that to one side for now, as the case I want to comment on concerns a simple dataset where there is limited scope for secondary analyses and where no human participants are involved.

 

My interest was piqued by comments on PubPeer about a paper entitled "Magnetic field screening in hydrogen-rich high-temperature superconductors ".  The thread on PubPeer starts with this extraordinary comment by J. E. Hirsch:

 

I requested the underlying data for Figs. 3a, 3e, 3b, 3f of this paper on Jan 11, 2023. This is because the published data for Figs. 3a and 3e, as well as for Figs. 3b and 3f, are nominally the same but incompatible with each other, and I would like to understand why that is. I asked the authors to explain, but they did not provide an explanation. Neither did they supply the data. The journal told me that it had received the data from the authors but will not share them with me because they are "confidential". I requested that the journal posts an Editor Note informing readers that data are unavailable to readers. The journal responded that because data were share with editors they "cannot write an editorial note on the published article stating the data is unavailable as this would be factually incorrect".

 

Pseudonymous commenter Orchestes quercus drew attention to the Data Availability statement in the article: "The data that support the findings of this study are available from the corresponding authors upon reasonable request".

 

J. E. Hirsch then added a further comment: 

 

The underlying data are still not available, the editor says the author deems the request "unreasonable" but it cannot divulge the reasoning behind it, nor can the journal publish an editor note that there are restrictions on data availability because the data were provided to the journal.  Springer Nature's Research Integrity Director wrote to me in September 2023 that "we recognize the right of the authors to not share the data with you, in line with the authors’ chosen data availability statement", and that "As Springer Nature considers the correspondence with the authors confidential, we cannot share with you any further details.

 

Now, I know nothing whatsoever about superconductors or J. E. Hirsch, but I think the editors, publisher and the authors are making themselves look very silly, and indeed suspicious, by refusing to share the data.  They can't plead patient confidentiality or ethical restrictions - it seems they are just refusing to comply because they don't want to.  

 

To up the ante, Orchestes quercus extracted data from the figures and did further analyses, which confirmed that J. E. Hirsch had a point - the data did not appear to be internally consistent.

 

Meanwhile, I had joined the PubPeer thread, pointing out

 

The authors and editor appear to be in breach of the policy of Nature Portfolio journals, stated here:
https://www.nature.com/nature-portfolio/editorial-policies/reporting-standards, viz:

An inherent principle of publication is that others should be able to replicate and build upon the authors' published claims. A condition of publication in a Nature Portfolio journal is that authors are required to make materials, data, code, and associated protocols promptly available to readers without undue qualifications. Any restrictions on the availability of materials or information must be disclosed to the editors at the time of submission. Any restrictions must also be disclosed in the submitted manuscript.

After publication, readers who encounter refusal by the authors to comply with these policies should contact the chief editor of the journal. In cases where editors are unable to resolve a complaint, the journal may refer the matter to the authors' funding institution and/or publish a formal statement of correction, attached online to the publication, stating that readers have been unable to obtain necessary materials to replicate the findings.

I also noted that two of the authors are based at a Max Planck Institute. The Max Planck Gesellschaft is a signatory to the BerlinDeclaration on Open Access to Knowledge in the Sciences and Humanities.  On the website it states:

the Max Planck Society (MPG) is committed to the goal of providing free and open access to all publications and data from scholarly research (my emphasis).

 

Well, the redoubtable J. E. Hirsch had already thought of that, and in a subsequent PubPeer comment made public various exchanges he had had with luminaries from the Max Planck Institutes.

Jos Lelieveld, director of the Max Planck Institute for Chemistry responded to my request for data with:

"Our institute fully supports the work of Mikhail Eremets and his team.”

Claudia Felser, Vice President of the MPG, responded to my request for data with:

"First of all, may I point out that the data on which the publication is based are available at the Institute. At the same time, I can confirm that, in response to your inquiry, it has been verified that the data support the results presented in the publication.”

and

"I do not consider myself responsible for your request for access to the raw data.”

and

"The President has asked me, as the responsible Vice-President of the Section, to give you a final answer on this matter….We had already informed you that Mr Eremets had expressed concerns about making his research data available. We believe that Mr Eremets has sufficiently justified these concerns. Given that the editors of Nature Communications, and therefore independent third parties, have also considered your concerns and have not granted you access to the data, we see no need to appoint a further panel of external experts to assess the situation.”

and

"In the absence of concrete evidence of misconduct, we urge you, in the spirit of fair science, to refrain from making allegations that the authors have transformed and manipulated the measured data”

Patrick Cramer, President of the MPG, did not respond to my emails.

 

All I can say to the Max Planck Gesellschaft is that this is not a good look. Hirsch has noted an inconsistency in the published figures.  This has been confirmed by another reader and needs to be explained. The longer people dig in defensively, attacking the person making the request rather than just showing the raw data, the more it looks as if something fishy is going on here.

 

Why am I so hung up on data-sharing? The reason is simple. The more I share my own data, or use data shared by others, the more I appreciate the value of doing so. Errors are ubiquitous, even when researchers are careful, but we'll never know about them if data are locked away.

 

Furthermore, it is a sad reality that fraudulent papers are on the rise, and open data is one way of defending against them. It's not a perfect defence: people can invent raw data as well as summary data, but realistic data are not so easy to fake, and requiring open data would slow down the fraudsters and make them easier to catch.

 

Having said that, asking for data is not tantamount to accusing researchers of fraud: it should be accepted as normal scientific practice to make data available in order that others can check the reproducibility of findings. If someone treats such a request as an accusation, or deems it "unreasonable", then I'm afraid it just makes me suspicious.  

 

And if organisations like Springer Nature and Max Planck Gesellschaft won't back up their policies with action, then I think they should delete them from their websites. They are presenting themselves as champions of open, reproducible science, while acting as defenders of non-transparent, secret practices. As we say in the UK, fine words butter no parsnips.   

 

 P.S. 27th May:  A comprehensive account of the superconductivity affair has just appeared on the website For Better Science.  This suggests things are even worse than I thought.   

 

In addition, you can see Jorge Hirsch explain his arduous journey in attempting to access the data here.    

 

NOTE ON COMMENTS: Many thanks to those who have commented. Comments are moderated to prevent spam, so there is a delay before they appear, but I will accept on-topic comments in due course.

Some thoughts on eLife's New Model: One year on

 

I've just been sent an email from eLife, pointing me to links to a report called "eLife's New Model: One year on" and a report by the editors "Scientific Publishing: The first year of a new era". To remind readers who may have missed it, the big change introduced by eLife in 2023 was to drop the step where an editor decides on reject or accept of a manuscript after reviewer comments are received. Instead, the author submits a preprint, and the editors then decide whether it should be reviewed. If the answer is yes, then the paper will be published, with reviewer comments. 

Given the controversy surrounding this new publishing model, it seems timely to have a retrospective look at how it's gone, and these pieces by the journal are broadly encouraging in showing that the publishing world has not fallen apart as a consequence of the changes. We are told that the proportion of submissions published has gone down slightly from 31.4% to 27.7% and the demographic characteristics of authors and reviewers are largely unchanged. The ratings of quality of submissions are similar to those from the legacy model. The most striking change has been in processing time: median time from submission to publication of the first version with reviews is 91 days, which is much faster than previously. 

As someone who has been pushing for changes to the model of scientific publishing for years (see blogsposts below), I'm generally in favour of any attempt to disrupt the conventional model. I particularly like the fact that the peer reviews are available with the published articles in eLife - I hope that will become standard for other journals in future. However, there are two things that rather rankled about the latest communication from the journal. 

First, the report describes an 'author survey' which received 325 responses, but very little detail is given as to who was surveyed, what the response rate was, and what the overall outcome was. This reads more like a marketing report than a serious scientific apprasal. Two glowing endorsements were reported from authors who had good experiences. I wondered though about authors whose work had not been selected to go forward to peer review - were they just as enthusiastic? Quite a few tables of facts and figures about the impact of the new policy were presented with the report, but if eLife really does want to present itself as embracing open and transparent policies, I think they should bite the bullet and provide more information - including fuller details of their survey methods and results, and negative as well as positive appraisals. 

Second, I continue to think there is a fatal flaw in the new model, which is that it still relies on editors to decide which papers go forward to review, using a method that will do nothing to reduce the tendency to hype and the consequent publication bias that ensues. I blogged about this a year ago, and suggested a simple solution, which is for the editors to adopt 'results-blind' review when triaging papers. This is an idea that has been around at least since 1976 (Mahoney, 1976) which has had a resurgence in popularity in recent years, with growing awareness of the dangers of publication bias (Locasio, 2017). The idea is that editorial decisions should be made based on whether the authors had identified an interesting question and whether their methods were adequate to give a definitive answer to that question. The problem with the current system is that people get swayed by exciting results, and will typically overlook weak methods when there is a dramatic finding. If you don't know the results, then you are forced to focus on the methods. The eLife report states:

 "It is important to note that we don’t ascribe value to the decision to review. Our aim is to produce high-quality reviews that will be of significant value but we are not able to review everything that is submitted." 

That is hard to believe: if you really were just ignoring quality considerations, then you should decide on which papers to review by lottery. I think this claim is not only disingenuous but also wrong-headed. If you have a limited resource - reviewer capacity - then you should be focusing it on the highest quality work. But that judgement should be made on the basis of research question and design, and not on results. 

Bibliography 

Locascio, J. J. (2017). Results blind science publishing. Basic and Applied Social Psychology, 39(5), 239–246. https://doi.org/10.1080/01973533.2017.1336093 

Mahoney, M. J. (1976). Scientist as Subject: The Psychological Imperative. Ballinger Publishing Company. 

Previous blogposts

Academic publishing: why isn't psychology like physics? 

Time for academics to withdraw free labour.

High impact journals: where newsworthiness trumps methodology

Will traditional science journals disappear?

Publishing replication failures

Just make it stop! When will we say that further research isn't needed?

 

I have a lifelong interest in laterality, which is a passion that few people share. Accordingly, I am grateful to René Westerhausen who runs the Oslo Virtual Laterality Colloquium, with monthly presentations on topics as diverse as chiral variation in snails and laterality of gesture production. 

On Friday we had a great presentation from Lottie Anstee who told us about her Masters project on handedness and musicality. There have been various studies on this topic over the years, some claiming that left-handers have superior musical skills, but samples have been small and results have been mixed. Lottie described a study with an impressive sample size (nearly 3000 children aged 10-18 years) whose musical abilities were evaluated on a detailed music assessment battery that included self-report and perceptual evaluations. The result was convincingly null, with no handedness effect on musicality. 

What happened next was what always happens in my experience when someone reports a null result. The audience made helpful suggestions for reasons why the result had not been positive and suggested modifications of the sampling, measures or analysis that might be worth trying. The measure of handedness was, as Lottie was the first to admit, very simple - perhaps a more nuanced measure would reveal an association? Should the focus be on skilled musicians rather than schoolchildren? Maybe it would be worth looking at nonlinear rather than linear associations? And even though the music assessment was pretty comprehensive, maybe it missed some key factor - amount of music instruction, or experience of specific instruments. 

After a bit of to and fro, I asked the question that always bothers me. What evidence would we need to convince us that there is really no association between musicality and handedness? The earliest study that Lottie reviewed was from 1922, so we've had over 100 years to study this topic. Shouldn't there be some kind of stop rule? This led to an interesting discussion about the impossibility of proving a negative and whether we should be using Bayes Factors, and what would be the smallest effect size of interest.  

My own view is that further investigation of this association would prove fruitless. In part, this is because I think the old literature (and to some extent the current literature!) on factors associated with handedness is at particular risk of bias, so even the messy results from a meta-analysis are likely to be over-optimistic. More than 30 years ago, I pointed out that laterality research is particularly susceptible to what we now call p-hacking - post hoc selection of cut-offs and criteria for forming subgroups, which dramatically increase the chances of finding something significant. In addition, I noted that measurement of handedness by questionnaire is simple enough to be included in a study as a "bonus factor", just in case something interesting emerges. This increases the likelihood that the literature will be affected by publication bias - the handedness data will be reported if a significant result is obtained, but otherwise can be disregarded at little cost. So I suspect that most of the exciting ideas about associations between handedness and cognitive or personality traits are built on shaky foundations, and would not replicate if tested in well-powered, preregistered studies.  But somehow, the idea that there is some kind of association remains alive, even if we have a well-designed study that gives a null result.  

Laterality is not the only area where there is no apparent stop rule. I've complained of similar trends in studies of association between genetic variants and psychological traits, for instance, where instead of abandoning an idea after a null study, researchers slightly change the methods and try again. In 2019, Lisa Feldman Barrett wrote amusingly about zombie ideas in psychology, noting that some theories are so attractive that they seem impossible to kill. I hope that as preregistration becomes more normative, we may see more null results getting published, and learn to appreciate their value. But I wonder just what it takes to get people to conclude that a research seam has been mined to the point of exhaustion. 

The world of Poor Things at MDPI journals

At the weekend, the Observer ran a piece by Robin McKie entitled "‘The situation has become appalling’: fake scientific papers push research credibility to crisis point". I was one of those interviewed for the article, describing my concerns about a flood of dodgy papers that was polluting the scientific literature.

Two days later I received an email from the editorial office of MDPI publishers with the header "[Children] (IF: 2.4, ISSN 2227-9067): Good Paper Sharing on the Topic of" (sic) that began:

Greetings from the Children Editorial Office!

We recently collected 10 highly cited papers in our journal related to Childhood Autism. And we sincerely invite you to visit and read these papers, because you are an excellent expert in this field of study.

Who could resist such a flattering invitation? MDPI is one of those publishers that appears to be encouraging publication of low quality work, with a massive growth in special issues where papers are published with remarkably rapid turnaround times. Only last week it was revealed that the journal is affected by fake peer review that appears to be generated by AI. So I was curious to take a look.

The first article, by Frolli et al (2022a) was weird. It reported a comparison of two types of intervention designed to improve emotion recognition in children with autism, one of which used virtual reality. The first red flag was the sample size: two groups each of 30 children, all originally from the city of Caserta. I checked Wikipedia, which told me the population of Caserta was around 76,000 in 2017. Recruiting participants for intervention studies is typically slow and laborious and this is a remarkable sample size to recruit from such a small region. But credibility is then stretched to breaking point on hearing that the selection criteria required that the children were all aged between 9 and 10 years and had IQs of 97 or above. No researcher in their right mind would impose unnecessary constraints on recruitment, and both the age and IQ criteria are far tighter than would usually be adopted. I wondered whether there might be a typo in this account, but we then hear that the IQ range of the sample is indeed remarkably narrow: 

"The first experimental group (Gr1) was composed of 30 individuals with a mean age of 9.3 (SD 0.63) and a mean IQ of 103.00 (SD 1.70). ...... The second experimental group (Gr2) was composed of 30 individuals with a mean age of 9.4 (SD 0.49) and mean IQ of 103.13 (SD 2.04)...."

Most samples for studies using Wechsler IQ scales have SD of at least 8, even if cutoffs are applied as selection criteria, so this is unbelievably low.

This dubious paper prompted me to look at others by the first author. It was rather like pulling a thread on a hole in a sweater - things started to unravel fast. A paper published by Frolli et al (2023a) in the MDPI journal Behavioral Sciences claimed to have studied eighty 18-year-olds recruited from four different high schools. The selection criteria were again unbelievably stringent: IQ assessed on the WAIS-IV fell between 95-105 "to ensure that participants fell within the average range of intellectual functioning, minimizing the impact of extreme cognitive variations on our analyses". The lower IQ range selected here corresponds to z-score of -0.33 or 37th percentile. If the population of students covered the full range of IQ, then only around 25% would meet the criterion (between 37th and 63rd centile), so to obtain a sample of 80 it would be necessary to test over 300 potential participants. Furthermore, there are IQ screening tests that can be used in this circumstance that are relatively quick to administer, but the WAIS-IV is not one of them. We are told all participants were given the full test, which requires individual administration by a qualified psychologist and takes around one hour to complete. So who did all this testing, and where? The article states: "The data were collected and analyzed at the FINDS Neuropsychiatry Outpatient Clinic by licensed psychologists in collaboration with the University of International Studies of Rome (UNINT)." So we are supposed to believe that hundreds of 18-year-olds trekked to a neuropsychiatry outpatient clinic for a full IQ screening which most of them would not have passed. I cannot imagine a less efficient way of conducting such a study. I could not find any mention of compensation for participants, which is perhaps unsurprising as the research received no external funding. All of this is described as happening remarkably fast, with ethics approval in January 2023, and submission of the article in October 2023.

Another paper in Children in 2023 focused on ADHD, and again reported recruiting two groups of 30 children for an intervention that lasted 5 months (Frolli et al., 2023b). The narrow IQ selection criteria were again used, with WISC-IV IQs in the range 95-105, and the mean IQs were 96.48 (SD =1.09) and 98.44 (SD = 1.12) for groups 1 and 2 respectively. Again, the research received no external funding. The report of ethics approval is scanty "The study was conducted in accordance with the Declaration of Helsinki. The study was approved by the Ethics Committee and the Academic Senate of the University of International Studies of Rome."

The same first author published a paper on the impact of COVID-19 on cognitive development and executive functioning in adolescents in 2021 (Frolli et al, 2021). I have not gone over it in detail, but a quick scan revealed some very odd statistical reporting. There were numerous F-ratios, but they were all negative, which is impossible, as F is a ratio between two positive numbers. Furthermore, the reported p-values and degrees of freedom didn't always correspond to the F-ratio, even if the sign was ignored.

At this point I was running out of steam, but a quick look at Frolli et al (2022a) on Executive Functions and Foreign Language Learning suggested yet more problems, with the sentence "Significance at the level of 5% (α < 0.001) has been accepted" featuring at least twice. It is hard to believe that a human being wrote this sentence, or that any human author, editor or reviewer read it without comment.

If anyone is interested in pulling at other related threads, I suspect it would be of interest to look at articles accepted for a Special Issue of the MDPI journal Disabilities co-edited by Frolli.

In his brilliant film Poor Things, Yorgos Lanthimos distorts familiar objects and places just enough to be disturbing. Lisbon looks like what I imagine Lisbon would be in the Victorian age, except that the colours are unusually vivid, there are strange flying cars in the sky, and nobody seems concerned at the central character wandering around only partially clothed (see, e.g., this review).  The combined impression is that MDPI publishes papers from that universe, where everything looks superficially like genuine science but with jarring features that tell you something is amiss. The difference is that Poor Things has a happy ending.

References 

Frolli, A.; Ricci, M.C.; Di Carmine, F.; Lombardi, A.; Bosco, A.; Saviano, E.; Franzese, L. The Impact of COVID-19 on Cognitive Development and Executive Functioning in Adolescents: A First Exploratory Investigation. Brain Sci. 2021, 11, 1222. https://doi.org/10.3390/brainsci11091222

Frolli, A.; Savarese, G.; Di Carmine, F.; Bosco, A.; Saviano, E.; Rega, A.; Carotenuto, M.; Ricci, M.C. Children on the Autism Spectrum and the Use of Virtual Reality for Supporting Social Skills. Children 2022a, 9, 181. https://doi.org/10.3390/children9020181

Frolli, A.; Cerciello, F.; Esposito, C.; Ciotola, S.; De Candia, G.; Ricci, M.C.; Russo, M.G. Executive Functions and Foreign Language Learning. Pediatr. Rep. 2022b, 14, 450-456. https://doi.org/10.3390/pediatric14040053

Frolli, A.; Cerciello, F.; Ciotola, S.; Ricci, M.C.; Esposito, C.; Sica, L.S. Narrative Approach and Mentalization. Behav. Sci. 2023a, 13, 994. https://doi.org/10.3390/bs13120994

Frolli, A.; Cerciello, F.; Esposito, C.; Ricci, M.C.; Laccone, R.P.; Bisogni, F. Universal Design for Learning for Children with ADHD. Children 2023b, 10, 1350. https://doi.org/10.3390/children10081350

An (intellectually?) enriching opportunity for affiliation

Guest Post by Nick Wise 

 

A couple of months ago a professor received the following email, which they forwarded to me.

 

"Dear esteemed colleagues,

We are delighted to extend an invitation to apply for our prestigious remote research fellowships at the University of Religions and Denominations (URD). These fellowships offer substantial financial support to researchers with papers currently in press, accepted or under review by Scopus-indexed journals. We welcome scholars from diverse academic disciplines to seize this intellectually enriching opportunity.

Fellowship Details:
Fellowship Type: Remote Short-term Research Fellowship.
Research Focus: Diverse fields, spanning humanities, social sciences, interdisciplinary studies, and more.
Research Output: Publication of research articles in Scopus-indexed journals.
Affiliation: Encouragement for researchers to acknowledge URD as their additional affiliation in published articles.
Remuneration: Project-based compensation for each research article.
Payment Range: Up to $1000 USD per article (based on SJR journal ranking).
Eligibility: Papers in press, accepted, or under review by Scopus-indexed journals.

Preference: Priority for indexing before December 30, 2023.

Application Process:   

To express your interest in securing a fellowship, kindly submit your curriculum vitae to  Ahmad Moghri at moghri.urd@gmail.com. When emailing your application, please use the subject line: "Research Fellowship, FULL NAME."

Upon Selection:
Successful applicants will receive formal invitations to join our esteemed fellowship program. Invitation letters and collaboration contracts will be dispatched within a maximum of 5 days.

We firmly believe that this fellowship program provides an invaluable platform for scholars to make substantial contributions to their fields while collaborating with the distinguished University of Religions and Denominations. We encourage all eligible individuals to seize this exceptional opportunity.

For inquiries or further information, please do not hesitate to contact moghri.urd@gmail.com.

Warmest Regards,”

Why would the institution pay researchers to say that they are affiliated with them? It could be that funding for the university is related to the number of papers published in indexed journals. More articles associated with the university can also improve their placing in national or international university rankings, which could lead directly to more funding, or to more students wanting to attend and bringing in more money.

The University of Religions and Denominations is a private Iranian university specialising, as the name suggests, in the study of different religions and movements. Until recently the institution had very few published papers associated with it according to Dimensions and their subject matter was all related to religion. However, last year there was a substantial increase to 103 published papers, and so far this year there are already 35. This suggests that some academics have taken them up on the offer in the advert to include URD as an affiliation.

Surbhi Bhatia Khan is a lecturer in data science at the University of Salford in the UK since March 2023 and a top 2% scientist in the world according to Stanford University’s rankings. She published 29 research articles last year according to Dimensions, an impressive output, in which she was primarily affiliated to the University of Salford. In addition though, 5 of those submitted in the 2nd half of last year had an additional affiliation at the Department of Engineering and Environment at URD, which is not listed as one of the departments on the university website. Additionally, 19 of the 29 state that she’s affiliated to the Lebanese American University in Beirut, which she was not affiliated with before 2023. She is yet to mention her role at either of these additional affiliations on her LinkedIn profile.

Looking at the Lebanese American University, another private university, its publication numbers have shot up from 201 in 2015 to 503 in 2021 and 2,842 in 2023, according to Dimensions. So far in 2024 they have published 525, on track for over 6,000 publications for the year. By contrast, according to the university website, the faculty consisted of 547 full-time staff members in 2021 but had shrunk to 423 in 2023.  It is hard to imagine how such growth in publication numbers could occur without a similar growth in the faculty, let alone with a reduction.

How many other institutions are seeing incredible increases in publication numbers? Last year we saw gaming of the system on a grand scale by various Saudi Arabian universities, but how many offers like the one above are going around, whether by email or sent through Whatsapp groups or similar?

The Committee On Publication Ethics held a forum on claiming institutional affiliations in December 2023, in recognition of the fact that guidance for what merits affiliation to an institution is lacking and there are no accepted standards for how many affiliations an author should give. It looks like such guidance can’t come soon enough.

Nick Wise is a researcher at the University of Cambridge, UK.

Note: Comments are moderated to prevent spam and abuse, so please be patient if you post a comment and it does not appear immediately

P.S. 3rd Feb 2024

Someone on social media queried the "top 2% rating" for Khan. Nick tells me this is based on an Elsevier ranking for 2022: https://elsevier.digitalcommonsdata.com/datasets/btchxktzyw/6

Low-level lasers. Part 2. Erchonia and the universal panacea

 

 

In my last blogpost, I looked at a study that claimed continuing improvements of symptoms of autism after eight 5-minute sessions where a low-level laser was pointed at the head.  The data were so extreme that I became interested in the company, Erchonia, who sponsored the study and in Regulatory Insight, Inc, whose statistician failed to notice anything odd.  In exploring Erchonia's research corpus, I found that they have investigated the use of their low-laser products for a remarkable range of conditions. A search of clinicaltrials.com with the keyword Erchonia produced 47 records, describing studies of pain (chronic back pain, post-surgical pain, and foot pain), body contouring (circumference reduction, cellulite treatment), sensorineural hearing loss, Alzheimer's disease, hair loss, acne and toenail fungus. After excluding the trials on autism described in my previous post, fourteen of the records described randomised controlled trials in which an active laser was compared with a placebo device that looked the same, with both patient and researcher being kept in the dark about which device was which until the data were analysed. As with the autism study, the research designs for these RCTs specified on clinicaltrials.com looked strong, with statistician Elvira Cawthon from Regulatory Insight involved in data analysis.

As shown in Figure 1, where results are reported for RCTs, they have been spectacular in virtually all cases. The raw data are mostly not available, and in general the plotted data look less extreme than in the autism trial covered in last week's post, but nonetheless, the pattern is a consistent one, where over half the active group meet the cutoff for improvement, whereas less than half (typically 25% or less) of the placebo group do so. 

FIGURE 1: Proportions in active treated group vs placebo group meeting preregistered criterion for improvement (Error bars show SE)*

I looked for results from mainstream science against which to benchmark the Erchonia findings.  I found a big review of behavioural and pharmaceutical interventions for obesity by the US Agency for Healthcare Research and Quality (LeBlanc et al, 2018). Figures 7 and 13 show results for binary outcomes - relative risk of losing 5% or more of body weight over a 12 month period; i.e. the proportion of treated individuals who met this criterion divided by the proportion of controls. In 38 trials of behavioural interventions, the mean RR was 1.94 [95% CI, 1.70 to 2.22]. For 31 pharmaeutical interventions, the effect varied with the specific medication, with RR ranging from 1.18 to 3.86. Only two pharmaceutical comparisons had RR in excess of 3.0. By contrast, for five trials of body contouring or cellulite reduction from Erchonia, the RRs ranged from 3.6 to 18.0.  Now, it is important to note that this is not comparing like with like: the people in the Erchonia trials were typically not clinically obese: they were mostly women seeking cosmetic improvements to their appearance.  So you could, and I am sure many would, argue it's an unfair comparison. If anyone knows of another literature that might provide a better benchmark, please let me know. The point is that the effect sizes reported by Erchonia are enormous relative to the kinds of effects typically seen with other treatments focused on weight reduction.

If we look more generally at the other results obtained with low-level lasers, we can compare them to an overview of effectiveness of common medications (Leucht et al, 2015). These authors presented results from a huge review of different therapies, with effect sizes represented as standardized mean differences (SMD - familiar to psychologists as Cohen's d). I converted Erchonia results into this metric*, and found that across all the studies of pain relief shown in Figure 1, the average SMD was 1.30, with a range from 0.87 to 1.77. This contrasts with Leucht et al's estimated effect size of 1.06 for oxycodone plus paracetamol, and 0.83 for Sumatriptan for migraine.  So if we are to believe the results, they indicate that the effect of Erchonia low-level lasers is as good or better than the most effective pharmaceutical medications that we have for pain relief or weight loss. I'm afraid I remain highly sceptical.

I would not have dreamed of looking at Erchonia's track record if it were not for their impossibly good results in the Leisman et al autism trial that I discussed in the previous blogpost.  When I looked in more detail, I was reminded of the kinds of claims made for alternative treatments for children's learning difficulties, where parents are drawn in with slick websites promising scientifically proven interventions, and glowing testimonials from satisfied customers. Back in 2012 I blogged about how to evaluate "neuroscientific" interventions for dyslexia.  Most of the points I made there apply to the world of "photomodulation" therapies, including the need to be wary when a provider claims that a single method is effective for a whole host of different conditions.  

Erchonia products are sold worldwide and seem popular with alternative health practitioners. For instance, in Stockport, Manchester, you can attend a chiropractic clinic where Zerona laser treatment will remove "stubborn body fat". In London there is a podiatry centre that reassures you: "There are numerous papers which show that cold laser affects the activity of cells and chemicals within the cell. It has been shown that cold laser can encourage the formation of stem cells which are key building blocks in tissue reparation. It also affects chemicals such as cytochrome c and causes a cascade of reactions which stimulates the healing. There is much research to show that cold laser affects healing and there are now several very good class 1 studies to show that laser can be effective." But when I looked for details of these "very good class 1 studies" they were nowhere to be found. In particular, it was hard to find research by scientists without vested interests in the technology.  

Of all the RCTs that I found, there were just two that were conducted at reputable universities. One of them, on hearing loss (NCT01820416) was conducted at the University of Iowa, but terminated prematurely because intermediate analysis showed no clinically or statistically significant effects (Goodman et al., 2013).  This contrasts sharply with NCT00787189, which had the dramatic results reported in Figure 1 (not, as far as I know, published outside of clinicaltrials.gov). The other university-based study was the autism study based in Boston described in my previous post: again, with unpublished, unimpressive results posted on clinicaltrials.gov.

This suggests it is important when evaluating novel therapies to have results from studies that are independent of those promoting the therapy. But, sadly, this is easier to recommend than to achieve. Running a trial takes a lot of time and effort: why would anyone do this if they thought it likely that the intervention would not work and the postulated mechanism of action was unproven? There would be a strong risk that you'd end up putting in effort that would end in a null result, which would be hard to publish. And you'd be unlikely to convince those who believed in the therapy - they would no doubt say you had the wrong wavelength of light, or insufficient duration of therapy, and so on.  

I suspect the response by those who believe in the power of low-level lasers will be that I am demonstrating prejudice, in my reluctance to accept the evidence that they provide of dramatic benefits. But, quite simply, if low-level laser treatment was so remarkably effective in melting fat and decreasing pain, surely it would have quickly been publicised through word of mouth from satisfied customers. Many of us are willing to subject our bodies to all kinds of punishments in a quest to be thin and/or pain-free. If this could be done simply and efficiently without the need for drugs, wouldn't this method have taken over the world?

*Summary files (Erchonia_proportions4.csv) and script (Erchonia_proportions_for_blog.R) are on Github, here.