My aim here is to provide some objective criteria that can be used. First, there is scientific evaluation: does the intervention have a plausible basis, and how has it been tested? Where claims are made about changing the brain, are they based on solid neuroscientific research? Second, there are red flags, some of which I listed in a previous post on ‘Pioneering treatment or quackery?” Here I've gathered these together so that there is a ready checklist that can be applied when a new intervention surfaces.
1. Who is behind the treatment and what are their credentials?
What you should look for here are relevant qualifications, particularly a higher degree (preferably doctorate) from an academic institution with a good reputation. Red flags are:
- No information about who is involved ▶#1
- Intervention developed by someone with no academic credentials ▶#2
- Citation of spurious credentials; affiliation with organisations that have very lax membership criteria, e.g., Royal Society of Medicine ▶#3
- A lack of publications in peer-reviewed journals. Publications only in books counts as a red flag, because there is no quality control. ▶#4
As far as #4 is concerned, relevant information can be obtained checking an author against a database such as Web of Science. However, access to such databases is largely restricted to academic institutions. Google Scholar is widely available, though its results are not restricted to peer-reviewed literature.
2. Is there a credible scientific basis to the treatment?
This is often difficult for a lay person to evaluate. Google Scholar may be helpful in tracking down articles that discuss the background to the intervention. Ideally, one is looking for a review by someone who is independent of those who developed it and who has good academic credentials. If no relevant journal articles are found on Google Scholar this is a red flag ▶#5. If a journal article is found, try to find whether the journal is a mainstream peer-reviewed publication.
3. Who is the intervention recommended for?
It is implausible that the whole gamut of neurodevelopmental problems has a single underlying cause, and it is unlikely that they will all respond to the same intervention. If an intervention claims to be effective for a host of diverse disorders, then this is a red flag ▶#6.
4. Is there evidence from controlled trials that the intervention is effective?
If there is such evidence, the main website for the intervention should describe it and provide links to the sources. No mention of controlled trials ▶#7, and heavy reliance on testimonials ▶#8 are both red flags. Chldren's progress should be measured on standardized and reliable psychometric tests, i.e. measures that have been developed for this purpose where normal range performance has been established. Failure to provide such information is another red flag ▶#9. It is not uncommon to find reference to trials with no controls, i.e. children’s progress is monitored before and after the intervention, and improvements are described. This is not adequate evidence of efficacy, for reasons I have covered in detail here: essentially, improvement in test scores can arise because of practice on the tests, maturation, statistical variation or expectation effects. If scores from before and after treatment are presented as evidence for efficacy, with no reference to control data, this is another red flag ▶#10, because it indicates that the practitioners do not understand the basic requirements of treatment evaluation.
If the evidence comes solely from children tested by people with a commercial interest, there may even be malpractice, with scores massaged to look better than they are. When there were complaints about an US intervention, Learning RX, ex-employees claimed that they had been encouraged to alter children's test scores to make their progress look better than it was (see comment from 6th Dec 2009). One hopes this is not common, but it is important to be alert to the possibility and to ensure those administering psychological tests are appropriately qualified, and if necessary get an independent assessment.
The strongest evidence for effectiveness comes from randomised controlled trials, which adopt stringent methods that have become the norm in clinical medicine. Where several trials have been conducted, then it is possible to combine the findings in a systematic review, which uses rigorous standards to evaluate evidence to avoid bias that can arise if there is ‘cherrypicking’ of studies. This level of evidence is very rare in behavioural interventions for neurodevelopmental disorders because the studies are expensive and time-consuming to do.
5. What is the attitude of those promoting the intervention to conventional approaches?
The question that an advocate for a new treatment has to answer is, if this is such a good thing, why hasn’t it been picked up by mainstream practitioners?
An answer that implies some kind of conspiracy by the mainstream to suppress a new development is a red flag ▶#11. This kind of argument is widely used by alternative medicine practitioners who maintain that others have vested interests (e.g. payments from pharmaceutical companies). This doesn’t hold water: basically, if a treatment is effective, then it makes no financial sense to reject it, given that people will pay good money for something that works.
Another red flag ▶#12 is if the new intervention is promoted alongside other alternative medicine methods that do not have good supportive evidence. This suggests that the practitioners do not take an evidence-based approach.
6. Are the costs transparent and reasonable?
Lack of information about costs on the website is a red flag ▶#13, especially if you can only get information by phoning (hence allowing the practitioner to adopt a hard sell approach). Is there any provision for a refund if the intervention is ineffective? If someone tells you their treatment has a 90% success rate, then they should be willing to give you your money back if it doesn't work. Another red flag is if you are asked to sign up in advance for a long-term treatment plan ▶#14. For example, in the case of the Dore programme, there were instances of families tied into credit agreements and forced to pay even if they don’t continue with the intervention.
Overall, you can see that SAS earns nine red flags on my evaluation
I suspect no intervention is perfect, and if you have a child who is struggling at school you may want to go ahead and try an intervention regardless of red flags. My goal here is not to stop people trying new interventions, but to ensure that they do so with their eyes open. If practitioners make claims about changing the brain, then they can expect to have those claims scrutinised by neuroscientists. The list of red flags is intended to help people make informed decisions: it may also serve the purpose of indicating to practitioners what they need to do to win confidence of the scientific community.
Sinha, Y., Silove, N., Wheeler, D., & Williams, K. (2006). Auditory integration training and other sound therapies for autism spectrum disorders: a systematic review Archives of Disease in Childhood, 91 (12), 1018-1022 DOI: 10.1136/adc.2006.094649
P.S. 6th March 2013: Here are some additional tips for spotting bad science more generally: