My aim here is to provide some objective criteria that can be used. First, there is scientific evaluation: does the intervention have a plausible basis, and how has it been tested? Where claims are made about changing the brain, are they based on solid neuroscientific research? Second, there are red flags, some of which I listed in a previous post on ‘Pioneering treatment or quackery?” Here I've gathered these together so that there is a ready checklist that can be applied when a new intervention surfaces.
1. Who is behind the treatment and what are their credentials?
What you should look for here are relevant qualifications, particularly a higher degree (preferably doctorate) from an academic institution with a good reputation. Red flags are:
- No information about who is involved ▶#1
- Intervention developed by someone with no academic credentials ▶#2
- Citation of spurious credentials; affiliation with organisations that have very lax membership criteria, e.g., Royal Society of Medicine ▶#3
- A lack of publications in peer-reviewed journals. Publications only in books counts as a red flag, because there is no quality control. ▶#4
As far as #4 is concerned, relevant information can be obtained checking an author against a database such as Web of Science. However, access to such databases is largely restricted to academic institutions. Google Scholar is widely available, though its results are not restricted to peer-reviewed literature.
2. Is there a credible scientific basis to the treatment?
This is often difficult for a lay person to evaluate. Google Scholar may be helpful in tracking down articles that discuss the background to the intervention. Ideally, one is looking for a review by someone who is independent of those who developed it and who has good academic credentials. If no relevant journal articles are found on Google Scholar this is a red flag ▶#5. If a journal article is found, try to find whether the journal is a mainstream peer-reviewed publication.
3. Who is the intervention recommended for?
It is implausible that the whole gamut of neurodevelopmental problems has a single underlying cause, and it is unlikely that they will all respond to the same intervention. If an intervention claims to be effective for a host of diverse disorders, then this is a red flag ▶#6.
4. Is there evidence from controlled trials that the intervention is effective?
If there is such evidence, the main website for the intervention should describe it and provide links to the sources. No mention of controlled trials ▶#7, and heavy reliance on testimonials ▶#8 are both red flags. Chldren's progress should be measured on standardized and reliable psychometric tests, i.e. measures that have been developed for this purpose where normal range performance has been established. Failure to provide such information is another red flag ▶#9. It is not uncommon to find reference to trials with no controls, i.e. children’s progress is monitored before and after the intervention, and improvements are described. This is not adequate evidence of efficacy, for reasons I have covered in detail here: essentially, improvement in test scores can arise because of practice on the tests, maturation, statistical variation or expectation effects. If scores from before and after treatment are presented as evidence for efficacy, with no reference to control data, this is another red flag ▶#10, because it indicates that the practitioners do not understand the basic requirements of treatment evaluation.
If the evidence comes solely from children tested by people with a commercial interest, there may even be malpractice, with scores massaged to look better than they are. When there were complaints about an US intervention, Learning RX, ex-employees claimed that they had been encouraged to alter children's test scores to make their progress look better than it was (see comment from 6th Dec 2009). One hopes this is not common, but it is important to be alert to the possibility and to ensure those administering psychological tests are appropriately qualified, and if necessary get an independent assessment.
The strongest evidence for effectiveness comes from randomised controlled trials, which adopt stringent methods that have become the norm in clinical medicine. Where several trials have been conducted, then it is possible to combine the findings in a systematic review, which uses rigorous standards to evaluate evidence to avoid bias that can arise if there is ‘cherrypicking’ of studies. This level of evidence is very rare in behavioural interventions for neurodevelopmental disorders because the studies are expensive and time-consuming to do.
5. What is the attitude of those promoting the intervention to conventional approaches?
The question that an advocate for a new treatment has to answer is, if this is such a good thing, why hasn’t it been picked up by mainstream practitioners?
An answer that implies some kind of conspiracy by the mainstream to suppress a new development is a red flag ▶#11. This kind of argument is widely used by alternative medicine practitioners who maintain that others have vested interests (e.g. payments from pharmaceutical companies). This doesn’t hold water: basically, if a treatment is effective, then it makes no financial sense to reject it, given that people will pay good money for something that works.
Another red flag ▶#12 is if the new intervention is promoted alongside other alternative medicine methods that do not have good supportive evidence. This suggests that the practitioners do not take an evidence-based approach.
6. Are the costs transparent and reasonable?
Lack of information about costs on the website is a red flag ▶#13, especially if you can only get information by phoning (hence allowing the practitioner to adopt a hard sell approach). Is there any provision for a refund if the intervention is ineffective? If someone tells you their treatment has a 90% success rate, then they should be willing to give you your money back if it doesn't work. Another red flag is if you are asked to sign up in advance for a long-term treatment plan ▶#14. For example, in the case of the Dore programme, there were instances of families tied into credit agreements and forced to pay even if they don’t continue with the intervention.
I’ll illustrate by applying the criteria to Sensory Activation Solutions. This is just one example of neuroscientific interventions on offer on the
web. I've singled it out because I was recently asked my opinion after a new SAS Centre opened in Milton Keynes this month.
1. Who is behind the treatment and what are their credentials?
The SAS website states Sensory Activation Solutions (SAS) is the 'brainwave' of Steven Michaëlis
and Kaśka Gozdek-Michaëlis and is the culmination of over 30 years of
study and work relating to how we learn and how we can be more effective
in life. I tried various approaches to search terms but was not
able to find any publications by either person on Google Scholar. This
is worrying: one would expect 30 years of study to yield some
peer-reviewed papers.
The
biography of Steven Michaëlis does not mention any academic
qualifications. He has a background in sound processing and computer
technologies and has trained as a group counsellor. The website states
that: Kaśka Gozdek-Michaëlis is an inter-faith, cross-cultural lecturer,
writer, psychotherapist and life-coach with over 25 years experience.
She gained a Master Degree in Oriental Studies at the prestigious
University of Warsaw, Poland. She is the author of two books in Polish, 'Develop your genius mind' and 'Super-possibilities of your mind'.
Overall,
the originators of the treatment are up-front about their background
and do not hide behind spurious qualifications. However, neither of them
appears to have any training in brain science or neurodevelopmental disorders, and their
methods have not been subject to peer review. Two red flags:▶#2 ▶#4
2. Is there a credible scientific basis to the treatment?
There were no entries in Google Scholar for "Sensory activation solutions", so I read the section on The science behind the SAS programs.
This provided a quite complex story, about how playing sounds through
headphones "activates the auditory processing centres in the brain...
leading to less sensory overload, faster understanding, better verbal
expression and improved reading and writing." It is a truism that
playing sounds to people will activate auditory centres of the brain:
that's what hearing is. The key question is whether the sounds used by
SAS do anything special. There are numerous components to
the SAS package, including use of vision, touch, taste, smell and
proprioception "to reduce sensory overload." Sensory overload is a
problem for some children, notably a subset of those with autistic
spectrum disorder. But it's not generally viewed as problematic for
children with dyslexia. It's also claimed that by presenting different auditory stimuli to the left and right ears, the SAS method can
promote right-ear dominance and inter-hemispheric integration.
In a video presentation, Michaëlis
explains this aspect of the theory further, picking up on some old ideas about
cerebral lateralisation, interhemispheric communication and rapid
auditory processing. To those who don't know the literature, this will sound convincing, but his account is
oversimplistic, and makes leaps from theory to intervention with no
evidence. For example, with current methods of imaging it would be possible to test whether SAS stimuli alter children's cerebral lateralisation, but there's no indication of any studies investigating this. Overall, the account of the brain bases of dyslexia is out of line with contemporary neuroscientific research. One red flag: ▶#5
3. Who is the intervention recommended for?
SAS is described as appropriate for attention deficit
disorder, hyper-activity, dyslexia, dyscalculia, hearing and speech
disorders, stammering, autism, Asperger's Syndrome, Down Syndrome,
global developmental delay, Cerebral Palsy, eating disorders, sleeping
disorders. In the video it is also recommended for acquired aphasia. One red flag: ▶#6
4. Is there evidence from controlled trials that the intervention is effective?
The "research" section of the website cites descriptive statistics
only, largely based on parent satisfaction indices. There is no evidence
that psychometrically sound measures were used to evaluate progress.
There
is a small scientific literature on Auditory Integration training (AIT), which has
many features in common with aspects of the SAS package; most studies focussed on autism, where there is little evidence of efficacy (Sinha et al, 2006). The American Speech-Language-Hearing Association concluded a review of AIT thus: Despite approximately one decade of practice in this country, this
method has not met scientific standards for efficacy and safety that
would justify its inclusion as a mainstream treatment for these
disorders. Four red flags: ▶#7 ▶#8 ▶#9 ▶#10
5. What is the attitude of those promoting the intervention to conventional approaches?
The 'resources' section of the website contained a wealth of
information about other kinds of intervention, both mainstream and
alternative.
6. Are the costs transparent and reasonable?
The website was quite complicated to navigate, and I may have missed
something, but I could not find any information about costs of treatment, only a
phone number. It's not possible therefore to say if costs are reasonable. It seems unlikely that clients would be tied in to long-term contracts, as treatment duration seems quite short, lasting weeks rather than months. One red flag: ▶#13
I suspect no intervention is perfect, and if you have a child who is struggling at school you may want to go ahead and try an intervention regardless of red flags. My goal here is not to stop people trying new interventions, but to ensure that they do so with their eyes open. If practitioners make claims about changing the brain, then they can expect to have those claims scrutinised by neuroscientists. The list of red flags is intended to help people make informed decisions: it may also serve the purpose of indicating to practitioners what they need to do to win confidence of the scientific community.
Reference
Sinha, Y., Silove, N., Wheeler, D., & Williams, K. (2006). Auditory integration training and other sound therapies for autism spectrum disorders: a systematic review Archives of Disease in Childhood, 91 (12), 1018-1022 DOI: 10.1136/adc.2006.094649
P.S. 6th March 2013: Here are some additional tips for spotting bad science more generally:
Posts
