Publishers, psychological tests and greed

© CartoonStock.com

There was an intriguing piece in the New England Journal of Medicine this week about a commonly used screening test that indicates if someone is likely to have dementia. The Mini Mental State Examination (MMSE) is widely used throughout the world because it is quick and easy to administer. The test is very simple: you need no equipment, and the eleven items, involving questions to test orientation (e.g. “Where are we?”) and language (e.g. “What is this?” while showing the patient a wristwatch) are reproduced at the end of the original article about the MMSE, which was published in 1975.

The problem is that now the authors have taken steps to license the test, so that it has to be purchased from Psychological Assessment Resources. The cost is modest, $1.23 per test, but nevertheless more than the cost of photocopying one side of paper, which is what people have been doing for years. And of course, if people have to use only officially purchased copies of MMSE there are the additional costs of raising purchase orders, postage, storing packs of forms, and so on.

I’ve got a particular interest in this story, as I have published psychological tests, both off my own bat, and through a test publishing company. I started out in the late 1970s, when I developed a test of children’s comprehension called the Test for Reception of Grammar (TROG). This was more complicated than MMSE in two important respects. It involved lots of brightly coloured pictures as well as a record form, and in order to decide if a child had comprehension problems, I needed to establish how well typical children performed at different ages. The latter process, known as test standardisation, is not a trivial task, because you have to test lots of children to get a good estimate of the range of scores as well as the average score at different ages. This early work was done as part of a study funded by the Medical Research Council (MRC), but I assumed that, if the project worked out, we’d need a test publisher, and so I contacted one. The project involved two big costs. First there was the cost of my time and effort in devising the test, finding reliable people to test hundreds of children nationwide, analyse the results and write the manual. The other cost was printing colour test booklets. I had assumed that the test publisher would be willing to cover this, but they weren’t. They suggested that the MRC should find another several thousand pounds to cover printing. Now this made me cross. The publisher would get for free a fully standardised test that they could sell, no doubt at vast profit, but they wanted someone else to foot the bill for production costs. MRC were actually making quite positive noises about finding the money, but I was irritated enough to explore other options. I found a local printer and learned about the arcane world of different colour separation processes, and came away with a reasonable quote. I also discovered something quite interesting. The costs were all in the initial process of creating plates: the actual printing costs were trivial. This meant that it cost no more to print 1,000 picture books than the 100 copies I needed. And the costs of printing record forms were trivial. I returned to MRC and suggested we left the publisher out of the equation, and they agreed. All proceeded very smoothly, but once the standardisation was completed, I had a problem. There were 900 unused copies of the picture book. I discussed with MRC what we should do. They suggested I could give them away, but this would mean the test would become obsolete as soon as all the copies were used up. In the end, we reached an agreement that I could sell the test in a kind of cottage industry, and share any profits with MRC. And so I did for about the next 15 years. I didn’t bother to copyright the test because it was cheaper to buy it from me than to photocopy it. Nevertheless, I made a nice profit, and took considerable pleasure in telling the publisher to piss off some years later when they approached me expressing interest in TROG.

My next foray into test publishing was with a four-page questionnaire, the Children’s Communication Checklist (CCC). As with TROG, I hadn’t set out to devise an assessment: it came about because there wasn’t anything out there that did what I wanted, so I had to make my own instrument. I published a paper on the CCC in 1998, and listed all the items in an Appendix. I had a problem, though. I was getting busier all the time. For some years I had been paying graduate students to look after TROG sales: the weekly trip to the post office with heavy parcels had become too much of a chore. And every time I moved house, there was the question of what to do with the stock: boxes of picture books and record forms. I also realised that TROG was getting out of date - it’s well recognised that tests need restandardising every ten years or so. I also wanted to develop a test of narrative language.  And the CCC was far from perfect and needed revamping and standardising. So I took the big step: I contacted a test publisher. A different one from before. To cut a long story short, they put money into the standardisation, covered production costs, and offered highly professional editorial support. There are now three of my tests in their catalogue. 

The upside for me? The tests are actually marketed, so sales are massive compared with my cottage industry activities. And I no longer have to keep a cellar full of cardboard boxes of stock, or concern myself with organising printing and despatching tests, or dealing with complaints from someone whose finger was cut by an injudiciously placed staple. There is a downside, though. The tests are far more expensive. Having done the publishing myself, I know a little secret of the test publishing business: they don’t make their profits from actual test materials such as coloured picture books or IQ test kit. The profits are all in the record forms. These cost peanuts to produce and are sold at a mind-boggling mark-up.

I went into the deal with the publisher with my eyes open. They are a business and I knew they’d make profit from my academic work - just as journal publishers do. I reckon they’ve done more to deserve that profit than most journal publishers, as they put money into test development. That involved taking a gamble that the tests would sell. I have benefited from having a large professional organisation promoting my work, and I do get royalties on the tests. I recycle these back to a relevant charity, and there’s something pleasing about profits from testing children’s language being ploughed back into helping children with language problems.

But my publisher’s situation is very very different from the situation with MMSE. The only people who could plausibly argue they deserve to make money from the test are its authors: the publisher has put no money into development of the test and taken no risks. The authors appear to be claiming that the test items are their intellectual property, and that anyone who attempts to develop a similar test is infringing their copyright. But where did the MMSE items come from? A quick read of the introduction to the 1975 paper gives an answer. Most of them are based a longer assessment described in a 1971 article by Withers and Hinton. It would seem that the main contribution of Folstein et al was to shorten an existing test. I wonder if the British Journal of Psychiatry should go after them for copyright infringement?

Newman, J., & Feldman, R. (2011). Copyright and Open Access at the Bedside New England Journal of Medicine, 365 (26), 2447-2449 DOI: 10.1056/NEJMp1110652

P.S. Another post that includes some information on how MMSE was developed.

You can read more by scrolling down to "The Mini Exam with Maximal Staying Power" on this site from 2007.

NHS research ethics procedures: a modern-day Circumlocution Office

In Little Dorritt, Charles Dickens rails against the stifling effects of bureaucracy:

No public business of any kind could possibly be done at any time without the acquiescence of the Circumlocution Office.… the Circumlocution Office was down upon any ill-advised public servant who was going to do it, or who appeared to be by any surprising accident in remote danger of doing it, with a minute, and a memorandum, and a letter of instructions that extinguished him.

 Substitute “NHS research ethics procedures” for Circumlocution Office, and “researcher” for public servant, and you have a perfect description of a contemporary problem.

December 2010

My programme grant has been running now for over a year, and it’s time to gird up my loins to tackle NHS ethics. I’ve had plenty of other research to keep me busy, but I’m aware that I’ve been putting off this task after earlier aversive experiences. “Come on,” I tell myself, “you deal with unpleasant and bureaucratic tasks regularly - reviewing grants, responding to reviewer comments, completing your tax return. You really just have to treat this in the same way.”

It starts well enough. I track down a website for the Integrated Research Application System IIRAS). I start to have misgivings when it tells me that it’ll take approximately an hour to work through its e-learning training module. To my mind, any web-based form that requires training in its use needs redesigning. But I bite the bullet and work through the training. Not too bad, I think. I can handle this. I start to complete the form. I’m particularly happy to find little buttons associated with each question that explain what they want you to say. A definite improvement, as in the earlier versions you spent a lot of time trying to work out what the questions were getting at. It also cleverly adapts so that it excludes questions that aren’t relevant to your application. This turns out to be a two-edged sword, as I discover some weeks later. But at present I am progressing and in a cheerful mood.

The process is interrupted by need to travel from Australia to UK, Christmas, snow, massive revision to do to address reviewer comments on a paper, etc.

January 2011

Input more information, design information sheets, consent forms, etc, etc. Still feeling buoyant. The form is virtually complete, except for some information from collaborators and bits that need to be completed by Oxford R&D. I realise we want an information video for kids who can’t read, but it’ll need to be approved, but we don’t want to go to all the trouble and expense of making it before getting approval. Discuss with helpful person from Oxford R&D, who suggests I write a script for approval. I also book in the film crew, shortlist and interview candidates for research assistant posts on the project, send draft to all collaborators for approval, and ask geneticist collaborator for help with some details. Am finding that progress is slower and slower, because navigating the form is so difficult: it displays one page at a time and does not scroll. You can specify a question to go to, but it’s not easy to remember which questions correspond to which numbered item, and so you end up repeatedly printing out the whole form and shuffling through a mountain of paper to find the relevant question. Keeping things consistent is a big headache.

February 2011

Two weeks’ holiday, then enter final details that were sent to me by collaborators and send the whole lot off to R&D.

The dynamic form starts to reveal its diabolic properties when I enter a new collaborator from Cardiff, only to find that the form now pops up with a new question, along the lines of “How will you meet the requirements of the Welsh Language Act 1993?”. I won’t. We’re studying language, and all our tests are in English, so only English speakers will be recruited. Explain that, and hope it works out.

But now it gets seriously worse. I’ve entered lots of clinical colleagues as “NHS Sites”, but it turns out they aren’t sites. They are Patient Identification Centres. I have to delete them all from the form. Well, I think, at least that makes life simpler. But it doesn’t. Because now they aren’t sites any more, new questions pop up. Who will do the patient recruitment, and how will we pay for it? This one is a Catch 22. Previously our research assistants have been supervised by a consultant to go through records to find relevant cases. Some places required that you get honorary NHS status, and that could necessitate fulfilling other requirements. I actually had to get vaccinated for tetanus as part of getting an NHS contract some years ago. They said it was in case I got bitten by a child, something that has not happened to me in 35 years of researching. But I digress.  Now, it seems, even a fully vaccinated, child-proofed, police-checked researcher is not allowed to go through medical records to identify cases unless patients have given prior consent. Which, of course, they won’t have, since they don’t know about the study.

“Help!” I say to my lovely clinical colleagues. “What do we do now?”. Well, they have a suggestion. If I can register with something called CLRN, then they can help with patient recruitment. I’m given contact details for a research nurse affiliated with CLRN who soothes my brow and encourages me to go the CLRN route. I have to fill in something called a NIHR CSP Application Form which apparently goes to a body called the “portfolio adoption team” who can decide whether to adopt me and my project. All of these forms want a project start date and duration. I did have early April as notional start date, but that’s beginning to look optimistic.

Late February: comments back from R&D. Have been through application with a fine toothcomb and picked up various things they anticipate won’t be liked by the ethics committee. Impressed with the thoroughness and promptness of the response, and found the people at R&D very helpful over the phone, but my goodness, there is a lot to cope with here:

First, it seems I am still in a muddle about the definition of NHS sites, so have filled in bits wrongly that need to be entered elsewhere. Am also confused about the distinction between an “outcome” and an “outcome measure”.

Then there is the question of whether I need “Site specific forms”. The word “site” is starting to cause autonomic reactions in me. Here’s what I’m told: “Please supply an NHS SSI form for each research site; Please note for Patient Identification Centres (PICs)  R&D approval is required but you do not need an SSI form for these provided no research activity takes place on that site – taking consent to take part in the project is a research activity, giving out information on the study/advertising the study is not considered a research activity.”

I also baulk at the suggestion that I should add to the information sheet: “The University has arrangements in place to provide for harm arising from participation in the study for which the University is the Research Sponsor. NHS indemnity operates in respect of the clinical treatment with which you are provided.”  Since I don’t understand what this means, I doubt my participants will, and the participants aren’t receiving any clinical treatment. Out of curiosity, I paste these two sentences into a readability index website. It gives the passage a Flesch-Kincaid Grade Level of 22, with readability score of 4 (on a scale of 0 to 100, where 100 is easy). I try to keep my information sheets at maximum 8th grade level, so reword the bits I do understand and delete the bits that seem irrelevant or incomprehensible.

I reluctantly went along with the idea that I should devise an “Assent form” for children. This is like a kiddie consent form, but with easier language, to be signed by both child and researcher. They seem to be a blanket requirement these days, regardless of the level of risk posed by research procedures. I dislike the Assent form because I am not sure what purpose it serves, other than to make children nervous about what they are getting themselves into. It has no legal status, and we can’t gather psychological test data from unco-operative children. Others share my view that this requirement is incoherent and wrong. But I want to do this study, so feel I have no choice. I had a look on the web and NHS guidance sites to look at suggested wordings, and did not like them, so did a modified and simplified version I hoped would be approved. It would be interesting to do some research on Assent forms to see how they are perceived by children.

March 2011

Hooray! By the start of March, I’m ready to submit my forms. Since  IRAS is all electronic, I had assumed I would do it with a button press, but that would be too simple. Multiple copies must be sent by snail mail within a specific time frame. There has been serious research on the environmental impact of this. But first there is the question of booking an appointment with an ethics committee. There’s a whole centre devoted to this task, and they have standard questions that they ask you about the nature of the research. I was doing well with these until we got to the question about children. Yes, I was going to do research with children. Ah, well then I couldn’t go to any old ethics committee, I had to go to one with a paediatrician. And, unfortunately, there weren’t any slots on committees in Oxfordshire with paediatricians. But, said the helpful girl on the phone, I could try calling the Oxfordshire people directly and they might be able to book me in. At 12.05 I call the number I’ve been given, only to get an automated message saying the office is only open from 10 to 12. Since the following morning I’m busy (I am trying to do my regular job through all this), despair starts to set in. But I break out of a meeting to call them the next morning. The phone rings. And rings. Back to my meeting. Break out again, repeat experience. Eventually I get through. Person at end of phone takes me through the same list of questions about type of research, and finds a convenient slot with an Oxfordshire committee, which I can make if I move an appointment. Move the appointment. Get called back to say that committee can’t unfortunately take me, because they don’t do proposals with children. Am offered another slot on a day when I have arranged to examine a PhD in London. Next one in Oxford is a month later, well after the proposed start date for the research. Best they can do is to offer me a slot with a Berkshire committee, who do have a paediatrician and are just one hour’s drive away, and which is later than the original slot, but sooner than the Oxford one. I decide to go for it. I then receive a remarkable document with a lot of multicoloured writing, which gives me a booking confirmation number, and a lot of instructions.

This triggers a frantic process because you then have seven days to get all the material delivered to the ethics committee. This may not seem difficult, except that all the information sheets and consent forms need to have little header put on them with the booking number and date, and they also want copies of things like a CV, copies of test forms and suchlike, and worse still, there have to be signatures not just from me but also from R&D, who are in a hospital a couple of miles away up a hill. Unfortunately coincides with a period when my PA is absent, and so I rush around like a demented cockroach getting this all together. I’d not budgeted much time for this bit, as I’d assumed submission would involve pressing a button on my computer and uploading some attachments and my diary was full. Somehow I had to find a couple of hours for fiddling with forms, a trip up the hill for a signature the next day, and a journey to the post office to ensure it would all get delivered on time.

I also needed to get the documents to CLRN. This could be done by email, but that soon bounced back. Once more the critical distinction between sites and centres eluded me, and I was told that I had to submit corrected documents because:

 “In Part C, if the only research site is the University of Oxford and the other organisations listed are Participant Identification Centres (PICs), there should be listed under the heading Participant Identification Centre(PIC)Collaborator/Contact immediately below the University of Oxford entry, and not separately.”

So back to the form again to alter this bit. At last it is accepted. But this now triggers new emails, including one from London saying:

“We have been notified that you may be participating in the above study. If the Chief Investigator or Study Coordinator confirms this, Central and East London CLRN will be supporting you locally through the NIHR CSP process and we look forward to working with you on this project.

If this is confirmed, please email all relevant documents to me when you submit your SSI Form through IRAS. The documents you need to submit are listed on the Checklist tab within your SSI Form in IRAS…..etc etc”

The SSI form was one I thought I didn’t have to complete, so I phoned the number given on the email, who said they couldn’t comment and I should ask Oxford, so I asked Oxford, who agreed I didn’t need to do anything.

Meanwhile, there’s yet another form that has popped up that wants to know what training in ethics the researchers have had. Since I haven’t had formal training, I’m told I can either go on a half-day course, or take an on-line course in five modules, each lasting around 45 minutes. I try the online course, but find most of the material is not relevant to me. It starts with pictures of concentration camp victims to emphasise why people need to be protected from reseachers, then goes on to give information focussed on clinical trials. I’m not doing a clinical trial. The quizzes at the end of each module don’t seem designed to check whether you have mastered the subtleties of ethical reasoning, so much as whether you know your way around the bureaucratic maze that is involved in ethical approval, and in particular whether you understand all the acronyms.

April 2011

The six weeks from early March to mid April were joyfully free from communications with ethics people, and normal life resumed. My new staff took up their posts and we made a start on filming for an information DVD for the project, and decided that we would delay the editing stage until after the Berkshire meeting. The day of the committee meeting dawned sunny and bright and I drove off to Berkshire, where I had a perfectly reasonable chat with the ethics committee about the project for about 15 minutes. The Paediatrician was absent. I explained I wanted to assemble the information video, but was told I had to wait until I received a letter documenting changes they’d want me to make. When this arrived, about a week later, they wanted some minor rewording of one sentence. This would be trivial for a written information sheet, but entailed some refilming and careful editing. In addition, the committee raised a point that had not been discussed when I met with them, namely that they were concerned at a statement we had made saying we would give feedback to parents about their children’s language assessment if we found difficulties that had not previously been detected. This, I was told, was an incentive, and I should “soften” the language. This was seriously baffling, as you either tell someone you’ll give them feedback or you don’t. I could not see how to reword it, and I also felt the concern about incentives was just silly. I sent them a copy of a paper on this topic for good measure.

May 2011

Oh frabjous day! At last I receive a letter giving consent for the study to go ahead. I think my troubles are over, and we swing into action with those parts of the project that don’t involve NHS recruitment. But joy is short-lived. I am only just beginning to understand the multifarious ways in which it is possible to Get Things Wrong when dealing with the Circumlocutions Office. I now start to have communications with the CLRN, who want copies of all documentation (including protocol, consent forms, the information video, etc etc - a total of 15 documents) and then tell me:

“The R&D Signature pages uploaded to the doc store on 27th June 2011 do not marry up with the R&D Form uploaded on 15th March 2011”

Requests for new form-filling also come in from the CCRN Portfolio. I’m getting seriously confused about who all these people are, but complete the form anyway.

And, worst still, in August I get a request from TVCLRN for a copy of  the letter I sent to Berkshire in which I responded to their initial comments. I had written it at a time when my computer was malfunctioning so it’s not with other correspondence. I spend some time looking on other computers for an electronic copy.  It seems that without a copy of this letter, they will not be satisfied. Anyhow, I think this will be simple to sort out, and phone the Berkshire ethics committee to ask if they could please send me a copy of the letter that I had written to them. Amazingly, I’m told that “due to GCP guidelines” the Berkshire ethics committee cannot give me a copy. Stalemate. I can’t actually remember how we dealt with this in the end, as my brain started to succumb to Circumlocution Overload.

The last 6 months

We have a meeting with the clinical geneticists with whom we’re collaborating, and I find that most of them are as confused as I am by the whole process. We discuss the Catch 22 situation whereby we aren’t allowed to help go through files to identify suitable patients because of ethical concerns, which means they have to take time out of their busy schedules to do so. This is where the CLRN is supposed to help, by providing research nurses who can assist, but only if we complete loads more paperwork. And having done this, after months of to-ing and fro-ing with requests for documentation or clarification, one of the CLRN centres has just written this week to say they can’t help us at all because they are a Patient Identification Centre and they need to be a PI, whatever that is. I’m currently trying to unravel what this means, and I think it means that they have to become an NHS Site - which was what I  had originally assumed when I started filling in the forms. But in order for them to do so, there are yet more forms to complete.

Meanwhile, in October, I had a request from the UKCRN saying I needed to upload monthly data on patient recruitment in a specific format, and sending me a 35 page manual explaining how to do this. Fortunately, after several exchanges on email, I was able to establish that we did not need to do this, as the hospitals we were dealing with were Patient Identification Centres rather than Sites. But now we have a PIC that wants to become a Site, who knows what new demands will appear?

And then, this week, a new complication. The geneticists who are referring to our study need to check with a child’s GP that it is appropriate to send them the recruitment materials. But an eagle-eyed administrator spotted that this letter “was not an ethically approved form”. I was surprised at this. This is not a letter to a patient; it is a standard communication between NHS professionals. Nevertheless, my R&D contact confirmed that this letter would need approval, and that I’d have to fill in a form for a “substantial amendment”, which would then need to be approved by all the R&D sites as well as the Berkshire ethics committee.

When I expressed my despair about the process on Twitter, I had some comments from ethicists, one of whom said “If you're doing research on ppl then someone has to look after them, no?” Of course, the answer is “yes”, and in fact the project I’m working on does raise important ethical issues. As another commentator pointed out, the problem is not usually with the ethics procedures, and it is true that the IRAS form is much better  than its predecessor and guides you through issues that you need to think about and offers good advice. But the whole process has got tangled up in bureaucratic legal issues and most of my problems don’t have anything to do with protecting patients and have everything to do with protecting institutions against remote possibilities of litigation.

Concluding thoughts

1. In the summer, I was contacted by a member of the public who was concerned about the way in which a medical project done at Oxford University was being used to promote unproven diagnostic tests and treatment for a serious medical condition. I recommended that my contact should write to the relevant person dealing with ethics in the University. I was sanguine that this would be taken seriously: here was an allegation of serious infringement of ethical standards and all my dealings with our R&D department indicated they were sticklers for correct procedures. A month or so passed; they didn’t reply to the complainant. I was embarrassed by this and so wrote to point out that a serious complaint had gone uninvestigated. After a further delay we both got a bland reply that did not answer the specific questions that had been raised and just reassured us the matter was being investigated. This just confirms my cynicism about the role of our systems in protecting patients. As Thomas Sowell pointed out: “You will never understand bureaucracies until you understand that for bureaucrats procedure is everything and outcomes are nothing.” 

2. The current system is deterring people from doing research. The problem is not with the individuals running the system: they’ve mostly been highly professional, helpful and competent, but they are running a modern Circumlocution Office. I’ve interacted with at least 27 people about my proposal, and that’s not counting the Research Ethics Committee members. I’m a few years off retirement and I’ve already decided that I won’t tangle with NHS Ethics again. I’m in the fortunate position that I can do research studies that don’t involve NHS patients, and I want to spend the time remaining to me engaged in the activity I like, rather than chasing pieces of paper so that someone somewhere can file them, or waiting for someone to agree that an innocuous letter from a Consultant to a GP is ethically acceptable.

3. To end on a positive note: I think there is another way. The default assumption seems to be that all researchers are unscrupulous rogues who’ll go off the rails unless continuously monitored. The system should be revamped as a mechanism for training researchers to be aware of ethical issues and helping them deal with difficult issues. For research procedures that are in common use, one can develop standard protocols that document how things should be done to ensure best practice. On this model, a researcher would indicate that their research would follow protocol X and be trusted to do the research in an ethical fashion. The  training would also ensure that researchers would recognise when a study involved ethically complex or controversial aspects that fell outside a protocol, and would be expected to seek advice from the Research Ethics Committee. The training would not revolve around learning acronyms, but would rather challenge people with case studies of ethical dilemmas to ensure that issues such as confidentiality, consent and risk were at the forefront of the researcher’s mind. This is the kind of model we use for people engaged in other activities that could pose risks to others - e.g.,  medical staff, teachers, car-drivers. Life would come to a standstill if every activity they undertook had to be scrutinised and approved. Instead, we train people to perform to a high standard, and then trust them to get on with it. We need to adopt the same approach to researchers if we are not to stifle research activity with human participants.

Further reading

Kielmann T, Tierney A, Porteous R, Huby G, Sheikh A, & Pinnock H (2007). The Department of Health's research governance framework remains an impediment to multi-centre studies: findings from a national descriptive study. Journal of the Royal Society of Medicine, 100 (5), 234-8 PMID: 17470931

Knowles, R. L., Bull, C., Wren, C., & Dezateux, C. (2011). Ethics, governance and consent in the UK: implications for research into the longer-term outcomes of congenital heart defects. Archives of Disease in Childhood, 96(1), 14-20.

Robinson, L., Drewery, S., Ellershaw, J., Smith, J., Whittle, S., & Murdoch-Eaton, D. (2007). Research governance: impeding both research and teaching? A survey of impact on undergraduate research opportunities. Medical Education, 41(8), 729-736.

Warlow, C. (2005). Over-regulation of clinical research: a threat to public health. Clinical Medicine, 5(1), 33-38.

Wilkinson, M., & Moore, A. (1997). Inducement in research. Bioethics, 11, 374-389.

Pioneering treatment or quackery? How to decide

My mother was only slightly older than I am now when she died of emphysema (chronic obstructive pulmonary disease). It’s a progressive condition for which there is no cure, though it can be managed by use of inhalers and oxygen. I am still angry at the discomfort she endured in her last years, as she turned from one alternative practitioner to another. It started with a zealous nutritionist who was a pupil of hers. He had a complicated list of foods she should avoid: I don’t remember much about the details, except that when she was in hospital I protested at the awful meal she’d been given - unadorned pasta and peas - only to be told that this was at her request. Meat, sauces, fats, cheese were all off the menu. My mother was a great cook who enjoyed good food, but she was seriously underweight and the unappetising meals were not helping. In that last year she also tried acupuncture, which she did not enjoy: she told me how it involved lying freezing on a couch having needles prodded into her stick-like body. Homeopathy was another source of hope, and the various remedies stacked up in the kitchen. Strangely enough, spiritual healing was resisted, even though my Uncle Syd was a practitioner. That seemed too implausible for my atheistic mother, whose view was: “If there is a God, why did he make us intelligent enough to question his existence?”

From time to time, friends and relatives of mine have asked my advice about other treatments that are out there. There is, for instance, the Stem Cell Institute in Panama, offering treatment for multiple sclerosis, spinal cord injury, osteoarthritis, rheumatoid arthritis, other autoimmune diseases, autism, and cerebral palsy.  Or nutritional therapist Lucille Leader,  who has a special interest in supporting patients with Parkinson's Disease, Multiple Sclerosis and Inflammatory Bowel Disease. My mother would surely have been interest in AirEnergy, a “compact machine that creates 'energised air' that feeds every cell in your body with oxygen that it can absorb and use more efficiently”.

Another source of queries are parents of the children with neurodevelopmental disorders who are the focus of my research. If you Google for treatments for dyslexia you are confronted by a plethora of options. There is the Dyslexia Treatment Centre, which offers Neurolinguistic Programming and hypnotherapy to help children with dyslexia, dyspraxia or ADHD. Meanwhile the Dore Programme markets a set of “daily physical exercises that aim to improve balance, co-ordination, concentration and social skills” to help those with dyslexia, dyspraxia, ADHD or Asperger’s syndrome. The Dawson Program offers vibrational kinesiology to correct imbalances in the body’s energy fields.  I could go on, and on, and on….

So how on earth can we decide which treatments to trust and which are useless or even fraudulent? There are published lists of warning signs (e.g. ehow Health, Quackwatch), but I wonder how useful they are to the average consumer. For instance, the cartoon by scienceblogs will make skeptics laugh, but I doubt it will be much help for anyone with no science background who is looking for advice. So here’s my twopennyworth. First, a list of things you need to ignore when evaluating a treatment.

1. The sincerity of the practitioner. It’s a mistake to assume all purveyors of ineffective treatments are evil bastards out to make money of the desperate. Many, probably most,  believe honestly in what they are doing. The nutritionist who advised my mother was a charming man who did not charge her a penny - but still did her harm by ensuring her last months were spent on an inadequate and boring diet. The problem is if practitioners don’t adopt scientific methods of evalulating treatments they will convince themselves they are doing good, because some people get better anyway, and they’ll attribute the improvement to their method.

2. The professionalism of the website. Some dodgy treatments have very slick marketing. The Dore Treatment, which I regard as of dubious efficacy, had huge success when it first appeared. Its founder, Wyford Dore was a businessman who had no background in neurodevelopmental disorders but knew a great deal about marketing. He ensured that if you typed ‘dyslexia treatment’ into Google his impressive website was the first thing you’d hit.

3. Fancy-looking credentials. These can be misleading if you aren’t an expert - and sometimes even if you are. My bugbear is ‘Fellow the Royal Society of Medicine’, which sounds very impressive - similar to Fellow the Royal Society (which really is impressive).  In fact, the threshold for fellowship is pretty low, so much so that fellows are told by the RSM that they should not use FRSM on a curriculum vitae. So when you see this on someone’s list of credentials, it means the opposite of what you think: they are likely to be a charlatan. It’s also worth realising that it’s pretty easy to set up your own organisation and offer your own qualifications. I could set up the Society of Skeptical Quackbusters and offer Fellowship to anyone I choose. The letters FSSQ might look good, but carry no guarantee of anything.

4. Testimonials. There is evidence (reviewed here) that humans trust testimonials far more than facts and figures. It’s a tendency that’s hard to overcome, despite scientific training. I still find myself getting swayed if I hear someone tell me of their positive experience with some new nutritional supplement, and thinking, maybe there’s something in it. Advertisers know this: it’s one thing to say that 9 out of 10 cats prefer KittyMunch, but to make it really effective you need a cute cat going ecstatic over the food bowl. If you are deciding whether to go for a treatment you must force yourself to ignore testimonials. For a start, you don’t even know if they are genuine: anyone who regards sick and desperate people as a business opportunity is quite capable of employing actors to pose as satisfied customers. Second, you are given no information about how typical they are. You might be less impressed by the person telling you their dyslexia was cured if you knew that there were a hundred others who paid for the treatment and got no benefit. And the cancer patients who die after a miracle cure are the ones you won’t hear about.

5. Research articles. Practitioners of alternative treatments are finding that the public is getting better educated, and they may be asked about research evidence. So it’s becoming more common to find a link to ‘research’ on websites advertising treatments. The problem is that all too often this is not what it seems. This was recently illustrated by an analysis of research publications from the Burzynski clinic, which offers the opportunity to participate in expensive trials of cancer treatment. I was interested also to see the research listed on the website of FastForword, a company that markets a computerized intervention for children’s language and literacy problems. Under a long list of Foundational Research articles, they list one of my papers that fails to support their theory that phonological and auditory difficulties have common origins. More generally, the reference list contains articles that are relevant to the theory behind the intervention, but don’t necessarily support it. Few people other than me would know that. And a recent meta-analysis of randomized controlled trials of FastForword is a notable omission from the list of references provided. Overall, this website seems to exemplify a strategy that has previously been adopted in other areas such as climate change, impact of tobacco or sex differences, where you create an impression of a huge mass of scientific evidence, which can only be counteracted if painstakingly unpicked by an expert who knows the literature well enough to evaluate what’s been missed out, as well as what’s in there. It’s similar to what Ben Goldacre has termed ‘referenciness’, or the ‘Gish gallop’ technique of creationists. It’s most dangerous when employed by those who know enough about science to make it look believable. The theory behind FastForword is not unreasonable, but the evidence for it is far less compelling than the website would suggest.

So those are the things that can lull you into a false sense of acceptance. What about the red flags, warning signs that suggest you are dealing with a dodgy enterprise? None of these on its own is foolproof, but where several are present together, beware.

1. Is there any theory behind the intervention, and if so is it deemed plausible by mainstream scientists? Don’t be impressed by sciency-sounding theories - these are often designed to mislead. Neuroscience terms are often incorporated to give superficial plausibility: I parodied this in my latest novel, with the invention of Neuropositive Nutrition, which is based on links between nutrients, the thalamus and the immune system. I suspect if I set up a website promoting it, I’d soon have customers. Unfortunately, it can be hard to sort the wheat from the chaff, but NHSChoices is good for objective, evidence-based  information. Most universities have a communications office that may be able to point you to someone who could indicate whether an intervention has any scientific credibility.  
2. How specific is the treatment? A common feature of dodgy treatments is that they claim to work for a wide variety of conditions. Most effective treatments are rather specific in their mode of action.
3. Does the practitioner reject conventional treatments? That’s usually a bad sign, especially if there are effective mainstream approaches.
4. Does the practitioner embrace more than one kind of alternative treatment? I was intriguted when doing my brief research on Fellows of the Royal Society of Medicine to see how alternative interventions tend to cluster together. The same person who is offering chiropractic is often also recommended hypnotherapy, nutritional supplements and homeopathy.  Since modern medical advances have all depended on adopting a scientific stance, anyone who adopts a range of methods that don’t have scientific support is likely to be a bad bet.
5. Are those developing the intervention cautious, and interested in doing proper trials?  Do they know what a randomised controlled trial is? If they aren’t doing them, why not? See this book for an accessible explanation of why this is important.
6. Does it look as though those promoting the intervention are deliberately exploiting people’s gullibility by relying heavily on testimonials? Use of celebrities to promote a product is a technique used by the advertising industry to manipulate people’s judgement. It’s a red flag.
7. Are costs reasonable?  Does the website give you any idea of how much they are, or do you have to phone up for information? (bad sign!). Are people tied in to long-term treatment/payment plans? Are you being asked to pay to take part in a clinical trial? (Very unusual and ethically dubious). Do you get a refund if it doesn’t work? If yes, read the terms and condition very carefully so you understand exactly the circumstances under which you get your money back. For instance, I’ve seen a document from the Dore organisation that promised a money-back guarantee on condition there was ‘no physiological change’. That was interpreted as change on tests of balance and eye movements. These change with age and practice, and don’t necessarily mean a treatment has worked. Failing to improve in reading did not qualify you for the refund.
8. Can the practitioner answer the question of why mainstream medicine/education has not adopted their methods? If the answer refers to others having competing interests, be very, very suspicious. Remember, mainstream practitioners want to make people better, and anyone who can offer effective treatments is going to be more successful than someone who can’t.