Thursday, 26 September 2013

Raising awareness of language learning impairments


A couple of years ago I did a Google search for ‘Specific language impairment’. I was  appalled by what I found. The top hit was a video by a chiropractor who explained he’d read a paper about neurological basis of language difficulties; he proceeded to mangle its contents, concluding that cranial osteopathy would help affected children.

I’ve previously described how I got together with colleagues in 2012 to try and remedy this situation, culminating in a campaign for Raising Awareness of Language Learning Impairments (RALLI). The practicalities have sometimes been challenging but I’m pleased to say that the collection of videos on our RALLI site has now attracted over 90,000 hits, providing an accessible and evidence-based source of information about developmental language impairments. As well as research-based films we have videos with practical information for parents, children and teachers.

So here, for those of you interested in this topic, is an index of what we have so far:

Background to RALLI

Research topics

Information for teachers

Support for parents and children

International
     Spanish translations/subtitled versions
Reference  
Bishop, D. V. M., Clark, B., Conti-Ramsden, G., Norbury, C., & Snowling, M. J. (2012). RALLI: An internet campaign for raising awareness of language learning impairments Child Language Teaching & Therapy, 28 (3), 259-262 DOI: 10.1177/0265659012459467
at 7 comments:

Thursday, 12 September 2013

Evaluate, evaluate, evaluate


© www.CartoonStock.com
When I was starting out on a doctorate, I’d look at the senior people in my field and wonder if I’d ever be like them. It must be great, I thought, to reach the advanced age of 40. By then you’d have learned everything you needed to know to do great science, and you could just focus on doing it. I suspect today’s crop of grad students are a bit more savvy than I was, but all the same, I wonder if they realise just how wrong that picture is – for two reasons.

First, you never stop learning. The field moves on. Instead of getting easier, it gets harder. I remember when techniques such as functional brain imaging first came along. The most competent people in that area were either those who had developed the methods, or young people who learned them as grad students. If you were of the generation above, you had three choices: ignore the methods, spend time learning them, or hire junior people who knew what they were doing. As the methods evolve, they get ever more complex, and meanwhile, your own brain starts to shrink. So if you are anticipating making it to a tenured post and then settling down in your armchair, think again.

Second, the more senior you get, the more of your time is spent, not on doing your own research, but on evaluation. You learn that an email entitled ‘invitation’ should not make your spirits rise: it’s just a desperate attempt to put a positive spin on a request for you to do more work for no reward. You get regular ‘invitations’ to review papers and grants, write job references, appraise promotion bids, sit on interview panels and examine theses. If you are involved in teaching, you’ll also be engaged in numerous other forms of appraisal.

I was prompted to think about this when someone asked on an electronic forum what was a reasonable number of doctoral theses to examine each year. The general consensus was two: though it will obviously depend on what other commitments someone has. It also varies from country to country. There are some jolly places in Europe where a PhD viva is just an excuse for a boozy party with a lot of dressing up in funny gowns and hats. In UK psychology, the whole thing is no fun at all: you have to read a document of 50,000-70,000 words reporting a body of work based on a series of experimental studies. You then write a report on it and see the candidate for a face-to-face viva, which is typically 2 to 3 hours long. Although failure is uncommon, it is not assumed that the candidate will pass (unlike in the viva-as-party countries), and weeping or catatonic candidates are not unheard of. Taking into account travel, etc., if you are going to do a proper job, you are probably talking about three days’ work. For this you get paid around the minimum wage – the fee for examining is typically somewhere between £120 and £200.

So why do we do it? The major reason is because the entire academic enterprise depends on reciprocity: we want people to examine our students and review our papers and grants. In addition, it’s important to maintain standards, and to ensure that degrees, promotions, publications and grants go to those who merit them. But the demands keep growing. In the 37 weeks of this year I’ve been asked to review 76 papers and six grants. I agreed to review 16 papers and three of the grants. This, of course, is nothing compared with being a journal editor or serving on a grants board, something that most of us will do at some point.

Clearly, if I agreed to do everything I was asked, I’d have no time for anything else. Of course, one learns to say no. But awareness of these pressures has made me look with rather a critical eye at how we use evaluation. There is, for instance, research suggesting that job interviews aren’t very useful at identifying good candidates:  we tend to be seduced by immediate impressions, which may not be a good indicator of a person’s suitability. Like most people, I’d be reluctant to take on an employee I hadn’t interviewed, but if Daniel Kahneman is to be believed, this is just because I am a victim of the Illusion of Validity.

I’m a supporter of the peer review system used by journals, and here I feel  I’m on more solid ground, because I can point to instances where my papers have been improved by input from reviewers. Nevertheless, where reviewing is used simply to reject/accept papers or grant proposals,  and where fine-grained decisions have to be made between many high-quality submissions, agreement between experts may be little better than chance (e.g. Fogelholm et al, 2012). Nevertheless, we stick with it, because it’s hard to know what to put in its place.

I’ve written a fair bit about that expensive and time-consuming evaluation process that UK academics engage in, the REF. It requires experts to make judgements of whether, for instance, papers are of 3* or 4* quality, a distinction based on whether the research is “world leading” or “internationally excellent…. but falls short of the highest standards of excellence.” The reliability of such judgements has not, to my knowledge, been evaluated, yet large amounts of funding depend on them. Those on REF committees are in the same situation as Pavlov’s poor dogs, having to make distinctions that are on the one hand impossible (discriminating circles and ellipses that become increasingly similar) and on the other hand very important (get it wrong and you get a shock).

There is one good thing about doing so much evaluation. You have the opportunity to see what others are doing – you may be the first person to read an important new paper, or examine a ground-breaking thesis. You may be forced to engage with different ways of thinking, and confronted with new topics and ideas. You may be able to provide useful input to authors. And since you yourself will be evaluated, it can be useful to see life from the other side of the table, as the person doing the evaluating. But all too often, even these advantages fail to compensate for the fact that as a senior academic you will spend more and more time on evaluation of others and less and less doing your own research.

Reference
Fogelholm, Mikael, Leppinen, Saara, Auvinen, Anssi, Raitanen, Jani, Nuutinen, Anu, & Väänänen, Kalervo (2012). Panel discussion does not improve reliability of peer review for medical research grant proposals Journal of Clinical Epidemiology, 65 (1), 47-52 DOI: 10.1016/j.jclinepi.2011.05.001
at 1 comment: