Friday, 18 April 2014

Data analysis: ten tips I wish I'd known sooner

I enjoy analysing data and I've been doing it for years, but I still do things inefficiently. All the same, I have learned some things along the way. As I work through another complex dataset, I thought it worth sharing some of the most useful tips I've picked up to make life simpler and smoother.  Some are very elementary and they will mostly be of relevance to psychologists who use Excel and SPSS to do fairly straightforward analyses, though some points are more generic.

A lot of these tips have to do with deploying those frontal lobes and thinking ahead: you need to be aware of three things:
  • How you set up a dataset can make a big difference to how easy it is to analyse later on
  • You will not remember anything about your data in a few years (or even months) time
  • There are increasing moves towards data-sharing - i.e. making your data available on a public repository and you need to plan for that too.

1. Label your subjects consistently and anonymously

You will almost certainly be required to anonymise data collected from human subjects. It is amazing how often people sign up to this in their ethics application but then forget all about it. I've seen supposedly anonymised data identified by people's names, initials and/or dates of birth. Don't do it!

I find simple consecutive number codes work fine, with a prefix denoting which study the subjects  come from. There are two things to think about, in addition to anonymisation. First, will it be useful to be able to sort subjects by code number into specific groups? For instance, if you have three groups identified as young, middle-aged and old, you might think of labelling them with Y1, Y2, .... M1, M2 etc.  But in certain computing systems, this will mean that when you generate output, e.g. means for each group, they will occur in alphabetic order, so M then O then Y.  So it may be  better to use codes that will follow a natural sequence - this means when you want to paste output into a table you don't need to fiddle about with it.

Another thing to note is that if you have a program that treats codes as alphabetic, then if you have, say, 20 subjects, and you sort them, they will come out in the order: S1, S10, S11, S12, S13, S14, S15, S16, S17, S18, S19, S2, S20, S3, S4, S5, S6, S7, S8, S9. This mildly irritating feature can be avoided if you ensure all codes are the same length, e.g. S01, S02, and so on.

On the other hand, if you are generating raw data files that you want to process automatically through some analysis package, make sure you have codes that can be easily read in a loop. It's much easier to tell a program to sequentially analyse files beginning with S and ending in numbers 1 to 20, than it is to type in each subject code separately.

And if you are going to analyse a whole set of files using an automated procedure, use a logical folder structure to organise your files. I've been stymied by finding that someone has carefully organised data files so that all the data for one child on various tasks are in one folder, and these folders are then within other folders that group them by age. I prefer it if all the files that are to be analysed together are kept together: provided the file-naming system is well-constructed, there should not be any chance of confusing who is who, and you can then point the analysis program just to one relevant folder, without a lot of if statements.

2. Label your variables in a consistent and intuitive fashion that will work across platforms

If you are going to end up analysing your results in SPSS, start out with variable names that SPSS will accept - i.e. no blanks, leading numbers or prohibited characters should be included. Think ahead to the paper you plan to write about the results and consider the variable names you will use there. I seldom obey my own advice here, but it's a common source of irritation to reviewers if you are inconsistent in how you refer to a variable. Much better to start as you mean to go on as far as is possible.

3. Use 'freeze panes' in Excel

This is about as basic as it gets, but I'm surprised at how many people don't know about it. In Excel, when you scroll down or across your file, the variable names or subject IDs scroll off the screen. If you place your cursor in the cell just below the variable names and just to the right of the subject IDs (i.e. the first cell of data), and select View|Freeze panes, the rows and columns above the current cell will stay put when you scroll. If you get it wrong, you can always unfreeze.

There's a rather clunky method that lets you freeze panes in SPSS, which can be useful if you want to see subject IDs while scrolled over to the right: see here for instructions

4. Keep an explanatory list of your variables

In SPSS, the 'labels' field can be useful for keeping a record of what the variable is, but it is limited in length and sometimes more detail is needed. It's worth keeping a data coding file which lists all variables and gives a brief description of what they are, what missing value codes are, and so on. This is absolutely critical if you plan to deposit data in an archive for sharing. It's good practice to work as if that is going to be the case.

5. Use one big file, rather than lots of little files, and hide variables that you aren't currently using

If you have a large dataset, it's tempting to break it up to make it more manageable. You may have hundreds of variables to contend with. You may therefore be tempted to pull out variables of current interest and store in a separate file. The problem is that if you do that, it's easy to lose track of where you are.

Suppose you have a giant master file, and you select a few variables to make a new file, and then in the new file you find an error. You then need to correct it in both files, which is tedious - so tedious that you may not bother and will then end up with different versions and be unsure which is correct.

Also, you may want to look at relationships between the variables you have extracted and other variables in the master file. This involves more fiddling with files, which is a good way of generating errors, especially if you use cut and paste.  So my advice is stick with one master file, which is scrupulously labelled with a version number when you update it. You can avoid the 'too much data' problem by just hiding bits of file that aren't currently in use.

In Excel, it's simple to just hide rows or columns that you aren't using. You just select the columns you want to hide and select View|Hide. If you want to see them again, you select the columns adjacent to the hidden columns and select View|Unhide.

You can do a similar thing in SPSS by defining Variable Sets. It's a bit more fiddly than the Excel operation, but once you have defined a variable set, it is saved with the file and you can reselect it very easily. Instructions for doing this are here.

This website, incidentally, is a treasure trove of useful advice, including instructions on  how to produce APA formatted tables from SPSS output.

6. Never name a file with the suffix 'final' and always back up key data

It's just tempting fate to call a file 'final'. You will end up with files called 'final final' or 'really final' or 'final v.10'. Better to use dates to identify the most recent version.

The back-up advice is blindingly obvious but even the most seasoned of us still forgets to do it. If you've put in several days' work on something, you need to have it stored safely in case your computer dies or gets stolen.

7. Look at your data

Before rushing into an analysis, it is important to plot your data. This can be a good way of picking up gremlins in the dataset, such as values which are out of range. For instance, if you have raw scores and scaled scores on a variable, plot one vs the other in a scatterplot - depending on whether there is age variation in the sample, scores should either fall on a straight line, or at least cluster around it.
You should also check whether variables are normally distributed, and if not consider how to deal with this if your analysis assumes normality. Andy Field's textbook Discovering Statistics Using SPSS gives helpful advice on this general topic (and many more!).

8. Keep a log of your analysis

I find it helpful to think through what I want to do, then make a list of the analyses and outputs that I need and work through them systematically. In practice, things seldom go smoothly. You may find that a subject's file is missing, or corrupted, or that data need transforming in some way. It's traditional to work with a lab book in which such things are recorded, though I prefer to do this electronically when at the analysis stage, by just keeping a Word document open, and making a note of everything as I go along, under the relevant date.

In this log I note the names of the files I've created and their location and what they do. This really is necessary if you are to avoid a future time when you have to wade through numerous versions of numerous files trying to find the one you can only dimly remember creating.

I also record problems in the log. For instance, suppose you have a task that was wrongly administered in a couple of cases so the data will have to be excluded. The log is where you record the date at which this was noted, which data were affected and what action was taken. If you don't do this, you may well find that you, or someone else, comes back to the dataset at a later time and cannot make sense of it. Or, as I have done all too often, spends a morning discovering the same problem that was already found months previously.

9. Use scripting in SPSS

The best advice of all is to keep a script of your analysis if you are working in SPSS. Even if you know nothing about scripting, this is very easy to do. Set up your analysis using the menu, and instead of hitting OK, you hit Paste. This opens a script window showing the script-version of the commands you have selected. You can then select all or part of the script and run it by hitting the big green arrow at the top of the script window. If you then run another analysis, and again hit Paste, the new syntax will be appended at the bottom of the script.

There are four advantages to doing things this way:
  • You can save the script and thus have a permanent record of the analysis you have carried out. This complements the log you are keeping (and you can save its name in the log).
  • You can annotate the script. Any text you type in that begins with an asterisk and ends with a full stop is treated as a comment. You can use this to remind yourself of what the analysis does.
  • You can come back and re-run the analysis. Suppose you find that you had an error in your dataset and you had to correct one person's data. Instead of having to laboriously reconstruct the whole analysis, you just  re-run the script.
  • This is a good way to learn about scripting. A lot can be picked up by just taking the auto-generated script as a model and tweaking things like the variables that are analysed. If you want to do a parallel analysis on several variables, it is much easier to copy and paste the relevant section of a script and change one or two variable names than to go through a menu.

10. Check all analyses before publishing

It is really important to check all your analyses before you submit a paper. Always go through a paper and make sure that all reported tables and analysis are reproducible. It is amazing how often you re-run an analysis and things come out differently. It can seem sometimes that a malign spirit inhabits your computer and is just teasing you. Even if the  differences are not serious, they can still be a source of worry. In my experience, they often have to do with things like forgetting to select the correct subset of participants for a specific analysis - if, - for instance, there are exclusionary criteria. Or a missing data code may have been  inadvertently been treated as data. In the worst case you may find you cut and pasted something into a file forgetting that the file had been sorted in a different order. And sometimes, just sometimes, both Excel and SPSS files can get corrupted. It can be useful to have data in both formats for an overall consistency check on things like mean values.

This is where SPSS scripting comes into its own. The whole business of re-running the analyses becomes much more tractable with a script. Furthermore, in a final check, you can further annotate the script, noting which table in the paper corresponds to a particular output and so have a nice, clear account of what you did to refer back to.

Saturday, 12 April 2014

WeSpeechies: A meeting point on Twitter

As an avid Twitter user, I'm always interested in new developments in social media, and so I was intrigued when Caroline Bowen (@speech_woman) and Bronwyn Hemsley (@bronwynhemsley) invited me to curate a session of @WeSpeechies for a week.
I said yes immediately, though I had no real idea (and still have no real idea) of what I was getting myself into. But I knew that Caroline had done an enormous amount to encourage 'Speechies' (i.e., Speech and Language Therapists in the UK/Speech-Language Pathologists elsewhere) to use the internet. In 1998, she started a web-based resource: this is a mine of useful information, including links to assessments, interventions and evidence-based information.
I'm not a speechie myself –  I'm a full-time researcher with a Principal Research Fellowship from the Wellcome Trust and my background is neuropsychology -   but I've always worked closely with the profession, because of my interest in children's communication impairments. So it's great to have the opportunity to interact with those who work at the coalface, and @WeSpeechies seems like a great idea.
But how does it work? Well, there's a description here.
In brief, the idea is that the Twitter handle @wespeechies is taken over by someone – the 'curator' – for a week, with the idea that they can foster interesting exchanges on Twitter. What's more, the interactions can be preserved: this can be helped if those interacting with @wespeechies remember to add the hashtag #WeSpeechies to their tweets.This way we can build an archive of useful and interesting interactions.
Now, of course, most of us are not glued to Twitter all day, even though we may give that impression, and there can be difficulties in interacting across time zones. As curator, I plan to check in a couple of times every day during the week to respond to tweets, but Caroline and Bronwyn wanted also to ensure that there is an opportunity for some live chat. They therefore include in the course of the week a "Tuesday Chat", which involves me chairing a Q&A session. Caroline and Bronwyn have kindly given me an easy topic for my first attempt, namely Apprehensive Academics on Twitter – inspired by this blogpost.
I'm going to do this at 8 to 9 in the morning on Tuesday 15th April 2014 on British Standard Time. This means that the live chat should appeal to night owls in North America, and larks in UK and Europe, whereas for those in Australasia it will be afternoon or early evening.
If you'd like to take part, please feel free to respond to any message from @wespeechies, and remember to include #WeSpeechies in your message. I've generated some questions to get the ball rolling; if you want to answer one of these, please make it clear which one by including, for instance, A1, A2, A3 etc at the start of your Tweet.
Q1 What makes for an effective/ineffective tweet? #WeSpeechies

Q2 Have you got any tips for attracting followers? #WeSpeechies

Q3 Are there downsides to Twitter? #WeSpeechies 
Q4 What unexpected benefits of Twitter have you found?  #WeSpeechies

Q5 Does your employer have an issue or policy on staff being on Twitter, that you know about?  #WeSpeechies
Please note that the idea of #WeSpeechies is to engage and connect with those who work as SALTs/SLPs or who have shared interests in problems affecting communication and/or swallowing. You don't need to be an expert – on the contrary, part of the fun of the exercise is that we can all share experiences and maybe learn something new. The hope is that, if you take part, you may find new contacts around the globe: people with shared interests who can make you feel more of a world-wide community.
A final note of warning: WeSpeechies does not encourage advertisers, spammers or trolls. It is not the place to try and sell or endorse a product. Anyone who does not abide by the basic @WeSpeechies guidelines will be warned and then blocked if the advertising or unsuitable promo Tweets continue.

If you are interested in speechie issues, I do hope you will participate by following @WeSpeechies. I'll be taking over the twitter handle on Sunday 13th April for a week, so look forward to meeting you there!

P.S. 15th April 2014
The Live Chat was fun, if somewhat frenetic.
If you missed it, you can see the tweets collated here:

Friday, 4 April 2014

Bishopblog catalogue (updated 4th April 2014)


Those of you who follow this blog may have noticed a lack of thematic coherence. I write about whatever is exercising my mind at the time, which can range from technical aspects of statistics to the design of bathroom taps. I decided it might be helpful to introduce a bit of order into this chaotic melange, so here is a catalogue of posts by topic.

Language impairment, dyslexia and related disorders
The common childhood disorders that have been left out in the cold (1 Dec 2010) What's in a name? (18 Dec 2010) Neuroprognosis in dyslexia (22 Dec 2010) Where commercial and clinical interests collide: Auditory processing disorder (6 Mar 2011) Auditory processing disorder (30 Mar 2011) Special educational needs: will they be met by the Green paper proposals? (9 Apr 2011) Is poor parenting really to blame for children's school problems? (3 Jun 2011) Early intervention: what's not to like? (1 Sep 2011) Lies, damned lies and spin (15 Oct 2011) A message to the world (31 Oct 2011) Vitamins, genes and language (13 Nov 2011) Neuroscientific interventions for dyslexia: red flags (24 Feb 2012) Phonics screening: sense and sensibility (3 Apr 2012) What Chomsky doesn't get about child language (3 Sept 2012) Data from the phonics screen (1 Oct 2012) Auditory processing disorder: schisms and skirmishes (27 Oct 2012) High-impact journals (Action video games and dyslexia: critique) (10 Mar 2013) Overhyped genetic findings: the case of dyslexia (16 Jun 2013) The arcuate fasciculus and word learning (11 Aug 2013) Changing children's brains (17 Aug 2013) Raising awareness of language learning impairments (26 Sep 2013) Good and bad news on the phonics screen (5 Oct 2013) What is educational neuroscience? ( 25 Jan 2014) Parent talk and child language ( 17 Feb 2014) My thoughts on the dyslexia debate ( 20 Mar 2014)

Autism diagnosis in cultural context (16 May 2011) Are our ‘gold standard’ autism diagnostic instruments fit for purpose? (30 May 2011) How common is autism? (7 Jun 2011) Autism and hypersystematising parents (21 Jun 2011) An open letter to Baroness Susan Greenfield (4 Aug 2011) Susan Greenfield and autistic spectrum disorder: was she misrepresented? (12 Aug 2011) Psychoanalytic treatment for autism: Interviews with French analysts (23 Jan 2012) The ‘autism epidemic’ and diagnostic substitution (4 Jun 2012)

Developmental disorders/paediatrics
The hidden cost of neglected tropical diseases (25 Nov 2010) The National Children's Study: a view from across the pond (25 Jun 2011) The kids are all right in daycare (14 Sep 2011) Moderate drinking in pregnancy: toxic or benign? (21 Nov 2012)

Where does the myth of a gene for things like intelligence come from? (9 Sep 2010) Genes for optimism, dyslexia and obesity and other mythical beasts (10 Sep 2010) The X and Y of sex differences (11 May 2011) Review of How Genes Influence Behaviour (5 Jun 2011) Getting genetic effect sizes in perspective (20 Apr 2012) Moderate drinking in pregnancy: toxic or benign? (21 Nov 2012) Genes, brains and lateralisation (22 Dec 2012) Genetic variation and neuroimaging (11 Jan 2013) Have we become slower and dumber? (15 May 2013) Overhyped genetic findings: the case of dyslexia (16 Jun 2013)

Neuroprognosis in dyslexia (22 Dec 2010) Brain scans show that… (11 Jun 2011)  Time for neuroimaging (and PNAS) to clean up its act (5 Mar 2012) Neuronal migration in language learning impairments (2 May 2012) Sharing of MRI datasets (6 May 2012) Genetic variation and neuroimaging (1 Jan 2013) The arcuate fasciculus and word learning (11 Aug 2013) Changing children's brains (17 Aug 2013) What is educational neuroscience? ( 25 Jan 2014)

Book review: biography of Richard Doll (5 Jun 2010) Book review: the Invisible Gorilla (30 Jun 2010) The difference between p < .05 and a screening test (23 Jul 2010) Three ways to improve cognitive test scores without intervention (14 Aug 2010) A short nerdy post about the use of percentiles (13 Apr 2011) The joys of inventing data (5 Oct 2011) Getting genetic effect sizes in perspective (20 Apr 2012) Causal models of developmental disorders: the perils of correlational data (24 Jun 2012) Data from the phonics screen (1 Oct 2012)Moderate drinking in pregnancy: toxic or benign? (1 Nov 2012) Flaky chocolate and the New England Journal of Medicine (13 Nov 2012) Interpreting unexpected significant results (7 June 2013)

Journalism/science communication
Orwellian prize for scientific misrepresentation (1 Jun 2010) Journalists and the 'scientific breakthrough' (13 Jun 2010) Science journal editors: a taxonomy (28 Sep 2010) Orwellian prize for journalistic misrepresentation: an update (29 Jan 2011) Academic publishing: why isn't psychology like physics? (26 Feb 2011) Scientific communication: the Comment option (25 May 2011) Accentuate the negative (26 Oct 2011) Publishers, psychological tests and greed (30 Dec 2011) Time for academics to withdraw free labour (7 Jan 2012) Novelty, interest and replicability (19 Jan 2012) 2011 Orwellian Prize for Journalistic Misrepresentation (29 Jan 2012) Time for neuroimaging (and PNAS) to clean up its act (5 Mar 2012) Communicating science in the age of the internet (13 Jul 2012) How to bury your academic writing (26 Aug 2012) High-impact journals: where newsworthiness trumps methodology (10 Mar 2013) Blogging as post-publication peer review (21 Mar 2013) A short rant about numbered journal references (5 Apr 2013) Schizophrenia and child abuse in the media (26 May 2013) Why we need pre-registration (6 Jul 2013) On the need for responsible reporting of research (10 Oct 2013) A New Year's letter to academic publishers (4 Jan 2014)

Social Media
A gentle introduction to Twitter for the apprehensive academic (14 Jun 2011) Your Twitter Profile: The Importance of Not Being Earnest (19 Nov 2011) Will I still be tweeting in 2013? (2 Jan 2012) Blogging in the service of science (10 Mar 2012) Blogging as post-publication peer review (21 Mar 2013) The impact of blogging on reputation ( 27 Dec 2013)

Academic life
An exciting day in the life of a scientist (24 Jun 2010) How our current reward structures have distorted and damaged science (6 Aug 2010) The challenge for science: speech by Colin Blakemore (14 Oct 2010) When ethics regulations have unethical consequences (14 Dec 2010) A day working from home (23 Dec 2010) Should we ration research grant applications? (8 Jan 2011) The one hour lecture (11 Mar 2011) The expansion of research regulators (20 Mar 2011) Should we ever fight lies with lies? (19 Jun 2011) How to survive in psychological research (13 Jul 2011) So you want to be a research assistant? (25 Aug 2011) NHS research ethics procedures: a modern-day Circumlocution Office (18 Dec 2011) The REF: a monster that sucks time and money from academic institutions (20 Mar 2012) The ultimate email auto-response (12 Apr 2012) Well, this should be easy…. (21 May 2012) Journal impact factors and REF2014 (19 Jan 2013)  An alternative to REF2014 (26 Jan 2013) Postgraduate education: time for a rethink (9 Feb 2013) High-impact journals: where newsworthiness trumps methodology (10 Mar 2013) Ten things that can sink a grant proposal (19 Mar 2013)Blogging as post-publication peer review (21 Mar 2013) The academic backlog (9 May 2013) Research fraud: More scrutiny by administrators is not the answer (17 Jun 2013) Discussion meeting vs conference: in praise of slower science (21 Jun 2013) Why we need pre-registration (6 Jul 2013) Evaluate, evaluate, evaluate (12 Sep 2013) High time to revise the PhD thesis format (9 Oct 2013) The Matthew effect and REF2014 (15 Oct 2013) Pressures against cumulative research (9 Jan 2014) Why does so much research go unpublished? (12 Jan 2014) 

Celebrity scientists/quackery
Three ways to improve cognitive test scores without intervention (14 Aug 2010) What does it take to become a Fellow of the RSM? (24 Jul 2011) An open letter to Baroness Susan Greenfield (4 Aug 2011) Susan Greenfield and autistic spectrum disorder: was she misrepresented? (12 Aug 2011) How to become a celebrity scientific expert (12 Sep 2011) The kids are all right in daycare (14 Sep 2011)  The weird world of US ethics regulation (25 Nov 2011) Pioneering treatment or quackery? How to decide (4 Dec 2011) Psychoanalytic treatment for autism: Interviews with French analysts (23 Jan 2012) Neuroscientific interventions for dyslexia: red flags (24 Feb 2012)

Academic mobbing in cyberspace (30 May 2010) What works for women: some useful links (12 Jan 2011) The burqua ban: what's a liberal response (21 Apr 2011) C'mon sisters! Speak out! (28 Mar 2012) Psychology: where are all the men? (5 Nov 2012) Men! what you can do to improve the lot of women ( 25 Feb 2014)

Politics and Religion
Lies, damned lies and spin (15 Oct 2011) A letter to Nick Clegg from an ex liberal democrat (11 Mar 2012) BBC's 'extensive coverage' of the NHS bill (9 Apr 2012) Schoolgirls' health put at risk by Catholic view on vaccination (30 Jun 2012) A letter to Boris Johnson (30 Nov 2013) How the government spins a crisis (floods) (1 Jan 2014)

Humour and miscellaneous Orwellian prize for scientific misrepresentation (1 Jun 2010) An exciting day in the life of a scientist (24 Jun 2010) Science journal editors: a taxonomy (28 Sep 2010) Parasites, pangolins and peer review (26 Nov 2010) A day working from home (23 Dec 2010) The one hour lecture (11 Mar 2011) The expansion of research regulators (20 Mar 2011) Scientific communication: the Comment option (25 May 2011) How to survive in psychological research (13 Jul 2011) Your Twitter Profile: The Importance of Not Being Earnest (19 Nov 2011) 2011 Orwellian Prize for Journalistic Misrepresentation (29 Jan 2012) The ultimate email auto-response (12 Apr 2012) Well, this should be easy…. (21 May 2012) The bewildering bathroom challenge (19 Jul 2012) Are Starbucks hiding their profits on the planet Vulcan? (15 Nov 2012) Forget the Tower of Hanoi (11 Apr 2013) How do you communicate with a communications company? ( 30 Mar 2014)

Sunday, 30 March 2014

How do you communicate with a communications company?


Big companies exist to make profits. They don't like paying tax and they don't like paying employees. They've learned that in the modern world, they can outsource many of their customer services to countries with cheaper labour, but they are even happier when they can dispense with people altogether to move on to automated systems.  These can be really efficient for dealing with routine enquiries. Problem is, when it comes to non-routine enquiries, they really suck.

I discovered this when I got a new mobile and a contract with TMobile. I signed up for monthly payments by direct debit and was happy when I received my first bill and it all seemed in order.  But I then got a mysterious text message that purported to come from 150, which is the number for Tmobile enquiries. It read:

"We recently contacted you regarding your unpaid T-Mobile bill. Just to remind you that payment can be made simply by dialling 360 from your handset."

This was odd because (a) I had no memory of a previous contact from them and (b) I was paying by direct debit and could not see how I had incurred an unpaid bill. I decided it was probably some kind of scam, but thought I should just check it out with TMobile. But here my problems began.
I thought the easiest thing would be to register for online billing, and so I tried the website on the bill. This immediately brought up a scary warning screen, as follows:

It seemed odd to me that a major UK company would have a web address that did this, so I did a bit of snooping around with Google to make sure this was really OK before over-riding the warning and going ahead. Usual thing of entering all my details, password, security questions and so on. Seemed OK and I was told I'd receive a PIN by text. And at that moment, sure enough, a message with a PIN came through on the phone. Nearly there, I thought.

Unfortunately, though, the webpage with the box for the PIN was greyed out and would not accept any input. I tried a few times, both with my original browser, and then with another one. Same problem (once I got past the scary warning message). Tried on my ipad, and this time the whole thing froze before I even got to the PIN bit.

Well, the bill suggested that if I needed help I could ring 150, so I did. This gave me four automated options: 1) make a payment; 2) get details of my first bill; 3) help setting up voicemail; 4) get date of mobile number move. Since none applied, I pressed * for other options. This gave me two options: 1) help with my mobile phone; and 2) help with home broadband. If you select neither or those, you just get a patient voice saying "Let me go through those options again..."

I'd previously battled with the TMobile website to try and find a phone number for human help, and completely failed. All the encouraging tabs offering help lead you to a hopeless set of FAQs that are totally irrelevant to the problem at hand - or they try to lure you down the 150 cul-de-sac.

I tried Twitter. It began well, with this exchange:

However, when I then sent details of the two problems (suspicious looking text message and inability to enter PIN in website), I got this:

Now, to me the idea of a firm asking you to give them your full details "for security" over Twitter is absurd. When I suggested I'd rather communicate secure information over email, I was told:

So there we have it. A communications company who will do anything other than communicate. Because employing people costs money, and they clearly don't care if their customers get stuck in a phone options cul de sac, or spend hours on the web trying to find how to fix a problem.

I've resorted to writing a letter to Customer Services, but I suspect I won't get a reply, or if I do, it will be written by a bot. Will let you know if I do!

Update: 1st April 2014
Probably not a good day to relate my further attempts to communicate with TMobile, but here we go:

I happened to be in town so, armed with my contract and invoice, I dropped in at the friendly Oxford Phones4U store where I'd bought my TMobile contract and explained the problem. They were immediately able to tell me that others were having the same problem and it seemed to be a glitch in the network. So why couldn't the @TMobileUK twitter people tell me that yesterday? It would have immediately have reassured me.

But Phones4U went further. They felt that for my peace of mind, they should phone TMobile so I could hear it from the horse's mouth. The patient young man who was trying to help me sat dialling numbers and jotting down other numbers. "It's very frustrating," he said. "I can't get past the automated system." He dialled some mega-number where he gave someone his store code: but still the advice was to call 150, the communications black hole. After 10 minutes I had to leave. He gave me the last number he'd been given, which he thought would get me through to a human, but I had a busy afternoon and have not had a chance to try it.

I got back and found that TMobileUK had been active on Twitter, perhaps stung by adverse comments from another Tweeter:

I checked out the link re websites. It did not refer to the two problems I'd had - i.e. security warning followed by unresponsive screen, but at least they seemed to be doing something.
We continued:

Ah, the elusive 'dedicated customer services team'. This got short shrift from another user and further queries from me:

So far, this has not elicited any other response. But I did find this message that TMobileUK had broadcast to the world a couple of hours earlier:

So at last, an admission that there was an 'on-going online bill issue'. That's all I needed to know. And they don't need anyone's private details in order to tell us about it.

I'm still hoping that at some point I'll hear that they have fixed their website. But I suspect that they are much slower than most companies at responding to problems because there's no way anyone can actually tell them that there is a problem.

In this way, their cost-cutting strategy of replacing people with robots is counterproductive, as it can lead to major snarlups escalating much further than they need.

Update 3rd April 2014

Well, the good news is that my letter did receive a reply from one Phillip Wilson from EE Customer Service. The bad news is that it was no help at all.
He wrote:
I have reviewed your account and although there are no notes about an SMS being sent to you about your balance I can see that there is £xx outstanding which is now with our Collections Department. There is no Direct Debit set up on the account. Your number has been suspended for non payment since 29 March 2014. I am sorry to hear that you were unable to log onto the online account system. I would advise you to discuss your balance with the Collections Department on 08454122801.
Mr Wilson also rather sadly remarked: "I would have liked to discuss this issue with you, unfortunately I was unable to contact you so I am responding in writing." I know just how he feels.
I called the collections department and got through to a nice lady. I explained to her that I had all the paperwork for my direct debit, plus a first bill that stated "We will collect £11.00 on or around 22/4/14". Since she focused on debt collections, and did not apparently have any record of my account, she could not help. She said she'd put me through to the right department, but after five minutes of listening to awful tinny music I gave up and got back to the day job.
If anyone knows of a mobile phone company that has competent, efficient customer service personnel, please do let me know, as I rather urgently need to find one before I lose my sanity.

Update 9th April 2014
So, after the last abortive attempt to communicate with TMobile, I thought I'd give them one last chance. I wrote another letter, this time enclosing a copy of the direct debit mandate I had set up on March 6th, and also a copy of a bill from them that said they would be collecting payment for my next bill - implying they had a direct debit. 
Alas the only response I got was a peremptory demand for £11 and a statement that my phone had meanwhile been cut off for nonpayment.
I've had some suggestions from happy customers of alternative providers, but my only concern is that there seems a high probability that the minute I try and cancel the account, TMobile will suddenly discover and activate the direct debit. Watch this space....

11th April update
At last! A resolution!

When last in town, I'd found that O2 offered a good deal on a contract that would suit my phone, and I was keen to change. However, I realised it might not be easy to extricate myself from TMobile, and so I decided to ring the TMobile number I'd been given by Phones4U, which they claimed would lead to a person. It did begin with automated options, but one sounded relevant - changing my plan - and indeed this produced a human voice.

I explained the situation, only to be told that I was not in arrears, I had a direct debit set up, and my phone should now be working. Baffled, I asked about the date on which the direct debit had been activated. More waiting and tinny music, but eventually someone came on the line who seemed to know about my case. The direct debit had been activated the previous day and the suspension on my phone account lifted. And indeed, when I checked again, the phone was now operational.

I was surprised that nobody had bothered to tell me any of this, but later that afternoon, I did get another email from Phillip WIlson of EE Customer Services (my comments in red)

I am sorry for the delay in responding to you previously. I sent you a letter response yesterday which will arrive with you shortly. The letter I sent is below.
Thank you for your letter which I have received into the Customer relations Department today. The letter was in relation to the correspondence I sent you recently about the Direct Debit on your account. I would have liked to discuss this issue with you; unfortunately I was unable to contact you so I am responding in writing. [Would indeed be difficult to contact me as they had cut me off!]
I am sorry to hear that you were unable to speak with the Collections Department when calling the number I provided to you. [I did speak to the Collections Dept - they did not understand why I'd been given their number but tried to put me through to another dept]. I know it is frustrating when you have to wait on the line for an agent to answer. [Not just frustrating - quite impossible for anyone who has a demanding day job]. I can see from the account that the Direct Debit was set up when the account was created, however it failed to take the first bill as a Direct Debit does take 14 days to set up. [Bizarre - the direct debit was dated 6th March. I was cut off on 29th March]. The first bill always has to be paid manually using a Debit/Credit Card. [So why did nobody tell me that?!]. As the Direct Debit failed it was cancelled from the account. I have re-set the Direct Debit for you today. As a gesture for your issues I have also waived the £12 balance on the account.
I would be unable to disconnect your agreement for you as you are in contract until  6 March 2015. To early disconnect your agreement would cost  £15.61. As stated I have cleared the balance and set up your Direct Debit again for you.

I am delighted to be able to report this resolution of my problems, and I appreciate that TMobile at last has shown some recognition of the snarl-up by waiving the outstanding sum.

But really, it's not about money. It's about a catastrophic fail in customer relations that allowed a trivial issue to escalate into a prolonged exchange with numerous TMobile staff, damaging my sanity and TMobile's reputation - and presumably costing them far more than £12 in staff time.

In the sequence of events above, there are no less than six points at which this issue could have been resolved, yet TMobile failed to act appropriately every time:

  • The most pressing need is for a telephone help line that had a non-automated option for non-standard problems. TMobile's 150 number is hopeless in this regard. Their website is just as bad - everything automated and no opportunity to contact a person.
  • The staff manning the Twitter account @TMobileUK should have been able to sort out my problem, but they were unwilling or unable to communicate with me by phone or email, which severely limited their effectiveness, especially when they expected me to provide confidential information.
  • The @TMobileUK staff showed a striking lack of initiative in failing to respond to my request for instructions how to contact a human operator via 150. Even if they did not know how to do this, they could have explored the options themselves to give me the sequence of keystrokes I had requested. They did not.
  • The initial letter from Customer Services was remarkably unhelpful. Having said he had tried to contact me by phone, Mr Wilson should have given me a number where I could get back to him. Had he done so, I suspect this could have been sorted out promptly. Instead, I was referred to the debt collections department.
  • If Mr Wilson had told me in his first letter that there was a time lag on direct debits and that I needed therefore to pay by credit card I would have done so. Instead, he said I did not have a direct debit and implied I was untruthful in saying I had one.
  • When I called the Collections Department, I had given them all my details including a landline number where I could be reached. After I gave up waiting for a reply, they could have called me back but did not.
The final paragraph of the last letter from Mr Wilson does, of course, explain everything. Once you've signed the contract, you are stuck with them, unless you are willing to pay to terminate it. So they have absolutely no incentive to treat customers well, and their customer service department is set up like an obstacle course to deter all but the most persistent people.  

I am so sick of TMobile's lack of regard for its customers that I was tempted to terminate the contract anyhow, and blow the expense, but of course that would all take time and it's the one thing I don't have much of. So for the moment,  sticking with the current contract is the least bad option. I just hope that the increasing power of the internet enabling customers to document experiences like mine might eventually trickle through to affect the company's policy.