Why I still use Excel

The Microsoft application, Excel, was in the news for all the wrong reasons last week.  A paper in Genome Biology documented how numerous scientific papers had errors in their data because they had used default settings in Excel, which had unhelpfully converted gene names to dates or floating point numbers. It was hard to spot as it didn't do it to all gene names, but, for instance, the gene Septin 2, with acronym SEPT2 would be turned into 2006/09/02.  This is not new: this paper in 2004 documented the problem, but it seems many people weren't aware of it, and it is now estimated that the literature on genetics is riddled with errors as a consequence. 

This isn't the only way Excel can mess up your data. If you want to enter a date, you need to be very careful to ensure you have the correct setting. If you are in the UK and you enter a date like 23/4/16, then it will be correctly entered as 23rd April, regardless of the setting. But if you enter 12/4/16, it will be treated as 4^th December if you are on US settings and as 12^th April if you are on UK settings.

Then there is the dreaded autocomplete function. This can really screw things up by assuming that if you start typing text into a cell, you want it the same as a previous entry in that column that begins with the same sequence of letters. Can be a boon and a time-saver in some circumstances, but a way to introduce major errors in others.

I've also experienced odd bugs in Excel's autofill function, which makes it easy to copy a formula across columns or rows. It's possible for a file to become corrupted so that the cells referenced in the formula are wrong. Such errors are also often introduced by users, but I've experienced corrupted files containing formulae, which is pretty scary.

The response to this by many people is to say serious scientists shouldn't use Excel.  It's just too risky having software that can actively introduce errors into your data entry or computations. But people, including me, persist in using it, and we have to consider why.

So what are the advantages of keeping going with Excel?

Well, first, it usually comes for free with Microsoft computers, so it is widely available free of charge*. This means most people will have some familiarity with it –though few both to learn how to use it properly.

Second, you can scan a whole dataset easily: it's very direct scrolling through rows or columns. You can use Freeze Panes to keep column and row headers static, and you can hide columns or rows that you don't want getting in the way.

Third, you can format a worksheet to facilitate data entry. A lot of people dismiss colour coding of columns as prettification, but it can help ensure you keep the right data in the right place. Data validation is easily added and can ensure that only valid values are entered.

Fourth, you can add textual comments – either as a row in their own right, or using the Comment function.

Fifth, you can very easily plot data. Better still, you can do so dynamically, as it is easy to create a plot and then change the data range it refers to.

Sixth, you can use lookup functions. In my line of work we need to convert raw scores to standard scores based on normative data. This is typically done using tables of numbers in a manual, which makes it very easy to introduce human error. I have found it is worth investing time to get the large table of numbers entered as a separate worksheet, so we can then automate the lookup functions.

Many of my datasets are slowly generated over a period of years: we gather large amounts of data on individuals, record responses on paper, and then enter the data as it comes in. The people doing the data entry are mostly research assistants who are relatively inexperienced. So having a very transparent method of data entry, which can include clear instructions on the worksheet, and data validation, is important. I'm not sure there are other options of software that would suit my needs.

But I'm concerned about errors and need strategies to avoid them. So here are the working rules I have developed so far.

1. Before you begin, turn off any fancy Excel defaults you don't need. And if entering gene names, ensure they are entered as text.

2. Double data entry is crucial: have the data re-entered from scratch when the whole dataset is in, and cross-check the data files. This costs money but is important for data quality. There are always errors.

3. Once you have the key data entered and checked, export it to a simple, robust format such as tab-separated text. It can then be read and re-used by people working with other packages.

4. The main analysis should be done using software that generates a script that means the whole analysis can be reproduced. Excel is therefore not suitable. I increasingly use R, though SPSS is another option, provided you keep a syntax file.

5. I still like to cross-check analyses using Excel – even if it is just to do a quick plot to ensure that the pattern of results is consistent with an analysis done in R.  

Now, I am not an expert data scientist – far from it. I'm just someone who has been analysing data for many years and learned a few things along the way. Like most people, I tend to stick with what I know, as there are costs in mastering new skills, but I will change if I can see benefits. I've become convinced that R is the way to go for data analysis, but I do think Excel still has its uses, as a complement to other methods for storing, checking and analysing data. But, given the recent crisis in genetics, I'd be interested to hear what others think about optimal, affordable approaches to data entry and data analysis – with or without Excel.

*P.S.  I have been corrected on Twitter by people who have told me it is NOT free; the price for Microsoft products may be bundled in with the cost of the machine, but someone somewhere is paying for it!

Update: 2nd September 2016
There was a surprising amount of support for this post on Twitter, mixed in with anticipated criticism from those who just told me Excel is rubbish. What's interesting is that very few of the latter group could suggest a useable alternative for data entry (and some had clearly not read my post and thought I was advocating using Excel for data analysis). And yes, I don't regard Access as a usable alternative: been there tried that, and it just induced a lot of swearing.
There was, however, one suggestion that looks very promising and which I will chase up
@stephenelane suggested I look at REDcap.
Website here: https://projectredcap.org/

Meanwhile, here's a very useful link on setting up Excel worksheets to avoid later problems that came in via @tjmahr on Twitter
http://kbroman.org/dataorg/

Update 4th October 2016
Just to say we have trialled REDCap, and I love it.  Very friendly interface. Extremely limited for any data manipulation/computation, but that doesn't matter, as you can readily import/export information into other applications for processing. It's free but institution needs to be signed up for it: Oxford is not yet fully functional with it, but we were able to use it via a colleague's server for a pilot.

Alternative providers and alternative medicine

Jo Johnson thinks that the market in Higher Education is unfair. There are currently stringent procedures in place for any institution that wants to award degrees and call itself a University. One facet of this is the requirement that any new provider must initially have its degrees validated by an established University; The University of Suffolk, which gained University title this month, provides an example of how this can work, but many of those seeking to enter the market are unhappy with current arrangements.  Roxanne Stockwell, Principal of Pearson College, complained: “Until an institution has its own degree-awarding powers, it cannot offer degrees without being validated by an existing university. Under the current system, a new partner has to find a willing validating partner, and it is locked out if it cannot.”

Jo Johnson, in his role as Minister for Universities and Science last year, criticised this arrangement. He memorably said: “I know some validation relationships work well, but the requirement for new providers to seek out a suitable validating body from amongst the pool of incumbents is quite frankly anti-competitive. It’s akin to Byron Burger having to ask permission of McDonald’s to open up a new restaurant. .....I can announce that we will shortly be lifting the moratorium that has been in place for applications for new Degree Awarding Powers and for University Title. Once again, we are opening the doors to new entrants and challenger institutions, all in the interest of increasing the choices available to students.”

So how wide should the door be opened? This raises deep questions about what constitutes a University. Currently, British Universities have a strong international reputation. Historically, they have been subject to strict scrutiny in return for receiving government funding. They combine research and teaching to push forward the boundaries of knowledge, and have trained students to value knowledge for its own sake, not just as a means to an end.

Not everyone wants that kind of education: some students do not enjoy formal academic study and may prefer vocational courses or apprenticeships. It is important that our higher education system caters for them. The question confronting us now is how far we should extend the definition of a University. It is interesting to consider the Word Cloud I made of ‘alternative providers’, shown in Figure 1.

Figure 1: Alternative providers from http://www.hefce.ac.uk/reg/register/getthedata/. Key: Blue have University Title; Brown have Degree-Awarding Powers; Pink offer designated courses; Violet deliver HE as a franchise only. Taken from http://cdbu.org.uk/speedy-entrances-and-sharp-exits-letting-in-more-alternative-providers/.

Among those listed are six institutions providing training in various forms of complementary and alternative medicine. Two of these had progressed to the point of having degree-awarding powers, which is one step below having University Title.

Let us be clear: the subject matter that these institutions teach is not endorsed by serious scientists.  David Colquhoun highlighted a worrying trend for UK Universities to give degrees in ‘anti-science’ back in 2007. In Australia, where chiropractic has been taught within the regular University system, it has come under increasing attack by scientists, who note that by awarding degrees in these subjects, one gives credibility to procedures that are placebos at best and dangerous at worst.

Are these concerns just a sign of anti-competitiveness and elitism? Are scientists trying to squeeze out the alternative providers because they think they’ll poach students from them, just as McDonalds might try to ensure that Byron Burgers are denied space for development? I’d argue not, and furthermore that Jo Johnson’s analogy shows a startling lack of understanding of what a University is all about. Medicine has had many false leads and it would be the height of arrogance to assume that what we know now is the only truth. But the difference between medicine and alternative medicine is not just that there is evidence for effectiveness for medicine; it’s also that in medicine there is a continuous movement to develop and improve theory and practice, rigorously testing and debating ideas and using scientific methods to evaluate them. It’s difficult to do this well and it often goes wrong, but there is broad agreement about the importance of evidence.

Well, you might say, what about religion, another topic that features heavily among alternative providers? Should theology be banned from Universities because it is not evidence-based? The answer is no, and for similar reasons to those given above: the difference between our traditional Universities and the new providers is that Universities teaching theology consider a range of perspectives and teach students critical thinking. You do not need to be a believer in any God to study theology. In contrast, new providers are often narrow in their focus: many of those with religious affiliations look as if they train students in one religious viewpoint and one only.

Defining the difference between what is suitable and not suitable for inclusion in a University degree course is itself an interesting intellectual exercise. We should not assume that something is good just because it already exists, and is bad if it is new.  But if we just ignore this distinction and have a free-for-all whereby anything can be regarded as higher education provided that there are students willing to pay for it, we will end up with a system in which the terms University and Degree will count for very little, and where the survival of a higher education institution has more to do with its marketing skills than its academic standing. In the past, there were few institutions clamouring to become Universities because it was not easy to make a profit from Higher Education. That has all changed now that higher education providers can get their hands on money from the Student Loans Company. Experience to date suggests we need to have more, rather than less, scrutiny of alternative providers in the current financial climate.

One final point: the alternative providers I have discussed here could do very well on the Teaching Excellence Framework (TEF), which will rate higher education institutions according to three main criteria: student satisfaction, drop-out rates and employability. Indeed, if they recruit their students from among disadvantaged social groups, they might well achieve higher TEF scores than more selective institutions, because benchmarking is used to adjust the outcomes. So under Jo Johnson’s oversight, we could end up with a situation where the quality of teaching at the Anglo-European College of Chiropractic is deemed superior to that at the Universities of Oxford and Cambridge. An interesting thought.

Cost-benefit analysis of the Teaching Excellence Framework

©CartoonStock.com

The government’s new Higher Education and Research Bill gets its second reading this week. One complaint is that it has been rushed in without adequate scrutiny of some key components. I was interested, therefore, to discover, that a Detailed Impact Assessment was published in June, specifically to look at the costs and benefits of the various components of the Bill. What I found was quite shocking: we were being told that the financial benefits of the new Teaching Excellence Framework (TEF) vastly outweighed its costs – yet look in detail and this is all smoke and mirrors.

In particular, the report shows that while the costs of TEF to the higher education sector (confusingly described as ‘business’) are estimated at £20 million, the direct benefits will come to £1,146 million, giving a net benefit of £1,126 million (Table 1). How could the introduction of a new bureaucratic evaluation exercise be so remarkably beneficial? I read on with bated breath.

Well, sad to relate, it’s voodoo analysis.  This becomes clear if you press on to Table 12, which shows the crucial data from statistical modelling. Quite simply, the TEF generates money for institutions that get a good rating because it allows them to increase fees in line with inflation. Institutions that don’t participate in the TEF, or those that fail to get a good enough rating, will not be able to exceed the current 9K per annum fee, and so in real terms their income will decline over time. As far as I can make out, they are not included in Table 1. Furthermore, the increases for the compliant, successful institutions are measured relative to how they would have done if they had not been allowed to raise fees.

So to sum up:

• You don’t need the TEF to achieve this result. You could get the same outcome by just allowing all institutions to raise fees in line with inflation.
• As noted in the briefing to the Bill by the House of Commons: “the Bill is expected to result in a net financial benefit to higher education providers of around £1.1billion a year. This is in very large part due to the higher fees that providers with successful TEF outcomes will be able to charge students.” (p. 59)
• The system is designed for there to be winners and losers, and the losers will inevitably see their real income falling further and further behind the winners, unless inflation is zero.

The impact assessment does consider other options, including that of allowing fee increases in line with inflation provided the institution has a satisfactory Quality Assurance rating. This is rejected on the grounds that: “whilst QA is a good starting point, reliance on QA alone and in the longer-term will not enable significant differentiation of teaching quality to help inform student decisions and encourage institutions to improve their teaching quality.” (p. 37).  This makes clear that one consequence (and one suspects one purpose) of TEF is to facilitate the division into institutional sheep and goats, followed by starvation of the goats.

Another option, which was strongly recommended by many of those who responded to the consultation exercise on the Green Paper which preceded the bill, is to remove the link between TEF and fees. In other words, have some kind of teaching evaluation, where the motivation for taking part would be reputational rather than financial.  This too is rejected as not sufficiently powerful an incentive: “the Research Excellence Framework allocates £1.5bn a year to institutions. To achieve parity of esteem and focus between teaching and research the TEF will need to have a similar level of financial implications.” However, this is rather disingenuous. There is no pot of money on offer. We live in a country where we are used to government supporting Higher Education; now, however, the only source of income to universities for teaching is via student fees, but raising fees is unpopular.  The funding of universities will collapse unless they can either find alternative sources of income, or continue to raise fees in line with inflation, and TEF provides a cover story for doing that.

So we have a system designed to separate winners and losers, but the outcome will depend crucially on two factors: the rate of inflation and the rate of increase in students. The figures in the document have been modelled assuming that the number of students at English Higher Education Institutions will increase at a rate of around 2 per cent per annum (Table 12), and that annual inflation will be around 3 per cent. If either growth in numbers or inflation is lower, then the difference between those who do and don’t get good TEF ratings (and hence the apparent financial benefits of TEF) will decline.

What about the anticipated costs of the TEF?  We are told: “Institutions collectively will experience average annual costs of £22m as a result of familiarising, signing up and applying to the Teaching Excellence Framework, once the TEF covers discipline level assessments. This is equivalent to an average of £53,000 per institution, significantly less than the Research Excellence Framework (REF) at £230,000 per institution per year.” (p. 8). One can only assume that those writing this report have little experience of how academic institutions operate. For instance, they say that “Year One will not represent any additional administrative cost to institutions, as we will use the existing QA process.” I did a quick internet search and immediately found two universities who were advertising now for administrators to work on preparing for the TEF (on salaries of around £30-40K), as well as a consultancy agency that was touting for custom by noting the importance of being “TEF-ready”.

I have yet to get on to the section on costs and benefits of opening the market to ‘alternative providers’…..

If you are concerned at the threats to Higher Education posed by the Bill, please write to your MP - there is a website here that makes it very easy to do so.

Further background reading 
Shaky foundations of the TEF
A lamentable performance by Jo Johnson
More misrepresentation in the Green Paper
The Green Paper’s level playing field risks becoming a morass
NSS and teaching excellence: wrong measure, wrongly analysed
The Higher Education and Research Bill: What's changing?
CDBU's response to the Green Paper
The Alternative White Paper

Editorial integrity: Publishers on the front line

Thanks to some live tweeting by Anna Sharman (@sharmanedit), I've become aware that the 13th Conference of the European Association of Science Editors (EASE) is taking place in Strasbourg this weekend.

The topic is "Scientific integrity: editors on the front line", and the programme acknowledges Elsevier, who presumably have contributed funding for the conference.

It therefore seems timely to give a brief update of developments following three blogposts I wrote during February-March 2015, documenting some peculiar editorial behaviour at four journals: Research in Autism Spectrum Disorders (RASD: Elsevier), Research in Developmental Disabilities (RIDD: Elsevier), Developmental Neurorehabilitation (DN: Informa Healthcare) and Journal of Developmental and Physical Disabilities (JDPD: Springer).

To do the story full justice, you need to read these blogposts, but in brief, blogpost 1 described how Johnny Matson, the then editor of both RASD and RIDD had published numerous articles in his own journal, and engaged in frequent self-citation, leading to his receiving a 'highly cited' badge from Thomson Reuters. In the comments on that blogpost, another intriguing factor emerged, which was Matson's tendency to accept papers with little or no review. This was denied by Elsevier, despite clear evidence of very short acceptance lags that were incompatible with review.

Blogpost 2 was prompted by Matson defending himself against accusations of self-citation by pointing out that he published in journals that he did not edit. I checked this out and found he had numerous papers in two other journals: DN and JDPD, and that the median lag between a paper of his being submitted and accepted in DN was one day. (JDPD does not provide data on publication lags). I therefore looked at the editors of those journals, and found that they themselves were publishing remarkable numbers of papers in RASD and RIDD, again with extremely short publication lags. A trio of editors and editorial board members (Jeff Sigafoos, Giulio Lancioni and Mark O'Reilly), co-authored no less than 140 papers in RASD and RIDD between 2010 and 2014, typically with acceptance times of less than 2 weeks. Some of the papers in RIDD were not even in the topic area of developmental disabilities, but covered neurological conditions acquired in adulthood.

In blogpost 3, I turned the focus on to the publisher of RASD and RIDD, Elsevier, to query why they had not done anything about such irregular editorial practices. I did a further analysis of publication lags in RIDD, showing that they had dropped precipitately between 2008 and 2012, and that there was a small band of authors whose prolific papers were published there at amazing speed. I provided all the statistical data to support my case, including interactive spreadsheets that made it easy to determine which editors and authors had been benefiting from the slack editorial standards at these journals.

There was some interesting fall-out from all of this. The second blogpost drew fire from supporters of the editors I had "outed", accusing me of bad behaviour and threatening to complain to my university. Since everything I had said was backed by evidence, this did not concern me. I received heartfelt messages of support from people who were appalled that a particular approach to autism intervention had been promoted by this group of editors, who were in effect using their status to gain the veneer of scientific credibility for work which was not in fact peer-reviewed.  I was also contacted by several academics telling me that everyone knew this had been going on for years, but nobody had done anything; this level of passivity was surprising given that many were angry that  authors had reaped benefits from their staggeringly high publication rate, while those who were outside the charmed circle were left behind. I was urged to go further and raise my concerns with the universities employing those who were capitalising on, or engaging in, lax editorial behaviour. I do, however, have an extremely demanding job and I hoped that I had done enough by shining a light on dubious practices, and providing the full datasets that provided evidence. However, I now wonder if I should have been more pro-active.

I wrote to express my concerns to publishers of all four journals, and had my correspondence acknowledged. But then? Well, not a lot.

It's clear that Elsevier has taken some action. Indeed, my first blogpost was prompted by Michelle Dawson noting on Twitter that the editorial boards of RASD and RIDD had mysteriously disappeared from the online journals. She had previously noted Matson's pattern of mega-self-citation, and I had written directly to him, with copy to the publisher, some months previously to express concern, when I realised that I was listed as a member of the editorial board of RASD. Elsevier did not acknowledge my letter, but it is possible that the changes to the editorial boards that they had started were linked to my concerns.

The first direct response I had from Elsevier was some weeks after my final blogpost, when they explained that they were looking into the situation regarding unreviewed papers, but that this was a huge job and would take a long time. They were presumably disinclined to rely on the files that I had deposited on Open Science Framework, which show the identity and submission and acceptance data for every paper in RASD and RIDD.  They did appoint new editors and a small group of associate editors for both journals, all with good track records for integrity.

I have heard on the grapevine that they are now evaluating articles published in those journals that have been identified as not having undergone peer review; some of those approached to do these evaluations have mentioned this to me. It's rather unclear how this is going to work, given that, across the two journals, there are nearly 1000 papers where the available data indicate a lag from receipt to acceptance of under 2 weeks. I guess we should be glad that at least the publisher is taking some action, albeit at a snail's pace, but I am dubious as to whether there will be any retractions.

Meanwhile, Developmental Neurorehabilitation changed publisher around the time I was writing, and is now under the care of Taylor and Francis. I wrote to the publisher explaining my concerns and received a polite reply, but then heard no more. I note that the Editor in Chief is now Wendy Machalicek, who previously co-edited the journal with Russell Lang. Lang's doctoral advisor was Mark O'Reilly, editor of JDPD, and one of the prolific trio who featured in blogpost 2. Lang himself co-authored 24 papers in RASD and 13 in RIDD, and 35 of these 39 papers were accepted within 2 weeks of receipt. Machalicek has published 11 papers in RASD and 5 in RIDD, and 12 of these 16 papers were accepted within 2 weeks of receipt.  She also did her doctorate in O'Reilly's department, and several of her papers are co-authored with him. In an editorial last year, Lang and Machalicek announced changes to the journal, some of which seem to be prompted by a desire to make the reviewing process more rigorous under the new publisher. However, one change is of particular interest: the scope of the journal will be broadened to consider "developmental disability from a lifespan perspective; wherein, it is acknowledged that development occurs throughout a person's life and a range of injuries, diseases and other impairments can cause delayed or abnormal development at any stage of life." That will be good news for Giulio Lancioni, who was previously publishing papers on coma patients, amyotrophic lateral sclerosis, and Alzheimer's disease in RIDD. He and his collaborators – Jeff Sigafoos, Mark O'Reilly, as well as Russell Lang and Johnny Matson – are all current members of the editorial board of the journal.

It seems to be business as usual at the Springer title, Journal of Developmental and Physical Disabilities. Mark O'Reilly is still the editor, with Lang and Sigafoos as associate editors; Lancioni, Machalicek and Matson are all on the editorial board. Springer's willingness to turn a blind eye to editors playing the system becomes clear when one sees that a recent title, "Review Journal of Autism and Developmental Disorders" has as Editor-in-Chief no less a personage than Johnny Matson. And, surprise, surprise, the editorial board includes Lang, Sigafoos and Lancioni.

One of the overarching problems I uncovered when navigating my way around this situation was that there is no effective route for a whistleblower who has uncovered evidence of dubious behaviour by editors. Elsevier has developed a Publishing Ethics Resource Kit  but it is designed to help editors dealing with ethical issues that arise with authors and reviewers. The general advice if you encounter an ethical problem is to contact the editor. The Committee on Publication Ethics also issues guidance, but it is an advisory body with no powers. One would hope that publishers would act with integrity when a serious problem with an editor is revealed, but if my experience is anything to go by, they are extremely reluctant to act and will weave very large carpets to brush the problems under.