What is going on in Hindawi special issues?

A guest blogpost by Nick Wise 

 http://www.eng.cam.ac.uk/profiles/nhw24

The Hindawi journal Wireless Communications and Mobile Computing is booming. Until a few years ago they published 100-200 papers a year, however they published 269 papers in 2019, 368 in 2020 and 1,212 in 2021. So far in 2022 they have published 2,429. This growth has been achieved primarily by the creation of special issues, which makes sense. It would be nearly impossible for a journal to increase its publication rate by an order of magnitude in 2 years without outsourcing the massive increase in workload to guest editors.

Recent special issues include ‘Machine Learning Enabled Signal Processing Techniques for Large Scale 5G and 5G Networks’ (182 articles), ‘Explorations in Pattern Recognition and Computer Vision for Industry 4.0’ (244) and ‘Fusion of Big Data Analytics, Machine Learning and Optimization Algorithms for Internet of Things’ (204). Each of these special issues contains as many papers as the journal published in a year until recently. They also contain many papers that are flagged on Pubpeer for irrelevant citations, tortured phrases and surprising choices of corresponding email addresses.

However, I am going to focus on one special issue that is still open for submissions, and so far contains a modest 62 papers: ‘AI-Driven Wireless Energy Harvesting in Massive IoT for 5G and Beyond’, edited by Hamurabi Gamboa Rosales, Danijela Milosevic and Dijana Capeska Bogatinoska. Given the title of the special issue, it is perhaps surprising that only two of the articles contain ‘wireless’ in the title and none contain ‘energy’. The authors of the other papers (or whoever submitted them) appear to have realised that as long as they included the buzzwords ‘AI’, ‘IoT’ (Internet of Things) or ‘5G’ in the title, the paper could be about anything at all. Hence, the special issue contains titles such as:

• Analysis Model of the Guiding Role of National Sportsmanship on the Consumer Market of Table Tennis and Related IoT Applications 
• Evaluation Method of the Metacognitive Ability of Chinese Reading Teaching for Junior Middle School Students Based on Dijkstra Algorithm and IoT Applications 
• The Construction of Shared Wisdom Teaching Practice through IoT Based on the Perspective of Industry-Education Integration
  

Of the 62 papers, 60 give Hamurabi Gamboa Rosales as the academic editor and 2 give Danijela Milosevic. Why is the distribution of labour so lopsided? One can imagine an arrangement where the lead editor does the admin of waving through irrelevant papers and the other 2 guest editors get to say that they’ve guest-edited a special issue on their CV.

Of course, in addition to boosting publication numbers for the authors and providing CV points for the guest editors, every paper in the special issue has a references section. Each reference gives someone a citation, another academic brownie point on which careers can be built. An anonymous Pubpeer sleuth has trawled through the references section of every paper in this special issue and found that Malik Bader Alazzam of Amman Arab University in Jordan has been cited 139 times across the 62 papers. The chance that the authors of almost every article would independently decide to cite the same person seems small.

The most intriguing fact about the papers in the special issue however, is that only 4 authors give corresponding email addresses that match their affiliation. These 4 include the only 3 papers with non-Chinese authors. Of the other 58, 1 uses an email address from Guangzhou University, 6 use email addresses from Changzhou University, and 51 use email addresses from Ma’anshan University. All of the Ma’anshan addresses are of the form 1940XXXX@masu.edu.cn and many are nearly sequential, suggesting that someone somewhere purchased a block of sequential email addresses (you do not need to be at Ma’anshan University to have an @masu email address). The screenshot below shows a sample (the full dataset is linked here).

A subset of the titles from the special issue with their corresponding email addresses, all of the form 1940XXXX@masu.edu.cn

The use and form of the email addresses suggests that all of these papers are the work of a paper mill. It is hard to imagine otherwise how 51 different authors could submit papers to the same special issue using the same institutional email domain and format. Indeed, before 2022 only 2 papers had ever used @masu.edu.cn as a corresponding address according to Dimensions. It is equally hard to imagine how Hamurabi Gamboa Rosales is unaware. How can you not notice that, of the 19 papers you receive for your special issue on the 12th of July, 18 use the same email domain that doesn’t match their affiliation? This may also explain why Hamurabi has dealt with almost all the papers himself. This special issue should be closed for submissions and an investigation begun.

Stepping back from this special issue, this is not an isolated problem. There are at least 40 other papers published in Wireless Communications and Mobile Computing with corresponding emails from Ma’anshan, and Dimensions finds there are 46 in Computational Intelligence and Neuroscience, 38 in Computational and Mathematical Methods in Medicine and 30 in Mobile Information Systems, all published in 2022 and all in Hindawi journals. What are the chances that 18404032@masu.edu.cn is used in a special issue in Computational Intelligence and Neuroscience, 18404038@masu.edu.cn in Disease Markers and 18404041@masu.edu.cn in Wireless Communications and Mobile Computing?

Finally, masu.edu.cn is only one example of a commonly used email domain that doesn’t match the author’s affiliation. It is conceivable that the entire growth in publications of Wireless Communications and Mobile Computing, Computational Intelligence and Neuroscience (163 articles in 2020, 3,079 in 2022) and Computational and Mathematical Methods in Medicine (225 in 2020, 1,488 in 2022) is from paper mills publishing in corrupted special issues.

Nick Wise

*All numbers accurate as of the 12th October 2022.

Has the Society for Neuroscience lost its way?

The tl;dr version: The Society for Neuroscience (SfN) makes humongous amounts of money from its journal and meetings, but spends very little on helping its members, while treating overseas researchers with indifference bordering on disdain.

I first became concerned about the Society for Neuroscience back in 2010 when I submitted a paper to the Journal of Neuroscience. The instructions to authors explained that there was a submission fee (at the time about $50). Although I 'd never come across such a practice before, I reckoned it was not a large sum, and so went ahead. The Instructions for Authors explained that there was a veto on citation of unpublished work. I wanted to cite a paper of mine that had been ‘accepted in principle’ but needed minor changes, and I explained this in my cover letter. Nevertheless, the paper was desk-rejected because of this violation. A week later, after the other paper was accepted, I updated the manuscript and resubmitted it, but was told that I had to pay another submission fee. I got pretty grumbly at this point, but given that we'd spent a lot of time formatting the paper for J Neuroscience, I continued with the process. We had an awful editor (the Automaton described here), but excellent reviewers, and the paper was ultimately accepted.

But then we were confronted with the publication fee and charges for using colour figures. These were substantial – I can’t remember the details but it was so much that it turned out cheaper for all the authors to join the SfN, which made us eligible for reduced rates on publication fees. So for one year I became a member of the society. 

The journal’s current policy on fees can be found here.  Basically, the submission fee is now $140, but this is waived if first and last authors are SfN members (at cost of $200 per annum for full members, going down to $150 for postdocs and $70 for postgrads). The publication fee is $1,260 (for members) and $1,890 for non-members, with an extra $2,965 if you want the paper to be made open access.

There are some reductions for those working in resource-restricted countries, but the sums involved are still high enough to act as a deterrent. I used Web of Science to look at country of origin for Journal of Neuroscience papers since 2014, and there’s no sign that those from resource-restricted countries are taking advantage of the magnanimous offer to reduce publication fees by up to 50%.

Countries of origin from papers in Journal of Neuroscience (2014-2018)

The justification given for these fees is that ‘The submission fee covers a portion of the costs associated with peer review’, with the implication that the society is subsidising the other portion of the costs. Yet, when we look at their financial statements (download pdf here), they tell a rather different story. As we can see in the table on p 4, in 2017 the expenses associated with scientific publications came to $4.84 million, whereas the income from this source was $7.09 million.

But maybe the society uses journal income to subsidise other activities that benefit its nearly 36,000 members? That’s a common model in societies I’m involved in. But, no, the same financial report shows that the cost of the annual SfN meeting in 2017 was $9.5 million, but the income was $14.8 million. If we add in other sources of income, such as membership dues, we can start to understand how it is that the net assets of the society increased from $46.6 million in 2016 to $58.7 million in 2017.

This year, SfN has had a new challenge, which is that significant numbers of scientists are being denied visas to attend the annual meeting, as described in this piece by the Canadian Association for Neuroscience. This has led to calls for the annual meeting to be held outside the US in future years. The SfN President has denounced the visa restrictions as a thoroughly bad thing. However, it seems that SfN has not been sympathetic to would-be attendees who joined the society in order to attend the meeting, only to find that they would not be able to do so. I was first alerted to this on Twitter by this tweet:

This attracted a fair bit of adverse publicity for SfN, and just over a week later Chris heard back from the Executive Director of SfN who had explained that whereas they could refund registration fees for those who could not attend, they were not willing refund on membership fees. No doubt for an organisation that is sitting on long-term investments of $71.2 million (see table below), the $70 membership fee for a student is chicken feed. But I suspect it doesn’t feel like that to the student, who has probably also incurred costs for submitting an unsuccessful visa application.

Table from p 8 of SfN Annual Financial Report for 2017; The 'alternative investments' are mostly offshore funds in the Cayman Islands and elsewhere

There appears to be a mismatch between the lofty ideals described in SfN's mission statement and their behaviour. They seem to have lost their way: instead of being an organisation that exists to promote neuroscience and help their members, the members are rather regarded as nothing but a source of income, which is then stashed away in investments. It’s interesting to see that under Desired Outcomes, the Financial Reserve Strategy section of the mission statement has: ‘Strive to achieve end of year financial results that generate net revenues between $500,000 and $1 million in annual net operating surplus.’ That is reasonable and prudent for a large organisation with employees and property. But SfN is not achieving that goal: they are making considerably more money than their own mission statement recommends.

That money could be put to good use. In particular, given SfN’s stated claim of wanting to support neuroscience globally, they could offer grants for scientists in resource-poor countries to buy equipment, pay for research assistants or attend meetings. Quite small sums could be transformational in such a context. As far as I can see, SfN currently offers a few awards, but some of these are paid for by external donations, and, in relation to their huge reserves, the sums are paltry. My impression is that other, much smaller, societies do far more with limited funds than SfN does with its bloated income.

Maybe I’m missing something. I’m no longer a member of SfN, so it’s hard to judge. Are there SfN members out there who think the society does a good job for its membership?

Better control of the publication time-line: A further benefit of Registered Reports

I’ve blogged previously about waste in science. There are numerous studies that are completed but never see the light of day. When I wrote about this previously, I focused on issues such as reluctance of journals to publish null results, and the problem of writing up a study while applying for the next new grant. But here I want to focus on another factor: the protracted and unpredictable process of peer review that can lead to researchers to just give up on a paper.

Sample Gantt chart. Source: http://www.crp.kk.usm.my/pages/jepem.htm

The sample Gantt chart above nicely illustrates a typical scenario.  Let's suppose we have a postdoc with 30 months’ funding. Amazingly, she is not held up by patient recruitment issues, or ethics approvals, and everything goes according to plan, so 24 months in, she writes up the study and submits it to a journal. At the same time, she may be applying for further funding or positions. She may plan to start a family at the end of her fellowship. Depending on her area of study it may take anything from two weeks to six months to hear back from the journal*. The decision is likely to be revise and resubmit. If she’s lucky, she’ll be able to do the revisions and get the paper accepted to coincide with the end of her fellowship.  All too often, though, the reviewers suggest revisions. If she's very unlucky they may demand additional experiments, which she has no funding for.  If they just want changes to the text, that's usually do-able, but often they will suggest further analyses that take time, and she may only get to the point of resubmitting the manuscript when her money runs out. Then the odds are that the paper will go back to the reviewers – or even to new reviewers – who now have further ideas of how the paper can be improved. But now our researcher might have started a new job, have just given birth, or be unemployed and desperately applying for further funds.

The thing about this scenario, which will be all too familiar to seasoned researchers (see a nice example here), is that it is totally unpredictable. Your paper may be accepted quickly, or it may get endlessly delayed. The demands of the reviewers may involve another six month’s work on the paper, at a point when the researcher just doesn’t have the time. I’ve seen dedicated, hardworking, enthusiastic young researchers completely ground down by this situation, faced by the choice of either abandoning a project that has consumed a huge amount of energy and money, or somehow creating time out of thin air. It’s particularly harsh on those who are naturally careful and obsessive, who will be unhappy at the idea of doing a quick and dirty fix to just get the paper out. That paper which started out as their pride and joy, representing their best efforts over a period of years is now reduced to a millstone around the neck.

But there is an alternative. I’ve recently, with a graduate student, Hannah Hobson, put my toe in the waters of Registered Reports, with a paper submitted to Cortex looking at an electrophysiological phenomenon known as mu suppression. The key difference from the normal publication route is that the paper is reviewed before the study is conducted, on the basis of an introduction and protocol detailing the methods and analysis plan. This, of course takes time – reviewing always does. But if and when the paper is approved by reviewers, it is provisionally accepted for publication, provided the researchers do what they said they would.

One advantage of this process is that, after you have provisional acceptance of the submission, the timing is largely under your own control. Before the study is done, the introduction and methods are already written up, and so once the study is done, you just add the results and discussion. You are not prohibited from doing additional analyses that weren’t pre-registered, but they are clearly identified as such. One the study is written up the paper goes back to reviewers. They may make further suggestions for improving the paper, but what they can’t do is to require you to do a whole load of new analyses or experiments. Obviously, if a reviewer spots a fatal error in the paper, that is another matter. But reviewers can’t at this point start dictating that the authors do further analyses or experiments that may be interesting but not essential.

We found that the reviewer comments on our completed study were helpful: they advised on how to present the data and made suggestions about how to frame the discussion. One reviewer suggested additional analyses that would have been nice to include but were not critical; as Hannah was working to tight deadlines for thesis completion and starting a new job, we realised it would not be possible to do these, but because we have deposited the data for this paper (another requirement for a Registered Report), the door is left open for others to do further analysis.

I always liked the idea of Registered Reports, but this experience has made me even more enthusiastic for the approach. I can imagine how different the process would have been had we gone down the conventional publishing route. Hannah would have started her data collection much sooner, as we wouldn’t have had to wait for reviewer comments. So the paper might have been submitted many months earlier. But then we would have started along the long uncertain road to publication. No doubt reviewers would have asked why we didn’t include different control conditions, why we didn’t use current source density analysis, why we weren’t looking at a different frequency band, and whether our exclusionary criteria for participants were adequate. They may have argued that our null results arose because the study was underpowered. (In the pre-registered route, these were all issues that were raised in the reviews of our protocol, so had been incorporated in the study). We would have been at risk of an outright rejection at worst, or requirement for major revisions at best. We could then have spent many months responding to reviewer recommendations and then resubmitting, only to be asked for yet more analyses.  Instead, we had a pretty clear idea of the timeline for publication, and could be confident it would not be enormously protracted.

This is not a rant against peer reviewers. The role of the reviewer is to look at someone else’s work and see how it could be improved. My own papers have been massively helped by reviewer suggestions, and I am on record as defending the peer review system against attacks. It is more a rant against the way in which things are ordered in our current publication system. The uncertainty inherent in the peer review process generates an enormous amount of waste, as publications, and sometimes careers, are abandoned. There is another way, via Registered Reports, and I hope that more journals will start to offer this option.

*Less than two weeks suggests a problem!See here for an example.

Why does so much research go unpublished?

As described in my last blogpost, I attended an excellent symposium on waste in research this week. A recurring theme was research that never got published. Rosalind Smyth described her experience of sitting on the funding panel of a medium-sized charity. The panel went to great pains to select the most promising projects, and would end a meeting with a sense of excitement about the great work that they were able to fund. A few years down the line, though, they'd find that many of the funds had been squandered. The work had either not been done, or had been completed but not published.

In order to tackle this problem, we need to understand the underlying causes. Sometimes, as Robert Burns noted, the best-laid schemes go wrong. Until you've tried to run a few research projects, it's hard to imagine the myriad different ways in which life can conspire to mess up your plans. The eight laws of psychological research formulated by Hodgson and Rollnick are as true today as they were 25 years ago.

But much research remains unpublished despite being completed. Reasons are multiple, and the strategies needed to overcome them are varied, but here is my list of the top three problems and potential solutions.

Inconclusive results

Probably the commonest reason for inconclusive results is lack of statistical power. A study is undertaken in the fond hope that a difference will be found between condition X and condition Y, and if the difference is found, there is great rejoicing and a rush to publish. A negative result should also be of interest, provided the study was well-designed and adequately motivated. But if the sample is small, then we can't be sure whether our failure to observe the effect is because it is absent: a real but small effect could be swamped by noise. 

I think the solution to this problem lies in the hands of funding panels and researchers: quite simply, they need to take statistical power very seriously indeed and to consider carefully whether anything will be learned from a study if the anticipated effects are not obtained. If not, then the research needs to be rethought. In the fields of genetics and clinical trials, it is now recognised that multicentre collaborations are the way forward to ensure that studies are conducted with sufficient power to obtain a conclusive result.

Rejection of completed work by journals

Even well-conducted and adequately powered studies may be rejected by journals if the results are not deemed to be exciting. To solve this problem, we must look to journals. We need recognition that - provided a study is methodologically strong and well-motivated - negative results can be as informative as positive ones. Otherwise we are doomed to waste time and money pursuing false leads.  As Paul Glasziou has emphasised, failure is part of the research process. It is important to tell people about what doesn't work if we are not to repeat our mistakes.

We do now have some journals that will publish negative results, and there is a growing move toward pre-registration of studies, with guaranteed publication if the methods meet quality criteria. But there is still a lot to be done, and we need a radical change of mindset about what kinds of research results are valuable.

Lack of time

Here, I lay the blame squarely on the incentive structures that operate in universities. To get a job, or to get promoted, you need to demonstrate that you can pull in research income. In many UK institutions this is quite explicit, and promotions criteria may give a specific figure to aim for of X thousand pounds research income per annum. There are few UK universities whose strategic plan does not include a statement about increasing research funding. This has changed the culture dramatically;  as Fergus Millar put it: "in the modern British university, it is not that funding is sought in order to carry out research, but that research projects are formulated in order to get funding".

Of course, for research to thrive, our Universities need people who can compete for funding to support their work. But the acquisition of funding has become an end in itself, rather than a means to an end. This has the pernicious effect of driving people to apply for grant after grant, without adequately budgeting for the time it takes to analyse and write up research, or indeed to carefully think about what they are doing.  As I argued previously, even junior researchers these days have an 'academic backlog' of unwritten papers.

At the Lancet meeting there were some useful suggestions for how we might change incentive structures to avoid such waste. Malcolm MacLeod argued researchers should be evaluated not by research income and high-impact publications, but by the quality of their methods, the extent to which their research was fully reported, and the reproducibility of findings. An-Wen Chan echoed this, arguing for performance metrics that recognise full dissemination of research and use of research datasets by other groups. However, we may ask whether such proposals have any chance of being adopted when University funding is directly linked to grant income, and Universities increasingly view themselves as businesses.

I suspect we would need revised incentives to be reflected at the level of those allocating central funding before vice-chancellors took them seriously.  It would, however, be feasible for behaviour to be shaped at the supply end, if funders adopted new guidelines. For a start, they could look more carefully at the time commitments of those to whom grants are given: in my experience this is never taken into consideration, and one can see successful 'fat cats' accumulating grant after grant, as success builds on success. Funders could also monitor more closely the outcomes of grants: Chan noted that NIHR withholds 10% of research funds until a paper based on the research has been submitted for publication. Moves like this could help us change the climate so that an award of a grant would confer responsibility on the recipient to carry through the work to completion, rather than acting solely to embellish the researcher's curriculum vitae.

References

Chan, A., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gotzsche, P., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014). Increasing value and reducing waste: addressing inaccessible research Lancet (8 Jan ) : 10.1016/S0140-6736(13)62296-5

Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., . . . Glasziou, P. (2014). Biomedical research: increasing value, reducing waste. Lancet, 383(9912), 101-104.