IEEE Has a Pseudoscience Problem

Guest post by Solal Pirelli

The IEEE, full name Institute of Electrical and Electronics Engineers, is one of the main scientific publishers in domains related to its name. Many IEEE venues, such as ICSE in software engineering and IROS in robotics, are “top” venues that publish important research. While these are conferences and not journals, computer science and related fields are unusual in that conferences are typically the more prestigious option.

But as I’ve covered before in the case of another big computer science publisher, world-class research can coexist with world-class nonsense. Many not-so-top IEEE venues publish “AI gobbledegook sandwiches”, pointless papers that apply standard machine learning or artificial intelligence to basic data sets resulting in vague predictions supposedly improving on ill-defined baselines.

Unfortunately, bad science published by IEEE isn’t limited to boring applications of boring algorithms to boring data. In this blog post, I’ll present IEEE-published pseudoscience of various kinds, show how this correlates with other problems, and discuss why publishers don’t do enough about it. 

All kinds of quackery 

The IEEE has published numerous new “methods” to help providers or users of pseudoscientific disciplines. Ayurveda is enhanced with a “preprocessing framework” to detect diabetes, a neural network to classify herbs, and even an AI assistant. Astrology is automated with a machine learning model. Myers-Briggs personality type testing is granted another neural network.

Some IEEE papers are at the very fringe of pseudoscience, unconventional even by quack standards. A symposium on antennas and propagation published three papers by the same author on “scientific traditional Chinese medicine”, a variant based on electromagnetism and 5G with “supernatural potential” (see here, here and here). An Indian conference on electronics published four papers (here, here, here and here) by the same first author on “electro-homeopathy”, the brainchild of a 19th century Italian count that an Indian high court called “nothing but quackery” a decade before these papers were published.

Of course, no list of pseudoscience would be complete without perpetual motion. That’s right, the IEEE has published two papers on perpetual motion in 2017 and 2022! How these were not desk-rejected is anyone’s guess.

Even work that is not pseudoscientific in itself can propagate harmful or downright absurd stereotypes. Consider what IEEE-published and supposedly peer-reviewed papers have to say about autism: 

• “Children with autism require constant care because you never know what will trigger them” (source) 
• "If symptoms of autism are detected early, children with autism usually return to normal development after effective medical intervention" (source) 
• “A baby born with autism spectrum disorder may have a lower-than-average heart rate. Complete blockage of the heart at birth is rare. Abnormal heart rate leads to heart block. So, there is a high chance of the child's death due to permanent heart blockage at any time.” (source)

Why it matters

One may think that such papers won’t cause harm because they’re unlikely to be read, since they are mostly in unknown venues and unrelated to the IEEE’s domain. While I personally disagree since I believe publishing pseudoscience risks breaking the public’s trust in legitimate research, let me provide a more objective argument. Pseudoscience in papers is heavily correlated with other problematic practices that are more difficult to detect automatically. This makes searching for pseudoscience an effective way to find problematic venues, complementary to existing techniques.

The preprocessing framework to detect diabetes with Ayurveda? In a conference that accepts papers on the same day they are submitted, somehow speeding up the weeks or months usually necessary for proper peer review.

The neural network that classifies Ayurvedic herbs? In a conference that plagiarized its peer review policy from Elsevier’s “Transport Policy” journal. Look for fragments of this policy in your favorite search engine and you’ll find a surprising number of venues that have done so, seemingly without noticing the references to Transport Policy.

The four papers on electro-homeopathy? In a conference that published a mathematical “algorithm” amounting to high school mathematics. While exact definitions of “novelty” vary, no one could credibly claim that this paper is novel enough for a scientific conference.

The 2017 paper on perpetual motion? In a conference that didn’t notice an entirely plagiarized section in that paper, ironically from a source explaining why perpetual motion is impossible. How this is compatible with IEEE’s policy of checking all content for plagiarism is unclear.

The paper claiming “you never know what will trigger” autistic children? In a conference supposedly happening in a London office building, whose four IEEE-published editions only feature one paper from a European university among a sea of India-based authors. Did the authors of this conference’s papers really travel to the other side of the globe to present in a place not designed for presentations?

The neural network for Myers-Briggs? In a conference chaired by a professor whose Russian university is under sanctions from the US, the EU, Ukraine, and even Switzerland!

Action is rare 

The expected process here would be to report this nonsense to the publisher, who would investigate, quickly conclude these papers should never have been published, lose faith in the peer review process that led to their acceptance, and issue retractions. Barring extremely strong evidence from conference chairs that some cases were truly one-off exceptions, such retractions would cover entire editions of conferences.

This happens… sometimes. The IEEE has retracted papers before, such as this one after “only” five months. They have also retracted entire venues, such as this one totaling 400 papers, four years after it was reported.

 
But the IEEE frequently does not react at all to reports. Guillaume Cabanac, who specializes in scientific fraud detection, has repeatedly and publicly called them out. For instance, he’s reported telltale signs of ChatGPT as in this paper that includes “Regenerate Response” in the middle of text and this paper that includes “I am unable to […] due to the fact I am an AI language model”. He’s also reported “tortured phrases”, attempts at avoiding plagiarism detection that instead create nonsense such as “parcel misfortune” instead of “packet loss” in computer networking, in sometimes large concentrations. Cabanac and other sleuths have published “proceedings-level reports” on PubPeer, such as this one, when entire IEEE conferences have problems. None of the examples in this paragraph have led to any public reaction from the IEEE.

The IEEE occasionally issues “expressions of concern”, such one as for this paper over a year after concrete evidence of plagiarism was publicly reported. But expressions of concerns are not retractions. In mid-2023, Retraction Watch noted that hundreds of IEEE papers reported by Guillaume Cabanac and Harvard lecturer Kendra Albert were still up for sale. A year and a half later, that remains the case.

One case noted above is particularly noteworthy in terms of both reputation and IEEE awareness: The “scientific TCM” papers were published in the 2022 and 2023 editions of the “International Symposium on Antennas and Propagation”, a 6-decade-old conference whose 2024 edition boasted the IEEE President as a keynote speaker. Clearly, the IEEE is aware of the venue and its papers. What’s the point in “reporting” them?

Processes are inadequate 

The scale of publishers’ actions is nowhere near the scale of the problem. Creating a new conference or journal does not require that much time if the peer reviewing process is fake. As long as the average time it takes a publisher to retract a venue is higher than the time it takes to create a new venue, there won’t be meaningful progress.

Current publisher processes are designed to correct honest mistakes, not to fight malice. The time it takes to contact authors, wait for their response, wait for them to find original data, and so on is worth it when a single paper has a problem that can be explained by human error. But any such process is a waste of time when a paper contains blatant pseudoscience, has obviously been plagiarized, or uses terminology so bizarre no reviewer could have understood it.

To give an example of scale, here’s a collision of pseudoscience and tortured phrases. The paper on an AI assistant for ayurveda mentioned earlier is in the “2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT)”. Guillaume Cabanac’s Problematic Paper Screener currently lists 185 cases of tortured phrases manually confirmed by Cabanac himself, with another 160 pending assessment. These include “herbal language” instead of natural language, “system getting to know” instead of machine learning, “give-up-to-give-up” instead of end-to-end, and “0.33-celebration” instead of third-party.  

Individually contacting and waiting for hundreds of authors just in case they can explain why their paper talks about 0.33-celebrations isn’t going to cut it. Neither is individually contacting and waiting for dozens of conference editors just in case they can explain why their peer review process didn’t spot this nonsense. 

What can we do?

Given the incentives and processes at play, it’s not surprising to see the IEEE or any other big publisher publish pseudoscience. The authors of the papers mentioned in this post probably didn’t do anything illegal, except maybe for occasional plagiarism of copyrighted content, but nobody has the time and money to sue for such boring violations. This gives publishers a double excuse: they’re not publishing anything illegal, and retractions without a solid legal basis could backfire.

The scientific community needs to ban the “incompetence” defense from authors and stop associating with publishers that can’t be bothered to act quickly enough.  

Authors who publish obvious nonsense should not get a chance to explain themselves or “correct” their paper. 

Publishers make enough money from processing and selling articles. They can defend themselves from occasional lawsuits by angry authors, and they can hire scientific integrity specialists.

When I say “scientific integrity specialist”, that can unfortunately be as simple as “person looking for specific keywords in Google Scholar”. It’s what I did to find pseudoscience, and you can do that too. Report these on PubPeer, directly to publishers, or both. You can also go to the Problematic Paper Screener’s page listing articles that have not been manually assessed yet, and follow the instructions.

Finally, remember that most scientists have no idea this is going on. You can help by publicly calling out problematic papers and lack of action. Ask candidates for governance boards in more democratic publishers like the IEEE what they plan to do about fraud. Discourage institutions, especially public ones in democratic countries, from making blanket deals with publishers.

The Do It Yourself (DIY) conference

This blogpost was inspired by a tweet from Natalie Jester, a PhD student at the School for Sociology, Politics and International Studies at the University of Bristol, raising this question:

I agreed with her, noting that the main costs were venue hire and speaker expenses, but that prices were often hiked by organisers using lavish venues and aiming to make a profit from the meeting. I linked to my earlier post about the eye-watering profits that the Society for Neuroscience makes from its meetings.  In contrast, the UK's Experimental Psychology Society uses its income membership fees and the journal to support meetings three times a year, and doesn't even charge a registration fee.

Pradeep Reddy Raamana, a Canadian Open Neuroscience scholar from Toronto responded, drawing my attention to a thread on this very topic from a couple of weeks ago.

There were useful suggestions in the thread, including reducing costs by spending less on luxurious accommodation for organisers, and encouraging PIs to earmark funds for their junior staff to cover their conference attendance costs.

That's all fine, but my suggestion is for a radically different approach, which is to find a small group of 2-3 like-minded people and organise your own conference. I'm sure that people will respond by saying that they have to go to the big society meetings in their field in order to network and promote their research.  There's nothing in my suggestions that would preclude you also doing this (though see climate emergency point below). But I suspect that if you go down the DIY route, you may get a lot more out of the experience than you would by attending a big, swish society conference: both in terms of personal benefits and career prospects.

I'm sure people will want to add to these ideas, but here's my experience, which is based on running various smallish meetings, including being local organiser for occasional EPS meetings over the years. I was also, with Katharine Perera, Gina Conti-Ramsden and Elena Lieven,  a co-organiser of the Child Language Seminar (CLS) in Manchester back in the 1980s.  That is perhaps the best example of a DIY conference, because we had no infrastructure and just worked it out as we went along.  The CLS was a very ad hoc thing: each year, the meeting organisers tried to find someone who was prepared to run the next CLS at their own institution the following year. Despite this informality, the CLS – now with the more appropriate name of Child Language Symposium – is still going strong in 2019. From memory, we had around 120 people from all over the world at the Manchester meeting. Numbers have grown over the years, but in general if you were doing a DIY meeting for the first time, I'd aim to keep it small; no more than 200 people.

The main costs you will incur in organising a meeting are:

• Venue
• Refreshments
• Reception/Conference dinner
• Expenses for speakers
• Administrative costs
• Publicity

Your income to cover these costs will come from:

• Grants (optional)
• Registration fees

So the main thing to do at the start is to sit down and do some sums to ensure you will break even. Here's my experiences on each of these categories:

Venue

You do not need to hold the meeting at a swanky hotel. Your university is likely to have conference facilities: check out their rates. Consider what you need in terms of lecture theatre capacity, break-out rooms, rooms for posters/refreshments.  You need to factor in cost of technical support. My advice is you should let people look after their own accommodation: at most just give them a list of places to stay. This massively cuts down on your workload.

Refreshments

The venue should be able to offer teas/coffees. You will probably be astounded at what institutions charge for a cup of instant coffee and a boring biscuit, but I recommend you go with the flow on that one. People do need their coffee breaks, however humble the refreshments.

Reception/Conference dinner

A welcome reception is a good way of breaking the ice on the first evening. It need not be expensive: a few bottles of wine plus water and soft drinks and some nibbles is adequate. You could just find a space to do this and provide the refreshments yourselves: most of the EPS meetings I've been to just have some bottles provided and people help themselves. This will be cheaper than rates from conference organisers.

You don't have to have a conference dinner. They can be rather stuffy affairs, and a torment for shy people who don't know anyone. On the other hand, when they work well, they provide an opportunity to get to know people and chat about work informally. My experience at EPS and CLS is that the easiest way to organise this is to book a local restaurant. They will probably suggest a set meal at a set price, with people selecting options in advance. This will involve some admin work – see below.

Expenses for speakers

For a meeting like CLS there are a small number of invited plenary speakers. This is your opportunity to invite the people you really want to hear from. It's usual to offer economy class travel and accommodation in a good hotel. This does not need to be lavish, but it should have quiet rooms with ensuite bathroom, large, comfortable bed, desk area, sufficient power supply, adequate aircon/heating, and free wifi. Someone who has flown around the world to come to your meeting is not going to remember you fondly if they are put up in a cramped bed and breakfast. I've had some dismal experiences over the years and now check TripAdvisor to make sure I've not been booked in somewhere awful.  I still remember attending a meeting where an eminent speaker had flown in from North America only to find herself put in student accommodation: she turned around and booked herself into a hotel, and left with dismal memories of the organisers.

Pradeep noted that conferences could save costs if speakers covered their own expenses. This is true and many do have funds that they could use for this purpose. But don't assume that is the case: if they do have funds, you'd have to consider why they'd rather spend that money on coming to your meeting, than on something else. A diplomatic way of discussing this is to say in the letter of invitation that you can cover economy class travel, accommodation, dinner and registration. However, if they have funds that could be used for their travel, then that will make it possible to offer some sponsored places to students.

Administration

It's easy to overlook this item, but fortunately it is now relatively simple to handle registrations with online tools such as EventBrite. They take a cut if you charge for registration, but that's well worth it in my experience, in terms of saving a lot of grief with spreadsheets. If you are going for a conference dinner, then booking for this can be bundled in with registration fee.

In the days of Manchester CLS, email barely existed and nobody expected a conference website, but nowadays that is mandatory, and so you will need someone willing to set it up and populate it with information about venue and programme. As with my other advice, no need to make it fancy; just ensure there is the basic information that people need, with a link to a place for registration.

There are further items like setting up the Eventbrite page, making conference badges, and ensuring smooth communications with venue, speakers and restaurant. Here the main thing is to delegate responsibility so everyone knows what they have to do. I've quite often experienced the situation where I've agreed to speak at a meeting only to find that nobody has contacted me about the programme or venue and it's only a week to go.

On the day, you'll be glad of assistants who can do things like shepherding people into sessions, taking messages, etc. You can offer free registration to local students in return for them acting in this role.

Publicity

I've listed this under costs, but I've never spent on this for meetings I've organised, and given social media, I don't think you'll need to.

Grants

I've put optional for grants, as you can cover costs without a grant. But every bit of money helps and it's possible that one of the organisers will have funding that can be used. However, my advice is to check out options for grant funding from a society or other funder. National funding bodies such as UK research councils or NIH may have pots of money you can apply for: the sums are typically small and applying for the money is not onerous. Even if a society doesn't have a grants stream for meetings, they may be willing to sponsor places for specific categories of attendees: early-career people or those from resource-poor countries.

Local businesses or publishers are often willing to sponsor things like conference bags, in return for showing their logo. You can often charge publishers for a stand.

Registration

Once you have thought through the items under Expenditure, and have an idea of whether you'll have grant income, you will be in a good position to work out what you need to charge those attending to cover your costs. The ideal is to break even, but it's important not to overspend and so you should estimate how many people are likely to register in each category, and work out a registration fee that will cover this, even if numbers are disappointing.

What can go wrong?

• Acts of God. I still remember a meeting at the Royal Society years ago where a hurricane swept across Britain overnight and around 50% of those attending couldn't make it. Other things like strikes, riots, etc. can happen, but I recommend you just accept these are risks not under your control.
• Clash of dates. This is under your control to some extent. Before you settle on a date, ask around to check there isn't a clash with other meetings or with religious holidays.
• Speaker pulls out. I have organised meetings where a speaker pulled out at the last minute – there will usually be a good reason for this such as illness. So long as it is one person, this can be managed, and may indeed provide an opportunity to do something useful with the time, such as holding a mini-Hackathon to brainstorm ideas about a specific problem..
• You make a loss. This is a scary prospect but should not happen with adequate planning, as noted above. Main thing is to make sure you confirm what your speaker expenses will be so you don't get any nasty surprises at the last minute.
• Difficult people. This is a minor one, but I remember wise words of Betty Byers Brown, a collaborator from those old Manchester days, who told me that 95% of the work of a conference organiser is caused by 5% of those attending. Just knowing that is the case makes it easier to deal with.
• Unhappy people. People coming from far away who know nobody can have a miserable time at a conference, but with planning, you can help them integrate in a group. Rather than formal entertainment, consider having social activities that ensure everyone is included. Also, have an explicit anti-harassment policy – there are plenty of examples on the web.
• Criticism. Whatever you do there will be people who complain – why didn't you do X rather than Y?  This can be demoralising if you have put a lot of work into organising something.  Nevertheless, make sure you do ask people for feedback after the meeting: if there are things that could be done better next time, you need to know about them. For what it's worth, the most common complaints I hear after meetings are that speakers go on too long and there is not enough time for questions and discussion. It's important to have firm chairing, and to set up the schedule to encourage interaction.

What can go right?

• Running a conference carries an element of risk and stress, but it's an opportunity to develop organisational skills, and this can be a great thing to put on your CV. The skills you need to plan a conference are not so different from those to budget for a grant: you have to work out how to optimise the use of funds, anticipating expenses and risks.
• Bonding with co-organisers. If you pick your co-organisers wisely, you may find that the experience of working together to solve problems is enjoyable and you learn a lot.
• You can choose the topics for your meeting and get to invite the speakers you most want to hear. As a young researcher organising a small meeting, I got to know people I'd invited as speakers in a way that would not be possible if I was just attending a big meeting organised by a major society.
• You can do it your way. You can decide if you want to lower costs for specific groups. You can make sure that the speakers are diverse, and can experiment with different approaches to get away from the traditional format of speakers delivering a lecture to an audience. For examples see this post and comments below it.
• The main thing is that if you are in control, you can devise your meeting to ensure it achieves what scientific meetings are supposed to achieve: scholarly communication and interaction to spark ideas and collaborations. My memories of meetings I have organised as an early-career academic have been high points in my career, which is why I am so keen to encourage others to do this.

But! .... Climate emergency

The elephant in this particular room is air travel. Academics are used to zipping around the world to go to conferences, at a time when we are increasingly recognising the harm this is doing to our planet. My only justification for writing this post at the current time is that it may encourage people to go to smaller, more-focused meetings. But I'm trying to cut down on air travel substantially and in the longer term, suspect that we will need to move to virtual meetings.

Groups of younger researchers, and those from outside Europe and the UK, have a role to play in working out how to do this. I hope to encourage this by urging people to be bold and to venture outside the big conference arenas where junior people and those from marginalised groups can feel they are invisible. Organising a small meeting teaches you a lot of necessary skills that may be used in devising more radical formats. The future of conferences is going to change and you need to be shaping it.

-->