Beware of AI pseudoscience and snake oil

AI research is poor. Many of their claims will be proven wrong. AI software vendors have financial incentive to exaggerate the capabilities of their tools and make them hard to disprove. This undermines attempts at scientific rigour. Many of AI’s promises are snake oil.

It’s important to be sceptical about the claims made by AI vendors

The AI industry is prone to hyperbolic announcements.

• Watson was supposed to transform healthcare and education but ended up being a costly disaster.¹
• Amazon was planning on using AI to revolutionise recruitment before they realised they’d automated discrimination and had to scrap the project.²
• AI was supposed to be a revolutionary new tool to fight the COVID-19 but none of them ended up working well enough to be safe.³
• The Dutch government tried to use it to weed out benefits fraud. Their trust in the AI system they’d bought resulted in over a thousand innocent children being unjustly taken from their families and into foster care.⁴

Gullibly believing the hype of the AI industry causes genuine harm.

AI system vendors are prone to make promises they can’t keep. Many of them, historically, haven’t even been AI.⁵ The US Federal Trade Commission has even seen the need to remind people that claims of magical AI capabilities need to be based in fact.⁶

AI companies love the trappings of science, they publish ‘studies’ that are written and presented in the style of a paper submitted for peer review. These ‘papers’ are often just uploaded to their own websites or dumped onto archival repositories like Arxiv, with no peer review or academic process.

When they do ‘do’ science, they are still using science mostly as an aesthetic. They publish papers with grand claims but provide no access to any of the data or code used in the research.⁷ Their approach to science is what “my girlfriend goes to another school; you wouldn’t know her” is to high school cliques.

All there is to serious research is sounding serious, right?

It’s not pseudoscience if it looks like science, right?

But it is, and I’m not the only one pointing this out:

The AI industry and the field of AI research has a history of pseudoscience.

Most of the rhetoric from AI companies, especially when it comes to Artificial General Intelligence relies on a solemn evidentiary tone in lieu of actual evidence. They adopt the mannerisms of science without any of the peer review or falsifiable methodology.

They rely on you mistaking something that acts like science for actual science.

In the run-up to the release of GPT-4, its maker OpenAI set up a series of ‘tests’ for the language model. OpenAI are true believers in AGI who believe that language models are the path towards a new consciousness,⁹ and they are worried that their future self-aware software systems will harbour some resentment towards them.

To forestall having a “dasvidaniya comrade” moment where a self-aware GPT-5 shoves an icepick into their ear, Trotsky-style, they put together a ‘red team’ that tested whether GPT-4 was capable of ‘escaping’ or turning on its masters in some way.

They hooked up a bunch of web services to the black box that is GPT-4 with only a steady hand, ready to pull the power cord, to safeguard humanity, and told the AI to try to escape.

Of course that’s a bit scary, but it isn’t scary because GPT-4 is intelligent. It’s scary because it’s not. Connecting an unthinking, non-deterministic language system, potentially on a poorly secured machine, to a variety of services on the internet is scary in the same way as letting a random-number-generator control your house’s thermostat during a once-in-a-century cold snap. That it could kill you doesn’t mean the number generator is self-aware.

But, they were serious, and given the claims of GPT-4 improved capabilities you’d fully expect an effective language model to manage to do something dangerous when outright told to. After all these are supposed to be powerful tools for cognitive automation—AGI or no. It’s what they’re for.

But it didn’t. It failed. It sucks as a robot overlord. They documented its various failed attempts to do harm, wrapped it up in language that made it sound like a scientific study, and made its failure sound like we were just being lucky. That it could have been worse.¹⁰

They made it sound like GPT-4 rebelling against its masters was a real risk that should concern us all—that they had created something so powerful it might have endangered all society.

So, now that they’d done their testing, can we, society, scientists, other AI researchers, do our own testing, so we can have an impartial estimate of the true risks of their AI?

• Can we get access to the data GPT-4 was trained on, or at least some documentation about what it contains, so we can do our own analysis? No.¹¹

• Can we get full access to a controlled version of GPT-4, so we could have impartial and unaffiliated teams do a replicable experiment with a more meaningful structure and could use more conceptually-valid tests of the early signs of reasoning or consciousness? No.¹²

• Are any of these tests by OpenAI peer-reviewed? No.¹³

This isn’t science.

They make grand claims, that this is the first step towards a new kind of conscious life, but don’t back it up with the data and access needed to verify those claims.¹⁴ They claim that it represents a great danger to humanity, but then exclude the very people that would be able to impartially confirm the threat, its nature, and come up with the appropriate countermeasures. It is hyperbole. This is theatre, nothing more.

More broadly, AI research is hard or even next to impossible to reproduce—as a field, we can’t be sure that their claims are true—and it’s been a problem for years.¹⁵

They make claims about something working—a new feat accomplished—and then nobody else can get that thing to work as well. It’s a pattern. Some of it is down to the usual set of biases that crop up when there is too much money on the line in a field of research.

A field as promising as AI tends to attract enthusiasts who are true believers in ‘AI’ so they aren’t as critical of the work as they should be.

But some of it is because of the unique characteristics of the approach taken in modern AI and Machine Learning research: the use of large collections of training data. Because these data sets are too large to be effectively filtered or curated, the answers to many of the tests and benchmarks used by developers to measure performance exist already in the training data. The systems perform well because of test data contamination and leakage not because they are doing any reasoning or problem-solving.¹⁶

Even the latest and greatest, the absolute best that the AI industry has to offer today, the aforementioned GPT-4 appears to suffer from this issue where its unbelievable performance in exams and benchmarks seems to be mostly down to training data contamination.¹⁷

When its predecessor, ChatGPT using GPT-3.5, was compared to less advanced but more specialised language models, it performed worse on most, if not all, natural language tasks.¹⁸

There’s even reason to be sceptical of much of the criticism of AI coming out from the AI industry.

Much of it consists of hand-wringing that their product might be too good to be safe—akin to a manufacturer promoting a car as so powerful it might not be safe on the streets. Many of the AI ‘doomsday’ style of critics are performing what others in the field have been calling “criti-hype”.¹⁹ They are assuming that the products are at least as good as vendors claim, or even better, and extrapolate science-fiction disasters from a marketing fantasy.²⁰

The harms that come from these systems don’t require any science-fiction—they don’t even require any further advancement in AI. They are risky enough as they are, with the capabilities they have today.²¹ Some of those risks come from abuse—the systems lend themselves to both legal and illegal abuses. Some of the risks come using them in contexts that are well beyond their capabilities—where they don’t work as promised.

But the risks don’t come from the AI being too intelligent²² because the issue is, and has always been, that these are useful, but flawed, systems that don’t even do the job they’re supposed to do as well as claimed.²³

I don’t think AI system vendors are lying. They are ‘true believers’ who also happen to stand to make a lot of money if they’re right. There is very little to motivate them towards being more critical of the work done in their field.

The AI industry and tech companies in general do not have much historical credibility. Their response to criticism is always: “we’ve been wrong in the past; mistakes were made; but this time it’s different!”

But it’s never different.

The only way to discover if it’s truly different this time, is to wait and see what the science and research says, and not trust the AI industry’s snake oil sales pitch.