Trusting your own judgement on 'AI' is a huge risk
(This is loosely based on a couple of social media threads I posted last week, made longer and more tedious with added detail.)
One of the major turning points in my life was reading my dad’s copy of Robert Cialdini’s Influence: The Psychology of Persuasion as a teenager.
Other highlights of my dad’s library – he was a organisational psychologist before he retired – included books by Fanon, Illich, and Goffman and a bunch on systems thinking and systems theory so, in hindsight, I was probably never not going to be idiosyncratic.
But Cialdini’s book was a turning point because it highlighted the very real limitations to human reasoning. No matter how smart you were, the mechanisms of your thinkings could easily be tricked in ways that completely bypassed your logical thinking and could insert ideas and trigger decisions that were not in your best interest.
He documented tactics and processes used by salespeople, thieves, and con artists and showed how they could manipulate and trick even smart people.
Worse yet, reading through another set of books in my dad’s library – those written by Oliver Sacks – indicated that the complex systems of the brain, the ones that lend themselves to manipulation and disorder, are a big part of what makes us human.
But to a self-important asshole teenager, one with an inflated sense of his own intelligence, Cialdini’s book was a timely deflation as it was specifically written as a warning to people to be careful about manipulation.
He described it as a process where the mechanics of cognitive optimisation in the brain could be deceptively triggered into a semi-automatic sequence that bypassed your regular judgement – his very eighties metaphor was like that of a tape in your mind that went “click whirr” and played in response to specific stimuli.
These are what I tend to call psychological or cognitive hazards. Like the golfer’s sand trap, the only way to win is to avoid them entirely.
- Intelligence is not a defence. Most of these mechanisms and situations feel reasonable and intelligent in the moment.
- Willpower, under these circumstances, is a myth. You can’t “will” your way to resisting these tactics. The only defence is avoidance.
What made me especially receptive to this idea at the time was the recent experience of having been sold a shitty CD player – that was obviously a sub-par model – by an excellent salesman.
“I’m smart. I’m a nerd who knows electronics. Why did I fall for these tricks?”
Because we all do is why.
Of course Cialdini later went on to focus more on teaching people how to use manipulative tactics to convince and sell, but his later lapse of ethical judgement doesn’t disprove the value of his groundbreaking initial book.
The reason why I’m outlining just how weird I was as a teenager and a young man is that software developers in particular are prone to being convinced by these hazards and few in the field seem to have ever had that “oh my, I can’t always trust my own judgement and reasoning” moment that I had.
A recent example was an experiment by a CloudFlare engineer at using an “AI agent” to build an auth library from scratch.
From the project repository page:
I was an AI skeptic. I thought LLMs were glorified Markov chain generators that didn’t actually understand code and couldn’t produce anything novel. I started this project on a lark, fully expecting the AI to produce terrible code for me to laugh at. And then, uh… the code actually looked pretty good. Not perfect, but I just told the AI to fix things, and it did. I was shocked.
(If you don’t know what I mean with “an auth library” just know that it’s the most security sensitive and attacked point of any given web service. The consequences of a security flaw in this kind of library are potentially huge.)
Self-experimentation is gossip, not evidence #
The debate surrounding this has been fairly typical for software development. The authors claimed that an Large Language Model (LLM) agent let them build it faster and more reliably than otherwise, many in software dev are convinced that this is powerful evidence that these tools really work.
It’s not, for a good reason, but it’s also important to note the process here that bypasses the judgement of even smart people.
First off, that project is a single person acting without any control. It has the evidentiary value of a blog post claiming that echinacea cured their cold complete with bloodwork showing no cold virus. That neither proves or disproves the value of echinacea. It’s just gossip.
When all you have is gossip (software development research is not great as it’s genuinely a separate problem domain from computer science) you have to make do with it – the debate about types, TypeScript, and web development is, for example, largely anecdotal gossip not backed by much in terms of structured research – but when you’re trying to answer a question with huge ramifications, you really want proper research.
TypeScript, for those who aren’t familiar, is a Microsoft rewrite of JavaScript that’s incompatible with basic JavaScript in multiple ways and has a long history of breaking projects when updated.
It looks more like an “enterprise” programming language for large institutions, but we honestly don’t have any evidence that it’s genuinely more suitable for those circumstances than the regular JavaScript.
And it makes your project and business directly dependent on Microsoft, which is never ideal.
The decision to use TypeScript over JavaScript, despite there not really being any evidence available that doing so will make the overall system safer and more productive, is relatively harmless. It won’t kill people. It won’t disenfranchise anybody. It won’t lead anybody to being de-banked. It won’t deprive anybody of their right to an education.
Pretty much everything used to argue for or against TypeScript is either from self-experimentation or from anecdotal stories about other people’s self-experimentation.
And that’s where the psychological hazard comes in.
Self-experimenting with psychological hazards is always a bad idea #
Self-experimentation is exactly how smart people get pulled into homeopathy or naturopathy, for example. It’s what makes them often more likely to fall for superstitions and odd ideas. The smart person’s self-identity means they can’t believe their own psychological biases are fooling them.
Don’t self-experiment with psychological hazards! I can’t stress this enough!
There are many classes of problems that simply cannot be effectively investigated through self-experimentation and doing so exposes you to inflicting Cialdini-style persuasion and manipulation on yourself.
Consider homeopathy. You might hear a friend talk about “water memory”, citing all sorts of scientific-sounding evidence. So, the next time you have a cold you try it.
And you feel better. It even feels like you got better faster, although you can’t prove it because you generally don’t document these things down to the hour.
You come away feeling much less sceptical about homeopathy.
“Maybe there is something to it.”
Something seemingly working is not evidence of it working.
- Were you doing something else at the time which might have helped your body fight the cold?
- Would your recovery have been any different had you not taken the homeopathic “remedy”?
- Did your choosing of homeopathy over established medicine expose you to risks you weren’t aware of?
That last part is important as we have, as humans, an extremely poor understanding of probability.
Something that happens one per cent of the time doesn’t register with us. Those odds leave us with an impression that hardly anybody we know has encountered the phenomenon, while it is in fact quite quite common.
Poor treatment of a cold can lead to bronchitis or pneumonia and pneumonia can be lethal.
So a combination of an innocuous disorder and naive self-experimentation can literally kill you.
It’s unlikely, but the chances of it happening are not down to your intelligence or willpower, but is largely dependent on luck.
One of the reasons why I wrote the original LLMentalist post two years ago was I wanted people to understand that chatbots and the like are a psychological hazard.
Experimenting with them can lead to odd beliefs and a serious misunderstanding of both how you and the chatbots work.
You can’t trust your own instincts or judgement about Large Language Models and chatbots because they trigger a number of cognitive biases and psychological “effects” that short-circuit our judgement.
Again, never self-experiment with psychological hazards. It can take you years to unwind the damage.
We need science #
Our only recourse as a field is the same as with naturopathy: scientific studies by impartial researchers. That takes time, which means we have a responsibility to hold off as research plays out, much like we do with promising drugs, but the research is also stalled by the bubble.
Impartial research on “AI” is next to impossible at the moment. It’s like we had a new wonder drug on the market but we had no way of knowing if its risk dynamic is Thalidomide (brutal and horrible deformities), Paracetamol (low lethal overdose threshold), or Penicillin (systemic overuse destroys its effect).
Coming back to the project I cited at the start, there are a number of questions that it can’t answer that are kind of important for you to gauge its validity:
- There are no controls or alternate experiments. We only have the experimenter’s word for this being faster than if he had done it himself.
- We also don’t know if he’d gotten better or worse results by forking a related or similar library and rewriting it to do what he needed.
- We also have to take their (Cloudflare’s) word for it that this is actually code of an equal quality to what they’d get by another method. There’s reason to believe it was not, given the beginner nature of the security defects that have already surfaced.
- And, finally, we don’t know if they were just plain lucky playing roulette with a nondeterministic system.
This is the original sin of software dev: it’s a pop culture where we’re trained to accept gossip as evidence.
That’s fine if you’re debating largely meaningless details like “JavaScript, threat or menace?” but it’s very risky when the system in question is a psychological hazard built out of a Jenga tower of biases and Forer effects.
When that cluster of psychological hazards is being adopted as a solution for everything, everywhere, all at once, the consequences of letting yourself be conned by it are potentially enormous.
Marks become hazards in their own right #
A big risk of exposure to con artists, such as a psychic, is when a smart person is fooled by their own subjective experience and cognitive blindness to probabilities and distribution, refuses to believe they were fooled, and becomes and evangelist for the con.
This happens all the time with psychics, homeopathy, and faith healers. I wrote The LLMentalist Effect with a warning about exactly this problem:
Remember, the effect becomes more powerful when the mark is both intelligent and wants to believe. Subjective validation is based on how our minds work, in general, and is unaffected by your reported IQ.
If anything, your intelligence will just improve your ability to rationalise your subjective validation and make the effect stronger. When it’s coupled with a genuine desire to believe in the con—that we are on the verge of discovering Artificial General Intelligence—the effect should both be irresistible and powerful once it takes hold.
This is why you can’t rely on user reports to discover these issues. People who believe in psychics will generally have only positive things to say about a psychic, even as they’re being bilked. People who believe we’re on the verge of building an AGI will only have positive things to say about chatbots that support that belief.
And it’s not just limited to instilling a belief in the imminent arrival of Artificial General Intelligence. Subjective validation can be triggered by self-experimentation with code agents and chatbots. From the ever useful Wikipedia:
Subjective validation, sometimes called personal validation effect, is a cognitive bias by which people will consider a statement or another piece of information to be correct if it has any personal meaning or significance to them. People whose opinion is affected by subjective validation will perceive two unrelated events (i.e., a coincidence) to be related because their personal beliefs demand that they be related.
Using these tools in a highly subjective context as a self-experiment will make the result feel correct simply by virtue of it being born from your personal environment. Coincidences and random sequences will be seen as cause and effect.
This is why it’s unsurprising that somebody who personally stands to benefit from the “AI” Bubble – he is an early employee for a hosting startup – sincerely believes his self-inflicted con:
I think this is going to get clearer over the next year. The cool kid haughtiness about “stochastic parrots” and “vibe coding” can’t survive much more contact with reality.
I don’t recommend reading it, but you can if you want. It is full of half-baked ideas and shoddy reasoning.
- This harm is fine because we regularly hurt people that way anyway.
- False statements about quality and reliability that have been disproven numerous times in studies.
- Legitimate concerns are countered simply with a command to shove it up your arse.
- Mediocrity is fine. Normal even. No need for anything better.
There’s more. Much more. It’s mostly nonsense, which is why I don’t recommend reading it.
But one reason to highlight the shoddiness of it’s argument is that calls from authority figures are a cognitive hazard in and of themselves and if you aren’t familiar with how deceptive personal experience is when it comes to health, education, and productivity, you might find the personal, subjective experience of a notable figure in the field inherently convincing.
Even otherwise extremely sensible people fall for this, like Tim Bray:
I keep hearing talented programmers whose integrity I trust tell me “Yeah, LLMs are helping me get shit done.” The probability that they’re all lying or being fooled seems very low.
The odds are not low. They are, in fact, extraordinarily high. This is exactly the kind of psychological hazard – lot to gain, subjective experiences, observations free of the context of its impact on other parts of the organisation or society – that might as well be tailor-made to trick developers who are simultaneously overwhelmingly convinced of their own intelligence and completely unaware of their own biases and limitations.
Many intelligent and talented people believe in homeopathy, psychics, or naturopathy. It isn’t a question of their integrity as they are sincere in their beliefs. Nor is it a question of their intelligence as subjective validation is not dependent on a lack of intelligence. Education will only protect you insofar as some education – none of which engineers or software developers will ever encounter – teaches you to not subject yourself to situations where your own biases and subjective experiences can fool you.
This is not a low probability risk. It’s an extremely high probability risk.
And make a note of who is publishing the blog post. Fly is an hosting company so if the dual argument of “AI lets us go faster” and “AI makes mediocre code but we’re slowing down to make it usable” (paraphrased, but that’s essentially one of the arguments made in the post) is their self-contradictory policy, that puts a question mark on hosting with them.
The fundamental argument of the pro-AI crowd in coding is that it makes you go faster and, after you’ve gone faster, you can go slower a little bit to catch the errors caused by going faster. They back this with personal anecdotes that are largely subjective validation of what they want to be true.
The problem, though, with responding to blog posts like that, as I did here (unfortunately), is that they aren’t made to debate or arrive at a truth, but to reinforce belief. The author is simultaneously putting himself on the record as having hardline opinions and putting himself in the position of having to defend them. Both are very effective at reinforcing those beliefs.
A very useful question to ask yourself when reading anything (fiction, non-fiction, blogs, books, whatever) is “what does the author want to believe is true?”
Because a lot of writing is just as much about the author convincing themselves as it is about them addressing the reader.
The only sensible action to take – which was also one of the core recommendations I made in my book The Intelligence Illusion – is to hold off. Adoption of “AI” during a bubble, without extensive study as to the large-scale impact of adoption, is the cognitive, productive, and creative equivalent to adopting a new wonder drug at scale without testing for side effects, complications, or adverse drug interactions.
It’s outright irresponsible.
Again, most who say we need “better AI critique” are either not paying attention to the actual critics or simply do not like being told something they like has flaws that could be catastrophic is adopted at scale.
They inherently don’t accept any critique as valid, otherwise they’d engage with existing critique – such as that of Emily Bender and Alex Hanna – to find points they can act on.
There is no winning in a debate with somebody who is deliberately not paying attention.
Moreover, there is a simple test, a dividing line that explains the schism among developers on the usefulness of “AI” tools for software development and it has nothing to do with “AI” itself.
It’s a question of differing worldviews about what the state of software development was before the “AI” bubble.
Before LLMs arrived, the critics believed that existing software dev was flawed, largely inadequate, and a recipe for future crises, whereas the fans thought things were great but needed to be faster.
The LLM tools are all geared towards making existing – highly flawed – approaches to software development go faster, without addressing any of the fundamental issues that have caused recurring software crises over the years.
On that principle alone, even if you believed in the usefulness of LLMs in general, their application to software development is obviously and inherently a bad idea.
Making a broken thing go faster only makes it worse.
These two groups never agreed before chatbots and copilots became a thing and they are not going to agree today.
Those are two fundamentally different worldviews that will never be compatible.
That’s even before we get into the issues with how rampant cognitive hazards and the mechanics of manipulation are among “AI” tools and within the field of “AI” research.
The beliefs these two groups hold differed before the advent of LLM coding tools and aren’t likely to converge now, no matter how well reasoned the argument is.
We are also being let down by the epidemiology of our beliefs #
What “AI”, homeopathy, naturopathy, and psychic cons have in common isn’t just that they tap into a number of biases and “effects” that the human mind is vulnerable to.
They all belong to larger, more complex fields into topics and phenomena that even expert practitioners often only half-understand.
We still have a lot to learn about the human body, especially the human brain.
We only understand a fraction of what nature has to offer in terms of medicine and biology.
And even “AI” academics regularly talk about how they don’t fully understand how many of their larger models work.
These are perfect conditions for the spread of superstitious beliefs.
As Dan Sperber wrote in Explaining Culture in his attempt to explain the epidemiology of belief:
Half-understood or mysterious reflective beliefs are much more frequent and culturally important than scientific ones. Because they are only half-understood and therefore open to reinterpretation, their consistency or inconsistency with other beliefs, intuitive or reflective, is never self-evident, and does not provide a robust criterion for acceptance or rejection. Their content, because of its indeterminacy, cannot be sufficiently evidenced or argued for to warrant their rational acceptance. But that does not make these beliefs irrational: they are rationally held if there are rational grounds to trust the source of the belief (e.g. the parent, the teacher or the scientist). (p. 91)
This is why Tim Bray wasn’t being irrational when he cited the faith of his peers as a reason for his belief. This is how beliefs work. We rely on our trust of the sources to complete the loop from half-understanding to belief.
And because these ideas are only half-understood and vague, we can fit them in with our other ideas without problems or conflicts. There’s always a way for a developer, for example, to explain away conflicting details or odd events. The vagueness creates room to accommodate contradiction and furthers belief.
It’s no surprise that an almost religious faith in “AI” spreads fast among those open to the idea.
It’s not irrational to belief any of these things and trust your peers, but it is a mistake.
In our current environment – destruction of science funding, a massive “AI” bubble, the normalisation of gossip and hearsay as “facts” in software development – trusting the claims made by those advocating the adoption of “AI” is outright a mistake.
This is specifically the kind of large scale technology that needs thorough scientific testing because, on a micro-level, it might as well be purpose-designed to fool our judgement.