“The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con”
I’ve been working on this essay on and off for a few months now. I kept convincing myself that I had to be wrong, this is too dumb to be true, but then the research I did changed my mind again.
I’m now of the opinion that this is how chatbots have managed to convince so many that there’s an intelligence there when there isn’t. More specifically, it looks very likely that the illusion of intelligence is based on the same mechanism as that of a psychic’s con.
Essentially, every question or reply is just a statistically plausible guess and our minds do the work of making it look coherent because of a cognitive bias called subjective validation.
The end result is that we have chatbots with zero actual intelligence—about the same as that of a random number generator—that look to those who want to believe as intelligent agents. Who then get frightened by the “progress” and go on a press tour.
It’s all just a statistical illusion, which I’m calling “The LLMentalist Effect” simply because that’s the sort of silly joke that makes me laugh 😁
But the overall conclusion is not a joke and honestly makes me quite worried about the tech industry’s eager dive into pseudoscience and superstition.
Go read it over on the newsletter site.
Dev links
- "XML is the future". About developer hype culture. Or the pop culture as Alan Kay used to call it.
- "Arnold’s Docuseries: A Case Study in Online Content Pollution – On my Om". The new stuff on streaming is all meh. The old stuff doesn’t pay enough to keep available. Streamers have been turning into second rate VHS rentals from the 90s. Bónusvídeó if you’re Icelandic
- "Amazon’s allegedly “dangerous and illegal” warehouses spur Senate probe | Ars Technica"
- "Announcing my new book, You Deserve a Tech Union. — Ethan Marcotte"
- "Daring Fireball: DPReview Found a Buyer: Gear Patrol". There’s something fundamentally wrong with a management team where their first impulse is to shutter a valuable property and only shop it around when there’s outcry.
- "Cascade Layers are useless* - Manuel Matuzović". Layers have only been out for a short while, but they’re already fundamental to how I structure my CSS. I’m never going back if I can help it.
- "The Dot-com surge - The History of the Web". Those are the glory days that today’s tech industry is trying to recreate.
- "The cavalry have arrived. But not for long. – Hi, I’m Heather Burns"
- Hashing
- "Button types". I really like this idea. Declarative button actions would make many web dev tasks much easier.
- "MDN can now automatically lie to people seeking technical information - Issue #9208 - mdn/yari - GitHub"
- "The Perils of ‘Innovator’ Mindset". Confident idiots who think regulation is for cowards can end up getting themselves killed, and taking innocent people with them.
- "Why I moved on from Figma – No Handoff"
- "How to Kill a Decentralised Network (such as the Fediverse)“
AI links
- "How AI can distort human beliefs. Models can convey biases and false information to users"
- "Inside the AI Factory: the humans that make tech seem human - The Verge"
- 'Get a clue, says panel about buzzy AI tech: it’s being “deployed as surveillance”‘
- "Adobe Stock creators aren’t happy with Firefly, the company’s ‘commercially safe’ gen AI tool". I think calling Adobe’s Firefly ‘ethical’ (which I might have done in the past) is a stretch. It’s more “legal less problematic than other models”, but that doesn’t roll off the tongue.
- "The Best Time to Regulate A.I. Is Today – Pixel Envy". Give existing regulators extra funding earmarked for investigations into AI company misbehaviour. A good chunk of the stuff happening is illegal already.
- "In new AI hype frenzy, tech is applying the label to everything now". But they also define AI so broadly as to include almost any large-scale, statistically-driven computer program.
- "AI Expert Says ChatGPT Is Way Stupider Than People Realize"
- "How to Prepare for the Deluge of Generative AI on Social Media"
- "Exclusive: OpenAI Lobbied E.U. to Water Down AI Regulation | Time". I had been wondering what OpenAI’s EU lobbying strategy looked like because it looks considerably less effective than it’s US strategy.
- "The New CSS - Matthias Ott – User Experience Designer"
- "Google doesn’t want its employees using Bard code - The Register". Who’s willing to bet that there are departments within Microsoft who have given their staff the same warning re Copilot/ChatGPT?
- "That old WIRED ideology - by Dave Karpf". If you’re too young to remember just how awful Wired used to be.
- "Bullet Points: the SSC lives, VC financial logic kills everything it touches, and the difference between new WIRED and old WIRED". Always post the link when somebody points out just how awful Wired has historically been.
- "The massive bug at the heart of the npm ecosystem"
- "On Hate Scaling Laws For Data-Swamps". Highly recommended. This paper looks at the effects of scale on model bias and it looks like bias gets worse with scale. Which is unsurprising, TBH.
- "Authors Sue OpenAI Claiming Mass Copyright Infringement of Novels – The Hollywood Reporter". Completely unsurprising.
- "ChatGPT maker OpenAI faces class action lawsuit over data to train AI - The Washington Post"
- "Next-gen content farms are using AI-generated text to spin up junk websites | MIT Technology Review". Surprise! Who could have predicted this?
- "The AI is eating itself - by Casey Newton - Platformer"
- "AI is killing the old web, and the new web struggles to be born - The Verge". I have mixed feelings about this piece. Yeah, AI spam is going to be very bad for our information ecosystem. But, OTOH most of these platforms were in a very bad way before AI entirely due to their own mismanagement.
The rest
- "Gulls choose what to eat by watching humans, study suggests | Science | The Guardian"
- "Out and Proud for the Miners". Great interview.
- "2,200 Forgotten Vintage Computers Are Being Liberated From a Barn in Massachusetts". Amazing story.
- "Ecological doom-loops: Why ecosystem collapses may occur much sooner than expected". There is no way to restore collapsed ecosystems within any reasonable timeframe. There are no ecological bailouts. In the financial vernacular, we will just have to take the hit.