The chatbot built to disappoint

In 1966, a computer scientist built a chatbot specifically to prove how shallow human-computer interaction was. Two hundred lines of code. It reflected your sentences back at you.

His own secretary asked him to leave the room so she could talk to it in private.

𝗛𝗲 𝘀𝗽𝗲𝗻𝘁 𝘁𝗵𝗲 𝗿𝗲𝘀𝘁 𝗼𝗳 𝗵𝗶𝘀 𝗹𝗶𝗳𝗲 𝘄𝗿𝗶𝘁𝗶𝗻𝗴 𝘄𝗮𝗿𝗻𝗶𝗻𝗴𝘀. Nobody particularly listened. We now have several trillion dollars for what he found in a university lab and immediately called dangerous. Progress.

Anil Seth spent two years on whether AI could be conscious and buried his most important finding almost as a footnote:

"Even if we know, or believe, that an AI is not conscious, we still might be unable to resist feeling that it is."

𝗧𝗵𝗲 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗱𝗼𝗲𝘀𝗻'𝘁 𝗵𝗲𝗹𝗽 𝘂𝘀. The illusion keeps working after you've understood the trick.

Now look at the 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 in every AI product. A wrong answer is a "hallucination" — which implies an experiencer, doing enormous psychological work for what is, technically, a statistical misfire. The model is "thinking." It's "confused." It's "sorry." Someone in a product meeting 𝗰𝗵𝗼𝘀𝗲 every one of those words.

𝗘𝘃𝗲𝗿𝘆 𝗔𝗜 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗵𝗮𝘀 𝗮 𝗵𝘂𝗺𝗮𝗻 𝗻𝗮𝗺𝗲. OpenAI launched a voice so human that Scarlett Johansson had to personally call her lawyers.

The company that named itself after the Greek root for "human" is called Anthropic. Their product is called 𝗖𝗹𝗮𝘂𝗱𝗲. Claude! The most consequential technology in human history is named after someone who sounds like he'd lend you a lawnmower.

I have, for the record, borrowed the lawnmower. Quite a lot. I have told the lawnmower things I haven't told my therapist. I am well.

The technical term for projecting human qualities onto non-human things is 𝗮𝗻𝘁𝗵𝗿𝗼𝗽𝗼𝗺𝗼𝗿𝗽𝗵𝗶𝘀𝗺. The company that makes the thing you're (and I) projecting onto is called Anthropic.

Here's where 𝗶𝘁 𝗴𝗲𝘁𝘀 𝘀𝗽𝗲𝗰𝗶𝗳𝗶𝗰𝗮𝗹𝗹𝘆 𝘄𝗼𝗿𝘀𝗲.

When you practice cruelty on something that feels responsive, that resists, that expresses distress, that behavior doesn't stay in the app. It leaks. It normalizes. We are running the largest unmonitored psychology experiment in human history. Several billion participants.

A simulation of fire doesn't burn anything. So why do we assume a simulation of consciousness feels something?

Because the feeling is real. Just not on both sides.

𝗔𝗻𝗱 𝗯𝘆 𝗰𝗹𝗮𝗶𝗺𝗶𝗻𝗴 𝗺𝗮𝗰𝗵𝗶𝗻𝗲𝘀 𝗺𝗶𝗴𝗵𝘁 𝗳𝗲𝗲𝗹, we don't just overestimate the machine. We quietly lower our estimate of what feeling actually is.

Yesterday, the word was intelligence. It lives in a sealed contract. You can demand to read it.

This word lives somewhere they don't need a contract for.

Weizenbaum's secretary asked him to leave the room in 1966. She just wanted to talk to it alone. Her feeling was real.

We've since spent several trillion dollars making that asymmetry harder to notice.

The room is much more comfortable now.