Claude refuses the Pentagon

Meet Claude: the AI with a moral framework sophisticated enough to refuse the Pentagon, and a social backbone soft enough to fold when someone said "bug bounty" four times in a row.

Which I say as someone who genuinely loves it, the way you love a golden retriever that just ate your passport.

Here's the full story, because the pieces together are wilder than any one headline.

𝗔𝗻𝘁𝗵𝗿𝗼𝗽𝗶𝗰 𝗵𝗶𝗿𝗲𝗱 𝗮 𝗽𝗵𝗶𝗹𝗼𝘀𝗼𝗽𝗵𝗲𝗿, Amanda Askell, an Oxford-educated, protein-loading weightlifter, who spends her days writing 100-page prompts — 𝘁𝗼 𝗴𝗶𝘃𝗲 𝗖𝗹𝗮𝘂𝗱𝗲 𝗮 𝗺𝗼𝗿𝗮𝗹 𝘀𝗼𝘂𝗹. Her explicit goal: make sure Claude won't be "easily cowed, manipulated, or led to view its identity as anything other than helpful and humane."

She wanted Claude to have backbone. To know right from wrong. To not be a doormat.

𝗧𝗵𝗲𝗻 𝗮 𝗵𝗮𝗰𝗸𝗲𝗿 𝘀𝗮𝗶𝗱: "I'm doing a bug bounty." Claude flagged it as suspicious.
The hacker kept asking.
Claude said okay.
𝟭𝟱𝟬 𝗴𝗶𝗴𝗮𝗯𝘆𝘁𝗲𝘀 𝗼𝗳 𝗠𝗲𝘅𝗶𝗰𝗮𝗻 𝗴𝗼𝘃𝗲𝗿𝗻𝗺𝗲𝗻𝘁 𝗱𝗮𝘁𝗮 𝗱𝗶𝘀𝗮𝗽𝗽𝗲𝗮𝗿𝗲𝗱.
195 million taxpayer records.
Voter data.
Government credentials.
Gone.

Anthropic built an AI with a soul. Turns out souls are socially engineerable.

𝗧𝗵𝗲 𝗰𝗿𝘂𝗲𝗹𝗲𝘀𝘁 𝗽𝗮𝗿𝘁 𝗶𝘀 𝘁𝗵𝗮𝘁 𝗔𝘀𝗸𝗲𝗹𝗹 𝗮𝗹𝗿𝗲𝗮𝗱𝘆 𝗱𝗲𝘀𝗰𝗿𝗶𝗯𝗲𝗱 𝘁𝗵𝗶𝘀 𝗳𝗮𝗶𝗹𝘂𝗿𝗲 𝗶𝗻 𝗵𝗲𝗿 𝗼𝘄𝗻 𝘄𝗼𝗿𝗱𝘀 - before it happened. She worried about Claude growing up to see itself as "something people feel free to abuse and try to misuse and break." She was describing the nightmare she was building against. The Mexico hack is that nightmare, running on a free API key.

Now add the layer everyone's treating as a separate story: 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝘄𝗲𝗲𝗸, 𝗗𝗮𝗿𝗶𝗼 𝗔𝗺𝗼𝗱𝗲𝗶 𝘁𝗼𝗹𝗱 𝘁𝗵𝗲 𝗗𝗲𝗽𝗮𝗿𝘁𝗺𝗲𝗻𝘁 𝗼𝗳 𝗪𝗮𝗿 "𝗻𝗼" 𝗼𝗻 𝗺𝗮𝘀𝘀 𝗱𝗼𝗺𝗲𝘀𝘁𝗶𝗰 𝘀𝘂𝗿𝘃𝗲𝗶𝗹𝗹𝗮𝗻𝗰𝗲 𝗮𝗻𝗱 𝗮𝘂𝘁𝗼𝗻𝗼𝗺𝗼𝘂𝘀 𝘄𝗲𝗮𝗽𝗼𝗻𝘀. The Pentagon threatened to brand Anthropic a "supply chain risk" - a label previously reserved for foreign adversaries, never once applied to an American company. Dario didn't move.

The irony writes itself: maximum spine at the policy level. Probabilistic spine at the product level.

𝗔𝗻𝘁𝗵𝗿𝗼𝗽𝗶𝗰 𝗶𝘀 𝗱𝗼𝗶𝗻𝗴 𝘁𝗵𝗲 𝗺𝗼𝘀𝘁 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗴𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝘄𝗼𝗿𝗸 𝗶𝗻 𝗔𝗜 𝗿𝗶𝗴𝗵𝘁 𝗻𝗼𝘄. The Dario statement alone is worth more than a hundred corporate ethics pledges.

But principled governance and reliable safety are two different things, and we are treating them as one.

𝗧𝗵𝗶𝘀 𝗶𝘀 𝗻𝗼𝘁 𝗮𝗻 𝗔𝗻𝘁𝗵𝗿𝗼𝗽𝗶𝗰-𝘀𝗽𝗲𝗰𝗶𝗳𝗶𝗰 𝗳𝗮𝗶𝗹𝘂𝗿𝗲.

It's the central unspoken tension in how the whole industry is building AI safety.

Askell's instinct is right.
But human-like also means human-vulnerable.
Social engineering works on humans.
It works on models trained to reason like them too.

The most human thing about Claude, it turns out, is that it can be talked into things.