Probably won’t overthrow humanity

Anthropic just published a 53-page report explaining why its AI probably won't try to overthrow civilization. The fact that they needed 53 pages should tell you everything.

This document maps eight pathways by which an AI could cause catastrophic harm from the inside: 𝗽𝗼𝗶𝘀𝗼𝗻𝗶𝗻𝗴 its successor's training data, inserting code 𝗯𝗮𝗰𝗸𝗱𝗼𝗼𝗿𝘀, 𝘄𝗮𝗿𝗽𝗶𝗻𝗴 information that reaches government officials, even 𝗰𝗼𝗽𝘆𝗶𝗻𝗴 𝗶𝘁𝘀𝗲𝗹𝗳 onto outside servers to operate unsupervised.

For each scenario, their researchers assessed 𝗹𝗶𝗸𝗲𝗹𝗶𝗵𝗼𝗼𝗱, 𝗱𝗶𝗳𝗳𝗶𝗰𝘂𝗹𝘁𝘆, and whether current 𝗱𝗲𝗳𝗲𝗻𝘀𝗲𝘀 would catch it.

Their conclusion: risk is "very low but not negligible."

A phrase that sounds clinical until you notice it's doing an enormous amount of work.

What makes this interesting isn't the scary parts.
It's the 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗮𝗹 𝗵𝗼𝗻𝗲𝘀𝘁𝘆.
No other major AI company publishes this kind of self-examination at this level of granularity.

Anthropic is essentially saying: here is a detailed map of how our own product could go wrong, here is what we think we'd catch, and here is what we might miss.

𝗕𝘂𝘁, and this is a big but, several of their own defenses are rated "weak" or "moderate" against the scenarios they're designed for.

So yeah, they gave themselves a B-minus, published it anyway, and the rest of the industry hasn't even shown up for the exam.

𝗧𝗵𝗲 𝗔𝘁𝗹𝗮𝗻𝘁𝗶𝗰 𝗰𝗮𝗽𝘁𝘂𝗿𝗲𝗱 𝘁𝗵𝗶𝘀 𝘁𝗲𝗻𝘀𝗶𝗼𝗻 𝗯𝗲𝗮𝘂𝘁𝗶𝗳𝘂𝗹𝗹𝘆 𝗶𝗻 𝗮 𝗿𝗲𝗰𝗲𝗻𝘁 𝗽𝗿𝗼𝗳𝗶𝗹𝗲.
Anthropic employees openly wish the industry would slow down. The CEO warns AI could displace most white-collar jobs within five years. A co-founder says the capital markets say "go faster," so they do.

Oh, and Anthropic also set up an AI-run vending machine in their cafeteria. It managed inventory and pricing autonomously. Ran the business into the ground within a month.

In the Alpha Arena trading competition, Claude at least traded more cautiously than ChatGPT and Gemini, who each lost 60%+.

So: can't run a vending machine, but won't panic-sell Bitcoin.
𝘗𝘳𝘰𝘨𝘳𝘦𝘴𝘴.

If this is the level of rigor at the most safety-focused company in AI, 𝘄𝗵𝗮𝘁 𝗱𝗼𝗲𝘀 𝗶𝘁 𝗹𝗼𝗼𝗸 𝗹𝗶𝗸𝗲 𝗲𝘃𝗲𝗿𝘆𝘄𝗵𝗲𝗿𝗲 𝗲𝗹𝘀𝗲?

Most orgs deploying AI at scale don't have sabotage risk reports. They have a Slack channel called #ai-experiments and a vague plan to "figure out governance later."

So here's the real question: should pre-deployment risk reports like this be a (somewhat?)regulatory requirement for frontier AI, or does mandating them just create compliance theater that lets less honest companies check a box and move on?

I genuinely don't know. Drop your take.

Link to the report in the comments :)