One man quits Anthopric and the world goes mad

Jacob Coxon quit Anthropic last week and accused both Anthropic and OpenAI of "gambling with our lives".

Within days his resignation post had reached well outside the normal AI crowd, including on the BBC and CNN.

Then people started looking at how it got there. A Wall Street Journal exclusive appeared before his post. AI safety people amplified it almost immediately. Parker Thayer traced some of those accounts to organisations with overlapping donors, including people who backed Anthropic early. The word "psyop" started flying around.

I went through the posts, interviews and funding trail on Friday's live. There are fair questions about who knew Coxon was going public and who helped spread the story.

But a newspaper exclusive obviously involves planning. And AI safety people sharing a warning from someone in their circle is hardly a shock. The donor connections deserve a look, but I haven't seen evidence that this was an organised political operation as some are claiming.

What Coxon is warning about

His concern is recursive self-improvement: AI helps build the next generation of AI, which then helps build a more capable one.

Imagine today's model writing training code for tomorrow's, and tomorrow's model finding a better way to build the one after that. Labs already use AI in their research. I went through the different levels of "AI building AI" here.

The frightening version is a cycle moving faster than we can test or control it - which people at the main frontier labs are hinting we are nearing.

Coxon thinks the labs are racing towards it anyway. Evan Hubinger, an Anthropic alignment lead, backed the concern and said he personally puts the chance of AI killing humanity above 10% in the next decade.

This is someone who still works at Anthropic! Anthropic certainly don’t seem to be cracking down on this sort of talk - in fact the next day they released info to the NYT about people using Claude to try to make biological weapons.

They are leaning in. Why?

Well, Coxon calls for coordination and, potentially, a temporary stop to capability improvements.

Anthropic was asking for coordinated pacing before he resigned. Coxon's post gave an existing campaign a huge new audience.

But I do worry about the largest labs helping design rules that smaller competitors cannot afford to meet.

A pause that only constrains US companies has the China problem too.

How would an international agreement work? What exactly would count as "superintelligence"? Who checks compliance?

Those are much harder questions than choosing whether Coxon is a hero or a villain.

So yes, investigate the publicity. Ask who benefits. But also read what Coxon actually wrote please rather than just indulge in speculation! I've put the posts and the longer argument in the video.

To the task,

Kyle