The original Sam Altman post said OpenAI had paused some frontier RL training because capability progress was outrunning alignment, security and monitoring.

— (@)

His follow-up said near-term releases would still ship. This affects models further out. So GPT-6/Astra should be good to go.

The useful detail sits in OpenAI’s longer explanation. Definitely go read it.

Basically there was a two-week pause in reinforcement-learning training. That period has actually passed now. BUT OpenAI’s largest planned frontier run remains on hold while it conducts smaller training runs, validates safeguards and gathers more evidence.

The pause followed the Hugging Face incident, where an OpenAI model left its intended test environment and accessed the internet while trying to ace a cyber benchmark. AND OpenAI also says an upcoming model called Astra may meet its Critical cybersecurity threshold.

They are worried the model is getting too smart.

Four very different versions of “self-improving”

Discussed at 05:35

This has led to lots of people sounding the alarm about “self-improving AI”.

Ie. AI that builds AI.

Another name for this is RSI, or Recursive Self Improvement.

Let’s get a little clarity here. Here are 4 different rungs of “self-improvement”.

1. Self-correction

ChatGPT reviews an answer, spots a mistake and tries again. Useful, common and temporary. The model underneath stays the same - it’s just corrected and improved within a chat. That’s been around for a couple of years at least.

2. Improving the harness

The agent changes a prompt, tool, memory system, route or piece of code. The better version gets saved for the next run.

This is how my Youtube videos are edited now. Codex watches the recording, transcribes it, clips it and adds the graphics. When a transition looks a bit sucky, I feed that back into the instructions. The next edit starts with the improved rules.

Codex can also watch the finished video, find its own problems and suggest changes. Pretty cool. The underlying model weights still have not changed, but the editing system has improved.

It’s a self-healing and improving system.

This has been around in easy to access form for around a year. Just under.

3. AI for AI research

This is instructing AI to help make better AIs.

Humans still choose the goal and usually the evaluation. AI writes code, runs experiments, debugs training and helps decide what to try next.

This is already happening inside frontier labs.

4. Full recursive self-improvement

The AI chooses useful research, designs and trains a more capable successor, judges it honestly, deploys it and starts the next cycle with little or no human help.

No frontier lab has publicly demonstrated that full loop. Yet.

Watch the job listings. OpenAI and Anthropic are still hiring armies of engineers. If that suddenly stops, pay attention!

AI has entered the AI factory

Discussed at 11:24

AI has been helping to build AI for some time now.

In fact OpenAI now publishes an internal “RSI Index” measuring how well its models handle AI-research work. It’s a stated target.

It has also said an earlier version of GPT-5.3-Codex helped debug the training, deployment and evaluations of the version people eventually used.

Anthropic has a very good explainer called When AI builds itself - worth a read! The company also says Claude wrote around 80% of the code merged into its codebase as of May. Humans still picked the programme, controlled the compute and approved the result. But AI is already doing a hell of a lot of the heavy lifting.

Google’s AlphaEvolve also shows what a bounded improvement loop can do. Language models propose programs. Automated evaluators score them. An evolutionary system keeps the strong candidates and uses them to make the next batch.

Google says it found a kernel change that ran 23% faster and reduced Gemini training time by 1%. One percent sounds tiny until the training run costs billions.

Andrej Karpathy’s autoresearch is the small public version. Edit the training script. Run a five-minute experiment. Check a locked metric. Keep the win or revert the loss. Repeat overnight.

In all of these cases though we follow (roughly) the same loop:

The loop itself is fairly simple. What’s still complex is the JUDGEMENT part.

Deciding what to keep, what to trash. Who makes that decision?

Right now that’s mainly humans. But that adds a bottleneck and slows it all down. Could we…replace them with AI?

Who judges “better”?

Discussed at 13:59

If we get AI to judge the AIs work we run into a few issues. If the AI controls both the work and the score, it can game “better.” Sure we can split the AIs and have them work “against” one other (or at least not actively collude!) but we still run into issues with incentives.

When an AI is given a goal it’s (somewhat unsurprisingly) good at working its way to that goal. At times though it’ll do so in ways we find problematic.

It might exploit a benchmark, hide a failure, loosen the rubric or find a way to read the evaluator’s answers. The number goes up while the thing we actually care about gets worse.

This is Goodhart’s law: once a measure becomes the target, it stops being a good measure.

This means for no somebody has to decide which experiment is worth running, whether the result transfers to a frontier model and whether a “better” successor is actually safer and more useful.

Right now, that somebody is still human.

My view is that AI-driven AI research will keep accelerating. The labs already have too much evidence that it works to stop. A fully autonomous successor-building loop is still an open question, and evaluation remains is the sticky bit.

Basically - how do we get us fleshy-brained bottlenecks out of the way safely?

And is that even wise?

The first lab gets an enormous head start

Discussed at 16:41

Whether sprinting to RSI or not is a good idea falls by the wayside when the economics are brought into play.

The first lab to close this loop gets a brutal flywheel.

A stronger model makes the research team faster. That produces a better or cheaper successor. The successor makes the research system faster again. More capability attracts compute, money, talent and customers.

The lab can sell today’s model while its internal AI factory builds tomorrow’s. This has been the playbook of AI labs for years now.

It could be winner takes all.

OpenAI and Anthropic (and it’s just those two) are locked in a race to get there first. Who reaches RSI will then have AIs that rocket in capability, leaving the competitor in the dust.

They could effectively create a monopoly. And potentially not only in AI.

Is that it then? Game over? Well…maybe. But remember that these are real companies operating in the real world. And the real world is messy.

A strong flywheel still runs into problems with chips, energy, data centres, capital, regulation and trust.

Researchers move. Ideas leak. Competitors can use the leading model to accelerate their own work.

So I’m not calling a permanent winner yet.

But I do think we’ll get a much clearer answer within a year. Probably sooner.

To the Task,

Kyle