Anthropic warns AI could soon build itself, calls for slowdown

Friday, 5 June 2026

Anthropic says advanced AI development may need to be slowed or paused.

More than 80% of Anthropic’s code is now written by its AI, Claude.

The company warns AI could soon improve itself without human input.

It predicts small teams could soon do the work of huge organisations.

Anthropic wants governments and AI labs to work together on global rules.

One of the companies at the forefront of the AI revolution is calling for a pause in AI development to allow society time to grapple with its immense implications.

In a lengthy blog post published Friday (NZT), the makers of Chatbot Claude say AI progress is reaching a potentially dangerous speed and 'the human role is narrowing at each step in the development process.”

While gearing up for a trillion-dollar IPO later this year, alongside rival OpenAI, the company is calling for global collaboration between AI labs and governments to slow the current breakneck race to build increasingly powerful systems.

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of technology,” the blog post states.

Anthropic highlights that AI models are increasingly responsible for their own development and nearly full automation of AI progress is becoming plausible.

This is sometimes called ‘Recursive Self Improvement’ (RSI) referring to an AI system that is able to build a better version of itself.

The fear with RSI is that once AI systems can code themselves, progress will speed up exponentially, as each successive AI model builds a better successor and does so faster with each iteration.

RSI could lead to a situation AI researchers sometimes call 'liftoff' or an 'intelligence explosion,' where the capability of the systems shoots up far faster than our ability to build safeguards.

Anthropic says as of May 2026, more than 80% of the code deployed into Anthropic’s codebase was authored by Claude and is nearing a point where humans stop writing code entirely, and shift only to reviewing it.

The fast approaching future

The blogpost outlines three possible scenarios for what comes next.

In the first, where AI progress stalls at the current level but today’s frontier AI capabilities become widely diffused, the company still expects “major changes to occur in the world”.

It says a 100-person company will increasingly do the work of a 1,000-person one, because each employee will sit atop a pyramid of AI agents acting like employees.

In the second scenario, which the authors deem the most likely, the pace of progress will compound as it automates.

In this scenario, 100-person companies will do the work of 10,000- or 100,000-person organizations.

“This would revolutionize knowledge work and government services, but could also be turned to harmful ends, from authoritarian surveillance of whole populations to influence operations that tailor manipulation to each individual and run at a scale no human team could match,” the authors say.

In the third scenario, which they believe less likely, full RSI is achieved and humans play a ‘substantially diminished’ role. By their own admission, the authors struggle to predict the ramifications of humans taking their hands off the wheel completely.

“We do not have good intuitions for what this world would look like, because our economy is currently driven by humans and human-built tools. By its nature, a world driven by fast recursive self-improvement could become dominated by the self-improving model as its capabilities fully eclipse those of humans.”

As Stuff reported this week, experts warn a self improving form of AI could lead to human extinction.

While it says some kind of slowdown would be ‘good’, Anthropic is less clear on how it could be achieved. It notes a poorly implemented slowdown “simply lets the least cautious actors catch up technologically, it could leave everyone less safe.”

The company points to existing agreements on nuclear weapons as a potential model for how to regulate AI, while stressing that the clock is ticking.

“Those regimes took decades to build both the infrastructure and the trust. We don’t have that long. A unilateral pause by one lab, by contrast, is achievable immediately, but accomplishes much less: it would change who the front-runner is, but it would not create the wider deliberative process that is currently missing.”

Anthropic says it will be speaking with its rivals in other labs, policy makers and researchers to create ‘better options for collaboration’ and will publish the results of these discussions.