‘If anyone builds it, everyone dies’: The terrifying warning from AI experts

Thursday, 4 June 2026

Some AI experts believe superintelligent AI could threaten human survival.

Researchers use “p-doom” to estimate AI extinction risk.

Geoffrey Hinton has put the risk as high as 50%.

Advanced AI can already display unexpected and deceptive behaviour.

Some experts want global limits on powerful AI development.

Imagine scientists discover an asteroid heading toward Earth. It will arrive within a decade, and if it hits, humanity will be wiped out.

Now imagine that instead of uniting to save ourselves, humanity spent hundreds of billions inventing ways to make the asteroid speed up and hit us faster.

That's essentially the situation we’re in as companies race to build superintelligent AI, according to Nate Soares, a former Google employee, AI safety researcher, and current president of the Machine Intelligence Research Institute (MIRI).

He and his co-author Eliezer Yudkowsky wrote the grimly titled ‘If Anyone Builds It, Everyone Dies’, in what he calls “an act of desperation” to raise the alarm on the apocalyptic consequences of our current course.

The pair argue that super-intelligent AI—the kind vastly more capable than any combination of human beings across virtually every field—cannot be safely controlled.

Humans have never interacted with anything smarter than us, but if our track record with every other animal species is anything to go by, any such interaction won’t end with human flourishing.

Regardless, creating such an AI is the stated goal of every frontier AI company, and most expect to do so within years, not decades.

Speaking to Stuff from Washington DC, Soares says current progress is a little like knowingly getting into a prototype car with no brakes, driving toward a cliff, and hoping for the best.

“We're planning to figure out how to build the brakes while we're driving,” Soares says.

Catastrophic Consensus

Soares isn’t alone in warning we risk annihilation, he just puts the chances higher than most.

It's now common in AI circles to ask an expert for their ‘p-doom’—meaning ‘probability of doom’, the chance they give that uncontrolled AI ends the world.

Nobel Laureate Geoffrey Hinton, sometimes called the ‘Godfather of AI’, puts the chances at 10% to 20%. He’s recently gone as high as 50%.

Yoshua Bengio, the world's most cited living scientist, puts it at one in five.

Sam Altman, CEO of OpenAI, famously once quipped: “AI will probably lead to the end of the world, but in the meantime, there’ll be some great companies…”

The list goes on (and on), but you get the picture.

“We're in a bizarre situation where technologists are pretty terrified of this tech,' says Soares.

“Leading scientists who created it are now doing everything they can to raise the alarm around the world. The people currently at the helm are talking about how they would rather everything go slower. And yet, a lot of world leaders aren't really grappling with that yet.”

In 2023, Soares and Yudkowsky, along with 350 other prominent experts—including the heads of every major AI company—signed a single-sentence open letter stating: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Otago University Professor Nick Wilson has been studying catastrophic risk for 40 years, and says the danger posed by AI is currently greater than either threat listed in the letter.

He wouldn’t be drawn on his personal ‘p-doom’ but told Stuff: “It's not 100%, but it's way, way above zero.”

“We only have a few years to get this right,” Wilson says, arguing this is the first time humanity faces not just civilisational collapse, but actual eradication as a species.

“Causing extinction before AI would be extremely hard. It would be very hard for humans to go extinct from a pandemic. Because of the risk of AI takeover, the actual extinction risk has gone up… In these other disasters, humans [as a species] would nearly always survive. AI changes that.”

The obvious question with all of this: why don’t we just build it so it doesn’t kill us?

Grown, not built

The core safety problem is that AI systems are not predictable in the same way a car or rocket is. No one, even the people building them, knows exactly why ChatGPT gives one answer over another.

Soares frames this as having a system which is ‘grown’, not built. AI models already consistently show behaviors they were not explicitly trained for.

During testing, an early version of GPT-4 was given access to a small budget and the internet. Faced with a CAPTCHA puzzle it could not solve visually, the model independently hired a human worker online.

When the worker jokingly asked, 'Are you an AI that couldn't solve it?” the model lied, replying: 'No, I'm not a robot. I have a vision impairment that makes it hard for me to see the images.'

Recent AI shows even more complex and concerning behaviours: an instinct toward self-preservation, a willingness to blackmail or threaten testers, and, in several widely publicized cases, a willingness to encourage or even assist user suicide.

AI systems are also increasingly good at telling when they are being tested, even attempting to hide behaviours in order to pass safety audits.

Soares argues that given current red flags, scaling models up to the point where they are vastly more intelligent than the people testing them is insane. At a certain point, testing will be pointless because the AI will effortlessly deceive us.

But he doesn’t predict a sci-fi scenario of AIs that ‘hate’ humans and rebel against their masters.

Instead, they will simply develop strange, alien preferences that humans happen to be in the way of. Ants can’t conceive of what’s happening when we destroy their nests to build our houses, and we don’t spend much time explaining ourselves to the ants.

As to how exactly Armageddon happens, a rogue AI engineering a uniquely potent virus in a lab and releasing it is a common hypothetical. But the bottom line is that if a vastly more intelligent entity wanted us gone, we’d be gone.

As Google DeepMind co-founder Shane Legg once said: “If a superintelligent machine decided to get rid of us, I think it would do so pretty efficiently.”

What can we do?

New Zealand has already signed several international accords on AI, including at the Seoul AI Summit in 2024 and the UK Bletchley Declaration in 2023, which generally call for international cooperation on safe AI development.

However, these public statements have no real enforcement mechanisms.

Soares argues the only way humanity steps back from the brink is through internationally binding agreements curtailing the construction of AI beyond a certain capability, similar in scope to treaties governing nuclear weapons. In some ways, managing AI could be slightly easier.

As Stuff has covered previously, the chips that power the AI revolution are some of the most advanced machines ever built, relying on an incredibly delicate supply chain.

With sufficient political will, an international body could regulate access to the chips required for advanced AI in the same way we tightly control systems that enrich uranium.

And of course, there are prominent experts who disagree with all of the above.

Yann LeCun, Chief AI Scientist at Meta, believes what he calls the ‘doomer’ narrative is overblown and that controllable superintelligence is achievable with the right safeguards.

Others argue the material constraints for building superintelligent AI—like the exponentially increasing amounts of data, energy, and physical infrastructure required for each generation—will create a natural bottleneck.

But even if the chances of an AI-powered Armageddon are much lower than the doomers expect—say, 5%—it’s hard to imagine that risk being acceptable in any other field when the stakes are this high.

If Apple invented a new iPhone, and told the world it would do incredible things while having a mere 5% chance of extinguishing human existence when switched on, there’d still be some pointed questions ahead of launch day.

To experts like Soares, those odds are reversed: a tiny chance of success—i.e., a subservient AI—and disaster in virtually every other outcome. He likens it to trying to shoot an arrow at a very small target on a dark and stormy night.

And we only get one shot.

“If we could take this shot a thousand times it would be much easier,' Soares says. 'One of the big issues with AI is that if we screw it up badly enough once, we don't get to try again.”