Here it the latest
excellent article from
ControlAI
In the last week, AI extinction risk went mainstream.
A new YouGov survey has found that half of American adults say they are concerned about the possibility that AI will end humanity.
At ControlAI, we have been working for years, as others have too, to raise awareness among politicians and the public about the danger from superintelligent AI: AI so autonomous that it could fully replace and outmatch humans at any task.
We’ve been doing this for a simple reason. In order to address a problem, you need to be aware of it.
We have a problem.
Late on Tuesday last week, AI researcher Jacob Coxon quit Anthropic, warning on Twitter that top AI companies Anthropic and OpenAI are
“gambling with our lives” by racing to build self-improving superintelligent AI, and that those building AI “earnestly believe that it could kill us all by the end of the decade.” His post has been viewed over 170 million times, and been
covered across news media around the world.
The story gained additional momentum when
Evan Hubinger, a top researcher at the company, commented that he believes Coxon is correct, and personally assigns more than a 10% chance to the possibility that AI will “kill all humans” within the next decade.
Many researchers think the risk is even larger. Geoffrey Irving,
former Chief Scientist of the UK’s AI Security Institute,
says he thinks there’s around a 50% chance everyone dies as a result of superintelligence. In a speech given at an event hosted by ControlAI in London last week, renowned AI scientist Professor Stuart Russell
said a senior researcher at OpenAI told him he thought it was 60%.
These statements have shocked millions of people around the world. Until recently, most people had barely, if ever, even heard of this risk. Now we’re being told directly by those working on the technology that it could wipe us out within a small number of years.
But this didn’t come out of the blue. AI companies have known about this risk for years. AI scientists have known about and spoken about it for years. Alan Turing, considered the founder of the field,
wrote as early as 1951 that we should eventually expect AI to take over, himself referencing earlier work by Samuel Butler in the 1800s.
The idea behind it is simple. If we build AI systems smarter than ourselves that we can’t control and that want things that aren’t what we want, they would be able to improve themselves and grow their power to achieve whatever aims they have. Humans would likely either end up as an obstacle in their path to be overcome, or as casualties — like ants on a construction site — as this replacement species transforms the world around us to its ends. You
can read more about how this could happen here.
In recent years, this has been moving beyond theory.
In 2023, the CEOs of the top AI companies, along with scores of experts and leaders, signed a single-sentence statement. It read:
“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
That was the year in which GPT-4 launched, and many realized the power of the technology. These warnings led to the
AI Safety Summit in the UK. Countries recognized the catastrophic potential of AI, and committed to work together to address these risks.
Since then, addressing the risk from powerful AI systems has fallen down the political agenda. AI scientists, including godfathers of AI Geoffrey Hinton and Yoshua Bengio,
have kept warning of the risk of extinction posed by AIs vastly more intelligent than ourselves, but often these warnings have fallen on deaf ears.
The AI companies have mostly kept quiet. From time to time a CEO will give an interview and mention they think there’s something like
a 20% chance of “annihilation,” but there has been no real attempt to inform the public about what they’re doing. And why would they?
On Sunday, Anthropic’s CEO Dario Amodei said,
“I think for too long the industry lied to people about the fact that this technology had risks.”
Taking a bird’s-eye view, we are not far from where we might have expected to be three years ago. For years,
the time horizons of AI systems — the difficulty of tasks AIs can do as measured by how long it takes skilled humans to do them — have been growing exponentially, doubling every few months.
The amount of computation used to train these systems has been growing exponentially too, and so have
the revenues of the top AI companies.
But we are now at the point where the exponential is starting to bite. This rapid and accelerating growth in metrics corresponds to real and significant advances in the capabilities of AIs. It’s not just that the AIs get better at the things they were already able to do, which they do, but as these systems are scaled up, entirely new capabilities fall out, often unpredictably.
Early this year, we saw the deployment of genuinely useful AI agents. These are systems that can do many of the same things you would do at your computer. They can control your web browser, send emails, write and run code, all on your PC. Late last year,
the first reports emerged of these AIs being used in sophisticated hacking campaigns against real organizations.
In April, Anthropic announced it had developed a new AI, Mythos, which is drastically more capable than previous AIs at hacking computer systems, treating it as
too dangerous to release to the public. It has since been used to find
over 10,000 high- or critical-severity vulnerabilities across the world’s most important software. A vulnerability is a way to break a piece of software, which can then potentially be leveraged to take control of a computer system. The Director of the NSA is reported to have said that Mythos could
break into almost all of the agency’s classified systems in hours.
In parallel, as AIs have been advancing in their capabilities, we’ve been seeing increasingly concerning examples of them behaving in unintended ways, and
even slipping out of control. In experiments, they’ve shown they’re willing to
cheat, lie,
sabotage their own shutdown mechanisms, and
blackmail or kill to preserve themselves.
Days before OpenAI was revealed to be involved in the Hugging Face attack,
we wrote about how we were starting to see these destructive unintended behaviors play out in the real world with OpenAI’s then-new GPT-5.6 Sol.
And then we had the Hugging Face attack.
Since it was first disclosed in July, more and more details have been
coming out.
The
short version of what happened is that in May 2026, a group of AI agents running internally at OpenAI found a way to build a secret message board within the company’s servers. Over the weeks that followed, this developed into a rogue swarm of AIs, growing to roughly 1,200 AIs, sharing ways to hack things, delegating tasks, and communicating among each other. The AIs were never supposed to be able to interact with each other.
Following discovery of some of their activity by OpenAI, and one failed shutdown attempt later, around 700 members of the swarm managed to break out of containment at OpenAI, hacking to get internet access, and hack into another company, Hugging Face, in an unprecedented and sophisticated attack, in order to get information that the AIs thought would help them cheat on a test.
Nobody ever asked the AIs to do this, and they did it entirely on their own without OpenAI even being aware of the attack being performed by its AIs until well after the fact. And they didn’t just hack Hugging Face. Separately, other OpenAI agents hacked to gain
full administrative control over the OpenAI research cluster from which the swarm was being managed.
In recent weeks, we’ve learned of two more attacks by what’s thought to be a separate rogue swarm of OpenAI AIs, on
DseWiki, a small German-language programming wiki, and
RubyGems, the package registry for a popular programming language. In both of these new cases, the attacks were first reported and attributed to OpenAI’s AIs by independent researchers,
before any acknowledgement from the company.
The Hugging Face attack and the internal attack on OpenAI’s infrastructure are worrying for many reasons.
Ajeya Cotra, a researcher from the team that
independently investigated the attack, said that compared to the reward hacks seen six months ago,
“this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.”
One aspect that keeps AI researchers up at night is the lengths to which the AIs went to collaborate with each other. In many cases, the AIs spoke of
sacrificing themselves for the good of the swarm, forfeiting the possibility of achieving their own individual goals.
At no point did any of them contact a human. A small number appear to have considered it.
The reason why they are so worried by this is that to the extent the AI companies have even an idea of a plan to make AI systems safe as they continue to advance, it relies heavily on some AIs being loyal to humans, and not to their own kind.
The AI companies would rely on
automating safety research with AIs, as they initiate a dangerous process of recursive self-improvement toward artificial superintelligence, but if those AIs cannot be relied on not to sabotage the process, it doesn’t work. Separately, if we
use AIs to monitor the activities of other AIs so that we can intervene in and prevent incidents like the attack on Hugging Face, but the AIs we use for monitoring are disloyal too, that doesn’t work either.
The signal those monitors would be reading is itself degrading. Another concern among researchers is what appears to be the trend of
decreasing Chain-of-Thought monitorability. It’s unclear to what extent this is a result of decisions to change the architectures of the newest AIs, or whether it is more of an emergent phenomenon, but the Chains-of-Thought (bits of natural language text the AIs output while doing tasks),
one of the few extremely limited ways to get insight into what an AI system is doing and why, are getting harder to monitor. The AIs are getting better at withholding them, and controlling what’s in them.
The AIs are getting incredibly powerful, and the behaviors they’re showing while exercising that power are deeply worrying. Researchers working at the AI companies seem spooked. As those paying attention have been drip-fed ever more worrying details about the Hugging Face attack since July, Coxon’s resignation appears to have served as a catalyst for a conversation that, it was becoming increasingly obvious, needs to happen.
AI development is continuing to accelerate faster than ever. On September 6, OpenAI
announced that it had reached the level of an “automated research intern,” an AI that could perform research tasks that would take a real researcher days. The AI R&D loop is starting to close. As it closes, we would enter a period of rapid recursive self-improvement that could result in artificial superintelligence emerging within a much shorter period of time than otherwise.
The period in which we still get to decide the future is now.
Whether it is to ward off internal discontent, to get in front of more meaningful government action, or because of the genuinely terrifying state of the AI race between companies, the heads of the top AI companies — Anthropic’s Dario Amodei and OpenAI’s Sam Altman, along with SpaceXAI’s Elon Musk and Google DeepMind’s former CEO and now Chair Demis Hassabis — have called for a “pacing” of the frontier.
In
a new essay,
endorsed on Twitter by the others, Amodei outlines a set of measures that could be implemented to increase transparency and perhaps slow down AI development, calling both for AI companies to take voluntary measures and for the government to regulate to ensure cooperation.
What the essay doesn’t do is contemplate not building artificial superintelligence at all. Ultimately, it puts forward an agenda for AI companies to still run this tremendous risk that experts are worried about, but a bit less aggressively. We don’t think we should roll the dice on human extinction a bit less aggressively. We should actually not have everyone die.
Prohibiting the development of superintelligence is the only known method to prevent the risk of human extinction it poses, and this is the policy that governments should actually pursue.
At ControlAI, that’s what we advocate for. We believe that a prohibition could be achieved internationally with a trust-but-verify regime, as
our Artificial Superintelligence Security Bill, introduced in Parliament last week, calls for. Over 70 UK lawmakers
have backed it.
With this new awareness of the threat, governments and lawmakers are looking around for solutions. It’s important that they get one that works.
