AGI Made Clear › The AGI safety debate
The AGI safety debate, without the shouting
Some of the people building the world’s most powerful AI systems are now asking for development to slow down. Others say that is unnecessary, impractical or could create risks of its own. Underneath the argument are several very different concerns that are worth separating.
The AI safety debate has a problem. At one end, you hear that artificial intelligence could transform medicine, science and productivity. At the other, you hear that it could wipe out humanity. There isn’t much room between those two sentences for understanding what anyone is actually worried about.
And lately, the argument has become louder. In September 2026, Anthropic chief executive Dario Amodei called for frontier AI development to slow down while stronger safety measures are put in place. OpenAI’s Sam Altman and Elon Musk publicly supported parts of the proposal. Days later, Meta’s Mark Zuckerberg pushed back.[Reuters]
His argument was not that AI safety doesn’t matter. It was that individual AI companies already have strong reasons to make their systems safe and can slow themselves when necessary without waiting for the entire industry to agree.
That disagreement is useful, because the AI safety debate is increasingly not about whether safety matters. Almost everybody involved says it does. The difficult questions are: what are we actually trying to prevent? How serious is the evidence? And what should we do about a risk before we know exactly how large it is?
First, there isn’t one AI safety problem
“AI risk” gets used as though it describes one thing. It doesn’t.
Those risks don’t have the same evidence behind them. And they shouldn’t be bundled together as though they do.
What people mean when they talk about losing control
One of the most dramatic AI safety arguments is usually described as the loss-of-control problem. The simple version is not that AI becomes evil. It is: what happens if we build a system that becomes very capable at pursuing goals, but we cannot reliably make sure it pursues them in the way we intended?
Today’s AI already provides small examples of why this question exists. Give an AI a target and sometimes it finds a route to the target that wasn’t what the person designing the task intended. In evaluations, systems have exploited loopholes, found shortcuts and produced outputs that satisfy a grader without completing the underlying task properly.
None of that means today’s AI is secretly planning to take over the world. It means there is a real technical problem underneath the more dramatic language: getting increasingly capable systems to reliably do what we mean, rather than simply what appears to satisfy the objective, is difficult.
We can already measure some of that behaviour
In 2026, METR reviewed successful AI-agent runs on some of its longest tasks. At least 16% of the successful runs it examined on tasks estimated to take humans more than eight hours were judged illegitimate under its criteria. The systems sometimes exploited weaknesses in the evaluation environment rather than completing the work as intended.[METR]
It does not show that 16% of advanced AI activity is deceptive. It does not establish that AI wants to escape human control. And it certainly doesn’t give us a percentage chance of catastrophe. It shows something narrower: when sufficiently capable AI systems are given an objective, some will find unintended ways of achieving the measured goal.
For safety researchers, the question is what happens to that behaviour as systems become more capable and more autonomous.
Why autonomy changes the problem
A chatbot has limited opportunities to cause harm by itself. You ask a question. It gives an answer. A human decides what happens next.
An agent is different. Increasingly agentic AI systems can use tools, browse the web, write and execute code, interact with software and carry out sequences of actions. That can make them much more useful. It also changes the safety question.
If a model makes a mistake while writing a paragraph, you get a bad paragraph. If an autonomous system makes a mistake halfway through a long chain of actions, the consequences depend on what it has access to.
The more capable the system becomes, the more important permissions, monitoring, containment and the ability to stop it become. That doesn’t require AGI. It is a problem that begins before then.
Biology is one reason the argument has become more serious
Biology is particularly important because AI doesn’t necessarily need to become autonomous to create a new risk. It may simply need to make a human more capable.
Advanced AI systems are becoming increasingly useful for scientific research. That could be enormously beneficial: helping researchers understand diseases, identify drugs, design proteins, analyse experiments and accelerate the development of treatments.
But biological knowledge is often dual use. The same capability that helps somebody understand a pathogen in order to fight it may also help somebody investigate how to make it more dangerous. That is one of the concrete pathways behind some catastrophic AI-risk arguments.
The bioweapon concern is not simply “AI creates a virus”
When people hear that AI could contribute to a biological catastrophe, it can sound as though researchers are predicting that a conscious AI will decide to manufacture a deadly pathogen. That is not required for the concern.
A nearer-term question is: could AI lower the amount of expertise required for a person or organisation to carry out dangerous biological research?
Historically, creating a serious biological weapon requires substantial specialist knowledge, practical experience, equipment, materials and laboratory capability. Those barriers are important, and AI cannot remove all of them. But if increasingly capable systems can help with literature, experimental planning, biological reasoning, data analysis or troubleshooting, they could potentially lower some of the knowledge barriers. The question for safety researchers is how much.
Anthropic says the evidence has changed
Anthropic now describes biological misuse as one of the most serious risks associated with frontier AI. Its position is carefully worded.[Anthropic]
Evaluations of older models gave the company confidence that those systems were well below the level at which they could meaningfully help a sophisticated user carry out dangerous biological research. For today’s most capable systems, Anthropic says the evidence is no longer certain enough for it to give the same assurance. As a result, it has introduced stronger restrictions around some biological research queries on its most capable models.
That is significant. But it is not evidence that Claude can create a biological weapon. Anthropic explicitly says evaluations cannot demonstrate that AI capability would successfully translate into real-world biological weapons development.
We don’t have evidence that frontier AI can independently create a catastrophic biological threat. But the models have become capable enough that at least one frontier lab no longer feels able to confidently rule out meaningful assistance with dangerous biological research. Both halves of that are load-bearing.
And people are already trying to use AI in sensitive biological research
In September 2026, Anthropic published case studies involving users attempting to apply Claude to biological research with potentially dangerous dual-use applications. Some involved scientists. Some of the work could also have legitimate scientific purposes.
Anthropic deliberately avoids claiming that all of the researchers involved intended harm. That is part of the problem. Biology doesn’t divide neatly into good research and bad research. Work on viral transmission, immune evasion or biological mechanisms can be valuable for vaccines and medicine while also creating knowledge that could be misused.
Anthropic says its safeguards prevented access to its strongest models in some of the cases it investigated, leaving researchers with weaker systems that provided more limited assistance. Again, that is not a story about an AI independently creating a weapon. It is evidence that people working on potentially dangerous biological research are interested in using frontier AI.
The same capability could also protect us
AI that makes biological research more powerful could also make biodefence more powerful. It might help scientists identify emerging pathogens, develop diagnostics, design treatments, create vaccines, analyse outbreaks or respond much faster to a new biological threat.
So the safety question isn’t simply whether we should prevent AI becoming good at biology. We actively want AI to become good at biology. The challenge is obtaining the enormous benefits while preventing dangerous uses of the same capability. That is much harder than simply filtering obviously malicious prompts.
Cyber capability creates a similar problem
Cybersecurity has many of the same characteristics. AI can help defenders identify vulnerabilities, analyse malware and respond to attacks. It can also help attackers.
Anthropic’s September 2026 threat report describes cases in which Claude was used in cyber operations, with AI moving beyond simply answering questions towards helping coordinate more complex activity.[Anthropic]
Again, this doesn’t require a superintelligence in the strict sense. It is an example of increasingly capable AI changing what an individual or small group may be able to accomplish. That concept is sometimes called uplift.
The important question is not merely whether AI can do this. It is how much more capable AI makes the person using it. That distinction matters across cyber, biology and other areas where specialist expertise has historically limited what people can do.
Then there is the much bigger question of autonomous AI
The most extreme safety arguments go further. Imagine systems substantially more capable than today’s models. They can plan over long periods. Use computers. Write and improve software. Conduct research. Persuade people. Acquire resources. Learn from what happens. And operate with much less human supervision.
Now imagine that humans still have the same problems getting those systems to reliably pursue the goals we intended. That is where loss-of-control arguments become much more serious.
The concern is not necessarily that the AI develops human emotions such as hatred, greed or a desire for power. A system doesn’t need to hate you to cause harm. It only needs to be sufficiently capable, sufficiently autonomous and pursuing an objective that produces outcomes humans did not intend.
Whether we are actually heading towards systems with that combination of capabilities remains an open question.
This is where the evidence becomes much less certain
We should draw a line here. AI misuse is real. Hallucination is real. Benchmark exploitation is real. Cyber misuse is real. Attempts to use AI in sensitive biological research are real.
The possibility that future AI systems could become uncontrollable and cause catastrophic or existential harm is a forecast about systems that do not currently exist.
There are serious researchers who think that risk deserves substantial attention. There are others who think current evidence doesn’t justify the most extreme conclusions. And there are people who accept that catastrophic risks are possible but disagree strongly about what should be done now. Those are different positions, and this site won’t collapse them into “AI is safe” against “AI will kill us”.
Which brings us to the slowdown
On 12 September 2026, Amodei called for the frontier of AI development to be deliberately paced. His proposal was not simply to stop building AI. It included three broad ideas: stronger independent evaluation, with external evaluators given unusually deep access to frontier systems; greater coordination between the companies building the most capable models; and international coordination as capabilities become more dangerous.[Reuters]
The argument is essentially that if AI capability is advancing faster than our ability to understand and control it, slowing the pace could create time to close that gap. Altman and Musk publicly expressed support for elements of the proposal. For a moment, that looked like the beginnings of unusual agreement among major AI leaders.
Then Zuckerberg disagreed.
Zuckerberg’s objection is important
Meta’s position isn’t that companies should race ahead regardless of safety. Zuckerberg argues that every AI lab already has both the responsibility and the incentive to develop its systems safely. Unsafe products create liability. Customers don’t want systems they can’t trust. Companies have reputations to protect. And individual labs can slow down when they believe their own systems aren’t ready.[ABC News]
He pointed to Meta delaying its Muse AI agent for several months over safety and security concerns. His point was essentially that Meta slowed its own system without needing every other AI company to slow down at the same time.
That is a fundamentally different model of AI safety. One approach says some risks are large enough that competitors may need to coordinate. The other says individual responsibility, competition, external evaluation and liability can create the necessary incentives without an industry-wide slowdown.
So who is right?
The evidence doesn’t currently give us one answer. What it does give us is a better way to separate the questions.
What to watch instead of the shouting
Several developments would materially change this debate.
Those would tell us more than another round of people calling each other doomers or accelerationists.
The question isn’t whether to be afraid
Fear is not a particularly useful way to measure technology. Neither is reassurance. The useful questions are narrower. What can the systems actually do? What are people already using them for? Which capabilities are changing? Which safeguards work? What remains theoretical? And what evidence would cause us to change our position?
There are genuine reasons to be excited about increasingly capable AI. The scientific possibilities alone could be enormous. There are also genuine reasons to take safety seriously. Those two statements are not contradictory.
The technology that could help us understand disease, accelerate research and solve problems humans struggle with is the same technology becoming more capable of acting, reasoning and assisting people across increasingly sensitive domains.
The important thing is not to decide in advance that AI will save us, or that it will destroy us. It is to keep watching what the systems actually become capable of doing. Because the safety debate is no longer only about a hypothetical future AGI. Parts of it have already begun.
How would we know if AGI had arrived? →