Should we slow down AI?

First, nobody is proposing that we stop using AI

This distinction gets lost remarkably quickly. “Slow down AI” sounds as though somebody wants companies to switch off their assistants and stop developing artificial intelligence. That isn’t what is being proposed.

The argument is about the frontier: the small number of organisations training the world’s most capable systems and trying to make the next generation significantly more powerful than the last.

AI products would continue to exist. Companies could continue building applications. Businesses could continue adopting AI. Researchers could continue working on safety and other areas.

The proposal is about deliberately reducing the speed at which the maximum capability of frontier models advances, if safety systems are failing to keep pace. That is a much narrower idea than “pause AI”.

Why would somebody building AI want to slow it down?

Because capability and control may not be improving at the same speed.

The systems are getting better at coding, research, mathematics, using computers, operating tools, carrying out longer pieces of work, scientific reasoning and, increasingly, acting with less direct human involvement.

Those capabilities can be enormously useful. But each one can also create a different kind of risk. Better coding can help software engineers and cyber attackers. Better biological reasoning can help drug discovery and potentially lower some barriers to dangerous biological research. Greater autonomy can make agents more useful while also allowing mistakes to propagate further before a human notices. Longer-horizon systems can accomplish more, but become harder to monitor.

The concern is not simply that AI is becoming more capable. It is that capability may eventually move faster than our ability to evaluate, monitor and control it.

The risks themselves, and how strong the evidence is →

The proposal has three broad parts

The first has a straightforward logic. If the company building the system is also the company deciding whether it is safe enough to release, there is an obvious conflict. Independent evaluation doesn’t remove that conflict completely, but it creates another source of evidence.

The second part is where it becomes hard

Imagine one lab discovers that a new capability is developing faster than its safety systems can handle. It decides to slow down. But its competitors continue.

That lab may have reduced its own risk. It may also have handed a competitive advantage to somebody else. Which creates a powerful incentive not to be the company that slows down first.

Coordination is the proposed answer. In theory. In practice, it requires competitors to trust one another.

And the third problem is countries

Even if every major American AI company agreed, there would still be another question: what about everybody else?

AI is strategically important. Governments increasingly view advanced AI as an economic, military and geopolitical capability. So a slowdown involving companies in one country could potentially give companies elsewhere an advantage.

That may be the hardest part of the entire proposal, because the AI safety question is now tangled up with another one: who controls the world’s most capable AI?

Then Zuckerberg said no

On 15 September, Meta’s chief executive publicly rejected the need for a coordinated slowdown. His argument is worth understanding properly.[Reuters]

He didn’t say safety doesn’t matter. He argued that every AI laboratory already has both the responsibility and the incentive to make sure its systems are safe. If an AI product causes serious harm, the company may face legal liability. Customers may stop trusting it. Businesses may refuse to use it. Its reputation may suffer. And a company can simply delay its own product when it believes the system isn’t ready.

He pointed to Meta’s own Muse AI agent, whose release Meta delayed for several months while it worked on safety and security. The point was essentially that Meta slowed down when it thought it needed to, without needing everybody else to slow down too.

This is the real disagreement

It is tempting to describe this as safety against speed. That isn’t quite right. A more useful description is coordination against individual responsibility.

One position is that some risks may become too large for individual competitors to manage independently. The other is that companies already have strong incentives to manage those risks themselves.

Both approaches assume safety matters. They disagree about the mechanism that produces it.

Huang makes a similar argument from competition

Nvidia’s Jensen Huang has also pushed back against calls for additional restrictions. The argument is broadly that markets already create strong incentives for safe systems. Businesses don’t want unreliable AI. Consumers don’t want unsafe products. Companies deploying AI need customers to trust it.

From this perspective, safety is not something markets inevitably undermine. It can become a competitive advantage. An AI agent that businesses trust is more valuable than one they don’t, which gives companies a commercial reason to improve alignment, reliability and security.

The question is whether those incentives are strong enough when the possible harm extends beyond the company’s customers.

Where ordinary product safety stops being a good analogy

Suppose a company releases a bad accounting application. Customers dislike it. They stop buying it. The company suffers. The market has done some of the work.

But imagine a technology where a failure could affect people who never chose to use the product. Cyber capability is one example. Biological capability is another.

If an AI system significantly lowers the expertise needed to carry out a dangerous cyberattack or biological experiment, the people harmed may never have been customers of the AI company. That is why critics of purely market-led safety argue that liability and reputation may not be sufficient. The possible costs aren’t always contained inside the transaction between company and customer.

There is another problem with waiting for proof

Most regulation happens after we understand a risk. A product causes harm. We measure it. Rules follow.

Frontier AI creates a more difficult problem, because some of the risks being discussed are potentially very large but highly uncertain. Nobody has demonstrated that an advanced AI system will take control of critical infrastructure. Nobody has demonstrated that a future autonomous system will escape human control. Nobody has demonstrated that a frontier model will enable someone to create a catastrophic biological weapon. Those are possible future scenarios.

Slowing down has costs too

AI isn’t only producing risks. It is being developed for medicine, scientific research, education, accessibility, productivity, drug discovery, cyber defence, climate modelling and countless ordinary business applications.

If more capable AI can help scientists develop treatments faster, deliberately slowing capability has a cost. If AI can make companies more productive, slowing progress may have an economic cost. If one country slows while another continues, there may be a strategic cost.

And if regulations become so expensive that only the largest technology companies can comply with them, safety regulation could inadvertently strengthen the companies already dominating frontier AI.

There is no option here with zero consequences.

There is also a question of who gets to make the rules

Should the companies building frontier AI decide when development needs to slow? Should independent evaluators? Governments? International organisations? Scientists? Some combination of them?

Microsoft’s Satya Nadella has supported deliberate pacing and independent evaluation while arguing that decisions cannot be controlled by a handful of frontier AI companies.[AP]

That is an important position. Even if the major labs agreed among themselves, handing a small group of technology companies the power to decide the future pace of AI development raises obvious questions of accountability. The debate isn’t simply about whether AI should be controlled. It is also about who controls the control system.

And now governments are taking sides

The disagreement has already moved beyond the AI companies. The US administration has pushed back against calls for greater restrictions, arguing that excessive caution could damage America’s position relative to China. France’s finance minister has also questioned the slowdown argument, suggesting that calls from leading US AI companies could have the effect of protecting companies that are already ahead. Meanwhile, the United Nations has called for greater international coordination around AI safety.

So the divide isn’t simply one company against another. It is becoming a broader argument about national competitiveness, regulation, corporate power and global coordination. That makes a universal slowdown considerably harder than it sounds.

The China problem doesn’t have an easy answer

China appears repeatedly in this debate for a reason. Imagine the United States imposed strict limits on frontier AI development while China did not. If advanced AI becomes as strategically important as its advocates believe, that could have major economic and national-security consequences.

Now reverse it. Imagine every country refuses to slow down because it fears another country might continue. Then even governments that believe the technology is becoming dangerous may feel compelled to keep accelerating.

This is a familiar coordination problem. Everyone’s safest collective outcome may not be the outcome each participant is individually incentivised to choose. That doesn’t prove international coordination will work. It explains why the issue keeps appearing.

What would actually trigger a slowdown?

This is perhaps the most important question of all. “Slow down if AI becomes dangerous” sounds sensible. But what counts as dangerous enough?

A benchmark score? A biological capability? A successful cyberattack? An autonomous system circumventing its controls? An AI substantially improving AI research? A model becoming capable of operating independently for days? A safety evaluation showing that safeguards can be bypassed?

Unless those thresholds are defined in advance, a commitment to slow down risks becoming subjective. Different companies can look at the same evidence and reach different conclusions.

That is why independent measurement matters so much. A slowdown policy is only as useful as the measurement system that tells us when to use it.

Can we trust the tests being used to make those judgements? →

Recursive self-improvement could become one of those thresholds

One capability appears repeatedly in the current discussion: AI helping to improve AI. This already happens in limited ways, with AI systems helping researchers write code, analyse experiments and evaluate models.

That is not the same thing as recursive self-improvement. The stronger version would involve AI materially improving its own capabilities, then using the improved system to make further improvements. If that process became powerful enough, the rate of AI development itself could change.

Zuckerberg has specifically argued that Meta is concentrating the significant majority of its computing resources on AI that serves people rather than racing towards recursive self-improvement. That distinction is worth watching, because if AI begins materially accelerating the development of more capable AI, deciding when to slow down becomes considerably harder.

So should we slow down AI?

There isn’t one factual answer, because the decision depends on how we weigh risks that remain uncertain against benefits that are already appearing. But several parts of the argument can be separated.

The argument may ultimately be about evidence

The two sides sound very different. One says we need to slow down before capability outruns safety. The other says build safely, evaluate carefully and slow individual systems when necessary, but don’t impose a coordinated brake on progress.

What would help resolve the disagreement isn’t another dramatic prediction. It is better evidence.

Can independent evaluators reliably identify dangerous capabilities before deployment? Are safety systems keeping pace with capability? Can models meaningfully assist dangerous biological or cyber work? Are increasingly autonomous agents becoming harder to control? Can companies reliably police themselves when commercial pressure is high? Do coordinated safety standards actually reduce risk? And can international agreements survive the incentives created by competition?

Those questions are becoming measurable. The answers should determine what happens next.

This debate has changed

For years, “should we slow down AI?” sounded like a philosophical question about a distant technology. In September 2026, it became a disagreement between the people actually building the frontier systems.

Some of them now believe the gap between capability and safety is becoming important enough to deliberately reduce the pace of progress. Others believe individual companies, independent evaluation, liability and competition can manage the risks without coordinated restraint.

Neither side is arguing that AI should simply be unsafe. They are arguing about what happens when safety and speed come into conflict. And that may become one of the defining questions of the AGI era.

The AGI safety debate, without the shouting →

Have we entered the AGI era? →

Is AI progress slowing down? →

Is AGI dangerous? The plain-English version →