AGI Made Clear › Who is responsible for AI safety?
Who is responsible for keeping increasingly powerful AI safe?
As AI systems become more capable and autonomous, one of the biggest questions is not simply what they can do. It is who decides when they are safe enough to deploy, and whether commercial incentives are enough to keep development responsible.
The AI companies have an answer to this, and it is a serious one rather than a brush-off. It runs roughly like this: we already have powerful reasons to build systems that behave as intended, because a system that does not is a worse product, a legal liability and a commercial embarrassment. Safety is not a tax on the business. It is part of the business.
That argument is stronger than its critics usually allow. It is also incomplete in a specific way, and the gap is where the real disagreement sits. This page sets out both.
The argument: the incentives already point the right way
Start with the version that is plainly true. A company selling access to an AI system is selling reliability. If the system invents facts, ignores instructions, leaks what it was told or behaves differently on Tuesday than it did on Monday, customers leave. Enterprise buyers in particular do not purchase impressive. They purchase predictable.
So the firms building these systems spend heavily on making them behave. They publish evaluation results, run red-team exercises, ship model cards and documentation, and hold releases back when testing goes badly. None of that requires regulation to explain. It is what any company does when its product is judged on whether it can be trusted.
The claim goes further than reputational risk, though, and this is the part worth understanding properly.
Alignment as a capability, not a constraint
Alignment is the problem of getting a system to pursue what was actually intended, rather than what was literally specified or what it inferred from its training. It is usually discussed as a safety topic. The commercial argument is that it is also a capability.
The case is easiest to see with AI agents: systems given a goal rather than a single instruction, which then take many steps to pursue it. An agent that does not reliably do what its user meant is not merely unsafe. It is useless. Every step it takes on a misread intention compounds, and the person supervising it ends up doing more work than if they had done the task themselves.
The more autonomy a product has, the more its value depends on alignment. A chatbot that misreads you wastes a minute. An agent that misreads you spends an hour going the wrong way, and you pay for the compute either way. On this reading, alignment research is not a cost centre attached to a product team. It is the thing that makes the product work at all.
If that is right, then a great deal of safety work gets funded without anybody having to be altruistic about it. That is a genuinely reassuring thought, and it is the strongest form of the industry's position.
The counter-question: enough for whom?
Here is where the argument becomes incomplete rather than wrong.
Commercial incentives are excellent at policing harms that land on the customer. A system that is unreliable, insecure or embarrassing hurts the people paying for it, they notice, and they leave. The market handles that.
Commercial incentives are much weaker on harms that land somewhere else. If a failure mode imposes costs on people who are not customers, who never agreed to anything and who have no commercial relationship with the company, then the feedback loop that makes the first argument work is simply absent. Nobody cancels a subscription they never had.
Liability is the usual answer to that, and it is a partial one. Liability works where harm is attributable, quantifiable and arrives soon enough to sue over. It works much less well where the harm is diffuse, where causation is contested, or where it exceeds what any company could pay. Those are exactly the categories that people worried about advanced AI tend to be worried about.
So the honest formulation is not “incentives work” or “incentives fail”. It is that incentives work well for one class of risk and poorly for another, and the argument between the labs and their critics is largely about which class the important risks fall into.
Independent evaluation, and what it currently is
Most of what the public knows about frontier system capability comes from the organisation that built the system, evaluated it and published the report. That is not an accusation of dishonesty. It is a structural fact: these systems are expensive enough that only a handful of organisations can build them, and access good enough to evaluate them properly is mostly held by those same organisations.
Independent evaluation means somebody outside that arrangement testing the system, with enough access to test it meaningfully, and able to publish what they find whether or not the result is flattering. All three conditions matter, and the third is the one most often missing.
The question is not whether external testing exists. It does, and there is more of it every year. The question is whether an evaluator whose access depends on the goodwill of the organisation being evaluated can deliver the thing the word independent is doing.
The compute question, which is the interesting one
This is the part of the argument most likely to be skimmed past, and it is the part with the most in it.
Compute is the processing power used to train and run these systems. It is finite, it is expensive, and how a company chooses to spend it is a real decision with real consequences. Most of it currently goes on two things: training new systems, and serving the people using the existing ones.
There is a third possible use. A sufficiently capable system could be pointed at improving itself: making itself more efficient, more capable, better at the work of building the next version. That is recursive self-improvement, and it is the mechanism behind almost every scenario in which AI capability moves faster than the people around it can react.
A company that allocates the overwhelming majority of its compute to serving users is making a statement about pace, whether or not it frames it that way. Compute spent answering customer questions is compute not spent on a system improving itself. The ratio is a policy choice, and unlike most safety commitments it is a quantity rather than an intention.
The concern that sits underneath it is concentration. If one system were given a large and growing share of available computing power to improve itself, the gap between it and everything else would not close. It would widen, and it would widen faster as it went. That is the scenario the compute allocation question is really about, and it is why a sentence about resource allocation is doing more work than it looks like it is.
It is also, usefully, checkable in principle. Capability claims are contested and hard to verify. A commitment about how compute is apportioned is the kind of thing an auditor could look at.
The bigger issue: who sets the standard
Everything above resolves into one question. As capability increases, should each lab decide for itself when a system is safe enough to release, or should there be common standards and external oversight that apply across all of them?
The case for lab-by-lab is speed and competence. The people who built the system understand it best, they can move at the pace the technology moves, and a regulator writing rules for a field that reorganises itself every 18 months will write rules for the field as it was.
The case for common standards is that self-governance holds until it is expensive. It works well in ordinary conditions and is tested precisely when a competitor ships first, when a funding round depends on a launch, or when holding a system back costs a great deal of money. Standards exist for the moments when the incentives point the other way, which are the only moments they are needed.
There is also a coordination problem that neither side disputes. Frontier capability is less concentrated than it was, more of it is released as open weights, and a meaningful share of it now sits outside any single jurisdiction. A standard that binds three companies in two countries is a standard with a gap in it.
Why this matters now
None of this is a question about AGI arriving. It is a question about the road there, and it becomes more pressing at every step along it, not only at the end.
Each increment of autonomy shifts where the responsibility sits. A system that answers a question puts the human in the loop by default. A system that carries out a task over hours does not. The further that goes, the more the safety question stops being about what the system can do and starts being about who was supposed to be watching.
That is the argument worth following, and it is not settled. What we would say is that the claim “our incentives already handle this” should be treated as a testable proposition rather than a reassurance. It predicts specific things: published evaluations, external access that does not depend on goodwill, compute allocations that can be checked. Those are the things to watch.
What alignment actually means, explained from scratch →
Is AGI dangerous? The plain-English version →
How capability is measured, and the six ways a score misleads →