AGI Made Clear › Real-world impact
Real-world impact: what is measured, not what is predicted
There is no shortage of predictions about what AI will do to work, jobs and productivity. This page has a stricter rule: if nobody has measured it yet, it doesn’t belong here.
Ask what AI will do to the economy and you will find some enormous predictions. Millions of jobs changed. Entire professions transformed. Productivity surging. Entry-level work disappearing. Companies operating with dramatically fewer people.
Some of those predictions may turn out to be right. But predictions aren’t measurements. And right now there is a surprisingly large gap between what we hear about AI’s impact and what researchers have actually been able to observe.
So this page deliberately asks a narrower question: what can we say has happened already? Not what a chief executive expects. Not what a consultancy forecasts for 2030. Not what somebody thinks their productivity increased by. What has actually been measured?
The recurring pattern
Measured here. Predicted elsewhere.
AI can now produce some remarkably good professional work
One of the more useful attempts to measure this is GDPval. Rather than testing AI on abstract questions, it gives models tasks based on economically valuable work across 44 occupations, and the outputs are compared with work produced by experienced professionals.[OpenAI]
When OpenAI introduced the benchmark in September 2025, its best model at the time produced work that expert graders preferred or rated equally to the human reference 47.6% of the time. By April 2026, the headline figure for GPT-5.5 had risen to 84.9%. That is a huge movement in a short period.
But it needs context. GDPval tasks are designed to be clearly specified and independently gradable. Real jobs usually aren’t.
A professional isn’t simply handed a perfectly described task, given everything needed to complete it and then judged on the final deliverable. Real work involves figuring out what somebody actually wants. Asking questions. Finding missing information. Dealing with colleagues. Changing direction. Managing competing priorities. Remembering what happened last week. And being responsible when something goes wrong.
It does not mean AI can now do 84.9% of those jobs. It means something narrower, but still important: when some professional tasks are clearly specified and the output can be judged independently, frontier AI can already produce work that compares surprisingly well with work produced by experienced humans. That is worth knowing. So are the limits of the test.
Coding shows why measuring real-world impact is difficult
Software development gives us one of the strangest examples of the difference between AI capability and actual productivity.
In 2025, METR ran a randomised controlled trial involving experienced open-source developers working on repositories they already knew. The developers were allowed to use AI tools for some tasks and had to work without them for others.[METR]
The result surprised almost everybody. With AI, they took 19% longer. Even more interestingly, the developers didn’t think AI had slowed them down. Afterwards, they estimated that AI had made them around 20% faster.
Measured result: 19% slower. Perceived result: about 20% faster. That gap matters. But it would be a mistake to turn the finding into “AI makes software developers slower”. METR doesn’t claim that. The experiment covered a particular group of experienced developers, working on mature open-source repositories they knew extremely well, using the AI tools available in early 2025.
AI changed quickly after that. And when METR tried to repeat the experiment with newer tools later in 2025, it ran into another problem.
The recurring pattern
Measured here. Predicted elsewhere.
By 2026, measuring AI productivity had itself become harder
The follow-up study produced raw estimates suggesting AI was now making participants faster rather than slower. For developers returning from the original study, METR estimated an 18% speedup. For newly recruited developers, the estimate was around 4%.[METR]
But METR doesn’t present those numbers as a clean measurement of how much AI speeds up developers today. Why?
Because people who found AI most useful increasingly didn’t want to participate in an experiment that sometimes required them to work without it. That creates selection bias. Some of the people who might benefit most from AI effectively disappear from the experiment. METR also encountered difficulties measuring time when developers were running several AI agents concurrently. The researchers therefore changed the design of the study.
That might sound like an annoying methodological footnote. It isn’t. It is part of the story. AI adoption is becoming sufficiently embedded in some kinds of technical work that controlled experiments comparing “AI” with “no AI” are becoming harder to run.
Then ask people how much AI helps them
The numbers become much bigger.
Between February and April 2026, METR surveyed 349 technical workers, including software engineers, researchers, academics, PhD students, founders and managers. The median respondent reported that AI made them roughly three times faster on the work where they used it.[METR]
That is enormous. But METR asked another question too. Instead of asking how much faster people could perform tasks, it asked how much additional value AI allowed them to produce. The answer was much lower: depending on how the question was framed, the median self-reported improvement in value was around 1.4x to 2x.
That is a fascinating difference. Doing something three times faster doesn’t necessarily mean producing three times as much useful work.
If AI makes a previously expensive task extremely quick, people may simply do more of that task. A researcher might build a dashboard they would previously have decided wasn’t worth the time. A developer might prototype three approaches rather than one. A manager might analyse information they previously would have skimmed.
That is useful. But speed, output and economic value are not interchangeable measurements, and public conversations about AI productivity often treat them as though they are.
So which number should we believe?
All of them, within the limits of what they actually measured. That is the important bit.
These aren’t competing answers to one question. They are answers to different questions. And putting them beside one another gives us a much more useful picture of where AI actually is.
The recurring pattern
Measured here. Predicted elsewhere.
Capability is moving faster than measurement
This may be the most important thing on the page. The technology is changing extremely quickly. Measuring its effect on real organisations is much slower.
A new model can be released today. A company can start using it tomorrow. But properly measuring whether it increased productivity, changed hiring, improved output quality or altered the structure of the organisation can take months or years. By the time a careful study is published, the AI tools being used may already have changed.
The most current information is often the least rigorous. The most rigorous information is often measuring technology that has already moved on. There isn’t an easy solution to that. But it does mean we need to be careful about what different kinds of evidence can tell us.
What about jobs?
This is where the gap becomes even bigger. There are plenty of forecasts about AI and employment. There are also anecdotes about companies reducing hiring, changing roles, reorganising teams or expecting AI to allow fewer people to do more work. Those things matter.
But establishing that AI has caused a measurable change in employment across an economy is much harder. Employment changes for lots of reasons. Interest rates. Economic growth. Company investment. Offshoring. Wages. Industry cycles. Automation that has nothing to do with generative AI. And AI itself. Separating one effect from all the others takes time and good data.
Those are three different claims. This page will treat them that way.
The same applies to entry-level work
One of the most plausible concerns about increasingly capable AI is that it could affect junior work first. A lot of entry-level knowledge work involves tasks AI is becoming good at: research, drafting, summarising, basic analysis, documentation and routine coding. That makes the argument worth investigating.
But plausibility isn’t measurement. If we want to know whether AI is actually reducing entry-level opportunities, we need evidence showing the change, over time, while accounting for the other things affecting hiring.
Until we have that, we will describe it as a risk, trend or forecast rather than a measured fact. The distinction may sound pedantic. It isn’t. It is the difference between saying something could be happening and saying we know that it is.
What has actually been established?
At the moment, the evidence gives us a more complicated picture than either the optimists or the pessimists usually present.
None of that means AI isn’t having a major economic effect. It means measuring that effect properly is harder than announcing it.
Why this matters for AGI
There is a reason this page sits on AGI Made Clear. One influential way of defining AGI is economic. OpenAI’s charter, for example, describes AGI in terms of highly autonomous systems outperforming humans at most economically valuable work.
Seven definitions, and what each one would require →
If we are going to use economic capability as part of the definition, then real-world economic evidence matters. Not just benchmark scores. Not just what a model can demonstrate once. We need to know what happens when the technology meets actual work.
Does output improve? Does the same work require fewer people? Do organisations become more productive? Do jobs change? Do new kinds of work appear? Does the benefit survive outside controlled tasks? And does it happen across enough of the economy to justify describing the technology as genuinely general?
Those questions aren’t answered yet. But we are finally starting to get data that lets us ask them properly.
The number we will keep watching
There probably won’t be one statistic that tells us AI has transformed work. The more useful signal will be when different measurements start pointing in the same direction.
Controlled experiments. Independent evaluations. Company productivity. Employment data. Wages. Output per worker. Changes in the kinds of jobs people actually do.
If those begin moving together, we will have much stronger evidence that AI is changing the economy rather than simply becoming more capable inside a benchmark. Until then, this page will keep the rule it started with.
The recurring pattern
Measured here. Predicted elsewhere.
Does AI actually make people more productive? →
Can AI make scientific discoveries? →