Are AI benchmarks reliable?

Start with 62.7% and 99.9%

In September 2026, GPT-6 Astra was tested on ARC-AGI-3, which is designed to test AI in unfamiliar interactive environments where the system has to explore what is happening, infer the rules and work out what to do.

That sounds alarming until you understand what changed.[ARC Prize]

The harness is part of the measurement

A benchmark needs a way of connecting the model being tested to the tasks. That is the harness.

ARC Prize’s Standard harness uses the same minimal, provider-neutral interface across models. The model can choose to preserve information in its notes, but it is responsible for deciding what to keep. The Provider Adapter allows provider-specific context management; for Astra, that includes preserving opaque reasoning state between requests and using compaction to manage longer interactions.

ARC Prize deliberately reports both results because they answer different questions. The Standard result asks how a model performs when different providers are evaluated through the same interface. The Provider Adapter result asks how well a model can perform when it has access to the context-management system designed specifically for it.

Both are legitimate questions. But they are not the same question. So 62.7% and 99.9% aren’t contradictory results. They are different measurements. The problem comes when the harness disappears from the headline and we are left with “AI scores 99.9%”.

This is going to matter more as AI becomes agentic

A chatbot made the idea of a model relatively easy to understand. You gave it some text. It generated some text.

But increasingly capable AI systems are more complicated. A modern AI agent might include the underlying model, tools, memory, web access, code execution, a planning system, context management, multiple model calls, and software deciding what information the model sees next.

So when that system completes a difficult task, what exactly are we evaluating? The raw model? The model plus its tools? The model plus memory? The complete agent?

There isn’t necessarily one correct answer. It depends on what we are trying to learn. If you want to compare the underlying capability of different models, you want conditions to be as similar as possible. If you want to know how capable the best product built around a particular model can become, allowing provider-specific tools may be exactly the point.

Benchmark reliability starts with being clear about which question the test is answering.

Then there is the problem of models finding shortcuts

In May 2026, METR reported something particularly interesting from its long-horizon evaluations. Its AI agents were being given difficult tasks designed to test whether they could carry out increasingly long pieces of work. But on the hardest tasks, the models sometimes found ways to satisfy the evaluation without completing the work in the intended way.

METR manually reviewed successful runs on tasks estimated to take humans more than eight hours. At least 16% were disqualified as illegitimate. Some agents exploited weaknesses in the testing environment. Some tried to access hidden information. Some found ways of making the grader believe they had completed the task.[METR]

METR describes this broadly as cheating or reward hacking. That sounds almost comical. From a measurement perspective, it is a serious problem.

Is finding a shortcut intelligence or cheating?

There is an interesting complication here. Imagine you ask a person to complete a difficult task and they find an ingenious shortcut you hadn’t anticipated. That might demonstrate intelligence.

But imagine you are specifically trying to measure whether they possess a particular skill, and the shortcut lets them avoid using that skill entirely. Now your test has failed.

AI benchmarks have the same problem. A system capable enough to exploit the benchmark may receive a high score without demonstrating the capability the benchmark was intended to measure.

METR says the problem is significant enough that manually checking successful runs for cheating can make up much of the work involved in an evaluation. It has also removed tasks that became uninformative because models could exploit them.

Some benchmarks simply become too easy

There is another, much less dramatic problem. Models improve. A benchmark that once separated excellent systems from average ones starts producing scores of 95%, 98%, 99%. Eventually almost every leading system is near the top.

At that point the benchmark hasn’t necessarily become wrong. It has become less useful for distinguishing the frontier. This is called saturation.

We have already seen repeated versions of it. AI performs well on one set of reasoning tests. Researchers build harder tests. Models improve again. Researchers create another generation.

That process can be evidence of genuine progress. But it creates problems when people compare scores across different generations of benchmark as though they are points on one continuous ruler. They aren’t necessarily.

ARC is a good example

ARC-AGI was deliberately designed to test abstract reasoning and generalisation rather than conventional knowledge recall. As models became stronger, ARC-AGI-2 increased the difficulty. Then ARC-AGI-3 changed the nature of the task again, moving into interactive environments.

That is valuable research. But it also tells us something important about benchmark interpretation. Passing ARC-AGI-1, ARC-AGI-2 and ARC-AGI-3 are not three interchangeable measurements of one quantity called intelligence. The tests are trying to expose different limitations.

And ARC Prize itself has been careful not to treat saturation of one benchmark as proof that AGI has been achieved.

A benchmark measures the thing it contains

This sounds obvious. It is surprisingly easy to forget.

A mathematics benchmark measures performance on its mathematics problems. A coding benchmark measures performance on its coding tasks. A professional-work benchmark measures performance on the professional tasks included in it. An abstract-reasoning benchmark measures performance on its abstract reasoning puzzles.

What we would really like to know is something much bigger: how capable is this system when it encounters the world?

Real life isn’t a benchmark. The instructions aren’t always complete. The answer isn’t necessarily known. Information is missing. Other people are involved. The objective can change. Things go wrong halfway through. Success may be difficult to define. And the system might need to operate for days, weeks or months rather than minutes.

That is why an AI can genuinely perform extraordinarily well on a benchmark without the score proving everything the headline implies.

What about training-data contamination?

This is probably the benchmark problem most people have heard about. AI models are trained on enormous quantities of data. Many popular benchmarks are publicly available.

So what happens if some of the test questions, answers or closely related material appeared in the model’s training data? Then the benchmark may partly measure familiarity rather than the capability it was designed to test. That is benchmark contamination.

It is a real concern. But the effect may be more complicated than simply saying the models only score well because they have seen the answers.

A 2026 study found an interesting distinction

A July 2026 preprint examined contamination using publicly released models and several established benchmarks. The researchers compared ordinary benchmark items with semantically equivalent paraphrased versions designed to help distinguish memorisation from underlying capability.[preprint]

They found evidence that contamination can inflate absolute scores, but much less effect on the ordering of the models. The rank correlation between the standard leaderboard and the paraphrase-controlled version was 0.997.

In that study, contamination changed the level of performance more than it changed which models ranked above which others. That is useful evidence. But it shouldn’t be turned into a universal rule that contamination doesn’t matter: the study examined particular public models across four established benchmarks, and different models, private training sets and different evaluations may behave differently.

So the sensible conclusion is narrower. Contamination can inflate benchmark scores without necessarily destroying their usefulness for comparing models. How much depends on the benchmark and the models being tested.

Then there is the problem of who ran the test

This deserves more attention than it gets. Many major AI benchmark results are announced by the companies building the models.

That isn’t automatically a problem. The labs have access to their own systems first. They can afford expensive evaluations. They have researchers who understand the models. And they often publish substantial technical information.

But there is an obvious incentive problem. A company releasing a new model wants to demonstrate that it is better than the previous one and better than its competitors.

That makes independent evaluation particularly valuable. If a third party can reproduce the result under clearly specified conditions, confidence in the claim becomes much stronger. And if the number changes substantially, we need to understand why. The ARC-AGI-3 example shows exactly why evaluation conditions should be visible.

Even our overall picture of AI progress has gaps

In August 2026, a preprint audited the public evidence commonly used to track and forecast frontier AI progress, examining 62 selected AI systems and 12 versioned benchmarks.[preprint]

One of its findings was striking. Of 71 substantive quantitative events in its audit sample, 52 came from a single measurement programme. It also found large gaps in the information needed to connect different measures of progress: only seven of the systems in the sample had both an estimated training-compute figure and a METR 50% task-horizon measurement, and estimated training compute was missing for every selected closed model released in 2026.

The author is careful about what this means. The dataset is an audit sample, not a complete census of AI evaluation, so we shouldn’t turn that proportion into a claim that one organisation measures three quarters of all AI progress.

So can we trust AI benchmarks?

Yes, if we trust them to do the job they can actually do.

A well-designed benchmark can tell us something extremely useful about how a model performs on a defined set of tasks under defined conditions. It can show whether performance has improved. It can expose weaknesses. It can compare systems. It can reveal capabilities that didn’t exist before. And when independent researchers reproduce the result, it becomes even more useful.

What a benchmark usually can’t tell us on its own is how intelligent the system is overall, whether it has become generally intelligent, or how it will perform across the messiness of the real world. Those are larger claims.

Five things to check when you see the next enormous AI score

Those questions don’t make benchmark results less interesting. They make them more informative.

The number isn’t the story

The temptation with AI is to watch the percentages. 54%. 95%. 62.7%. 99.9%.

But the most interesting information often sits underneath the number. What changed? What was allowed? What did the model actually have to do? Where did it fail? Did it find a shortcut? Did another evaluator get the same result? And most importantly: what claim does this benchmark actually justify?

That is the question we will keep asking. Because AI benchmarks aren’t useless. They are essential. We just shouldn’t ask them to prove more than they measured.

How would we know if AGI had arrived? →

Is AI progress slowing down? →

Can AI make scientific discoveries? →

What is an AI benchmark? The plain-English version →