AGI Made Clear › Measurement
Three of the hardest tests in AI fell in eighteen months. That is a problem for everyone.
Each was designed to resist the shortcut that beat the last one. Each fell anyway. The question is whether that means the systems got general, or the tests were never as general as we thought.
ARC-AGI-2 stood at 54% in November 2025. By September 2026 it was at 95%. ARC-AGI-3, launched in March 2026 specifically to test interactive novelty, had a best score of 0.5% at launch and 63% six months later.
These are not benchmarks anybody expected to be soft. It is worth being clear about what this kind of test actually measures. Each ARC generation exists because the previous one turned out to be beatable by pattern coverage rather than reasoning, and each was built by people whose explicit purpose was to make that impossible.
Two readings, both defensible
The first is that the systems really are becoming more general, and the benchmarks are recording it accurately. The second is that we keep building tests which turn out to be narrower than the thing we meant to measure - that each one, once enough effort is pointed at it, reveals itself to have a structure that can be learned.
The ARC Prize Foundation leans towards the second, which is notable given that it is their benchmark falling. Their position on the latest result is that saturating it “would not represent proof of achieving AGI”, and that the environments have “deterministic, closed-ended mechanics” which do not represent the open-endedness of the real world.
The harness problem, which is separate and worse
The vendor announcement for the latest model reports 99.9% on ARC-AGI-3. The benchmark’s own neutral harness reports 63% for the same model. Neither number is fabricated. The higher one comes from a provider-specific harness that preserves the model’s reasoning state between requests and reuses prior work; the lower one from a harness that treats every system identically.
They answer different questions, and only one of them is comparable across vendors. A headline that prints the first without naming the harness is reproducing a lab’s best case as though it were a measurement.
What replaced them
FrontierMath Erdős, announced in September 2026, sets 68 conjectures that were genuinely unsolved as of August 2026 and demands a machine-verified proof. The best score is 3% - two problems, one of which took fifteen hours and $218 of compute.
The same model scores 98% on a benchmark of exceptionally hard problems that have already been solved, and 3% on problems that have not. Whatever else that gap is, it is not noise.
The problem underneath the problem
An audit published in August 2026 looked at 62 AI systems across 12 benchmarks and found that 73% of the quantitative evidence about progress traces back to a single measurement programme, and that training-compute figures are missing for every closed system released in 2026. We are arguing about a curve that almost nobody is independently measuring.