Saad Al-TamariIn September, GPT-6 Astra posted 99.9% on ARC-AGI-3, the benchmark designed to measure general intelligence. In the same launch, its numbers on the coding agent indexes moved by about two points. Both facts are true. The gap between them is the most useful thing to understand about benchmarks right now.
What ARC is trying to measure
ARC starts from François Chollet’s definition of intelligence as skill-acquisition efficiency: how quickly a system learns new tasks relative to its priors and experience, not how much it already knows [1]. ARC-AGI-1 tested that with small grid puzzles, each a new rule to infer from a few examples. It held for five years, until a preview of OpenAI’s o3 scored 75.7% on the semi-private set at $26 per task, and 87.5% with about 172 times the compute, at $4,560 per task [2].
ARC-AGI-2 raised the difficulty and calibrated its tasks against a human panel [3]. ARC-AGI-3, launched on 25 March 2026, changed the format entirely: hundreds of turn-based game environments with no instructions, no rules and no stated goals. An agent has to explore, work out how the world behaves, discover what winning means, and carry that knowledge into harder levels. At launch, humans solved 100% of the environments and frontier AI scored 0.51% [4]. Scoring rewards action efficiency against a human baseline: for each level, the median action count of the roughly 500 members of the public who completed it [5].
Six months from 0.51%
Those are ARC Prize’s verified results [6, 7]. On ARC-AGI-2 the frontier is now in the 90s: ARC Prize lists Claude Opus 5 at 90.4% and GPT-6 Astra at 95.0% at maximum reasoning effort. ARC-AGI-2 launched in March 2025. It took about eighteen months to get from hard to nearly saturated.
The harness is part of the score
The same model at the same maximum effort scored 62.7% under ARC Prize’s standard harness and 98.6% under OpenAI’s provider-adapter harness [5].
| Reasoning effort | Standard | Cost | Provider adapter | Cost |
|---|---|---|---|---|
| Max | 62.7% | $26,098 | 98.6% | $17,332 |
| XHigh | 59.3% | $37,317 | 98.4% | $18,147 |
| High | 54.8% | $40,705 | 99.9% | $18,817 |
| Medium | 38.6% | $48,090 | 98.4% | $19,285 |
| Low | 17.5% | $38,166 | 98.0% | $21,298 |
| None | 35.2% | $49,791 | 96.7% | $23,457 |
The difference is memory. The standard harness is provider-neutral and lets the model carry forward notes it chooses to keep. The provider adapter preserves the model’s opaque reasoning state between requests and compacts long conversations. On the 167 game runs both harnesses solved, the adapter was about 3.66 times faster and used 49% fewer tokens. OpenAI’s footnote says its harness changes two settings to better match real-world performance and does not target ARC-AGI-3 specifically [8].
Three details in that table are easy to miss. Under the standard harness, no reasoning (35.2%) beat low reasoning (17.5%): effort is not monotonic once the model has to manage its own memory. The adapter runs were cheaper as well as better, because carrying state forward avoids re-deriving it every turn. And the 36-point gap between harnesses is larger than the gap between Claude Opus 5 and Astra on the standard harness (30.2% against 62.7%). A score belongs to a model, a harness, an effort setting and a budget, together.
ARC Prize’s own position is careful. It reports both harnesses, says it believes a future AGI should be able to solve ARC-AGI-3 under the standard conditions, and states that saturating the benchmark would not be proof of AGI [5].
Why the reasoning jump didn’t show up in coding
Those deltas come from OpenAI’s launch table [8]. Independently, Artificial Analysis placed Astra at 67 on its Coding Agent Index when run through Codex: roughly level with Claude Opus 5 and Fable 5, and three points behind Claude Fable 5.1 in Claude Code at 70 [9]. At launch, the coding picture was a tie at the top, not a leap.
That is not a contradiction. ARC-AGI-3 rewards inducing the rules of a small, novel world from scratch, with little state and few actions. Everyday software engineering rewards nearly the opposite: navigating a large existing codebase, following conventions nobody wrote down, keeping long chains of edits consistent, and knowing when a change is actually done. The bottleneck there is context management and engineering judgment inside a huge, specific world, not abstraction from nothing. The largest coding-adjacent gain in OpenAI’s table, Terminal-Bench, is also the task closest to ARC’s interactive explore-and-act format. That last point is my reading of the numbers, not a claim either lab makes.
Coding benchmarks have their own problems
The coding side is not a clean yardstick either. SWE-bench turned real GitHub issues into tasks [10], and its human-validated Verified subset became the standard score. In February 2026 OpenAI stopped reporting it: auditing 138 problems that o3 did not solve consistently, it found 59.4% had material problems in their tests or descriptions, and every frontier model it tested could reproduce gold patches or verbatim problem details for some tasks [11]. Earlier, Liang et al. showed models could identify the buggy file from the issue text alone up to 76% of the time on SWE-bench repositories, against up to 53% on repositories outside it, a gap consistent with memorisation [12].
Benchmarks can also disagree with people. In METR’s randomised controlled trial, 16 experienced open-source developers took 19% longer on 246 real tasks when allowed to use early-2025 AI tools, after forecasting a 24% speedup and while estimating afterwards that they had been 20% faster [13]. The tools have improved a lot since, so the number shouldn’t be generalised. The method should: self-reported productivity is not a measurement.
Leaderboards built on human votes have their own version of this. The Leaderboard Illusion documented 27 private variants tested by Meta ahead of the Llama 4 release, and estimated that Google and OpenAI received about 19.2% and 20.4% of all arena data, against 29.7% for 83 open-weight models combined [14]. Contamination-resistant designs exist: LiveCodeBench only scores problems published after a model’s training cutoff [15].
How I read a benchmark claim now
- Which set? Public, semi-private or private. Public sets leak into training data.
- Who ran it? Vendor-reported or independently verified, and under which harness and effort setting.
- What did it cost? Cost per task changes the meaning of a score. 87.5% at $4,560 a task is a research result, not a product.
- How many attempts? pass@1, pass@k or best-of-n, and whether the intervals overlap.
- Saturated or contaminated? Near-ceiling scores and pre-cutoff public data both shrink what a benchmark can tell you.
- Does it look like my work? If not, build a small private eval from my own repositories and tickets, and trust it over any leaderboard.
A benchmark measures a model, a harness and a budget on one kind of task. The skill is reading which of those actually moved.
References
- 01Chollet, 2019. On the Measure of Intelligence. arXiv.
- 02ARC Prize, 2024. OpenAI o3 Breakthrough High Score on ARC-AGI-Pub. ARC Prize blog, 20 Dec 2024.
- 03Chollet et al., 2025. ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems. arXiv.
- 04ARC Prize, 2026. ARC-AGI-3 Launch. ARC Prize blog, 25 Mar 2026.
- 05Kamradt, 2026. OpenAI’s GPT-6 Astra on ARC-AGI-3. ARC Prize blog, 3 Sep 2026.
- 06ARC Prize, 2026. Claude Opus 5: ARC-AGI Results. ARC Prize verified results, 24 Jul 2026.
- 07ARC Prize, 2026. GPT-6 Astra: ARC-AGI Results. ARC Prize verified results.
- 08OpenAI, 2026. GPT-6 Astra: A New Generation of Intelligence. OpenAI.
- 09AI Weekly, 2026. GPT-6 Astra Lands at 67 on Coding Index, Behind Fable 5.1’s 70 (reporting Artificial Analysis). AI Weekly, 4 Sep 2026.
- 10Jimenez et al., 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR 2024.
- 11OpenAI, 2026. Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities. OpenAI, 23 Feb 2026.
- 12Liang et al., 2025. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. arXiv.
- 13Becker et al., 2025. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR.
- 14Singh et al., 2025. The Leaderboard Illusion. arXiv.
- 15Jain et al., 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. ICLR 2025.