The Score Was Never About the Model

Key Takeaways

• OpenAI’s president called GPT-6 Astra’s ARC-AGI-3 result “the AGI era.” The number behind that line, 99.9%, came from a harness that lets the model carry compressed reasoning state across puzzles; the same model scored 62.7% under ARC Prize’s neutral harness.
• The clearest sign the harness did the work: switching that same harness’s reasoning effort to “none” still scores 96.7%, higher than the neutral harness gets even at maximum reasoning.
• This isn’t unique to OpenAI. Less than two weeks earlier, NVIDIA wrapped a bare Claude Opus 5 (about 30% alone) in its own harness and hit 100% on ARC-AGI-3’s public set. Nobody called that AGI.
• ARC Prize’s own writeup says saturating the benchmark “would not represent proof of achieving AGI.” That sentence ran in the same post as the 99.9% headline.
• What’s separately real: Astra building its own compact symbolic notation to model unfamiliar game mechanics from scratch. That capability doesn’t depend on which harness scored it.

OpenAI’s president ended the GPT-6 Astra press briefing with four words: “Welcome to the AGI era.” Pressed on whether he meant it, Greg Brockman didn’t walk it back. “I think it might be about this model.”

The number behind that sentence is 99.9%, Astra’s score on ARC-AGI-3, a benchmark built specifically to be trivial for humans and hard for AI. Real benchmark, ambitious claim, real number. What the sentence leaves out is which test harness produced it, and what that harness was allowed to do that the benchmark’s own design says it shouldn’t.

What Was GPT-6 Astra’s Actual ARC-AGI-3 Score?

Two numbers, from the same model, tested the same week: 62.7% and 99.9%.

ARC Prize evaluated Astra across twelve harness configurations, but the headline comparison is between two:

Read the left column first. Turn reasoning off entirely, and the Provider Adapter harness still scores 96.7%, ahead of what the Standard harness manages even with reasoning maxed out. Whatever that harness is measuring, it isn’t primarily “how well does this model reason.” It’s closer to “how much does letting a model keep what it already worked out, across puzzles it’s supposed to be meeting for the first time, help.”

ARC Prize describes the difference between the two harnesses plainly: the Standard harness gives every model “a provider-neutral interface” in which it can preserve visible notes it chooses to keep between requests, and nothing else. The Provider Adapter additionally lets the model “preserve opaque reasoning state between requests” and “use compaction for longer conversations,” reusing prior work the neutral harness simply discards.

What Is ARC-AGI-3 Actually Trying to Rule Out?

Prior exposure. That’s close to the entire design.

ARC-AGI-3 launched on March 25, 2026. At launch, frontier AI scored 0.51% across the benchmark’s environments. Humans scored 100%. The design goal, in ARC Prize’s own words, is closing “the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations,” through environments built around “no pre-loaded knowledge or hidden prompts” and “novelty that prevents brute-force memorization.” An agent is supposed to land in a game it has never seen, with no instructions, and build a working model of the rules from scratch.

That premise is exactly what carried state complicates. If a harness setting lets a model retain the compressed reasoning it already did on puzzle four when it reaches puzzle five, puzzle five is no longer met cold the way the benchmark’s own design requires. It’s closer to letting a test-taker keep scratch paper between different exam questions on a test that is specifically trying to measure whether they can work out an unfamiliar kind of question from nothing.

Has This Happened Before Astra?

Less than two weeks earlier, with a different model, and nobody called it AGI.

On August 21, 2026, NVIDIA wrapped a bare Claude Opus 5, which scores about 30% alone, in its own AVO harness (memory, tool use, supervision, feedback) and cleared all 183 levels of ARC-AGI-3’s 25-environment public set, at 100%, using 12% fewer actions than the previous leader. NVIDIA’s own conclusion: “The model matters, but the model is not the entire agent.” Evaluating a model, they wrote, is not the same as evaluating an agent.

That result was scored on the easier public set, not the semi-private set Astra’s headline numbers came from, so it isn’t a like-for-like comparison of the identical test. But the mechanism is the same: same underlying model, wildly different score, depending entirely on what gets built around it. When NVIDIA published that gap, the takeaway everywhere was that harness design matters more than people assumed. When OpenAI published a structurally similar gap about its own model less than two weeks later, the line at the press briefing was “welcome to the AGI era.” Same shape of result, different author, different frame.

So What’s Actually New Here?

Not nothing. ARC Prize’s own analysis credits Astra with turning unfamiliar game mechanics into compact working models, developing its own domain-specific notation on the fly, shorthand like L8: hub q2 (8↓) for tracking state and extend8 to3; retract10 to2 for planning moves, to represent rules nobody told it. That’s a real behavior, and it’s independent of which harness scored it: a model choosing to build and use its own symbolic language to compress an unfamiliar system is not a scaffolding artifact. OpenAI researcher Aidan Clark said the jump from the prior model to Astra was the largest capability increase they’d measured between two consecutive releases. That claim doesn’t depend on the harness fight either, and there’s no obvious reason to assume it’s false.

The disagreement isn’t about whether Astra is more capable than what came before it. It almost certainly is. The disagreement is about what a single number, quoted without its harness, is entitled to claim on that capability’s behalf.

So Is GPT-6 Astra AGI?

The people closest to the result are visibly less certain than the four-word soundbite. ARC Prize’s own writeup states plainly that saturating the benchmark “would not represent ‘proof of achieving AGI,’” because ARC-AGI-3’s environments, however novel, remain “deterministic” and “closed-ended” next to “the complexity and open-endedness of the real world.” That sentence ran in the same post as the 99.9% headline.

Brockman himself, off the soundbite, was more hedged than the briefing’s closing line suggested: “Everyone has a different definition of AGI… it’s a much more gray, fuzzy thing,” and on whether Astra is really the first AGI system or just another point on a curve, “if you want to say this is the first one, I think it’s reasonable. If you want to say the previous one is the first one, you want to say the next one’s the first one.” OpenAI’s chief scientist, Jakub Pachocki, struck a different register the same week: “progress in intelligence does not guarantee progress in alignment,” and the company would “pause scaling” before accepting further loss of its ability to monitor it. Gary Marcus, no fan of OpenAI’s framing generally, zeroed in on the same gap from the outside: “what we don’t know is how robust that capability is. That is the key question.”

None of that is a “no.” It’s four people close to the same result declining to answer the AGI question with the confidence the press-briefing line implied.

This is the same failure this site described in July, in The Model Is Not the Agent: a model’s intelligence isn’t a property sealed inside its weights, it’s a property of the whole system it runs inside, and asking “how smart is this model” without naming that system is an incomplete question. It’s also the same instrument problem from A Perfect Score Without Solving Anything: a benchmark score is a claim about the conditions under which it was produced, and here, the entity choosing which condition to read aloud is the same one that built both the model and the harness that flatters it. None of this requires bad faith from anyone involved. It requires only that whoever picks which number gets quoted has an obvious interest in which number that is.

Closing Thought

The interesting failure here isn’t that OpenAI reported the more flattering number. Any lab in that position would. The interesting failure is that “welcome to the AGI era” and “this would not represent proof of achieving AGI” were published in the same week, about the same result, and only one of them was said out loud at the microphone.

Further Reading

• ARC Prize: OpenAI’s GPT-6 Astra on ARC-AGI-3, September 2026
• ARC Prize: GPT-6 Astra ARC-AGI Results, September 2026
• ARC Prize: Announcing ARC-AGI-3, March 2026
• NVIDIA: NVIDIA AVO Reaches 100% on ARC-AGI-3, August 2026
• Gary Marcus: Hot Take on GPT-6 Astra
• VentureBeat: ‘Welcome to the AGI Era’: OpenAI Launches GPT-6 Astra, September 2026
• Hacker News discussion, 234 points

More essays at Call to Think, or start with the AI Evaluation collection.