ARC Prize Just Cut OpenAI's 99.9% Astra Score to 62.7% — Here's What the Harness Hid

By Muzamil Ahmad · Updated 2026-10-26 · Research

#research#arc prize astra#gpt-6 astra benchmark#arc-agi-3 score#openai astra 99.9#ai benchmark harness#astra vs gpt-5.6

Save

On September 3, 2026, OpenAI launched GPT-6 Astra with a number that stopped the AI world in its tracks: 99.9% on ARC-AGI-3, the benchmark built specifically to resist gaming, memorization, and brute-force scaling. Three days later, Nvidia CEO Jensen Huang cited that figure on X when he declared that "AGI has arrived," a post that cleared 10 million views. OpenAI president Greg Brockman, at the launch briefing, welcomed listeners to "the AGI era."

There is just one problem with the number that travelled the world. It is not the score every other model gets measured by.

ARC Prize, the foundation that designs and runs ARC-AGI-3, published not one Astra score but two — and on its provider-neutral Standard harness, the same model scores 62.7%. That is still the highest result ever recorded on the benchmark, and it is still an enormous jump over every rival. But it is 37 points below the headline, and the difference is not the model. It is the harness: the scaffolding wrapped around the model during the test.

This guide explains both numbers, what each one actually measures, why the adapter run is also faster and cheaper, what ARC Prize and OpenAI each say about the gap, and — most usefully — how to read the next benchmark headline so you never get caught by this trick again.

One honesty note before we start, because it matters for how you should read this story. The correction itself is not brand new. ARC Prize published both harness results in its own results table during launch week in September, and independent analysts wrote up the split within days. What is new this week is the spotlight: digests on October 25 re-examined the gap and framed it as ARC Prize rejecting the 99.9% figure. That framing needs a correction of its own. ARC Prize did not expose a fabricated number. It produced the 99.9% itself, publishes it, and labels it by harness. What it rejects is using that number to rank labs against each other — because no rival model has ever run under the same adapter. Keep that distinction in mind and the whole story clicks into place.

What ARC-AGI-3 Actually Tests

Most benchmarks you read about are static: a question goes in, an answer comes out, and the score is the percentage right. ARC-AGI-3 is a different animal, and that is exactly why the harness matters so much here.

ARC-AGI-3 drops an agent into unfamiliar, turn-based game environments it has never seen before. The agent has to explore the environment, infer the rules from what it observes, identify the goal, plan a sequence of actions, act, read the result, and adjust when it is wrong. It is scored on two things at once: whether it completes the level, and how efficiently it gets there — how many actions it wastes along the way.

That design is deliberate. The benchmark's creator, François Chollet, spent years arguing that static question-and-answer tests mostly measure memorization: a model that has seen something similar in training will answer correctly without reasoning anything new. An interactive test, where the rules are novel and the agent must learn them by playing, is much harder to game. A model cannot memorize its way through a game it has never played.

This is also why ARC Prize guards the test's conditions so fiercely. In a static test, the conditions barely matter — the question is the question. In an interactive test, the conditions are half the measurement: what the agent is allowed to remember between turns, what tools it can use, how much context survives from one action to the next. Change those conditions and you change what is being measured, even if the model never changes at all. That is precisely what happened with Astra.

The Two Harnesses, in Plain Language

A "harness" is the scaffolding that connects a model to a benchmark: how prompts are delivered, what state survives between requests, how long conversations are managed, and what the model is allowed to carry forward. Think of it as the difference between an exam taken with a blank sheet of paper each question versus an exam where you keep your worked notes from every previous question. Same student, same questions — very different scores.

The Standard harness is ARC Prize's provider-neutral setup, the one every model runs through. Under it, the model receives the information needed to solve the game and can preserve visible notes of its own choosing between interactions. What does not persist is the model's richer, opaque reasoning state — the internal chain of thought it produced while working. Each request starts effectively cold: the model keeps only what it chose to write down, and older moves get truncated out of view as the conversation grows. ARC Prize describes it as the setup that "enables a model to carry forward notes it chooses to keep with it throughout the environment." It is the harness that gives, in ARC Prize's words, "a consistent, apples-to-apples comparison across providers."

The Provider Adapter harness is the setup OpenAI built for Astra. Under it, Astra's opaque reasoning state persists between requests, and compaction manages longer conversations — the system compresses earlier reasoning instead of discarding it, so the model can reuse prior work instead of reconstructing it from whatever it happened to write into its visible notes. In ARC Prize's own definition, the adapter "preserves opaque reasoning state between requests and uses compaction for longer conversations, allowing the model to reuse prior work."

Neither harness is a cheat. The Standard harness deliberately throws away something real — Astra's ability to carry its own reasoning forward — and forces the model to "figure out the game anew" repeatedly, as one independent analysis put it. That is a harness limitation, not a capability limitation. The adapter, meanwhile, measures something real too: what Astra can do when its vendor builds the scaffolding around its strengths. The problem was never that either number is false. The problem is that only one of them was headlined, and the headlined one is the one no other model can be compared against.

The Full Score Table: Same Model, Two Tests

ARC Prize published Astra's results across six reasoning-effort settings on both harnesses. Read the table row by row, holding effort constant, and the harness effect is impossible to miss.

Two things jump out. First, the two headline figures everyone quotes — 62.7% and 99.9% — do not even use the same effort level. The 62.7% is at maximum effort; the 99.9% is at high effort. Hold effort constant and the gap is 35.9 points at maximum (62.7% vs 98.6%) and 45.1 points at high (54.8% vs 99.9%). Second, the adapter scores barely move across effort levels — 96.7% to 99.9% — while the Standard scores swing wildly, from 17.5% at low effort to 62.7% at maximum. That pattern itself tells you something: when the model can carry its reasoning state forward, effort level stops mattering very much, because the model is not starting cold each time. When it cannot, effort is everything.

The runs were also priced very differently, and the pricing tells the same story from the other side.

The adapter run is cheaper as well as higher-scoring — which is exactly what you would expect when preserved state removes repeated re-reasoning. The Standard harness forces the model to redo work it has already done, and redoing work costs tokens. The adapter lets it pick up where it left off, and that shows up both in the score and on the bill.

Why the Adapter Is Faster and Cheaper, Not Just Higher-Scoring

If the adapter only inflated the score, the story would be simple: vendor rigs test, benchmark cries foul, everyone moves on. The reason this story has lasted seven weeks is that the adapter result is genuinely interesting engineering.

ARC Prize reports that across the 167 game-and-reasoning pairs both harnesses solved, provider-adapter runs were about 3.66 times faster and used about 49% fewer total tokens. Stop and think about what that means. The adapter did not just let Astra solve more levels. On the levels both setups could solve, it solved them in roughly a third of the time with roughly half the tokens. State persistence is not a cosmetic advantage. It is a large, measurable efficiency gain, and it is the same advantage Astra has inside OpenAI's products, where conversations are long, multi-turn, and stateful by design.

This is why the most careful coverage of the gap refuses to call either number fake. One analysis put it exactly right: read 62.7% as the apples-to-apples number and 99.9% as the ceiling — not one as fake and the other as real. The Standard score answers the question "how good is this model compared to every other model, under identical conditions?" The adapter score answers a different question: "how good is this model-plus-system when the vendor builds the environment around the model's strengths?" Both questions are worth asking. The error is answering the second question and presenting it as the answer to the first.

That distinction also explains why ARC Prize publishes both numbers, clearly labeled by harness, on its official leaderboard. A benchmark that hid the adapter result would be hiding a real capability. A benchmark that let the adapter result stand in for the model would be abandoning comparability. Publishing both, labeled, is the honest middle — and it is the reason the "rejection" framing from this week's digests needs care. ARC Prize rejected the comparison, not the measurement.

The Comparison That Travelled vs the Comparison That Holds

The most-shared version of this story was never just "Astra scored 99.9%." It was "Astra scored 99.9% while GPT-5.6 Sol scored 7.8%." That pairing ran everywhere at launch, and it is the pairing that does the most damage — because those two numbers are not from the same test.

Astra's 99.9% came from the Provider Adapter harness. Sol's 7.8% came from the Standard harness. Put them side by side and you are comparing a model running with its reasoning state preserved against a model running cold, then presenting the gap as a capability gap. The like-for-like comparison, all on the Standard harness, is very different — and still very impressive for Astra:

On equal footing, Astra roughly doubles its nearest rival and scores about eight times Sol. That is a step-function lead by any honest measure, and it is the number the benchmark actually supports. Nobody needs the 99.9% to make Astra look extraordinary. The 62.7% does that on its own. Which raises the obvious question: if the honest number is already a record, why headline the other one? The charitable answer is that OpenAI was reporting what its product does in its own environment. The less charitable answer is that 99.9% reads as "solved," and "solved" is what turns a product launch into an AGI announcement. Both answers can be true at once.

The AGI Claim: What Each Side Actually Said

The reason this benchmark dispute escaped the AI press and reached the front pages is the word attached to it: AGI. Here, too, the record is more careful than the headlines.

ARC Prize said, when it launched ARC-AGI-3, that saturating the benchmark would not on its own represent proof of achieving AGI, and it has repeated that position ever since. Its statement on Astra is worth quoting in full, because it is the benchmark's owner setting the limit on what its own numbers mean: "we believe Astra represents meaningful progress towards generalization... [but] we are not claiming that it is AGI." Co-founder Mike Knoop put it more bluntly: "we lack evidence to call this AGI yet." And founder François Chollet, while calling Astra "a step-function change in model capability for interactive reasoning problems," noted that the model surpassed the human action-efficiency baseline on 96% of the benchmark's levels, using fewer actions than the median tested human to get there. Note what that 96% figure is: action efficiency, a different metric from the headline percentage score, and one more illustration of how easily a single benchmark run generates several differently flattering statistics.

OpenAI's own language was more careful than the coverage it received. The company's launch materials described Astra as its most intelligent and best-aligned model to date, and Brockman's "Welcome to the AGI era" line came at a press briefing, reported as a hedged personal reflection rather than a formal claim. The unqualified "AGI has arrived" declaration came from Huang, not from OpenAI — a GPU seller, as one wry account noted, stating the claim to the fullest extent for the model developer, while the developer itself left room and the benchmark's owner declined to sign off at all. You cannot find a phrase like "AGI has been achieved" on OpenAI's official Astra release page.

So the surreal scene of launch week was three parties, one model, and three different claims: a vendor's briefing remark, a partner's unqualified declaration, and the benchmark owner's refusal. Only the third party owned the test. Its verdict is the one that should have travelled with the number.

OpenAI's Defense, Stated Fairly

A guide like this one owes OpenAI's side a fair hearing, because the defense is not frivolous.

OpenAI's position, as reported in the recent digests, is that its harness changes two settings to better match real-world performance, not specifically to target ARC-AGI-3 — and that the adapter does reflect how the model performs inside OpenAI's products, where state-carrying is a feature, not a cheat. There is real force in that argument. In production, nobody runs a frontier agent cold on every turn. ChatGPT, Codex, and the API all preserve conversation state; a model that can reuse its reasoning across a long session is genuinely more useful than one that cannot, and a benchmark that strips that ability away is measuring a crippled version of the product. If you are buying Astra to do long, multi-turn work inside OpenAI's stack, the adapter number is arguably the more relevant one for you.

OpenAI's own footnotes also acknowledged the modified harness — the adjustment was disclosed, not hidden. The dispute, stated precisely, is not about whether Astra is capable. Both sides agree it is the strongest interactive reasoner the benchmark has ever seen. It is about which number buyers, investors, and rival labs should carry around in their heads. ARC Prize's answer is that the number you carry for comparisons must be the one every model can reproduce. OpenAI's answer is that the number you carry for product decisions can be the one its product actually achieves. Those answers are compatible. The launch coverage, which carried only one of them, was not.

One small wrinkle inside ARC Prize's own record is worth noting for completeness. Chollet gave the Standard-harness figure as 66% in one social post, against 62.7% in the published table. The published table is the primary record, and this guide uses 62.7% throughout. If you see 66% cited elsewhere, it traces to that post, not to the table — a useful reminder that even the benchmark's owner can generate a second number, and that the table, not the timeline, is where scores live.

What This Means for You: How to Read the Next Benchmark Headline

The Astra gap is not really a story about one model. It is a story about a measurement regime that is breaking under the weight of agentic models, and it gives you a practical checklist for every benchmark headline you will read from now on.

1. Name the harness before you quote the number. A benchmark score without its harness is not a score, it is a slogan. From now on, "Astra scored 99.9%" should sound as incomplete to you as "the car did 0–60" without saying whether it was downhill. If a launch post, a press release, or a viral post does not name the harness, treat the number as unquoted until you find it.

2. Hold effort constant. Astra's two headline figures used different effort levels — maximum for 62.7%, high for 99.9%. Any comparison that does not hold effort, harness, and task fixed is not a comparison. When a vendor shows you a table, check that every row in it was produced under the same conditions. The computing press made this exact error at launch, running 99.9% against Sol's 7.8% as if they were the same test.

3. Ask who ran it. The most trustworthy number in this story is not the biggest one. It is the one produced by the party with the least interest in the outcome: the benchmark's owner, running every model through the same setup. Independent scoreboards exist for exactly this reason, and they are worth more than any vendor table. Our Kimi K3.1 Pro guide shows the same discipline from the other side — an independent index putting an open-weights model fourth in the world, within points of the best proprietary system, on a scoreboard nobody at Moonshot controls.

4. Ask what the run cost. Cost is a truth serum for benchmarks. A run that costs $26,098 (Standard, maximum) versus $18,817 (adapter, high) tells you something about what the harness is doing even before you understand the mechanism. When a score looks too good, check whether the run was also suspiciously cheap — or suspiciously expensive. Either anomaly is information.

5. Separate the model from the system. The deeper lesson of the Astra gap is that the thing being measured is changing. For most of the last decade, the interesting question was whether a model could do X. Now the system around the model — its memory, its context management, its tools, its state persistence — matters as much as the weights. A benchmark that measures only the weights will underrate real products; a benchmark that measures the whole system cannot compare models fairly. The honest answer is to report both, labeled, the way ARC Prize does. Any vendor that reports only one is making a marketing choice, and you should read it as one.

For buyers, the practical upshot is simple. If you are choosing between models, use 62.7% — the Standard number — and compare it against 30.2% and 7.8% on the same harness. If you are deciding whether to build long, multi-turn work inside OpenAI's stack, the adapter number tells you something real about what that stack delivers. Confusing those two decisions is how you end up paying for a headline instead of a capability.

Where Astra Stands Now

The benchmark dispute is not the last word on Astra — it is barely the first chapter. Weeks after launch, the Astra line took a stranger turn: on September 28, 2026, the Wall Street Journal reported that OpenAI was scrapping the planned October release of GPT-6.1 Astra after it failed internal safety evaluations, and OpenAI confirmed it the same day. What shipped instead was GPT-6.1 Sol, positioned as near-Astra coding performance at roughly one-fifth of Astra's token price. The original Astra's benchmark record — 62.7% on the neutral harness, the highest ever on ARC-AGI-3 — still stands. But the model line that produced it never became the product OpenAI planned, and the safety story that followed is a reminder that capability scores, however carefully measured, say nothing about whether a model should ship.

That is the right place to leave the Astra story: a genuine capability record, a genuine measurement dispute, and a genuine safety cancellation, all attached to the same model family within eight weeks. If you want the broader GPT-6 picture, our GPT-6 intelligent UI guide covers what the GPT-6 family changed for everyday ChatGPT users, and our Gemini 4 Argon guide covers the rival frontier model most often measured on the same harness. For the general skill of comparing frontier models without getting fooled, the ChatGPT vs Claude vs Gemini comparison applies the same discipline — same conditions, or no comparison — across the whole chatbot field.

Frequently Asked Questions

Is OpenAI's 99.9% fake? No. ARC Prize produced it, publishes it on its leaderboard, and labels it as a Provider Adapter result. The issue is comparability, not fabrication: no rival model has run under the same adapter, so the number cannot rank labs against each other. A real measurement of the wrong thing for comparisons is still the wrong number to quote without its label.

Which number should I quote? For comparing models, quote 62.7% — the Standard harness score at maximum effort. For describing what Astra does inside OpenAI's own product environment, you may quote 99.9%, but only with the words "Provider Adapter harness" attached. A bare "99.9%" is the error this whole story is about.

Does 62.7% mean AGI has arrived? No, and the benchmark's owner says so explicitly. ARC Prize stated at ARC-AGI-3's launch that saturating the benchmark would not prove AGI, and repeated after Astra's results that it is "not claiming that it is AGI." The 62.7% is a record on one demanding interactive-reasoning test. It is not a general-intelligence certificate.

Why is the adapter run cheaper and faster? Because preserved reasoning state removes repeated work. On the Standard harness, the model must reconstruct its reasoning from its own notes each turn, which costs tokens and time. On the adapter, it reuses prior work through state persistence and compaction — about 3.66 times faster and 49% fewer tokens on the pairs both harnesses solved.

Did ARC Prize only discover this gap in October? No. ARC Prize published both harness results in its results table during launch week in September, and analysts wrote up the split within days. The gap returned to the spotlight on October 25, when new digests re-examined it and framed it as a rejection. The numbers are seven weeks old; the argument about them is current.

What happened to Astra after launch? The planned GPT-6.1 Astra release was cancelled on September 28, 2026, after failed internal safety evaluations, and GPT-6.1 Sol shipped in its place. The original Astra's 62.7% Standard-harness record on ARC-AGI-3 still stands as the benchmark's best result.

Sources

• ARC Prize Rejects OpenAI's 99.9% GPT-6 Astra Score, Putting the Model at 62.7% on Its Own Test
• ARC Prize rejects OpenAI's claim that Astra scored 99.9 percent on AGI test
• GPT-6 Astra: Pricing, Benchmarks, and the Harness Catch
• GPT-6 Astra Launch: Benchmarks, Pricing, Rollout (2026)
• Why ARC-AGI-3 Gives GPT-6 Astra Two Scores
• ARC-AGI-3: Two Scores, One Model
• OpenAI's AGI number came from a harness, not the model