Co-Design Bench: AI Coding Agents Fully Solve Just 26.4% of Real Tasks

By Muzamil Ahmad · Updated 2026-11-09 · Coding

#coding#co-design bench#ai coding agents#gpt-6 astra#ai benchmarks#claude opus 5#qwen3.8-max#coding agent benchmark

How we write our guides: we start from the vendor’s own announcement or documentation, not from other “best of” lists. Launch dates and features are checked against that primary source; anything reported only by coverage is labelled as reported. Prices and allowances are given only when the vendor publishes them — otherwise we say “not published” rather than guessing. We do not invent test results, prices, or rankings. Our guides are drafted with AI assistance, then reviewed and edited before publication; the final call on what we recommend is a human one.

Co-Design Bench: AI Coding Agents Fully Solve Just 26.4% of Real Tasks
Co-Design Bench: AI Coding Agents Fully Solve Just 26.4% of Real Tasks

Ask the best AI coding agent in the world to do what real users actually ask it to do — take an existing web app, read a request that may arrive with a screenshot, a receipt or a rambling description attached, and ship the change — and it finishes the job completely about one time in four. That is the headline finding of Co-Design Bench, a new benchmark built from tens of thousands of real coding sessions: the strongest of ten model endpoints, OpenAI's GPT-6 Astra, meets an average of 69.9% of each task's requirements, yet fully satisfies only 26.4% of the tasks. On the repository-scale work that fills real development days, the full-pass rate falls to 17.9%.

If that number feels familiar, it should. Last week we covered two benchmarks that landed on the same day and split the world the same way — one flattering, one brutal — in our look at LifeVal vs GDPx. Co-Design Bench is the coding chapter of that story. Older coding benchmarks have saturated: when agents can run, check and fix their own code, they post scores above 99% on tests like WebCompass-edit, and the leaderboard stops measuring anything. Co-Design Bench replaces the synthetic exercise with the messy original — real users, real apps, real requests — and the scores fall by more than 35 points. This guide works through what the benchmark measures, why the collapse happens, what it means for anyone choosing or using an AI coding agent, and the caveats you should keep attached to that 26.4% figure before you quote it.

Co-Design Bench replaces synthetic exercises with real user change requests — and full-pass rates fall from above 99% to 26.4% for the best AI coding agent.
Co-Design Bench replaces synthetic exercises with real user change requests — and full-pass rates fall from above 99% to 26.4% for the best AI coding agent.

The short version

What Co-Design Bench actually is

Most coding benchmarks hand a model a self-contained puzzle: a function to write, a bug to fix, a test suite to satisfy. Co-Design Bench starts from the opposite end — from what people genuinely type into AI coding tools. The authors collected real user sessions, deduplicated 37,463 queries from 26,314 sessions, and distilled them into 273 tasks that pair a normalised request with a runnable, pre-edit web application. The agent's job is to revise the app so the request is actually fulfilled — not to describe the change, sketch it, or produce plausible-looking code, but to leave a working application behind.

The benchmark's full title — "Multimodal Collaborative Coding Evaluation for Agentic Models" — points at its two tracks. Multimodal Editing (100 tasks) covers localised edits where the instruction may arrive as more than text: an image, an audio clip, a video or a document can carry part of the brief. A representative task from the paper's examples asks the agent to pull an amount from a receipt and update every dependent total in the app — the number lives in a picture, but the work is spread across the codebase. Repository Editing (173 tasks) covers coordinated revisions across a whole project: the kind of change that touches layout, behaviour and data at once.

Across both tracks, the tasks probe three capability dimensions the authors name Grounded Understanding (did the agent understand what the user and their attachments actually meant?), UI Transformation (can it reshape the interface as asked?) and Functional Implementation (does the thing work afterwards?). It is a deliberately unglamorous test. There are no algorithmic party tricks; there is a user who wants their app changed, and an app that either changed correctly or did not.

How the grading works: hidden criteria and a judge that gets challenged

The scoring design is where Co-Design Bench most clearly breaks with the saturated benchmarks. Each task ships with hidden acceptance criteria written by experts — 2,331 of them across the 273 tasks, roughly eight or nine checks per task. A task is not passed by vibes or by a single unit test. Every criterion is graded on evidence: the paper describes a "judge-then-refute" protocol in which an AI judge first assesses the finished app using source code and live browser evidence, and then a separate verifier model challenges every positive verdict, specifically hunting for the cheats agents learn to exploit — mock-ups, hardcoded results, surfaces that look finished but do nothing. Points go only to features that demonstrably work in the running app.

That protocol produces the benchmark's signature pair of numbers. The criteria rate — 69.9% for the leader — measures partial progress: most of what was asked for, done. The full-pass rate — 26.4% — measures completion: every criterion satisfied, nothing left dangling. The 43-point gap between them is, in a sense, the whole finding. As The Decoder put it in its coverage of the results, the agent usually handles the bulk of the work but leaves at least one detail unfinished — so the human almost always has to clean up by hand. On nearly three out of four tasks, "almost done" is where the best agent in the world stops.

The results: ten models, one clear leader, no finishers

Ten model endpoints ran the benchmark under a shared agentic harness. The paper's headline standings, as reported in the Co-Design Bench paper itself:

StandingModelWhat the paper reports
1stGPT-6 Astra (OpenAI)69.9% of criteria met on average; 26.4% of tasks fully passed; leads both tracks
2nd–3rdClaude Opus 5 (Anthropic), Qwen3.8-Max (Alibaba)Next places behind Astra; exact per-model figures in the paper's full tables
LastGrok 4.6 (xAI)Bottom of the ten endpoints
NotableQwen3.8-27B (Alibaba)Climbs to second on the largest repository overhauls

And by track, for the leader:

TrackTasksGPT-6 Astra full-pass rate
Multimodal Editing10041.0%
Repository Editing17317.9%
All tasks27326.4%

Two details deserve more attention than the ranking. First, efficiency is not a rescue: The Decoder's summary of the paper notes that GLM-5.2 burned through nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored significantly worse — throwing compute at these tasks does not close the completion gap. Second, the open-weight context matters for how you read the table. Qwen3.8-Max sitting in the top group, and its smaller sibling Qwen3.8-27B rising to second on big overhauls, continue the pattern we tracked in our LongCat vs Kimi vs DeepSeek comparison: the newest Chinese open models are no longer a budget tier; on real repository work they are in the leading pack, even if no model in that pack finishes the job reliably.

Why the scores collapse: three design choices

A drop from above 99% to 26.4% demands an explanation, and most of it is in how the test is built rather than in any sudden model regression. Three choices do the damage.

The requests are one-shot. Each task is a single fixed user request. The agent cannot ask which total the user meant, cannot show a draft and be corrected, cannot iterate the way a real session unfolds. Anyone who has used a coding agent knows how much of its real-world success arrives on turn two and three — the clarification, the "no, the other button," the rolled-back attempt. Co-Design Bench deletes all of that and grades the first attempt only. That is simultaneously the benchmark's harshest feature and its most debatable one, and the authors acknowledge it as a limit.

The bar is full completion. A task with nine acceptance criteria that satisfies eight is, in everyday terms, a good result — an hour of human cleanup saved. In Co-Design Bench's headline metric it is a fail, identical to a task where nothing worked. The 69.9% criteria rate is the number that reflects partial usefulness; the 26.4% is the number that reflects trust — how often you could have walked away. Both are honest. They answer different questions, and headlines will pick the scarier one.

The cases are deliberately hard. The researchers favoured sessions where users had to rephrase their request multiple times or roll back the agent's changes — tasks that had already given AI agents trouble in practice. This is selection bias by design: the benchmark samples from the distribution of work where agents struggle, not from all work. A random sample of real requests, including the easy majority, would score higher. What the benchmark measures is closer to a stress test than a census — which is a legitimate thing to build, as long as nobody reads it as the census.

Put together: one fixed request, no conversation, an all-or-nothing bar, and a task pool tilted toward known failures. The surprise is not that the number is low. The surprise would have been if it were high.

The multimodal–repository gap is the real story

The 41.0% versus 17.9% split between the two tracks tells you where today's agents actually break. Multimodal Editing — read the receipt, move the panel, restyle the page — is bounded work: the change is local, the evidence is visible, the criteria are checkable in one screen. Agents complete two in five of those tasks outright, and their criteria scores are higher still.

Repository Editing is a different species. A coordinated revision threads through files that depend on each other; change types multiply — layout, design, interaction, content inside a single request — and the paper finds that every one of the ten endpoints scores lower on repository tasks spanning three or four change types than on those spanning one or two. Coordination, not comprehension, is the bottleneck: keeping every dependent view consistent with a number pulled from a receipt, keeping a renamed concept renamed everywhere, keeping the app runnable at each step. That matches the texture of real development, where the hard part of a feature is rarely any single edit and almost always the edits' agreement with each other. It also explains why the small-model upset happens here: on the largest overhauls, the much smaller Qwen3.8-27B climbs to second place — a result the authors themselves decline to generalise, but a useful reminder that raw scale is not the binding constraint on this kind of work.

The harness matters as much as the model

Buried in the paper's secondary experiments is the finding with the most practical consequence for buyers. The researchers ran the same models inside different harnesses — the surrounding programs through which an agent reads files, writes code and executes commands — and the harness moved scores as much as the model did. Qwen3.8-27B, per The Decoder's account, scored 75% or just 56% on larger overhauls depending on the environment it ran in, and which of two models won could flip with the harness. The authors' recommendation follows directly: never report a model without its working environment, and never buy on a leaderboard that lists models alone.

This is the same lesson the market keeps relearning at each layer. A coding agent is a system — model, harness, tools, context management — and the benchmark number belongs to the system. It is why our guide to the best AI coding assistants evaluates tools as products rather than as model names, and why headline model rankings, including this one, should be read as "this model in this harness on these tasks" — all three qualifiers, every time.

What this means if you build with AI coding agents

Budget for cleanup, honestly. The most defensible reading of Co-Design Bench for a working developer: on real change requests, expect the best current agents to deliver roughly 70% of what you asked for, complete, about a quarter of the time without your intervention. Plan review time into every agent task the way you plan it into a junior hire's task — because the failure mode is the same one: confident, mostly right, one detail quietly wrong.

Decompose before you delegate. The change-type result is actionable today. Tasks combining three or four kinds of change degrade every model; tasks scoped to one or two hold up far better. Splitting "redesign the checkout and add discount logic and update the emails" into sequenced, single-kind requests is not prompt-engineering folklore — on this evidence it is the difference between the 41% regime and the 17.9% regime.

Write acceptance criteria like the benchmark does. The tasks agents fail are the ones where "done" was never made checkable. Eight or nine explicit, testable criteria per task — the benchmark's own average — is a good template for a real brief: what must work, where, and how you will know. Agents graded by hidden criteria performed exactly as well as their briefs allowed; yours will too.

Choose the pair, not the model. Astra leads, Qwen's models punch at the top, and the harness can swing a result by twenty points. Trial the actual product you would deploy — model and environment together — on your own repository tasks before committing, and weight completion rate over demo speed. If you are evaluating the open-weight route for that work, our LongCat-2.5 guide covers the current value pick in that tier.

Keep the 99% claims in their place. When a vendor quotes a near-perfect coding score, the question is now standard: on tasks built from real requests, with hidden criteria and full completion required — or on a saturated exercise? Co-Design Bench does not make the older benchmarks fraudulent; WebCompass-edit scores above 99% genuinely happened. It makes them finished — measuring a skill agents have already maxed, which is not the skill your backlog needs.

The caveats: read these before quoting 26.4%

We hold every benchmark we cover to the same standard, and this one has four limits that travel with its headline number.

It is a preprint. Co-Design Bench appeared on arXiv on October 26, 2026 and has not been peer-reviewed. Benchmarks routinely shift between preprint and publication — criteria get revised, errors surface, model lineups change. Treat the figures as the authors' reported results, carefully produced but not yet independently checked. The full text is public in the paper's HTML version, so the methods at least are open to inspection.

Follow the affiliations — in both directions. All three corresponding authors and the majority of the author list are affiliated with Alibaba, whose Qwen models place well. That conflict deserves stating plainly. But the simplistic version of the suspicion fails on the paper's own numbers: the benchmark's winner is OpenAI's GPT-6 Astra, not an Alibaba model, and the test was built to be unflattering to everyone, Alibaba's endpoints included. A rigged test does not crown a rival. The affiliation is a reason to want replication, not a reason to dismiss the result.

AI judges can err. The grading is a judge-then-refute pipeline of models evaluating models. The refutation step is a genuine improvement over single-judge grading — it exists to catch faked completion — but no automated judge fully substitutes for a human running the app, and borderline criteria will be misjudged in both directions. The scores should be read with error bars the paper's presentation cannot fully supply.

One-shot is not how anyone works. The deepest limit is the one the authors name themselves: a fixed request, no follow-ups, no corrections. Real co-design is a conversation, and agents are materially better inside one. Co-Design Bench measures the floor of agent performance — what you get if you fire and forget — not the ceiling of what a developer and an agent achieve together over an afternoon. Read 26.4% as the fire-and-forget number. It is the right number for judging autonomy claims, and the wrong number for judging assisted development.

What to watch next

Three developments will decide whether Co-Design Bench becomes the reference test for coding agents or a one-week headline. First, replication: an independent run — by a lab without models in the table — would settle the affiliation question and test the judges. Second, a conversational variant: the same task pool with follow-ups allowed would measure the number practitioners actually live with, and the gap between the two editions would price the value of iteration itself. Third, saturation watch: every benchmark that embarrassed the field has eventually been climbed, and the honest signal of progress will be full-pass rates on this task pool rising without the criteria softening. Until then, the standing summary of AI coding agents, late 2026 edition, is the one this benchmark earned: they do most of most tasks, finish about a quarter of them, and the distance between those two numbers is exactly where human developers currently earn their keep.

Repository edits fully pass at just 17.9% versus 41.0% for multimodal edits: coordinating changes across a codebase is where coding agents break.
Repository edits fully pass at just 17.9% versus 41.0% for multimodal edits: coordinating changes across a codebase is where coding agents break.

Frequently Asked Questions

What is Co-Design Bench?

Co-Design Bench is a benchmark of 273 tasks, published as a preprint on arXiv on October 26, 2026 by researchers from Fudan, Alibaba and four other Asian universities. Instead of synthetic puzzles, its tasks are distilled from 37,463 real user queries across 26,314 coding sessions: 100 Multimodal Editing tasks and 173 Repository Editing tasks, graded against 2,331 hidden expert acceptance criteria.

Which AI model scored best on Co-Design Bench?

OpenAI's GPT-6 Astra, the strongest of the ten endpoints tested. It met an average of 69.9% of each task's acceptance criteria but fully satisfied only 26.4% of tasks. Anthropic's Claude Opus 5 and Alibaba's Qwen3.8-Max placed next; xAI's Grok 4.6 placed last.

Why do AI coding agents score above 99% on other benchmarks but 26.4% here?

Three design choices explain most of the gap of more than 35 points. Co-Design Bench uses real user requests instead of exercises, requires every acceptance criterion to pass for a task to count (no partial credit in the headline rate), and deliberately favours hard cases where users had rephrased requests or rolled back changes. It also allows one fixed request per task, with no follow-up questions or corrections.

What is the difference between the 69.9% and the 26.4%?

The 69.9% is the criteria rate: on average, GPT-6 Astra satisfied about seven in ten of each task's expert-written requirements. The 26.4% is the full-pass rate: the share of tasks where every criterion passed and nothing was left unfinished. The gap between them is the "human cleanup" zone — work that is mostly done but not done.

Is Co-Design Bench reliable?

It is a serious, inspectable piece of work with declared limits. It is an unreviewed preprint; most authors are Alibaba-affiliated, though the winning model is OpenAI's; grading is done by AI judges under a judge-then-refute protocol that can still err; and its one-shot design measures fire-and-forget autonomy rather than assisted development. Trust its direction — agents complete far less real work than saturated benchmarks suggest — more than any decimal point.

Should I switch coding agents because of these results?

Not on this table alone. The same research found that the harness — the environment an agent runs in — can swing a model's score by as much as the model choice itself, and every model failed most repository tasks. Trial the exact product you would deploy, model and harness together, on decomposed tasks from your own backlog, and judge it on full completions rather than speed.

Reader reviews

–
Loading reviews…