Coding Agent "Success Rates" Lie — The Evaluation Blind Spot QuoteBench Exposed

---

TL;DR

• QuoteBench demonstrates empirically that “success rate” benchmarks for coding agents can easily conflate bugs in the transport layer that passes commands (serialization/parsing) with the model’s own generation quality. It concludes that raw generation capability is nearly saturated at the frontier, and the real differentiator is “boundary adaptation.” (Paper)
• In poster generation, AutoDesign—which lets the harness itself self-improve—outperformed the commercial Claude Design, while formally verified code generation (Vero) still can’t reliably solve problems at repository scale. The omni-modal AI scientist (OmniScientist) completed all 36 cases, but the paper’s abstract alone does not establish whether the output reached practical quality.
• A 1B-parameter model trained solely on permissible data achieved SOTA in Danish (DFM Mimir v1)—an example showing that upstream data curation and niche language selection, rather than model scale, can be a differentiating axis.

Main Story: What Do Coding Agent “Success Rates” Actually Measure? — QuoteBench’s Investigation

LLM coding agents don’t execute the Bash commands they generate directly; instead, execution passes through an interface that serializes, wraps, and re-parses them. QuoteBench used 56 one-shot tasks built from 14 real-incident-derived families to probe the boundary between the “generation contract” (what the model produced) and the “execution transport” (how that gets run), by adding a single parser deliberately missing escape handling. (Paper)

Technical Take

Simply re-running the same responses through the buggy parser dropped success rates by 55.4 to 73.2 points across all eight configurations — an effect entirely attributable to the transport layer, unrelated to the model’s generation capability. When the existence of this boundary was then disclosed to the model, 6 of the 8 configurations recovered 30.4 to 60.7 points, while the remaining 2 showed zero or slightly negative recovery. The paper concludes that “raw generation quality is becoming saturated at the frontier, and the differentiating factor has shifted to boundary adaptation.” In the GPT-5.6-sol example, the naive “match score” difference was only -3.6 points, but that figure was actually the result of -64.3 points of real harm offsetting +60.7 points of compensation. Looking only at the aggregate score makes this kind of large cancellation completely invisible.

Business Take

Selecting agents/models based on public benchmark “success rates” at face value risks missing failures specific to one’s own execution environment (shell wrappers, parsers, escape handling). The paper also shows that “changing the deployment configuration flips the ranking between models” — among 26 comparable pairs, 1 showed a clear reversal, and 4 were close enough that the outcome could flip depending on a single task’s result. This can be read as the flip side of T1 (the moat lies upstream of the model): the implication is that the design quality of the command-execution boundary (parsing, escape handling), not just model selection, affects real-world success. That said, this study is based on 56 tasks and one type of deliberately introduced bug, so caution is warranted before generalizing.

Implications and Positioning

Prompt designs that explicitly disclose the boundary to the model could be a low-cost improvement lever. This is grounds for updating one’s position — from “just use the strongest model, as ranked by public benchmark success rates” to “the implementation quality of your own transport layer determines success or failure.” Signal/noise: leans toward signal, given the concrete experimental design and clear numbers, but the scope is limited (56 tasks), so over-generalizing should be avoided. Confidence: medium-to-high.

Other Notable Topics

A Poster-Generation Harness That Optimizes Itself (AutoDesign / PosterBench) AutoDesign introduces “meta-harness optimization” for the long-horizon agentic process of generating posters from papers, where a code agent recursively improves the harness itself based on rollout results. It scored 78.32 on the PosterBench Main Track (100 posters across 5 fields), beating the commercial closed system Claude Design by 7.45 points. Across all seven code-agent/model configurations, incorporating the trained DesignHarness improved the average score from 54.99 to 67.39, a gain of 12.40 points. In the fully autonomous loop, it completed 253 tool calls and 11 edits in 40 minutes for under $3, reportedly reaching average conference-poster quality in human evaluation. (Paper) Take: This is a concrete example of T10 — that there’s significant room for optimization in harness design itself — but poster generation is also an adjacent area (T6) that general-purpose AI platforms could easily absorb as a future feature. So what: the position taken here is that standalone investment in poster generation carries high platform-absorption risk, and attention should instead focus on “generalizing the harness self-improvement method.”

Repository-Scale Formally Verified Code Generation Still Falls Short of Practical Use (Vero) Vero is the first benchmark to evaluate the ability to generate implementations and formal proofs simultaneously at repository scale (across multiple modules) rather than for single functions. It unifies 43 cases drawn from Python, Dafny, Verus, and Coq into Lean 4 format, covering domains from cryptographic protocols to distributed systems. Even the strongest agent configuration with access to the Lean toolchain fully solved only 27 of the 43 cases, and failed to solve a single specification in the hardest cases. (Paper) Take: single-function-level verified code generation has already been evaluated individually by existing benchmarks, but the scale barrier — maintaining consistent implementation and proof choices across multiple modules — is still high. So what: in domains where fatal bugs are unacceptable, such as finance, cryptography, or distributed systems, it would be risky to assume full automation is viable today. This is an example of T3 — capability demonstrations and real-world deployment should be evaluated separately — so investment decisions in this area should be deferred for now.

Direct Perception of Raw Data Outperforms Feature-Based Approaches (OmniScientist) OmniScientist is a research automation pipeline made up of three agents — ideation, experimentation, and writing — equipped with a perception layer that directly processes raw multimodal evidence such as images, signals, audio, video, 3D structures, and trajectories, not just text and labels. Across all 36 real-data cases spanning 5 fields, it completed the full pipeline from raw data to a finished manuscript, achieving an average overall score of 6.3. Compared to a blind version given only precomputed scalar features, direct perception reportedly outperformed on all 7 evaluated criteria. (Paper) Take: completing all 36 cases demonstrates breadth of execution, but the abstract does not state the upper bound of the 6.3 score or how it maps to practical quality, so the number alone does not establish the quality level. So what: rather than concluding that research can be handed off entirely to AI, test whether the finding that direct perception outperformed precomputed scalar features on all seven evaluated criteria transfers to one’s own analysis pipeline (T9).

A 1B Model Trained Solely on Permissible Data Achieves SOTA in Danish (DFM Mimir v1) Mimir v1 is a 1-billion-parameter model based on the Hierarchical Reasoning Model (HRM) architecture, trained from scratch using only a mix of 161 datasets that are license-clean (permissible). Across 20 benchmarks spanning English, math, code, and Danish, it outperformed the original HRM-Text 1B, was competitive with the larger Qwen 3.5 4B and Gemma 4 E2B, and achieved a new SOTA in Danish. (Paper) Take: this is a concrete example of T1 — that competing with major models is possible through upstream data curation and architecture rather than model scale. Achieving SOTA in a non-English language can also be read as an example of T7 — expanding into a gap where general-purpose models are underserved, in this case a specific language/domain. So what: this weakens the assumption that bigger models are always better, and adds niche-language × permissible-data curation as a viable differentiation axis. It could also serve as a selling point for enterprise products aimed at customers seeking to reduce licensing uncertainty around training data.

Worth Trying This Week / Hype to Ignore

Worth trying: in your own coding agent operations, test the actual final state via your real shell/parser yourself, rather than relying on public benchmark success rates. QuoteBench’s method — inserting a parser with escape handling deliberately removed and observing the difference — can be repurposed to surface vulnerabilities in your own harness.

Hype to ignore: the sweeping claim that “the era has arrived where AI can automatically generate posters, research papers, and formally verified code.” All four items above are demonstrations that work only under specific conditions, and remain some distance from real-world deployment.