"Match Rate" Lies: QuoteBench Exposes the Blind Spot in Coding Agent Evaluation

---

TL;DR

• Coding agents’ “match rate” scores conflate generation quality with flaws in the execution transport layer (command serialization and re-parsing), and QuoteBench quantitatively separates the two. At the frontier, raw command generation itself is nearly saturated; what actually differentiates models is their ability to adapt to quirks in parsing behavior.
• Both the AI scientist system (OmniScientist) and formally verified code generation (Vero) demonstrate expanded capabilities, but a gap remains between completion rates and output quality — both are still far from replacing human experts.
• The small open model Mimir v1 and the inference acceleration method DARTree are modest infrastructure-side advances that don’t differentiate on their own, but they shift the baseline assumptions around cost and jurisdictional compliance.

Top Pick: QuoteBench — What Does a Coding Agent’s “Success Rate” Actually Measure?

LLM coding agents execute Bash commands through a pipeline of model output → serialization → wrapping → re-parsing. The research team prepared 56 one-shot tasks derived from 14 real-world incidents, then modified this pipeline by inserting a single parser with escaping deliberately left out, measuring how execution results changed without altering the model’s own output (Source).

Technical Take

Simply re-running the same response from the same model through the broken parser caused success rates to drop by 55.4 to 73.2 points across all eight configurations — with zero change to the model’s output. On the other hand, when the model was told in advance about “this constraint (the parsing quirk),” six configurations recovered 30.4 to 60.7 points (two configurations showed no recovery or a slight decline). The paper notes that at the frontier, raw command generation quality is nearly saturated, and the real difference between models has shifted to “how well they can adapt to a disclosed constraint.” GPT-5.6-sol is emblematic: the gap in “match rate” between the raw condition and the constraint-disclosed condition was only -3.6 points, but underneath that, a -64.3 point loss from parser breakage was nearly canceled out by a +60.7 point recovery from the model’s adaptation. Looking at match rate as a single metric hides this swing entirely. Furthermore, simply changing the execution transport configuration clearly reversed the ranking in one of 26 comparable model pairs, with four more pairs close enough to flip.

Business Take

The “success rate” or “task completion rate” that coding agent products and vendors publish depends heavily not on the model itself, but on how the “wiring” — command serialization and re-parsing — is built. A vendor can update its wrapper layer alone and shift scores up or down without touching the model. Moreover, the experimental design of “a parser with escaping deliberately left out” reproduces an execution path vulnerable to quotation and interpolation hazards, and the finding that such wiring flaws directly translate into a 55-to-73-point degradation in execution results is a concrete warning to developers building their own shell-execution agents.

Implications and Position

If you’re building your own shell-execution agent harness, audit the escaping at the interpolation point where model output becomes a shell command before you touch prompt engineering. Don’t take a vendor’s “agent success rate of X%” at face value unless they disclose both their execution transport and how they verify final state. This kind of “transport layer robustness” is the sort of thing official harnesses will absorb within a few quarters, so betting on it as a differentiator is a poor strategy.

Signal/Noise verdict: Signal. The A/B comparison design of re-running identical responses, along with the quantitative results, is clear and concretely substantiates the argument that “agent evaluation can be distorted by metric design.” Confidence is high.

OmniScientist: An AI Scientist That “Looks at Raw Data” Completes Its Runs, But Quality Is Still a Work in Progress

The research team presented OmniScientist, a fully autonomous AI scientist that runs the entire pipeline from ideation through experimentation to writing while directly perceiving raw, heterogeneous data — including images, signals, audio, video, 3D structures, trajectories, tables, equations, and graphs — rather than relying solely on text or precomputed summaries. Evaluated on 36 real-data tasks spanning five academic discipline groups and four evidence formats, the system completed the full pipeline from raw data to a finished manuscript in all 36 cases, achieving an average score of 6.3 when using the reference reasoning model. The configuration that directly perceived raw data also outperformed a blinded version given only precomputed numerical features (Source).

A “100% completion rate (36/36)” measures pipeline coverage, not output quality. An average score of 6.3 suggests a level that, while not shoddy, can’t be called peer-review-ready either. Implications and position: If you’re evaluating research/R&D support tools, it’s premature to base an investment decision on the headline “AI writes the paper.” A more realistic position right now is to use this as a wedge — incorporating just the underlying technique of directly perceiving non-text raw data into existing workflows.

Vero: Formally Verified Code Generation Plateaus at a Win Rate of Just Over 60% at Repository Scale

Existing verified code generation benchmarks have been limited to single functions, or to settings where the implementation is given and only the proof needs to be generated. Vero introduces 43 multi-module instances drawn from real repositories in Python, Dafny, Verus, and Coq — spanning everything from cryptographic protocols to distributed systems — with API interfaces, specifications, and reference implementations aligned in Lean 4, and requires synthesizing implementation and proof together (or proof alone). Even the strongest agent configuration with access to the Lean toolchain fully solved only 27 of the 43 instances, and for the hardest category of tasks, it couldn’t close a single specification (Source).

Simply moving from the simplified setting of single functions and proof-only tasks to the realistic setting of repository scale with simultaneous implementation-and-proof consistency drops frontier coding agents’ success rate to just over 60%. This is a real-world visualization, using the strictest possible standard of formal verification, of the practical ceiling on the claim that “AI can write correct code.” Implications and position: If “AI writes verified code” is your selling point, this 27/43 result — with zero solves on the hardest problems — can be used to calibrate customer expectations. In domains that require proof of correctness, such as finance, infrastructure, or cryptographic protocols, the reasonable design right now isn’t one that removes proof engineers, but one where humans handle repository-scale consistency checks and AI is used for localized assistance.

Mimir v1: A 1-Billion-Parameter Model Trained Only on Permissible Data Holds Its Own Against Larger Models

A Danish Foundation Models team trained Mimir v1, a 1-billion-parameter model built on the HRM (Hierarchical Reasoning Model) architecture, entirely from scratch using 161 datasets deemed permissible on grounds including open licenses, individual agreements, and the EU text-and-data-mining exception for research institutions. Evaluated across 20 benchmarks in English, math and code, and Danish, the model outperformed the original HRM-Text 1B, delivered competitive performance against the larger Qwen 3.5 4B and Gemma 4 E2B, and set a new state of the art in Danish (Source).

This is an approach that competes on data quality and a clearly stated basis for data use rather than scale, and the technical takeaway is that it demonstrates competitiveness with 4B-class models even under the constraint of using only data the team deems permissible. For large enterprises and public institutions where the legal basis for training-data use can be a compliance issue, this expands the set of options that don’t require a major performance trade-off. The new state of the art in a non-English language (Danish) also points to an opportunity in smaller-language markets that frontier labs tend to deprioritize. Implications and position: The impact is minor for English-centric products, but if you’re in a position to select models for specific languages or regulated industries, a small model trained only on data with an explicit basis for use is worth adding to your shortlist. It’s a data point for reconsidering a one-size-fits-all reliance on the single strongest general-purpose model.

DARTree: Tree-Structuring Speculative Decoding for Up to 9.73x Lossless Speedup

Diffusion-model-based draft generation is fast at proposing candidates because it can predict token blocks in parallel, but it has a weakness: the probability distribution at each position isn’t conditioned on the surrounding context. DARTree is a training-free speculative decoding method that extends a pretrained autoregressive (AR) correction head from a single draft sequence to an entire tree structure. Across seven math, code, and dialogue benchmarks, it achieved the highest average acceptance length and speedup in all four model/temperature settings tested. In one setting, its acceptance length was 98.6% higher than DFlash and 27.9% higher than Domino; it accepted up to 12.97 tokens per verification round and reported up to a 9.73x lossless speedup over sequential AR decoding (Source).

This is a “lossless” speedup that doesn’t change the output distribution — an improvement in inference cost, not capability. Being training-free means existing AR correction head assets can be reused as-is. This kind of optimization is an infrastructure-side advance that only pays off once it’s incorporated into serving frameworks like vLLM, and it’s unlikely to serve as a differentiator for a standalone product. Implications and position: If your cost structure is dominated by token inference costs, checking on implementation status in major serving frameworks once a quarter and recalculating your cost assumptions once it lands is sufficient — this isn’t something worth investing in a custom implementation for right now.

Try This Week / Hype to Ignore

Try this week: If you have a shell-execution agent harness, audit the escaping at your command interpolation point and add a re-execution test through a broken parser to your CI. If you’re evaluating products for non-English languages or high-compliance requirements, add a small model like Mimir v1 that states its basis for permissible data use to your candidate list.

Hype to ignore: OmniScientist’s “100% completion rate” for “AI fully automates research from ideation to writing” is pipeline coverage, not quality — don’t let it drive investment decisions without accounting for the average score of 6.3. Likewise, “AI writes formally verified code” (Vero) is premature as a replacement for proof engineers, as shown by the measured 27/43 results with zero solves on the hardest problems.