Gynga AI Topics
JA EN
日本語

What Changes When You Pass Tool Calls as Code Instead of JSON?

TL;DR

  • Featured: A new empirical study shows Programmatic Tool Calling (PTC), which has models call tools as code, matches or outperforms JSON tool calling in 11 of 14 models, matches or outperforms it in 13 of 14 under parallel fan-out, and remains stable under context degradation.
  • Also this week: four papers covering a benchmark for measuring “selective trust” toward misinformation, a statistical method that makes poker-style agent evaluation 74x more efficient, video LM failures in event counting, and a method for pinpointing root causes in agent failure trajectories.
  • The common thread: a simultaneous push to measure agent reliability, evaluation cost, and debuggability. This is a week to watch whether these get absorbed into official SDKs rather than rushing to productize them.

The Bitter Lesson of Tool Calling compares Programmatic Tool Calling (PTC), which has models call tools as code, against conventional native JSON tool calling, across 14 language models on the established BFCL v4 benchmark. In PTC, tools are exposed to the model as typed Python stubs, and the model writes code to invoke them. Execution and result retrieval complete within a single agent turn.

The results: PTC matched or outperformed JSON tool calling in 11 of 14 models. For the GPT-5.6 family, it improved 10.6% over the JSON approach. Under parallel fan-out conditions (simultaneous calls to multiple tools), it matched or outperformed in 13 of 14 models. Furthermore, under context degradation (context rot) from longer contexts, JSON tool calling degraded by an average of 2.3%, while PTC remained stable.

Technical Take

What’s novel about this research is that it systematically validates, using an established benchmark across multiple model generations, the practitioner intuition that “calling tools as code is more flexible” (the source itself explicitly states that “a systematic evaluation… has not been conducted”). The stability under context rot conditions in particular suggests this benefit may compound the longer an agent runs.

Business Take

The source doesn’t directly measure unit economics or token cost, but the results under parallel fan-out and context rot can be read as pointing toward lower costs from failure-driven retries and manual fixes. That said, this technology choice is unlikely to create a proprietary moat on its own. If vendors’ official SDKs adopt code-execution-based tool calling as standard, any homegrown PTC execution infrastructure will quickly become commoditized. On the operational side, exposing tools as typed Python stubs may be amenable to standardization, and an organization could roll it out more broadly if it already has a code-execution-capable framework.

Contrarian Angle / What’s Overlooked

The view that calling tools as code is more flexible is close to practitioner intuition and not particularly surprising on its own; this paper’s value lies in its quantitative backing. What tends to get overlooked: 3 of the 14 models showed no improvement, and validation beyond BFCL v4 is outside the source’s scope — over-generalizing would be a mistake.

Implications and Position

If you’re building a new agent stack, or rebuilding an existing one, it’s worth making PTC the default choice. However, since it is unlikely to constitute a defensible moat, it’s more sensible to use a framework that already supports it, if one fits your needs, rather than over-investing in a homegrown PTC execution system. There’s no need to stick with JSON tool calling simply because “it’s the industry standard.”

Other Notable Topics

MIST: A Benchmark for Measuring Selective Trust, and the SCOPE Training Method

Learning When to Trust via Selective Context Preference Optimization addresses the problem where, when language models condition their answers on external context, a single misleading signal can flip a correct answer into an incorrect one. The naive countermeasure — training resistance to context — has a hidden failure mode: a model that ignores context entirely looks robust but becomes useless in situations where the context is actually trustworthy. The authors reframe this as a “selective trust” problem, introducing MIST, a human-annotated benchmark that constructs each item under four conditions (clean, misleading, correct-context, and irrelevant-context), and SC2W, a paired metric measuring how often a misleading signal flips a clean-correct answer to wrong. The proposed method, SCOPE, optimizes DPO in a balanced way across all four conditions rather than only on misleading items, and the authors report it reduces SC2W while maintaining accuracy on the other conditions.

Technically, what’s new is the quantification of how training a model to “be suspicious” produces a separate failure mode (excessive dismissal). From a business standpoint, if you’re operating retrieval-augmented or tool-result-injecting agents, resilience to noisy external context is an area directly tied to reliability, and a metric like SC2W can be referenced as a framework for measuring vulnerabilities in your own pipeline. So what: if your RAG or tool-injection pipeline relies solely on an instruction prompt like “be suspicious of questionable input” as a countermeasure, that itself can create a separate risk of discarding correct answers. It’s worth having an evaluation axis that measures both over-suspicion and over-trust.

AV-AIVAT: A Statistical Method That Makes Poker-Style Agent Evaluation 74x More Efficient

AV-AIVAT is research on reducing the evaluation cost (money, model inference, expert time) of determining which of two agents is stronger, using sequential statistical methods. The existing method, AIVAT, reduces variance in imperfect-information games through conditional correction (a median 54x reduction across 15 configurations and 71,439 HUNL hand pairs), but it doesn’t indicate when to stop evaluating. This study proposes AV-AIVAT, which combines AIVAT with continuously-monitored Confidence Sequences, and reports that at a nominal 95% level with a target precision of ±1BB, stopping decisions based on raw results alone require a median of 74x more hands compared to AIVAT-corrected results.

Technically, this is a sequential testing design that stops “once sufficient evidence has accumulated” while maintaining a statistically valid confidence level. From a business standpoint, this validation is limited to poker (HUNL/Leduc hold’em), an imperfect-information game, and applicability to other domains is not verified within the source’s scope. So what: if you’re running agent version comparisons or A/B tests with a fixed, predetermined number of trials, the underlying idea of sequentially determining “how many trials are enough” is worth referencing, though you can’t directly reuse this study’s empirical results.

Video LMs Can’t Count High-Frequency Transient Events

The Low Frequency Trap validates video LM event-counting ability against executable ground-truth traces. Across three tasks — bouncing-ball wall contacts, blinks, and categorical state transitions — using 2,190 videos with controlled event counts and frequencies, the study found that at an 80% reliability threshold, Gemini 3.6 Flash could reliably count sustained state transitions up to 12 events at 0.5Hz and 1.0Hz, but there was no reliable positive-count region for transient blink events. In the high-count, high-frequency region, final-count accuracy was just 0.2%, and true-event recall stayed at only 18.1%. Increasing the sampling rate improved Bounce Ball accuracy from 19.6% to 29.3%, but the reported event sequence agreed with the ground-truth trace only 3.7% of the time.

Technically, the key point is that “getting the final count right” and “recovering the correct event sequence” are separate things (3.7%), which points to a risk that existing benchmarks scoring only the final answer overestimate video understanding capability. From a business standpoint, product designs — such as video analytics or activity-log generation — that hand full responsibility to a video LM alone carry a high risk of false detections and missed events. So what: if detecting high-frequency, transient events is part of your requirements, you should consider a hybrid design combining a video LM with deterministic detection logic, rather than relying on a video LM alone.

TrajDebug: Pinpointing Root Causes in Agent Failure Trajectories

TRAJDEBUG proposes a method for identifying the earliest error step responsible for the ultimate failure within an LLM agent’s failure trajectory. It addresses two challenges: in long trajectories, the reasoning basis is scattered across distant instructions, observations, and past context, making individual errors hard to identify; and failed trajectories contain multiple local errors, only some of which are actually responsible for the final failure. The method addresses these via multi-granularity history compression, evidence-based error identification, and tracking each error’s resolution status and ultimate impact. The authors built TrajErrBench, a benchmark of 486 manually annotated failure trajectories drawn from Tau2Bench and SWE-Bench Pro, and report the best overall performance against existing baselines. Code and data are planned for release.

Technically, the design of tracking an error’s lifecycle — whether it was “resolved” and whether it “affected the final failure” — goes a level deeper than simple first-error detection. From a business standpoint, this falls within the agent-operations observability space, and if this kind of capability becomes standard in agent frameworks or debugging tools, the value of building a homegrown trajectory-analysis infrastructure diminishes. So what: large investment in a homegrown error-triage infrastructure isn’t warranted. Observability capabilities are likely to eventually become standard in frameworks, so if you do invest engineering effort in-house, it’s more sensible to limit it to evaluation against the soon-to-be-released TrajErrBench.

Worth Trying This Week / Hype to Ignore

Worth trying this week: if your agent stack currently uses JSON tool calling, it’s worth running a small-scale test of the switching cost and benefit of moving to code-execution-based tool calling (PTC). Hype worth ignoring: the optimism that “PTC is a silver bullet” or that “video LMs will soon become universally capable.” This week’s empirical data supports neither claim.

Sources