Gynga AI Topics
JA EN
日本語

Offensive AI Gets Cheaper, Defensive AI Doesn't Scale — The Asymmetry in Security Agent Evaluation

TL;DR

  • Offensive AI agents (CTF) get stronger the more compute you throw at them, and open-weight models can approach frontier-level performance cost-effectively. Defensive AI agents (SOC investigation), by contrast, don’t improve with more compute — success hinges on disciplined tool use and skillful telemetry exploration.
  • Investment in offensive agents tends to get absorbed by frontier model improvements anyway, while defensive agents reward investment in telemetry integration and tool-calling harness design.
  • Other papers this week covered expanding pretraining-data poisoning vectors, probabilistic self-contradiction in LLM aggregation, flaws in evaluating agentic search, and retrain-free recommendation updates.

Lead story: Offense gets cheaper, defense doesn’t scale

Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents / Interactive results site

Using Cybench (CTF, capture-the-flag style security challenges) for offense and Splunk BOTS v1 (SOC investigation scenarios simulating breach investigations by a security operations team) for defense, this paper compares language model security agents by “success rate at a fixed cost level” rather than “peak success rate under a generous budget,” decomposing spend into inference cost and tool cost.

Technical read

Traditional security agent evaluations have mostly measured peak performance — vulnerability discovery, exploit writing, CTF completion — given a generous inference budget. What’s new here is comparing success rates at fixed cost and discovering that offense and defense scale in completely different ways. On the offensive side (CTF), performance improves with more test-time compute, and scaled open-weight models can approach frontier proprietary models while remaining cost-competitive. On the defensive side (SOC investigation), performance doesn’t scale the same way — success depends less on raw inference budget and more on disciplined tool use, telemetry exploration, and selective enrichment (narrowing down additional information).

One caveat: Splunk BOTS v1 is a somewhat dated training dataset (released in 2016) and may not fully replicate the noisy alert-handling reality of production SOC environments. Cybench is also an artificially designed set of CTF challenges, and both benchmarks come from a single research group’s pre-peer-review preprint, so the results should be read with that in mind.

Business read

  • Unit economics / cost structure: Automated penetration testing has a cost structure where investment in test-time compute translates directly into lower costs. SOC automation is the opposite — pouring in more tokens doesn’t help, and results only improve through investment in tool-calling harness design.
  • Build in-house vs. get absorbed by the platform: If offensive performance gains are mainly explained by “throwing more compute at it,” dedicated pentest-AI products are vulnerable to being swallowed by general-purpose reasoning improvements in frontier models. Defensive performance, on the other hand, depends on the engineering discipline of tool orchestration, which won’t be automatically solved by general model improvements alone.
  • Operational overhead / transferability: The success factors on the defense side (telemetry exploration, enrichment selection) amount to embedding SOC-specific operational knowledge into the harness (tool-calling design and sequencing logic) — scaling this across customers requires adapting to each organization’s telemetry structure.
  • Defensible moat: For defensive agent products, the moat isn’t the model itself but the telemetry integration and tool-calling design/interpretation logic.
  • Market sizing / positioning: If defensive performance depends on integration quality with existing security telemetry (log platforms like Splunk), incumbent security operations vendors building an agent layer on top of their own data have an advantage, while thin-wrapper SOC automation products are at a disadvantage.
  • Where humans stay in the loop: Since the bottleneck in SOC investigation is judgment rather than compute, the investment should go toward “agent narrows down, human makes the final call” workflows rather than full autonomy.

Contrarian angle / what’s overlooked

The conclusion that “defense doesn’t scale” may not be a permanent property. If tool-use discipline itself is a capability that could improve through fine-tuning on tool-use trajectory data, then the real story might not be “defense is compute-independent” but rather “current harness designs simply haven’t been trained specifically for tool use yet.” Also worth flagging: the asymmetry where offensive automation becomes cheap and practical before defensive automation does is arguably less a market opportunity than a security risk for the ecosystem as a whole.

Implications and position

If you’re building or buying security agent products, offensive (pentest automation) investment should be valued with the assumption that it will likely get absorbed by frontier model improvements. Defensive (SOC automation) investment, conversely, pays off more from designing a tool-calling harness matched to your own telemetry structure than from model selection. If you’ve already bet on offensive automation products, you need to shift your differentiation away from model dependence and toward proprietary data and workflows.

Confidence: High confidence in the methodology (that comparisons should control for cost). Medium confidence in the specific finding that “defense doesn’t scale,” since it comes from a single benchmark and a single research group, and reproducibility on other SOC datasets or in real production environments remains unverified. Leans toward signal, but generalization should be approached cautiously.

Other notable papers

Expanding pretraining-data poisoning vectors

Pretraining Data Can Be Poisoned through Computational Propaganda

This paper demonstrates that public discourse venues (forums, comment sections, and other interfaces anyone can post to) can be used to inject poisoning at scale into pretraining datasets. Using an analysis method called HalfLife, it estimates how much malicious content survives post-crawl data curation pipelines (filtering, deduplication, etc.).

Technical read: Prior poisoning research targeted controlled sources like Wikipedia, but actual pretraining corpora draw from vast and diverse sources. This work is novel in showing that publicly writable discourse interfaces are a viable poisoning vector, and in quantifying the conditions under which poisoning survives curation.

Business read: For any player collecting web-scale data in-house for pretraining or continued pretraining, data provenance becomes a security requirement. If frontier labs have adequate defensive curation in place, this risk falls mostly on fully in-house players doing their own data collection through training.

So what / stance update: If you’re collecting your own data and training your own models end to end, you need to budget for provenance tracking and poisoning detection as an operational cost. This isn’t a direct threat if you’re just consuming a frontier API, but the underlying lesson — question the quality of your training data — also applies to your own RAG and evaluation datasets.

Self-contradiction in LLM aggregation (the macro fallacy)

Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

This paper finds that if you keep splitting a population into ever-finer subgroups, prompt an LLM about each, and aggregate the estimates, you get closer to actual human data than asking the LLM directly about the population as a whole — a phenomenon they call the macro fallacy. By the law of total probability, the two approaches should agree, but the contradiction shows up broadly across frontier models.

Technical read: In-context reasoning in LLMs is often treated as conditional probability estimation, but this paper quantitatively shows that LLMs fail to satisfy basic probabilistic consistency. Models do possess subgroup-level knowledge (asking about finer segments produces good estimates) but lack a mechanism for integrating that into a consistent population-level estimate.

Business read: This has direct relevance to products using LLMs for market research, persona estimation, or synthetic surveys. “Ask the model directly” is unreliable, while “partition finely, then aggregate” is a concrete prompt-design guideline that improves accuracy.

So what / stance update: If you’re building or using synthetic data generation or persona-based market research tools, it’s worth baking the partition-then-aggregate method directly into the product design. Conversely, crude usage patterns like “just ask the LLM for market size directly” should be expected to keep missing the mark.

Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search

For agents that search and reason across multiple turns, this paper shows that about 30% of documents scored as “unhelpful” by standard static retrieval evaluation are actually “bridge documents” whose removal breaks the subsequent search trajectory. Using a counterfactual experiment on HotpotQA — removing documents and re-running the trajectory — the correlation with static utility scores was essentially zero (Spearman ρ = -0.026), and the proportion of bridge documents came out to 27.2% under an alternative metric.

Technical read: Traditional RAG evaluation (does reading a document improve the answer?) is reasonable for single-turn QA but doesn’t fundamentally apply to multi-turn agentic search — a point this paper quantifies for the first time. It surfaces a value axis invisible to static evaluation: “documents that tell you what to search for next.”

Business read: This is directly relevant to evaluating and tuning agentic search products (internal RAG, search agent SaaS, etc.). Reusing existing search evaluation pipelines (static relevance scoring) to evaluate agents or train retrievers risks optimizing away documents that are actually critical to supporting the trajectory.

So what / stance update: If you’re building agentic search in-house, you should replace static relevance metrics with counterfactual trajectory impact — “does removing this document change the trajectory?” Continuing to rely on static relevance evaluation risks misoptimizing by overlooking trajectory-critical documents.

Retrain-free recommendation embedding updates

Mutable Low-Rank Sketches for Retrain-Free Recommendation

This method updates embeddings on the spot the moment a user submits a new rating, without retraining the whole model. On the KuaiRec dataset, it matches or beats ALS (full-data, full-retrain) RMSE — 0.810 vs. 0.822 — while reading only 1.8% of the data, delivers 8x faster batch updates, and gives new users personalized recommendations in under 1 millisecond after their first rating.

Technical read: This approach tackles the long-standing “embedding staleness” problem in two-stage recommendation systems (embeddings stay stale until the next retrain) by learning a low-rank projection once and then recomputing on the fly via a sparse segment tree. It offers a theoretical guarantee (Theorem 1) that new observations monotonically tighten the error bound — a property existing methods lack.

Business read: This directly reduces retraining batch compute costs for recommendation SaaS/in-house systems built around cold-start handling and real-time personalization. The reported 40–130% improvement in item coverage under sparse data conditions (under 1% density) is especially relevant for e-commerce/content platforms carrying long-tail inventory.

So what / stance update: If cold-start for new users or retraining lag is a pain point in your service, this line of methods is worth evaluating. That said, the results come from a single dataset (KuaiRec), and reproducibility on production data still needs verification. It’s worth questioning the premise itself that “personalization can’t update without retraining.”

SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration

This framework formulates open-domain information gathering as “grounded relational schema completion,” externalizing search progress as explicit shared state — a Frontier Task, Evidence Graph, Coverage Map, and Failure Memory — rather than leaving it implicit in conversation history. It uses pipeline-parallel scheduling to immediately reassign stuck sub-agent slots to unresolved coverage gaps.

Technical read: The design philosophy of externalizing progress as structured data is well-founded, addressing the “repeated search loop” failure mode common to search agents with long dialogue histories. That said, based on the published abstract, there’s no explicit quantitative comparison against existing multi-agent search systems — this looks like an architectural proposal that’s ahead of its empirical validation.

Business read: If you’re building search/research agent products, the “externalized progress state” design pattern is worth adopting in your own implementation regardless of this specific framework. That said, this kind of general-purpose orchestration layer is also the kind of thing likely to get absorbed into major agent frameworks eventually.

So what / stance update: The idea of explicitly externalizing progress state is something you can apply to your own agent design right now. Production investment in the framework itself, however, lacks sufficient justification until quantitative comparisons against existing methods are published.

Try this week / hype to ignore

Try this week

  • Add “success rate at fixed cost” as one axis in evaluating your own security, RAG, or recommendation agents (a direct application of the lead story’s lesson).
  • If you’re running agentic search in production, run a small-scale counterfactual test — “does removing this document break the trajectory?” — instead of relying on static relevance (checking for bridge documents).

Hype to ignore

  • Hype claiming “AI agents will fully autonomously replace penetration testing” conflates the finding that offense can cheaply approach frontier-level performance with an unfounded assumption that defense scales the same way.

Sources