The Race for Smarter Models Is Over — AI's Next Investment Target Is "Evaluation and Governance"

---

TL;DR

• Databricks’ chief scientist declared that “AI is already smart enough — the bottleneck is evaluation.” Article
• Palantir’s CEO singled out OpenAI’s and Anthropic’s business models for criticism, teaming with NVIDIA to pitch an alternative where “customers own the weights.” Article
• Hard numbers from AWS show token consumption is starting to come under corporate cost discipline. The conversation is shifting from “capability” to “economics and reliability” in multiple places at once. Article

Lead story: The race for smarter models is over — AI’s next investment target is “evaluation and governance”

The most substantive material today comes from an interview with Jonathan Frankle, Databricks’ chief AI scientist and co-founder of MosaicML. His argument is blunt: today’s AI models are already smart enough. Even if capability gains stopped right now, there’s decades of work left just figuring out how to apply existing models. The real bottleneck is evaluation and governance — how do you actually verify that AI did a good job?

Technical read

Frankle’s argument starts from a comparison with human employees. A human worker can get by with a qualitative performance review once every six months to a year, but that doesn’t work for AI. Self-driving cars make the reason clear: a human driver passes a licensing test once and can drive on public roads indefinitely, but self-driving software that passes a test once can’t be mass-deployed instantly — a single bug propagates simultaneously to every vehicle running that same software. That’s why AI needs evaluation that’s orders of magnitude more rigorous than what humans require.

The concrete bar he sets is a useful yardstick: “Being right 70% of the time isn’t good enough. Even 90% isn’t enough. It has to be genuinely trustworthy.” Hold your own agents or AI features up to that bar, and you’ll likely find that many products “pass the demo” but fall short of production-grade reliability.

The real difficulty isn’t technical — it’s articulating tacit knowledge. Translating the implicit human standard of “what counts as good work” into a checklist is itself hard. Frankle calls this “a far harder research problem than building the next giant model,” one that could take more than a decade to solve. His framing draws a clean line: model capability can be outsourced to providers like OpenAI or Anthropic, but defining “what counts as good work” is the user’s own job.

Business read

Looked at through unit economics, evaluation and governance aren’t a one-time model-selection cost — they’re an ongoing operational one. Labeling, rubric design, and monitoring — human judgment labor — become the new cost battleground.

There’s a real risk in the build-vs-platform-absorption question. Databricks is tying this argument directly to selling MLflow, its own open-source product. In other words, “evaluation infrastructure” as a differentiator could eventually be fully absorbed into Databricks’ or a competitor’s observability platform. Over-investing in building an evaluation stack from scratch risks over-investing in something that commoditizes within a few years.

That said, some assets are defensible. Even as evaluation tooling itself commoditizes, the rubric for “what counts as good work for us specifically” — and the accumulated history of failure cases behind it — isn’t easily copied. That’s closer to organizational judgment than software, and it can function as a moat upstream of the model itself.

There’s an ironic twist on operational load: evaluation isn’t the kind of task that gets easier by having AI evaluate AI — it’s precisely the area AI can’t yet bootstrap on its own. For now, it remains a labor-intensive bottleneck dependent on rubrics that humans define.

Market positioning is being redrawn. The arena of competing on model quality is heading toward commoditization, while investment shifts to the separate arena of evaluation, governance, and observability.

Contrarian angle / what’s overlooked

Some discount is warranted here. Frankle works for a company that literally sells evaluation and governance products, so “the bottleneck isn’t the model, it’s evaluation” doubles as justification for his own product line — the position-taking angle can’t be ignored. Also, “models are already smart enough” is itself a falsifiable, strong claim. If OpenAI and Anthropic keep delivering discontinuous capability leaps, the whole premise of securing reliability through evaluation weighting could loosen.

Implications and positioning

What to build and what to abandon is fairly clear. Abandon the assumption that switching to the next model will raise your quality. Build an evaluation pipeline — as lightweight as possible and not locked into any one vendor’s tooling — that defines what “good work” means for your specific business.

On whether this templatizes: the “mechanism” of evaluation infrastructure itself can be templatized and packaged (and will likely get absorbed the way MLflow has). But the company-specific yardstick of what to measure can’t be templatized. That’s where the investment opportunity remains.

It’s worth updating your stance accordingly: spending time building a system to measure your own output quality carries more leverage than spending time picking the “best” model.

Other notable stories

Palantir’s CEO criticizes OpenAI’s and Anthropic’s business models

Article

Palantir CEO Alex Karp named OpenAI and Anthropic directly, asking “why should customers hand over access to their data so [these companies] can build their own competitive advantage?” At the same time, he laid out a plan with NVIDIA to build “frontier-class” custom models for government agencies based on NVIDIA’s Nemotron open model, built in air-gapped environments isolated from outside access, with full ownership — including the weights — retained by the customer.

Technically, the claim that “an open model can be pushed to frontier-level performance” is still unproven — actual performance differences await public benchmarks. On the business side, this is a direct attack on the “charge per token + strengthen the model using customer data” business model, and it goes straight at the question of where the moat actually sits. It’s a differentiation strategy built around data sovereignty and IP protection.

So what: for industries that can’t tolerate the risk of customer data being used via API to strengthen a vendor’s model — government, critical infrastructure, finance — weight ownership as a differentiator will land. But for typical small-scale products, the cost and complexity of air-gapped operation is overkill, and it’s a choice not worth emulating. Confidence: moderate. Karp’s comments lean heavily toward positioning, and whether the NVIDIA partnership actually delivers remains unverified.

AWS: “There’s a move to rein in costs” — the token-consumption problem

Article

According to Amazon, Amazon Bedrock’s token processing volume in Q1 exceeded the cumulative total of all previous periods combined, with customer spending up 170% quarter-over-quarter. Against that backdrop, an AWS Japan executive said in an interview that “there’s a movement to treat token consumption as a cost to be reined in.” AWS is emphasizing that customers should balance not just unit price but total cost of ownership (TCO).

There’s nothing technically new here, but on the business side it reads as a sign that the era of “token consumption = unconditionally good” is ending, and that the stage of being held accountable for ROI has arrived.

So what: even for small-scale founders, if you can’t talk about AI spend in terms of TCO — including the actual revenue and profit your generated output produced, not just unit price — your AI budget is headed for scrutiny. An internal KPI that only tracks “token volume” is a warning sign; replace it with a metric tied to actual outcomes.

Program-as-Weights — compiling natural-language specs into cheap, executable code

arXiv

A proposal called “fuzzy-function programming” compiles natural-language specifications for tasks — log monitoring, repairing broken JSON, ranking search results — that are hard to write as rules but lose reproducibility and cost-efficiency when simply handed off to an LLM API — into small, locally executable neural artifacts. A 4B compiler model, trained on a self-built 10-million-example dataset called “FuzzyBench,” outputs parameter-efficient adapters for a frozen, lightweight interpreter. A 0.6B Qwen3 interpreter reportedly matched the performance of directly prompting Qwen3-32B, while using roughly 1/50th the inference memory and running at 30 tokens/second on a MacBook M3.

The technical core is a shift in the role of the foundation model — from a solver invoked for every problem to a tool-maker invoked once. The large model gets called a single time per function definition, and every subsequent call runs cheaply and locally.

So what: this is a concrete instance of the principle of routing tasks to the cheapest, most deterministic mechanism appropriate to their level of abstraction. If your product has spots where you’re just repeatedly handing the same kind of decision off to an API, this class of technique could eventually let you compile and localize it, changing your cost structure. That said, this is a single-paper research result, and generalizability and reproducibility remain unverified. Adoption right now should be avoided — treat it as signal, but leaning toward noise at this stage.

Microsoft cuts 4,800 jobs

The Verge

Microsoft is cutting about 4,800 jobs, roughly 2.1% of its workforce, concentrated mainly in Xbox and commercial sales. Xbox alone plans to cut about 20% of its division headcount by the end of the fiscal year. An internal memo stated that “the eliminated roles weren’t replaced by AI,” while also noting that “AI is changing how work gets done” — a phrasing that deliberately blurs the causal line.

So what: this isn’t the first time a major company has pursued layoffs and AI investment simultaneously, and generalizing this single case as proof that “AI took the jobs” is premature. The reorganization has plenty of sales- and gaming-business-specific causes of its own, and forcing it into a simple AI-causation narrative leans toward hype — close to something worth ignoring.

Worth trying this week / Hype worth ignoring

Worth trying this week: write out a 3-to-5-item checklist of “what counts as good work” for your own AI agent or feature, and manually score recent outputs against it. It’s a small-scale version of Frankle’s argument, and even just doing it gives you a read on where you actually stand — stuck at 70%, or past 90%.

Hype worth ignoring: the simplistic framing of “Microsoft’s mass layoffs = AI took the jobs.” Organizational restructuring is business as usual, and this material alone doesn’t provide solid grounds for pinning it on AI. Similarly, avoid jumping to “a paper like Program-as-Weights is ready for production.” The numbers are compelling, but it’s a single research result, and there’s real distance between that and production deployment.