Two AI Models Set to “stir government urgency”, But Will This Challenge Undo Them?

A pair of near-term model releases is forcing a hard tradeoff: scarce compute and high-stakes government relationships are shaping what gets shipped next—and what gets shelved. OpenAI has reportedly shut down its Sora app to free computing resources for the upcoming Spud model, while Anthropic is drawing renewed Pentagon attention to revive a Claude deal that the US government recently set to expire. The common thread is urgency: both companies are being pushed to deliver the next qualitative jump fast enough to matter, even as regulators and defense agencies tighten timelines and expectations.

That pressure is landing alongside a benchmark that’s designed to be brutally revealing. ARC AGI 3, a new installment in the ARC benchmark line, reports a striking gap between human performance and today’s frontier models: humans reach 100% while leading AI systems score under half a percent. The benchmark’s creators argue this gap matters because it measures residual deficiencies between frontier AI and human-level AGI—without relying on language, memorized knowledge, or cultural cues. ARC AGI 3 uses abstract, grid-based tasks where the goal often must be inferred rather than explicitly stated, and it tests multiple abilities at once: exploration, planning, memory, and goal setting.

ARC AGI 3 also tries to prevent “benchmark gaming,” a problem that already saturated earlier ARC AGI 1 and 2. Those earlier benchmarks improved rapidly once models gained publicly demonstrated chain-of-thought reasoning, but the paper warns that similarity between public demonstrations and private tests let training pipelines effectively attack the evaluation distribution. For ARC AGI 3, the public and private sets are designed to be more out-of-distribution, so models can’t rely on shortcuts that mimic the test’s hidden structure.

Scoring rules further shape what “progress” would mean. Performance is capped at 100% based on a human-derived baseline, so even perfect efficiency can’t exceed the ceiling. Turn-based mechanics also remove speed/reflex advantages from the equation, and action efficiency is central: attempts that use more than five times the number of actions taken by humans are scrapped, while inefficiency is quadratically penalized. Even so, the benchmark remains adversarial—human baselines are derived from the second-best human run, and the best first-run human performance can imply substantially different scores under the same rubric.

The benchmark’s low scores are not just a verdict on raw capability; they also reflect constraints on how models can be deployed. Systems built with special ARC AGI 3-specific harnesses are disallowed, and the evaluation emphasizes general-purpose API play with minimal context—no reminders about winning or action minimization. That helps explain why even strong models like Gemini 3.1 reportedly land around 0.37%.

Beyond ARC AGI 3, the broader AI trajectory looks uneven. OpenAI is reportedly pushing toward a fully automated AI researcher that can tackle complex problems with humans acting as reviewers, but historical productivity gains from AI drafting and human editing have been closer to incremental improvements than immediate takeoff. Meanwhile, agentic systems bring new security risks: a “vibe-coded” hack reportedly co-opted an open-source Python library in a way that could leak secrets and keys if updates weren’t handled correctly. The result is a “messy middle” moment—better at drafting and generalization across lower-level tasks, weaker on higher-level objectives like academic integrity and adaptive goal setting—making the next year a high-variance test of both capability and control.