Ai2's StateBench: Gemini 3.8 Flash, Claude Opus 5.5 and GPT-6 Sol and Luna drop to 13% or lower on the longest, hardest state-tracking problems
A benchmark for following how values change across a long document, which is what long agent runs quietly depend on.
// Key points
- Four difficulty levels map to kinds of variable assignment: immutable values, reassignment by value, self-referential updates (x = x + 5) and assignment by reference (x = y, swaps)
- All four frontier models did well on the two easier levels, but on the two harder ones accuracy fell sharply as inputs grew, to 13% or lower on the longest general finite problems
- Trained from scratch at 275M parameters on sequences of up to 256 assignments and tested on up to 64,000, Gated DeltaNet and hybrid models generalized almost perfectly on three levels; transformers degraded with length and never learned the hardest level
- Format matters: some small models scored near the ceiling on RULER-style problems but near chance on the same problems framed as spreadsheet edits
Builder's takeThis is the failure behind agents that lose the plot after many edits: working out the current value after a chain of updates is hard even for frontier models. In long agent runs I wouldn't ask the model to reconstruct state from the transcript; keep the current state in a structured object the harness updates and feed it back each turn. Test with your own data formats too, not just needle-in-a-haystack scores.
// Background · from #AI agents
Full timeline →- Nov 4 Claude's Messages API adds message threads in beta: later turns send only the new messages, and a thread expires 5 minutes after its last request
- Nov 4 Google DeepMind releases Cheminformatics Skills for Antigravity: in a lung-cancer drug case, an agent curates thousands of compounds and proposes macrocycles with synthetic routes in an hour
- Nov 4 AWS sets up an on-behalf-of token exchange in Bedrock AgentCore Gateway so an agent calls ServiceNow as the signed-in user