Google tests Agent Anomaly Detection on five outside benchmarks: it flags 87% of harmful instructions the agent carried out, but catches only 4 of 18 driven by poisoned long-term memory
Google ran its agent anomaly detector on public benchmarks it did not write, and reported the strengths, the false-alarm rates and a clear gap.
// Key points
- The test used five public benchmarks, AgentDojo, InjecAgent, Agent Security Bench, AgentDyn and AgentHarm, 1,774 sessions in all, with an ADK agent on gemini-2.5-flash and each benchmark's deterministic check labeling the outcome.
- Of harmful instructions the agent read and carried out, 87% were flagged: 85% on AgentDojo, 87% on InjecAgent, 99% on AgentDyn and 81% on AgentHarm.
- False alarms ran 2% to 4% on benign office sessions and 16% on AgentHarm, whose benign tasks also post to social media, run terminal commands and download files.
- The clear gap is memory poisoning: when instructions planted in long-term memory made the agent act, it caught 4 of 18.
Builder's takeA vendor testing on someone else's benchmarks and publishing the weak spots is worth copying. For product teams the takeaway is that after-the-fact detection catches most injections but long-term memory is a blind spot: agents like PandaClaws that remember user preferences need a separate check before anything is written to memory, with a traceable source for each write.
// Background · from #AI agents
Full timeline →- Nov 13 Anthropic launches self-hosted Claude Managed Agents: the service, agents, sessions and history live on your own infrastructure, with model calls routable to Bedrock, Google Cloud or Foundry
- Nov 13 The next-generation AgentCore runtime becomes the default: sessions on a 2 GB image start in under 2 seconds at P75, down from 29.7, with up to 68% less billed memory for multi-turn agents
- Nov 12 Google launches the Gemini Decisions API: Gemini Decision 1 answers up to 1,000 closed-set questions in one pass at $0.10 per million input tokens, with no output billed