Tools #AI agents Nov 12, 2026 · Originally published Nov 11

AWS details UAEF, a framework-agnostic agent evaluation system: traces from LangGraph, Strands, AgentCore and others are normalized and scored on 36 metrics across 7 dimensions

Rebuilding evals every time you switch frameworks is a real pain for many teams. How UAEF solves it with adapters, and which dimensions it covers.

Image: AWS Machine Learning Blog
Primary source AWS Machine Learning Blog · aws.amazon.com Read the original ↗

// Key points

  • Adapters convert the messages, tool calls and agent-to-agent handoffs of LangGraph, LangChain, Strands Agents and Bedrock AgentCore into one canonical trace, so every later step is framework-independent.
  • It has 36 built-in metrics across 7 dimensions: tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination and reasoning; tool-calling metrics need ground truth with the expected calls, order and arguments.
  • Traces are screened for PII such as emails, phone numbers and credit card numbers before storage, and runs are stored as experiments in DynamoDB and S3 for comparing versions.
  • AWS says 12 organizations in telecom, healthcare and financial services used it to compare agent versions before release, with runnable notebooks for each supported framework.

Builder's takeThe idea is worth copying even if the system is not: settle on one framework-neutral trace format and have evals read only that. I would first write a test set with expected tool-call order for the PandaClaws publishing agent and run it before and after any model or prompt change, which fits with the NeMo failures-to-tests loop from 11-05.

// Background · from #AI agents

Full timeline →
  1. Nov 12 Google launches the Gemini Decisions API: Gemini Decision 1 answers up to 1,000 closed-set questions in one pass at $0.10 per million input tokens, with no output billed
  2. Nov 12 Microsoft Research discloses Lazarus: reprogramming fewer than 10 dormant neurons, with no training, backdoors an agent built on the model
  3. Nov 12 Tether AI Research releases BotPsy, a 0.8B on-device agent model: BFCL v4 rises from 23.4 to 44.0, and it beats Qwen3.5-2B-Think, 2.5x its size, on five of six tool-use benchmarks

Every Monday, the week in AI in one email

Every Monday morning: the 6 AI stories that mattered most in the past seven days, with my take, in your inbox.

Your email is only used to send the weekly and is never shared. Every email has a one-click unsubscribe link. See the latest issue →