Tools #AI agents Oct 17, 2026 · Originally published Oct 16

AWS shares the evaluation loop behind its internal sales agents: three months took the pass rate from 93% to 96% and pure hallucinations from 138 to under 52 per 10,000 responses

An agent quality engineering case from a real production system, with concrete numbers and failures.

Primary source AWS Machine Learning Blog · aws.amazon.com Read the original ↗

// Key points

Built on Amazon Bedrock AgentCore: sellers ask questions in plain English in Amazon Quick, answered by sub-agents for customer insights, performance insights and text-to-SQL.
Before launch, evaluation caught the Customer Insights Agent reporting a customer’s $410.08 three-month revenue as “$410.08 Million.”
In one pre-production test, a mismatch window from data, tool and prompt changes landing apart dropped the pass rate from 78% to 38%.
A release gate over 200 questions blocks automatically if the score regresses against the candidate’s prior run, fails to beat production, or too many questions fall below the per-question threshold.

Builder's takeReporting $410.08 as $410.08 million is the scariest kind of error: confident, well formatted and hard to spot in manual checks. I’d copy the approach with a fixed question set for the dashboard Q&A in AI Cloud Drive, run on every prompt or data-pipeline change before release, and ship data, tool and prompt changes together, since the mismatch in between is where incidents happen.

// Background · from #AI agents

Full timeline →