Research #AI agents Oct 21, 2026 · Originally published Oct 20

IBM's ALTK-Evolve appends learned rules to the prompt, taking a GPT-4.1 support agent to 67.5% Pass^4 on τ²-bench retail without touching weights

Distill behavioral rules from an agent's successes and failures back into its system prompt, and an older model beats the retail leaderboard entries on reliability.

Primary source IBM Research on Hugging Face · huggingface.co Read the original ↗

// Key points

With GPT-4.1, an older non-reasoning model, on τ²-bench's stock ReAct scaffold, retail Pass^4 (all four independent attempts correct) rose from 57.5% to 67.5%, above every retail entry in the October 12 leaderboard snapshot.
Retail mean accuracy rose from 78.1% to 82.5%; airline from 57.5% to 65.0%.
How many rules to keep depends on the domain: retail did best with all 292; airline's Pass^4 peaked at 45% with only 14 high-support rules, against 40% with all 101.
The same model extracts the guidelines and runs the agent, with no weight updates or human annotation.

Builder's takeThis matches the approach in my checklist on fallback and human confirmation in agent systems: turning failures into explicit rules beats switching to a pricier model. I'll keep a failure log for every agent in production, distill it into rules for the prompt on a schedule, and watch every-time metrics like Pass^4, not just the average.

// Background · from #AI agents

Full timeline →