NVIDIA shows a NeMo loop that turns agent traces into tests and harness fixes; in one example held-out passes rose from 4/6 to 6/6 and tool calls fell from 28 to 15
A concrete recipe for the loop most agent teams run by hand: find a bad behaviour in traces, write an eval for it, change the prompt, check on unseen tasks.
// Key points
- NeMo Compass analysed 42 recorded traces of an enterprise assistant against an ETHOS.md of expected behaviour and found it sending messages without user confirmation
- NeMo Eval Author generated four Harbor evaluations, including one that fails any run calling send_message without human confirmation; a coding agent then edited the system prompt
- Development tasks went from 3/6 to 6/6 passes (tool calls 10 to 6); held-out tasks from 4/6 to 6/6 (28 to 15)
- In a Dassault Systèmes example, malformed tool calls dropped from 142 to zero over about 12 hours overnight, with 35% fewer tool calls per run
Builder's takeThis builds on the NemoClaw blueprint from 11-02, and the useful part is the method, not the vendor: every bad trace becomes a regression eval before anyone touches the prompt. For PandaClaws I'd start with one rule like "never publish without confirmation" written as a verifier, and only then let an agent edit prompts.
// Background · from #AI agents
Full timeline →- Nov 5 Amazon WorkSpaces for AI agents adds Linux and Windows 11 cloud desktops alongside Windows Server
- Nov 4 Claude's Messages API adds message threads in beta: later turns send only the new messages, and a thread expires 5 minutes after its last request
- Nov 4 Google DeepMind releases Cheminformatics Skills for Antigravity: in a lung-cancer drug case, an agent curates thousands of compounds and proposes macrocycles with synthetic routes in an hour