Research #AI agents Oct 23, 2026 · Originally published Oct 22

Microsoft open-sources ThinkingBox-Training: using final database state as the reward, 50 GRPO steps take Qwen3.8-27B from 51.70% to 60.78% average success

Grading an agent by what it leaves behind is now being used directly as a training reward. How it works, and what it costs.

Primary source Hugging Face Blog (Microsoft) · huggingface.co Read the original ↗

// Key points

Each of ThinkingBox-Bench's 507 tasks starts a real backend, gives the agent MCP tools and a simulated user with private context, and at the end runs executable checks on the final database state: whether the records are right, not which tools were called.
The team built a training pool from similar tasks outside the benchmark, choosing ones neither too hard nor too easy for the current model, based on success rates estimated once from the base policy.
After 50 GRPO steps, Qwen3.8-27B's average success rose from 51.70% to 60.78%, and tasks passed on all 20 attempts rose from 38 to 107, with a small decline in at-least-once coverage.
On a hotel downgrade task, the base model changed the booking without required manager approval in 15 of 20 attempts; after training it preserved booking and payment state in all 20.

Builder's takeEven without training, use this idea for evaluation first: when an agent organizes or moves files in AI Cloud Drive, acceptance should check what the folders and permissions end up as, not whether the agent says it's done. Tasks passed 20 out of 20 going from 38 to 107 tracks the production experience better than the average, and that metric is worth copying.

// Background · from #AI agents

Full timeline →