Microsoft Research evaluates Copilot on synthetic enterprises: three industry environments with hundreds of users each, saving about $100,000 per evaluation dataset
How to build near-realistic enterprise environments for evaluating agents without touching real customer data, and what it saved.
// Key points
- The Industry Evaluation Data Generation Platform builds synthetic enterprise environments and automated query pipelines to evaluate Microsoft Copilot without real customer data.
- Three environments, pharmaceutical, capital markets and IT/software, each with hundreds of users and full communication histories, grounded in O*NET occupational data, public job postings and SEC filings.
- A composite quality metric improved 39% over the initial generation approach, and some industry environments scored above 0.9 out of 1.0.
- Dataset creation went from a multi-day manual process to automated runs, saving an estimated $100,000 per evaluation dataset and about $300,000 overall; the platform and data are internal to Microsoft, with no public release.
Builder's takeThis is exactly where private-data products get stuck: AI Cloud Drive cannot evaluate on users' files, and AI Interview should not use real candidates. Following this approach, I would generate roles and candidates in bulk from public job postings and occupational data, assemble a fake company's drive from public documents, and build one fixed regression set to run before every model or prompt change.
// Background · from #AI agents
Full timeline →- Nov 1 AWS lays out an end-to-end Bedrock cost playbook: spend attributed to each IAM principal, over-budget users blocked in seconds to minutes, and more than half of coding-fleet calls on cheaper models
- Nov 1 Bedrock AgentCore Runtime adds session endpoints: preview a web app in your own browser while a coding agent edits it in the cloud
- Oct 31 Gemini API's Computer Use tool is generally available across the Interactions and GenerateContent APIs, with Gemini 3.8 Flash as the recommended model