OpenAI says it will honor a shared robots.txt signal that lets websites state their AI training preferences once
Websites will no longer need separate rules for each crawler to say whether their content may be used for training, which matters to anyone running a content product.
// Key points
- OpenAI will honor a shared robots.txt signal to help website operators communicate their AI training preferences.
- The signal is aimed at all participating providers: a site writes it once and its training preference reaches every participating AI provider.
Builder's takeFor products like PandaClaws that publish content outward, this is a switch that belongs on the launch checklist: decide whether you want your site used for training, then set it once in robots.txt using the syntax the providers specify. Conversely, if your product crawls the web, start treating this signal as part of your crawl rules now instead of only honoring a few named user agents.
// Background · from #OpenAI
Full timeline →- Oct 31 OpenAI explains how it detects frontier model misalignment: layered monitoring catches misaligned and unauthorized agent behavior so responders can step in before incidents escalate
- Oct 30 OpenAI updates its Model Spec around five core values, with explicit rules for agent autonomy scope and ignoring untrusted data by default
- Oct 30 Codex CLI 0.166.0 adds GPT-6.1 Terra to its bundled, Vertex AI and Bedrock catalogs and explains safety blocks