OpenAI explains scaling ChatGPT's Habitat storage with multi-tenancy, global caching and Azure Cosmos DB for 12x faster reads
At a billion users an AI product's bottleneck is storage too. This is the second engineering post in OpenAI's storage scaling series.
// Key points
- The post is Part II of OpenAI's engineering series on rapidly scaling online storage for over 1 billion ChatGPT users, covering a storage layer called Habitat.
- Per OpenAI's summary, Habitat was scaled with reliable multi-tenancy, global caching and Azure Cosmos DB.
- OpenAI says the work delivered 12x faster reads for ChatGPT.
Builder's takeConversation, memory and file state is the layer of an AI product that breaks first under load and gets the least attention. For a read-heavy product like AI Cloud Drive I'd settle tenant isolation and the cache layer before worrying about switching models. The full page couldn't be fetched, so the details here rest on OpenAI's own summary.
// Background · from #OpenAI
Full timeline →- Oct 25 OpenAI describes how it contains frontier model training and evaluation, including offline environments and high-compute red-teaming
- Oct 24 OpenAI shut down 12 legacy model snapshots including GPT-4, GPT-4 Turbo, o1 and o3-mini, plus 5 fine-tuning bases, on October 23
- Oct 23 OpenAI's Decisions API is generally available on gpt-6-luna, with up to 400 questions per request and prompt caching for shared input