Cloudflare discloses a GLM-5.3 cross-request leak: 155 responses carried other customers' data, caused by a KV pool smaller than the context window plus an SGLang bug
A full write-up of a tenant isolation failure in hosted inference. Teams serving models with SGLang get checks they can apply directly.
// Key points
- Between August 27 and September 9, 155 responses from GLM-5.3 on Workers AI contained data from other customers' earlier requests, all to prompts longer than 681,280 tokens. Cloudflare found no evidence of deliberate exploitation.
- Gadi Evron of Knostic reported it via HackerOne on September 8. Cloudflare took the model offline within 40 minutes of declaring an incident, restored it the same day, and published on October 26 in coordination with the researcher and the SGLang maintainers.
- Two causes: the model launched with a 1,048,576-token context while the prefill KV pool held only 681,280 tokens, and tests checked only HTTP status codes; and in disaggregated prefill/decode mode SGLang swapped a rejected request for a one-token placeholder, so decode read stale cache pages and the client got a 200 instead of a 400.
- After the fix the pool holds about 2.04 million tokens. SGLang patched the bug in v0.5.20 with follow-ups in v0.5.21 and v0.5.22, and Cloudflare recommends all operators upgrade to v0.5.22 or later.
Builder's takeThis is the kind of risk self-hosted open models miss most easily: no error, just data quietly crossing tenants. If you run SGLang, upgrade to 0.5.22 first. Before launch, read the KV pool size from the running engine and confirm it is at least the advertised context, and make long-context tests check the output itself, with canary strings, not just status codes. For multi-tenant file Q&A like AI Cloud Drive, this goes on my release checklist.
// Background · from #Open models
Full timeline →- Oct 26 Ollama 0.40.11 adds text-to-speech, starting with Phonon, 35% faster than Gradium's own runtime on an M5 Max
- Oct 26 Google details TB screening in the field: a partner device has read 25,000 chest X-rays, and open MedGemma has passed 10 million downloads
- Oct 25 Hugging Face launches a Trainium compatibility lookup: whether a Hub model runs unmodified on Trainium, with a script to run it yourself