A tenant-isolation failure in Cloudflare's Workers AI platform exposed data from one customer's inference requests to other customers for nearly two weeks, the company disclosed on October 26 in a coordinated report with the SGLang open-source inference engine1.
Between August 27 and September 9, 155 responses generated by the GLM-5.3 model contained data from other customers' requests made to the same model shortly before theirs. Every affected response was to a prompt longer than 681,280 tokens. Cloudflare found no evidence of deliberate exploitation, saying the affected requests were consistent in timing and volume with ordinary long-context use.
Root cause: undersized KV pool meets unguarded rejection path
The incident traced to two overlapping issues: a misconfiguration and a bug in SGLang's handling of over-length requests in disaggregated prefill/decode mode. Cloudflare deployed GLM-5.3 with a 1,048,576-token context window, but the KV cache pool on its prefill GPUs only had room for 681,280 tokens. SGLang sizes the KV cache pool at startup from whatever GPU memory remains after loading model weights, and the shared pool's fixed-size pages are not zeroed between requests. Workers AI runs GLM-5.3 with SGLang in a disaggregated configuration using prefill GPUs and decode GPUs connected by RDMA via Mooncake.
Cloudflare was aware of the mismatch between the KV cache pool size and the model's context length before launch. Its pre-launch testing validated HTTP response codes and performance at long contexts but not model output coherence.
Response and hardening
Security researcher Gadi Evron of Knostic reported the issue through HackerOne on September 8 at 23:11 UTC. Cloudflare reproduced the bug the following morning and took the model offline within 40 minutes of declaring an incident. The company deployed a corrected configuration and a patched inference engine before restoring service the same day, September 9 at 19:40 UTC. It adjusted the GLM-5.3 configuration to match GLM-5.2, freeing enough GPU memory that the prefill KV pool now holds approximately 2.04 million tokens. The patched engine retires a request carrying a pending rejection at the disaggregation admission queue before any bootstrap, allocation, or KV transfer occurs.
Cloudflare proposed the fix upstream as pull request #38704; the SGLang maintainers merged #38935, which rejects over-length requests earlier. The SGLang team addressed the bug in v0.5.21, and Cloudflare recommends operators running disaggregated SGLang update to v0.5.22 or backport the listed patches.
On September 10, Cloudflare met with SGLang maintainers at RadixArk to agree on upstream fixes. RadixArk would investigate the race conditions, upstream the integrity-checking idea, and build reproduction tests for this class of bug. Cloudflare applied the engine fix and hardening to every model it serves with SGLang and now checks KV pool size against context length directly from the running engine before any disaggregated deployment change ships.
Cloudflare confirmed no other model hosted on Workers AI was affected; every other disaggregated deployment already had KV pools larger than its context length on both prefill and decode. The company is directing formal-methods work toward KV cache transfer engines and plans to model and prove the safety of an RDMA-based KV cache transfer protocol.
ANALYSIS The incident exposes a gap specific to disaggregated inference serving: when prefill and decode stages run on separate GPU pools connected by RDMA, a mismatch between advertised context length and actual KV capacity can silently convert rejected-but-unretired requests into cross-tenant data leaks. The fact that unzeroed KV pages are the norm in SGLang's shared pool means the blast radius of any future admission-path bug extends to every tenant on the same engine instance.