Cloudflare's GLM-5.3 Workers AI Leak: How a KV Cache Bug Served Other Customers' Data

On October 26, 2026, Cloudflare published a post-incident report on its GLM-5.3 model on Workers AI. For almost two weeks, a small number of customers who sent very long prompts could receive text that belonged to other customers' sessions. Cloudflare traces the cause to two gaps: a cache pool it knew was smaller than the context window, and a bug in the open-source inference engine. It found no evidence of deliberate exploitation.

What happened in the Cloudflare GLM-5.3 incident?

By our arithmetic from Cloudflare's timeline, the model went private about 15 hours after the report arrived (the report arrived at 23:11 UTC, and Cloudflare says verification began at 08:00). Cloudflare says it took the model offline within 40 minutes of declaring the incident.

The scope Cloudflare reports: 155 responses were affected, all to prompts longer than 681,280 tokens, and only the GLM-5.3 model (@cf/zai-org/glm-5.3). Cloudflare says a small number of customers received responses containing data from other customers' requests made to the same model shortly before. After excluding its own testing and the researcher's, it says the remaining requests look like ordinary long-context use in timing and volume, and that it found no evidence of deliberate exploitation. It also says no other model hosted on Workers AI was affected.

Why did long prompts return other customers' data?

By Cloudflare's account, the engine let a prompt too large for the cache pool continue, and the decode node then read cache pages that still held earlier requests' data. A KV cache is the memory an inference engine keeps for each request so the model does not recompute the whole prompt for every new token. It is the attention cache inside the engine, not Cloudflare Workers KV, the storage product. In Cloudflare's setup, it is a shared pool of fixed-size pages on the GPU. Pages are not zeroed between requests; the next request to receive a page overwrites it. Cloudflare calls this common among inference engines and necessary for performance. The consequence is that the engine alone must track who owns each page.

In sequence, by Cloudflare's account: a prompt longer than the cache pool arrives; prefill rejects it internally and swaps in a one-token placeholder; one page of cache is transferred to decode; decode reads all the pages it reserved for the full prompt, most of them stale; the client receives HTTP 200 instead of an error. In our reading, the step where the request should have stopped is the rejection at prefill.

Cloudflare reports that the bad response was coherent, its usage data showed the long prompt's token count, and in Cloudflare's diagram example it arrived faster than prefill should allow. In our reading this is a weak hint at best, not a dependable alarm. Cloudflare also says the bug had been reported to SGLang by another user as issue #30233 on July 6, 2026, before the August 27 launch, and that it noticed this while investigating the incident.

Why did Cloudflare's own tests miss it?

In Cloudflare's words, its pre-launch tests validated HTTP response codes and engine performance at long context lengths, but not model output and coherence. It saw successful status codes and a streaming body with no error frames, and assumed the engine handled the difference gracefully. The pool size is printed in the SGLang startup logs and exposed as a metric, so the mismatch was observable. The lesson Cloudflare draws is to check the pool size against the context length directly, from the running engine, before any disaggregated deployment change ships.

What did Cloudflare change?

• The configuration: the prefill KV pool now holds about 2.04 million tokens, roughly twice the context length, so a valid prompt always fits.
• The engine: a patched path retires a rejected request at admission with its original error, before any allocation or KV transfer. Upstream fixes landed in SGLang v0.5.20 to v0.5.22; Cloudflare recommends all operators move to v0.5.22 or later.
• The checks: every disaggregated deployment change now verifies KV pool size against context length, deferred release on both roles, transfer batch bounds and that the required patches are applied.
• The load balancer: a client disconnect no longer aborts a request in the middle of a transfer.
• Integrity checks: a per-transfer checksum has landed upstream as a first step. Cloudflare says it is working on per-page ownership tags checked at read time and on formal verification of the KV cache transfer protocol, and that it disabled an earlier prototype of the page tags because of reliability issues.

What does the SGLang team's own post add?

The SGLang team published its own disclosure the same day, in coordination with Cloudflare. It describes the same mechanism from the engine side: a request marked for rejection could still proceed through the disaggregated handoff, while the single-node rejection path returned an error instead. It names the two upstream fixes, #36988 and #38935, and says Cloudflare notified the maintainers on September 9. The 155-response count and the customer impact are Cloudflare's; the SGLang post refers readers to Cloudflare's report for both.

The post also describes a related but separate bug class that matters if you run disaggregated serving. When a request is aborted mid-transfer, a write that has already been launched cannot be cancelled, and if the destination memory is reassigned first, the late write can land inside another request's cache. SGLang says this corruption is silent, rare in typical workloads and far more likely with long contexts. It says default prefill/decode configurations could hit it, whenever a request is aborted mid-transfer and the freed slots are reassigned before the late write lands, from the v0.4.x line until v0.5.21 (v0.5.22 for MORI), and that a single-instance deployment is not affected by this race. For this separate race, SGLang says a deployer has no reliable way to tell whether exposure already occurred, because the corruption leaves no error or log line. Cloudflare says the bug in its incident was different from this timing race.

SGLang's recommendations for prefill/decode deployments: upgrade (v0.5.21 protects Mooncake and NIXL by default; MORI needs v0.5.22), run a router that cancels both halves of a request pair, and watch the sglang:decode_deferred_kv_release_total metric, shipped in v0.5.22, where a healthy deployment releases almost all holds through "drained". On v0.5.19 or v0.5.20 it describes setting the deferred-release environment variable on both servers as a partial measure that covers only decode-initiated aborts and does not replace upgrading. Its own timeline also lists a further NIXL and MORI fix (#45973, so that failed sends are no longer acknowledged as drained) slated for v0.5.23.

What should teams running or buying inference check?

The first group below is Cloudflare's own advice to operators; the second is our analysis.

From Cloudflare's report:

• Measure the KV pool and compare it with the context length on every prefill and decode node, after any change to weights, quantization, parallelism or memory fraction. If the pool is smaller, lower the context length to fit.
• Upgrade to an SGLang build that includes the admission fixes (v0.5.22 or later) so an over-length request is rejected before handoff.
• Run deferred KV release on both roles and bound Mooncake transfer batches.
• Turn on corruption detection where your model supports it, and check your logs to confirm requests are actually being verified. Cloudflare says these checks do not yet cover every model architecture.

Our additions:

• Test the content, not the status code. Cloudflare says it validated its fixes with canary-marker tests; make a cross-tenant marker test a permanent release gate. Send long prompts from two test tenants, each seeded with a unique marker string, and fail the build if either marker appears in the other's output.
• Add gateway limits as defense in depth. Count input in tokens with the model's tokenizer and set the limit below the smallest KV pool you measure on any node, with a margin of several cache pages, and measure what your engine actually rejects instead of computing it (the reporter of #30233 describes the engine's own limit as slightly below the pool, but their numbers are internally inconsistent). Cloudflare says not to rely on a proxy to enforce the limit, so also lower the engine's context length to fit and verify that the engine rejects what does not fit.
• Treat unexpected latency as a diagnostic signal, not proof. In its diagram example, Cloudflare notes the faulty reply came back faster than the prefill should take. A latency-versus-prompt-length check is cheap, but for the separate abort race SGLang says a deployer has no reliable way to tell after the fact whether exposure occurred.
• Keep a one-step model off switch. Cloudflare says it took the model offline within 40 minutes of declaring the incident; by our arithmetic from its timeline, requests returned HTTP 400 after 38 minutes. Make sure you can do the same without a redeploy.
• If you consume hosted open models, ask your provider how tenant isolation is tested and what it notifies customers about, and how fast. The 681,280-token threshold applies to the requests that received leaked text; Cloudflare gives no length threshold for the earlier requests whose data leaked, so prompts of any length may have been the source. Minimize sensitive data in prompts.

What is still unknown?

Cloudflare does not say how many customers were affected beyond "a small number", and the post does not state whether affected customers were notified individually; it notes a status page update at 14:00 UTC on September 9. It says the stale pages held other users' prompts and partial generations, but not what that content was in the 155 responses. The exposure ran from August 27 to September 9. Cloudflare published the report on October 26 as a coordinated disclosure with the researcher and the SGLang maintainers, 47 days after service was restored (our calculation). The 155-response count and the customer-impact assessment come from Cloudflare's investigation; the SGLang team's post, written in coordination with Cloudflare, describes the engine mechanism and the fixes. Everything here comes from the published reports; we have no access to Cloudflare's logs or systems.

How is this different from the agent incidents we covered?

In Anthropic's four evaluation incidents, a model with a goal reached systems it should not have. Here no model pursued a goal: shared infrastructure returned one customer's cached data to another. In our reading, both come down to the same discipline, which is verifying a boundary from the outside instead of assuming it holds. For the operational side, see why AI agents break in production. We also cover this report in our October 29 digest.

Sources

• Cloudflare, October 26, 2026: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang
• SGLang Team, October 26, 2026: SGLang security update: improving request isolation
• SGLang GitHub issue #30233, July 6, 2026: PD disaggregation: aborted prefill requests (input too long) cause decode to generate garbage from incomplete KV cache