---
title: "A Too-Small Memory Pool and an SGLang Bug Put Strangers’ Prompts in Cloudflare Answers"
description: "Cloudflare launched GLM-5.3 with a memory pool 35 percent smaller than its advertised window, and a public report of the same failure had drawn no reply"
author: "rews special report"
published: 2026-10-26T22:01:23.889Z
modified: 2026-10-27T12:03:03Z
url: https://rews.cc/a/cloudflare-s-glm-5-3-model-returned-other-users-data-in-155--2f8840
language: en
type: special report
tags: ["security", "ai", "cloudflare", "glm", "llm", "tech"]
publisher: "Rews (https://rews.cc)"
---

# A Too-Small Memory Pool and an SGLang Bug Put Strangers’ Prompts in Cloudflare Answers

*Cloudflare launched GLM-5.3 with a memory pool 35 percent smaller than its advertised window, and a public report of the same failure had drawn no reply*

Special report · By rews special report · October 26, 2026 · https://rews.cc/a/cloudflare-s-glm-5-3-model-returned-other-users-data-in-155--2f8840

## In brief

- Cloudflare says 155 GLM-5.3 responses on Workers AI held other customers’ data between Aug. 27 and Sept. 9
- The prefill memory pool held 681,280 tokens, against an advertised 1,048,576-token context window
- An SGLang bug passed rejected over-length prompts to decode nodes, which read pages earlier requests had left
- A public SGLang report of the same failure, filed July 6, was closed by a bot on Sept. 5 with no reply
- Fixes shipped in SGLang v0.5.20 to v0.5.22, and a draft to zero freed pages was opened Oct. 25

For 13 days beginning Aug. 27, anyone who sent Cloudflare’s hosted copy of the GLM-5.3 coding model a prompt longer than 681,280 tokens got back an ordinary-looking answer built partly from other customers’ requests, the company disclosed on Monday.

By Cloudflare’s count, 155 responses carried pieces of strangers’ work before Gadi Evron, the chief executive of the security firm Knostic, reported the problem on Sept. 8. The company set out the failure in [an incident report](https://blog.cloudflare.com/kv-cache-exposure-remediation/) published as a coordinated disclosure with [a security update from SGLang](https://www.lmsys.org/blog/2026-10-26-cf-security), the open-source inference engine that runs the model. Cloudflare said it found no evidence of deliberate exploitation and that no other model on its Workers AI platform was affected.

Read alongside the researcher’s own account, the two posts describe a leak that needed two failures to coincide, and both were visible beforehand. Cloudflare knew before launch that the model’s memory pool was 35 percent smaller than the context window it advertised. Its tests checked status codes, not whether the answers made sense. The engine bug had been reported in public on July 6 by another operator running a GLM model with the same 1,048,576-token window. No one replied, and a bot closed the report on Sept. 5. Upstream, the code that finally stopped the leak had been written to fix a different failure, in models that read images.

Beneath both failures is a design choice. The GPU memory pages that hold a model’s working state are not wiped between customers, because wiping costs speed. Cloudflare is keeping that choice for its A.I. models. A few weeks earlier, after a separate leak in its container product, it turned wiping back on.

## Two numbers that never met

Cloudflare [announced GLM-5.3 on Workers AI](https://developers.cloudflare.com/changelog/post/2026-08-28-glm-5.3-workers-ai/) on Aug. 28, calling it “Z.ai’s flagship agentic coding model with a 1M token context window, reasoning, and function calling.” Its incident report dates the exposure from Aug. 27. The configured window was 1,048,576 tokens. The KV cache pool on the prefill GPUs, the memory where the model keeps what it has already read of a prompt, held 681,280.

That left a band of 367,296 tokens where a prompt got through the front door but did not fit in the room behind it. Knostic found that prompts above 1,048,576 tokens drew a clean HTTP 400 error. Prompts inside the band got HTTP 200 and an answer.

No one sets the pool size by hand. SGLang works it out at startup from whatever GPU memory remains once the model weights have loaded, then prints it in the logs and publishes it as a metric. The context length is a separate setting. Cloudflare wrote that keeping the two in line was its job.

> Nothing forces the two to agree, and it is our job to check that they do.
>
> — Cloudflare, in its Oct. 26 incident report, [blog.cloudflare.com](https://blog.cloudflare.com/kv-cache-exposure-remediation/)

The company saw the gap and judged it harmless. According to the report, its long-context benchmarks before launch returned successful status codes and streams with no error frames, and engineers assumed the engine handled the difference “via streaming or some other mechanism.” The tests measured response codes and speed. They never read the output.

Anthropic described a similar blind spot in [a September 2025 postmortem](https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues) on three infrastructure bugs that had degraded Claude’s answers, one of them a routing error involving servers set up for a 1-million-token window. It promised evaluations able to tell a working implementation from a broken one.

## How the handoff failed

Workers AI runs GLM-5.3 in what engineers call disaggregated mode. One pool of GPUs, the prefill nodes, reads the prompt and builds its KV cache. The cache then moves over remote direct memory access, through a transfer system called Mooncake, to a second pool, the decode nodes, which write the reply. The cache is stored in fixed-size pages that get recycled. Each new request writes over whatever the previous one left, so the engine alone keeps track of which page belongs to whom.

SGLang’s scheduler did reject prompts too long for the pool. Its rejection routine recorded an error for the client and replaced the prompt with a one-token placeholder, which let the request drain through the normal batching code without claiming real memory. That worked on a single machine. In the split setup, the admission queues on both sides checked only whether a request had finished. A rejected request kept its rejection in a field, to\_finish, that the finished() check never looked at. So it appeared to be still running.

Cloudflare’s report walks through an example with eight pages. The router gave the decode node the prompt’s full length, and decode reserved eight pages for it. Prefill ran a forward pass on the single placeholder token, sent one page and signaled that the transfer was complete. Decode started generating and read all eight. This request had written only the first. The other seven still held what earlier requests left behind: other users’ prompts and partial generations. The client received HTTP 200 instead of 400, a prompt\_tokens count matching its long prompt, and a coherent reply that came back faster than it should have. SGLang’s maintainers summed up the engine’s part in the failure.

> In SGLang’s disaggregated prefill/decode path, a request marked for rejection could still proceed through the handoff.
>
> — The SGLang team, in its Oct. 26 security update, [lmsys.org](https://www.lmsys.org/blog/2026-10-26-cf-security)

Every request over the threshold behaved this way, Cloudflare wrote. Timing had nothing to do with it. That distinguishes it from a related class of bugs the two teams had been chasing for months, where an aborted request’s late RDMA writes land in pages that have already gone to someone else. SGLang’s post calls those “a distributed use-after-free, where the dangling pointer is a write request sitting in a NIC queue.” The design that both kinds of bug live in, with prefill split from decode and the cache shipped between them, was the topic of an office-hours session on vLLM, another open-source engine, in 2024.

[![vLLM Office Hours - Disaggregated Prefill and KV Cache Storage in vLLM - November 14, 2024](https://i.ytimg.com/vi/FPr37jCOvrA/hqdefault.jpg)](https://www.youtube.com/watch?v=FPr37jCOvrA)

A recorded vLLM office-hours session from Nov. 14, 2024, on disaggregated prefill and KV cache storage, the serving design Cloudflare uses for GLM-5.3. Video: Neural Magic · YouTube

## A stranger’s invoices

Mr. Evron was not looking for the bug. In [Knostic’s account](https://www.knostic.ai/blog/hallucination-or-cross-tenant-when-an-llm-answers-with-another-users-context), published on Tuesday, he wrote that the oddity first showed up during routine work through OpenRouter, a service that routes requests among model hosts: file paths from a machine that was not his, and tutorials in Chinese. On Sept. 7, triage traced the responses to Cloudflare, which [OpenRouter lists](https://openrouter.ai/provider/cloudflare) as a host for GLM-5.3, and testing moved to Knostic’s own Workers AI account.

According to Knostic, the leaked text included personal names, client and company names, invoice numbers, sums in euros, payment dates, details of source code, working-directory paths and records of agent tasks. Showing that none of it was a hallucination took more work.

> The challenge was establishing that the returned material actually came from another workload.
>
> — Gadi Evron, chief executive of Knostic, [knostic.ai](https://www.knostic.ai/blog/hallucination-or-cross-tenant-when-an-llm-answers-with-another-users-context)

Knostic put a unique marker at the start and end of every prompt. The marker came back on prompts of up to 662,162 tokens and was gone from about 700,000 tokens on. Response time fell from 68.9 seconds for the 662,162-token request to 7.1 seconds for a 700,000-token one, which suggested the prompt was not being read. The filler was made only of NATO-alphabet words, and the leaked text had none of them. One response included a shell tool call matching a public repository with a signed commit “less than four minutes after our affected inference,” far too recent to be training data. On the same platform, GLM-5.3 Flash, GLM-5.2 and GLM-4.7 Flash either answered properly or rejected the oversized prompts. The affected request was still “billed in full at 103,604.64 neurons,” Cloudflare’s billing unit.

The report reached Cloudflare’s HackerOne program at 11:11 p.m. Coordinated Universal Time on Sept. 8, according to the company’s timeline. Staff started verifying it at 8 a.m. the next day, declared a security incident at 1:22 p.m. and reproduced the leak with two separate accounts within 25 minutes. The model went private at 2 p.m. By 2:35 p.m. engineers had traced the cause to the pool size, after finding that a canary node with a larger pool was not vulnerable. Public access came back at 7:40 p.m. with a corrected configuration and a patched engine. Mr. Evron called the handling “an example of how a bug bounty program should run.”

## The warning on the tracker

The failure had been described publicly two months before. On July 6, a GitHub user, kevin-14, opened [issue #30233](https://github.com/sgl-project/sglang/issues/30233) on SGLang’s tracker. It reported that in disaggregated mode a prefill request aborted for being too long was cut down to one token, that only that token’s cache reached decode, and that decode, “which allocated space for the full prompt length,” ended up “producing garbage output.” The reporter was running SGLang 0.5.11 with GLM-5.2 in FP8, a 1,048,576-token context length and a pool capped at 284,928 tokens. The issue came with reproduction steps and a proposed fix.

No maintainer responded. On Sept. 5, eight days after GLM-5.3 went live on Workers AI and three days before Mr. Evron’s report, a bot closed the issue for inactivity. Cloudflare wrote that it came across the issue while investigating its own incident, and that it had been raising “these types of issues” with SGLang since version 0.5.10.

The upstream code that ended the exposure came from an unrelated fix. [Pull request #36988](https://github.com/sgl-project/sglang/pull/36988), opened Aug. 29 by the contributor mickqian and merged Sept. 6, targeted vision-language models: when an image-encoder failure aborted a disaggregated prefill request, resources stayed attached to it and a checker restarted the scheduler. As part of the cleanup, aborted prefill requests are retired before their results reach decode, and Cloudflare and SGLang now say that is what closes this exposure. The pull request does not mention issue #30233 or security. It went out in v0.5.20 on Sept. 18, [according to SGLang’s release page](https://github.com/sgl-project/sglang/releases). A narrower change, [#38935](https://github.com/sgl-project/sglang/pull/38935), merged Sept. 16 and released in v0.5.21 on Oct. 2, turns rejected requests away at both admission queues before any memory is allocated, following Cloudflare’s draft. Its author, Shangming Cai, was direct about what it does.

That sequence explains an apparent contradiction in Cloudflare’s post, which says in one place that SGLang addressed the bug in v0.5.21 and in another that the patch shipped in v0.5.20. Each statement describes a different patch. Cloudflare tells operators to run v0.5.22, released Oct. 19, because it also contains the fixes for the related race, and [Unite.ai’s coverage](https://www.unite.ai/cloudflare-says-glm-5-3-bug-leaked-155-responses-across-tenant-sessions/) gave the same advice.

Cloudflare had already built a defense against this kind of misread. In [an Aug. 3 post](https://blog.cloudflare.com/smaller-faster-safer-models/) it described tagging every physical cache page so that a request reading a page it did not expect would be aborted, and wrote that at its scale “even a one-in-a-billion mistake would show up regularly.” According to the incident report, the prototype cost under 1 percent in throughput and tail latency and was tested on GLM-5.2 in mid-August, after which the company “disabled it shortly afterwards due to reliability issues.” GLM-5.3 launched later that month. The report does not say whether the tags would have caught this failure, in which decode read pages that had been reserved for the request but never written.

## Leaks before this one

Cloudflare has sent one customer’s data out in another’s responses before. In February 2017, Tavis Ormandy of Google’s Project Zero spotted pages served through Cloudflare that contained private data from unrelated sites. An old parser bug had become live when a buffering change let it read past the end of a buffer. The company’s [incident report](https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/), written by John Graham-Cumming, then its chief technology officer, gave Sept. 22, 2016, as the earliest date memory could have leaked. At the peak, from Feb. 13 to Feb. 18, it put the rate at about one in every 3,300,000 HTTP requests. Search engines had cached 770 pages with leaked memory, spread across 161 domains.

On March 20, 2023, a bug in redis-py, the open-source library OpenAI used to talk to its cache, let some ChatGPT users see the titles of other people’s conversations. It also exposed payment details for 1.2 percent of ChatGPT Plus subscribers who were active during a nine-hour window, [OpenAI said](https://openai.com/index/march-20-chatgpt-outage/). Sam Altman, the company’s chief executive, wrote about it on X two days later.

> we had a significant issue in ChatGPT due to a bug in an open source library, for which a fix has now been released and we have just finished validating.
>
> a small percentage of users were able to see the titles of other users’ conversation history.
>
> we feel awful about this.
>
> — **Sam Altman** @sama on X · [March 22, 2023](https://x.com/sama/status/1638635717462200320)

Sometimes shared state in model serving is deliberate. In audits run in September and October 2024, Chenchen Gu and colleagues at Stanford found prompt caching at eight of 17 model providers, and at seven of them the cache was shared across organizations. In principle, timing could then tell one customer what another had sent. [The paper](https://arxiv.org/pdf/2502.07776) appeared at the International Conference on Machine Learning in 2025, and the authors said they had told the companies before publishing.

The nearest hardware parallel came in January 2024. Tyler Sorensen and Heidy Khlaaf of Trail of Bits disclosed LeftoverLocals, a flaw in GPUs from Apple, Qualcomm, AMD and Imagination, which did not clear local memory between programs. Another process on the same machine could use the leftovers to reconstruct a chatbot’s answers, about 181 megabytes per query on an AMD card running a 7-billion-parameter model, [according to their paper](https://arxiv.org/html/2401.16603v1). Nvidia’s chips were not affected. Trail of Bits announced the disclosure on X.

> Today, we are disclosing LeftoverLocals, a vulnerability that allows listening to LLM responses through leaked GPU local memory created by another process on Apple, Qualcomm, AMD, and Imagination GPUs (CVE-2023-4969)
>
> — **Trail of Bits** @trailofbits on X · [January 16, 2024](https://x.com/trailofbits/status/1747302889163219303)

Cloudflare’s other leak this fall came from the same habit of not wiping memory. On Sept. 24 it [disclosed](https://blog.cloudflare.com/containers-cross-tenant-vulnerability/) that a storage pool behind Cloudflare Containers and Sandboxes had been set up with skip\_block\_zeroing, which let a new customer’s disk keep 60 KiB of an earlier customer’s data in each reused 64 KiB block. Oren Yomtov of Accomplish found leftover data on 18 of 24 container placements, [BleepingComputer reported](https://www.bleepingcomputer.com/news/security/cloudflare-fixes-containers-cross-tenant-flaw-exposing-customer-data/). Cloudflare’s first fix was to remove that option so the pool wiped blocks again. The GPU pages serving GLM-5.3 are still not wiped, a practice the incident report calls common among inference engines and “necessary to achieve the best performance.”

## The argument over blame

Cloudflare’s post drew 47 comments on Hacker News, and the first dispute was about its title, which names “the GLM-5.3 Workers AI model.” One reader objected to attaching the model’s name to the bug.

> This is an SGLANG and a Cloudflare bug long before it is a Z.AI model bug, I don’t see what they could have done about it. Cloudflare should own it instead of naming other (imo innocent) parties in the title.
>
> — jacquesm on [Hacker News](https://news.ycombinator.com/item?id=50253748)

A user posting as kflansburg, who wrote in the first person for Cloudflare, answered that the post never calls this a model bug and names the endpoint because “our customers need to know which of our endpoints was affected.” On the substance, the record backs both commenters. Cloudflare’s report puts the cause in its own configuration and in SGLang, Knostic’s tests found other GLM models on the platform working correctly, and neither post finds any fault in Z.ai’s weights.

Another reader took the episode as a lesson about programming languages.

> Upshot: even though sglang is written in Rust, they still got burned by undefined behavior from uninitialized memory.
>
> — octoberfranklin on [Hacker News](https://news.ycombinator.com/item?id=50254683)

Neither half of that matches the record. SGLang’s [repository](https://github.com/sgl-project/sglang) contains Python as well as Rust, and the functions Cloudflare names, set\_finish\_with\_abort and finished(), are Python methods in its scheduler. The memory also wasn’t uninitialized in any sense a compiler checks. The pages held valid data from earlier requests, and the engine read them because a status check gave the wrong answer. A memory-safe language stops a program from reading past a buffer. It has no idea which customer a page of GPU memory belongs to. Cloudflare’s report puts that bookkeeping on the engine.

A third reader argued that the trigger was rarer than the headline figure suggests.

> I read that as you can only do a single prefill up to 681k tokens, separate from cache hit or generation. I think people try pretty hard to avoid a cache miss on sessions that long and rarely do standalone requests that big so the situation triggering this condition is rare.
>
> — hedgehog on [Hacker News](https://news.ycombinator.com/item?id=50254347)

The count supports the rarity: 155 responses in 13 days. The mechanism is harder to pin down. Cloudflare says every request over 681,280 tokens behaved this way and does not separate fresh prompts from those resumed out of cache, so the record can’t confirm the reader’s idea that ordinary multi-turn sessions got through safely.

The strongest case against Cloudflare comes from its own report. It knew about the mismatch before launch, a public bug report described the failure, and it had switched off its page-tagging defense. Its finding that no one exploited the bug rests on its own traffic analysis. The report counts 155 responses but says neither how many customers had their requests show up in them nor whether those customers were told, gaps [DataStudios](https://www.datastudios.org/post/cloudflare-workers-ai-glm-5-3-kv-cache-exposure-155-responses-sglang) also noted. One limit works in Cloudflare’s favor: whoever sent the long prompt had no control over whose leftovers came back.

## What comes next

Before any change to a disaggregated deployment ships, Cloudflare now reads the pool size from each running engine and checks it against the context length. The GLM-5.3 prefill pool now holds about 2.04 million tokens, roughly twice the window. The company also holds prefill’s source pages until transfers drain, caps the size of Mooncake transfer batches and has changed its load balancer so that a client disconnecting no longer aborts a request in the middle of a transfer.

SGLang’s new corruption detectors, a per-transfer checksum flag and a KV canary, don’t yet cover every model architecture, and Cloudflare’s report includes GLM-5.3 among the models it is working with SGLang to add. Over the longer term, the company says it will apply formal verification, which it has used for its DNS systems, to model and prove the safety of an RDMA-based protocol for moving KV caches. Cloudflare wrote that none of this rules out the next failure.

> They do not make misuse impossible, and this incident is a reminder that the next bug will have a shape we have not seen.
>
> — Cloudflare, in its Oct. 26 incident report, [blog.cloudflare.com](https://blog.cloudflare.com/kv-cache-exposure-remediation/)

SGLang’s post lists one more fix for its next release, v0.5.23: failed NIXL and MORI transfers will stop reporting their writes as drained. The project has not backported any of the fixes to older release branches.

The bluntest fix is still a draft. On Oct. 25 the GitHub user kflansburg opened [pull request #46435](https://github.com/sgl-project/sglang/pull/46435), which would let operators zero KV pages as they are freed, so that a future bookkeeping error reads zeros instead of a stranger’s data. It is off by default, supports only some memory pools and has no performance measurements yet. Its author estimated that the zeroing traffic “roughly equals KV write traffic, plus one extra kernel launch and one event per free call.” As of Tuesday it was still a draft, and its test runs were failing.

## Join the discussion

- [Cross-request KV cache exposure in the GLM-5.3 Workers AI model (155 responses)](https://news.ycombinator.com/item?id=50252159) — Hacker News · 145 points · 47 comments
- [Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang](https://www.reddit.com/r/CloudFlare/comments/1xgb36l/crossrequest_kv_cache_exposure_in_the_glm53/) — r/CloudFlare · 2 comments
- [Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang](https://www.reddit.com/r/SecOpsDaily/comments/1xgc4t3/crossrequest_kv_cache_exposure_in_the_glm53/) — r/SecOpsDaily

## See also

- [Cloudflare Fixes Cross-Tenant Data Exposure in Containers (InfoQ)](https://www.infoq.com/news/2026/10/cloudflare-cross-tenant-exposure/) — infoq.com · Industry reaction to Cloudflare's September storage-layer leak
- [Auditing Prompt Caching in Language Model APIs (ICML 2025)](https://proceedings.mlr.press/v267/gu25b.html) — proceedings.mlr.press · Stanford audit of cross-customer prompt-cache sharing at 17 providers
- [CacheProbe: Auditing Prompt Cache Isolation in Gateway APIs](https://arxiv.org/abs/2605.30613) — arxiv.org · Follow-up study testing cache isolation through the OpenRouter gateway
- [Cloudbleed](https://en.wikipedia.org/wiki/Cloudbleed) — en.wikipedia.org · Overview of Cloudflare's 2017 memory leak and the dispute over its scope
- [The Cloudflare (Cloudbleed) Proxy Service Vulnerability Explained (Rapid7)](https://www.rapid7.com/blog/post/2017/02/24/cloudflare-data-leakage-or-dare-i-saycloudbleed/) — rapid7.com · Contemporary security-firm explainer of the 2017 leak
- [OpenAI data leak and the Redis race condition (Sonatype)](https://www.sonatype.com/blog/openai-data-leak-and-redis-race-condition-vulnerability-that-remains-unfixed) — sonatype.com · Technical account of the redis-py race behind the 2023 ChatGPT leak