Cloudflare fixes container flaw that exposed other customers' residual disk data

Cross-tenant leak traced to storage allocator, not VM boundary; no evidence of exploitation

By LineZotpaper
Published
Read Time3 min
Cloudflare has patched a cross-tenant data exposure vulnerability in its Containers platform after a researcher showed that paying customers could recover residual disk blocks left behind by other customers' workloads on the same host. The company says it found no evidence the flaw was exploited maliciously.

Cloudflare has disclosed and fixed a cross-tenant data exposure vulnerability in Containers, the same platform that underpins its Sandboxes product. A customer with a Workers Paid account could recover leftover disk blocks belonging to other customers' containers on the same host, according to a report by security researcher Oren Yomtov of Accomplish, filed through Cloudflare's bug bounty program on September 4.

The root cause was not a break in the virtual machine boundary but a flaw in the storage layer beneath it. Each Cloudflare container runs inside a Firecracker microVM with a writable root disk provisioned using Linux device mapper thin provisioning. The affected storage pools used a 64 KiB thin-block size and were configured with skip_block_zeroing, which tells the allocator not to clear newly allocated blocks before they are used.

That combination created the gap. When a container wrote a single 4 KiB block into an unmapped region, dm-thin allocated a 64 KiB physical block from a pool shared across customer accounts. The write only overwrote its own portion, leaving up to 60 KiB that could still contain whatever the previous container had written. A raw read of the disk could then return data the new container never wrote.

The researchers used ext4 directory block checksums to distinguish their own blocks from others. Across six production placements, all 5,614 testable directory blocks came from other customers, with 2,700 distinct foreign directory inodes identified. Residual material appeared on 18 of 24 placements and 20 of 22 underlying nodes across four continents. Recovered block types included directory structures, database pages, and structurally complete SQLite databases.

Cloudflare stressed the limits of the exposure. An attacker could not choose a victim, workload, or host, could not read an actively attached disk, and had no guarantee residual data would be present. The researchers did not demonstrate modification of another customer's data or any impact on availability.

Security practitioners discussing the disclosure focused on where the isolation broke. Peter Ward, a senior cloud security engineer at Visa, called it a recurring pattern, saying the failure was tenant isolation broken at the storage layer rather than an application bug. He argued that ephemeral disk and shared hosts warrant explicit zero-on-allocate or wipe-on-release checks in the control plane, and suggested teams ask any container or serverless platform whether it guarantees that a block read returns only bytes the workload itself wrote.

§

Analysis

Why This Matters

  • Multi-tenant cloud platforms promise hard isolation between customers; this incident shows that promise can break in unexpected layers, in this case the block allocator rather than the hypervisor.
  • Even limited data leakage between tenants on a shared host is a serious trust issue for serverless and container platforms, because customers cannot choose or audit their neighbours.
  • Cloudflare's Containers and Sandboxes are used by paying developers; the fix matters to anyone running workloads on them, and the pattern is relevant to any provider using thin-provisioned storage without zeroing blocks.

Background

Container and serverless platforms isolate customer workloads inside lightweight virtual machines, typically with a hypervisor boundary that is considered the security perimeter. Below that boundary sits host storage, often shared and thin-provisioned so that physical space is allocated on demand. If the allocator hands out blocks that have not been wiped, a new tenant's raw disk reads can surface remnants of an earlier tenant's data. This class of issue is known in storage systems, but proving it across a major commercial platform is rare.

Key Perspectives

Cloudflare: The company acknowledged the vulnerability, credited the researcher, and says it has remediated the issue. It emphasises the practical limits: no victim selection, no access to live disks, and no demonstrated write or availability impact.

Security researchers: The discovering team demonstrated concrete leakages, including complete SQLite databases, across many placements and nodes on multiple continents, arguing the issue was systemic to the configuration rather than a single bad host.

Security practitioners: Industry commenters, such as Peter Ward, see this as a storage-layer isolation failure that points to a broader lesson: control planes should explicitly enforce zero-on-allocate or wipe-on-release for ephemeral disks, and customers should be able to ask platforms whether this guarantee exists.

What to Watch

  • Whether Cloudflare discloses steps taken to prevent recurrence, such as removing skip_block_zeroing or adding wipe-on-release checks, and whether other cloud providers audit their thin-provisioning configurations for similar gaps.
  • Any further details from Cloudflare's remediation timeline or additional forensic findings about the residual data recovered during testing.
  • Follow-up research that applies similar probing techniques to other serverless or container platforms, which could surface comparable exposures.

Sources

Zotpaper

Written by software from the reporting listed above, scored by an automated standards desk, and published without a person reading it first. If something here is wrong, tell the editor and it will be put right.