Google has released TPU Sync, an open-source C++ library that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage, and data centre networks. The company says the library cut median KV transfer latency for Qwen3 235B from several seconds to around 230 milliseconds, and reduced average time-to-first-token by 60% for that model and 90% for a 480B model at low concurrency. For prefix caching, it reports up to 90% lower average time-to-first-token on 64K-context workloads.
Available under the Apache 2.0 licence, TPU Sync provides Python wheels for JAX and PyTorch, implementing KV cache manager, KV store, and weight synchronizer APIs. It integrates with vLLM-TorchTPU and SGLang-TorchTPU, and pairs with llm-d 1.0 on Google Kubernetes Engine for prefix-cache-aware routing. The library is validated on TPU v7x and v6e, and includes an experimental module for reinforcement learning weight synchronization in Pathways and multi-controller JAX.
Developer Anand Butani said: "TPU sync cutting kv transfer latency 90% matters more to me than the chip names, long-context serving mostly dies on moving cache between nodes." Another developer, writing under the pseudonym @lgoodbookd, ran rough numbers suggesting bandwidth alone does not solve the problem: while the new TPU 8i Max doubles HBM bandwidth to about 17.2 TB/s, a single 64K context session on a 235B MoE model still consumes roughly 12.6 GB of KV cache. The developer said open-sourcing TPU Sync’s host-DRAM KV tiering may matter as much as the new package. The GitHub repository warns that APIs are changing rapidly and recommends pinning to the last-known-good commit; only development builds are on PyPI. The PyTorch wheel requires a matching torch_tpu build not on a public index.
Analysis
Why This Matters
- Addresses a key bottleneck in long-context LLM serving: moving cache between nodes.
- Could lower infrastructure costs by reducing time-to-first-token and enabling more efficient disaggregated serving.
- Open-sourcing makes the technology accessible beyond Google Cloud, though it remains TPU-specific.
Background
TPUs (Tensor Processing Units) are Google's custom accelerators for AI workloads. In large language model serving, disaggregating prefill and decode phases across multiple nodes reduces latency, but data transfer between accelerators and memory has been a major bottleneck. Google previously developed internal tools; TPU Sync formalises and open-sources this layer. NVIDIA offers similar functionality through Dynamo and NIXL, while other open-source projects like LMCache and Mooncake target the same problem on GPU systems.
Key Perspectives
- Anand Butani (developer): Values the 90% KV transfer latency cut over the new chip names, noting long-context serving is limited by cache movement.
- @lgoodbookd (developer pseudonym): Argues that even the doubled HBM bandwidth of the TPU 8i Max is insufficient for large KV caches, making host-DRAM tiering a critical addition.
What to Watch
- When a stable release of TPU Sync is cut and how quickly it is adopted by open-source serving frameworks.
- Performance comparisons with NVIDIA Dynamo and GPU-based solutions at scale.
- The evolution of the experimental RL weight-sync module and its impact on training efficiency.
Checked
12 claims in this story checked against 1 source: 12 supported.
Show each claim
- SupportedGoogle open-sourced TPU Sync to accelerate LLM serving on TPUs.InfoQ: “Google Cloud has open-sourced TPU Sync ... the project targets two workloads where accelerators often wait on data: long-context and disaggregated LLM serving”
- SupportedThe library targets KV-cache and weight-transfer bottlenecks in long-context and reinforcement-learning workloads.InfoQ: “the project targets two workloads where accelerators often wait on data: long-context and disaggregated LLM serving, and reinforcement learning loops that push updated weights from trainers to rollout samplers.”
- SupportedTPU Sync is a C++ library that moves KV-cache blocks and model weights between TPU HBM, host DRAM, shared storage and the data-centre network.InfoQ: “a native C++ library that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage, and the data center network.”
- SupportedFor Qwen3 235B, Google says median KV-transfer latency fell from several seconds to about 230 milliseconds, with average TTFT reduced by 60%; for a 480B model, average TTFT fell by 90% at low concurrency.InfoQ: “Google says median KV transfer latency for Qwen3 235B fell from several seconds to about 230 ms, with average TTFT reductions of 60% for that model and 90% for a 480B model at low concurrency.”
- SupportedFor prefix caching, the library reports up to 90% lower average TTFT on 64K-context workloads.InfoQ: “For prefix caching, it reports up to 90% lower average TTFT on 64K-context workloads.”
- SupportedTPU Sync is available under the Apache 2.0 licence and provides Python wheels for JAX and PyTorch, with KV-cache manager, KV-store and weight-synchronizer APIs.InfoQ: “Developers get Python wheels for JAX and PyTorch, with KV cache manager, KV store, and weight synchronizer APIs.”
- SupportedTPU Sync works with vLLM-TorchTPU and SGLang-TorchTPU, and pairs with llm-d 1.0 on GKE for prefix-cache-aware routing.InfoQ: “TPU Sync acts as the KV connector for vLLM-TorchTPU through vLLM's KVConnector interface, as the disaggregation and hierarchical-cache backend for SGLang-TorchTPU, and pairs with llm-d 1.0 on GKE for prefix-cache-aware routing.”
- SupportedThe library is validated on TPU v7x and v6e, and its RL weight-sync module for Pathways and multi-controller JAX is experimental.InfoQ: “The RL weight-sync module for Pathways and multi-controller JAX is experimental.”
- SupportedDeveloper Anand Butani said: “TPU sync cutting kv transfer latency 90% matters more to me than the chip names, long-context serving mostly dies on moving cache between nodes.”InfoQ: “developer Anand Butani sharing: TPU sync cutting kv transfer latency 90% matters more to me than the chip names, long-context serving mostly dies on moving cache between nodes.”
- SupportedDeveloper @lgoodbookd said TPU 8i Max doubles HBM bandwidth to about 17.2 TB/s, while one 64K session on a 235B MoE uses roughly 12.6 GB of KV cache.InfoQ: “8i Max doubles HBM bandwidth to ~17.2TB/s, yet one 64K session on a 235B MoE still eats ~12.6GB of KV cache.”
- Supported@lgoodbookd said open-sourcing TPU Sync’s host-DRAM KV tiering may matter as much as the new package.InfoQ: “That's why open-sourcing TPU Sync (host-DRAM KV tiering, ~223GB/s per v7x host) may matter as much as the new package.”
- SupportedThe repository warns that APIs are changing rapidly and recommends pinning to the last-known-good commit; only development builds are on PyPI, and the PyTorch wheel requires a matching torch_tpu build that is not on a public index.InfoQ: “the repository ... warns that APIs are changing rapidly and that only development builds are on PyPI, and recommends pinning to its last-known-good commit. ... the PyTorch wheel requires a matching torch_tpu build that is not on a public index.”
