Google Launches TPU Sync : An Open Source KV Cache Transfer for TPUs

Google Launches TPU Sync : An Open Source KV Cache Transfer for TPUs
3 min read

TPU Sync, an open-source C++ library, enables moving KV caches and model weights on Cloud TPUs, Google’s processors specialized for AI computation. According to Google, KV cache transfer latency is reduced by over 90% on mixture-of-experts (MoE) models with 235 to 480 billion parameters.

The library treats TPU HBM memory, host DRAM, shared storage, and the datacenter network as a single memory hierarchy. It addresses, according to the article, a growing bottleneck: moving model state, rather than computing it. Its core is written in C++, with Python packages for PyTorch and JAX, and its repository is published under the Apache-2.0 license.

TPU Sync serves two purposes: KV cache during inference, and weight broadcasting between trainer and sampler in reinforcement learning (RL). It works on Cloud TPU v7x (Ironwood) and v6e, according to the repository’s README. Benchmarks were conducted on TPU v7x in a 2x2x1 configuration, with PCIe Gen5 and 200 Gbps network cards.

A direct transfer between memories

The library reads JAX PJRT buffers or PyTorch tensors in place, via direct memory access (DMA), without Python serialization or intermediate buffers. Transfers can occur during computation, and built-in pacing prevents large copies from delaying the launch of compute kernels. Google reports throughput reaching up to 223 GB/s per Ironwood host, compared to a 256 GB/s ceiling for PCIe Gen5. On the datacenter network (DCN), throughput reaches about 192 Gbps with one network card, and about 350 Gbps with two.

In a so-called “disaggregated” prefill/decode (P/D) service, the prefill node offloads KV blocks to host RAM. A controller then routes them to the decode node, where they arrive in HBM memory while decoding continues.

Results from internal tests

Google conducted its internal tests in July 2026, with tpu-inference v0.24.0, JAX 0.10.2 and libtpu 0.0.42.1, noting that actual results will vary. For the P/D service, the median KV transfer latency for Qwen3 235B dropped from several seconds to about 230 ms. The average TTFT (time to first token) decreased by 60% for the 235B and 90% for the 480B under low concurrency, while token output throughput increased by 30% to 70% for the 235B during traffic peaks.

For prefix caching, the effective cache capacity increased by an order of magnitude when MoE weights occupy most of the HBM. The average TTFT decreased by up to 90% on 64K-context workloads, throughput improved by over 50%, and P99 inter-token latency decreased by 30%.

Integrations and limitations

With PyTorch, the library works via TorchTPU, and natively with JAX. In vLLM-TorchTPU, it serves as the TPU KV connector behind vLLM’s KVConnector interface. In SGLang-TorchTPU, it acts as the P/D transfer backend and hierarchical cache I/O. On GKE, llm-d 1.0 uses it for affinity-based prefix cache routing. Inferact and RadixArk co-designed the vLLM and SGLang integrations, and Google states that Meta and Wayfair are evaluating the architecture.

Google does not publish figures for reinforcement learning, and the README indicates the library is not yet recommended for general use. The RL functions and multi-tier storage remain experimental, and all figures are self-reported. The article also compares TPU Sync to NVIDIA Dynamo KVBM + NIXL, LMCache and Mooncake, noting that their bandwidth figures do not measure the same paths: TPU Sync’s corresponds to host PCIe DMA, while Mooncake’s corresponds to a network transfer over RoCE.

S’abonner
Notification pour
guest

0 Commentaires
Le plus ancien
Le plus récent Le plus populaire
0
Nous aimerions avoir votre avis, veuillez laisser un commentaire.x