Google Cloud has open-sourced TPU Sync, a native C++ library that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage, and the data center network1. The release targets two workloads where accelerators frequently stall on data movement: long-context and disaggregated LLM serving, and reinforcement learning loops that push updated weights from trainers to rollout samplers.
For prefill/decode disaggregation, Google says median KV transfer latency for Qwen3 235B fell from several seconds to about 230 ms, with average time-to-first-token reductions of 60% for that model and 90% for a 480B model at low concurrency. For prefix caching, the company reports up to 90% lower average TTFT on 64K-context workloads.
The announcement arrived alongside Google's disclosure of the TPU 8tx and TPU 8i Max systems.
ANALYSIS Pairing a data-movement library with new TPU hardware positions the two releases as complementary: faster silicon matters less if the serving stack is bottlenecked on cache and weight transfers between memory tiers.
TPU Sync ships Python wheels for JAX and PyTorch, exposing KV cache manager, KV store, and weight synchronizer APIs. On the serving-framework side, it acts as the KV connector for vLLM-TorchTPU through vLLM's KVConnector interface, serves as the disaggregation and hierarchical-cache backend for SGLang-TorchTPU, and pairs with llm-d 1.0 on GKE for prefix-cache-aware routing. The RL weight-sync module for Pathways and multi-controller JAX is described as experimental.
ANALYSIS The breadth of framework hooks — spanning vLLM, SGLang, and Google's own Pathways — suggests Google is positioning TPU Sync as a cross-stack primitive rather than a single-framework feature, lowering the barrier for teams already committed to open-source serving stacks to adopt TPU-based infrastructure.