
The VectorIndustry ProUPDATED1mo ago
Disaggregated Inference Moves from Design Pattern to Production Default
Huawei is now working on tier-aware KV-cache routing for disaggregated inference, with Bong-hwa Park from Huawei describing a system called LARDD that "tracks where every block lives and sends each request to its cache," managing spillover…
SINGLE SOURCE