A PyTorch blog post published November 10 details serving Meta's Muse Spark Open, an 816B-parameter mixture-of-experts model with approximately 42B active parameters per token, on a single TPU v7 host of four chips using SGLang on the TorchTPU backend1. A megakernel running all 62 decoder layers as one Pallas program halves time per output token for one or two concurrent requests. On SemiAnalysis's 1M-context AgentX agentic-coding benchmark, the system reached 433 output tokens per second per request at 6,101 tokens per second per chip, scaling to 25,906 tokens per second per chip at 95 tokens per second per request.
SGLang Megakernel Serves 816B Muse Spark Open on Four TPU v7 Chips
A PyTorch blog post details serving Meta's 816B-parameter Muse Spark Open on a single four-chip TPU v7 host using SGLang's megakernel, halving decode time per token.
Was this useful?
The day’s vector, in your inbox.
Vector Wire Daily: one dispatch each publishing day.