Skip to content
VECTOR WIREAI INTELLIGENCE
UTC
Home

vLLM's inference pipeline splits to four stages, but only three are production-ready

vLLM's inference stack now decomposes into as many as four independently scaled stages: encoder, prefill, decode, and — experimentally — attention versus FFN. The project is no longer iterating on a two-stage prefill/decode split; it is building toward a pipeline architecture in which every computationally distinct phase of transformer inference can be deployed, routed, and scaled on its own hardware. The three-stage encoder/prefill/decode split is shipping in main. The fourth stage, attention-FFN disaggregation, remains an external plugin whose authors acknowledge it needs large-scale validation. The gap between the two matters.

The argument is not that disaggregation works in principle. Vector Wire covered that transition in October, when disaggregated prefill and decode moved from design pattern to production default[4]. The new claim is narrower: multimodal workloads are forcing a third stage into the pipeline, and Mixture-of-Experts architectures may force a fourth. vLLM is building support for both before most operators have finished deploying the first split, but it is doing so across three competing orchestration surfaces, not one unified control plane, and the fourth stage remains experimental. The direction is clear; the convergence is not.

Encoder / Prefill / Decode disaggregation, which vLLM calls EPD, separates the media-encoding stage that processes images, video frames, and audio into embeddings from the language-model prefill and decode stages1. The demand on encoding varies with media size, modality, and request mix, and can change independently of prefill and decode demand. DeepSeek's own inference deployment describes an E/P/D architecture with a separately deployed vision encoder, and vLLM main now supports metadata-only EPD for DeepSeek V4.1 Flash.

The engineering beneath EPD spans five areas: metadata-only requests that let downstream instances consume remote embeddings without repeating media preprocessing; a dedicated encoder execution path through a separate Model Runner V2 with encoder-only CUDA Graph support; remote embedding transfer through NIXL and Mooncake integrations; cross-encoder output reuse through Mooncake Store; and lower orchestration overhead in the Python EPD proxy and in external orchestrators including vLLM Router, SMG, and llm-d.

The latency and throughput numbers, drawn from vLLM's own benchmarks, are substantial. At concurrency 64–128, EPD showed P90 time-to-first-token 44–47% lower than an eight-GPU prefill/decode baseline. Input-plus-output throughput per GPU rose 6.6–9.5% at the same concurrency levels. In a shared-reuse experiment, P90 TTFT fell from 1.656 to 0.891 seconds, and P50 TTFT fell from 1.209 to 0.819 seconds. Across 128 measured requests containing 512 images, shared reuse reduced repeated image encodings from 512 to zero. A 30-frame video evaluation on separate 4-GPU GB200 nodes showed end-to-end throughput improvements of approximately 94–115% over three runs. On audio, metadata-only forwarding reduced mean latency from 980 to 853 ms and increased throughput from 62.21 to 73.13 requests per second, with accuracy on the MMAU test-mini benchmark changing from 75.90% to 75.65%.

Weight-loading efficiency also improved sharply. On a pruned Kimi K3 checkpoint, the encoder read 2 of 96 shards and weight loading dropped from 683 seconds to 0.31 seconds.

The fourth stage is more speculative. The vLLM AFD Plugin, published on July 23, separates attention and FFN computation inside MoE models into independently deployed services2. Attention is stateful and coupled to request scheduling and the KV cache, while the FFN path is dominated by routed expert computation and all-to-all communication; when both share the same worker topology, the system must make one set of scaling choices for workloads with very different requirements. The plugin currently supports NVIDIA GPUs and Ascend NPUs, synchronous and asynchronous connectors, and DeepSeek V2/V3-family model wrappers. vLLM itself labels the project experimental and notes it needs more large-scale testing.

Early AFD numbers on Ascend 910C hardware give a sense of the tradeoff space. At 32K fixed input length, an EP64 baseline achieved 168.2 tokens per second per die; a 48-attention, 16-FFN AFD configuration achieved 151.4; and a 64-attention, 16-FFN configuration achieved 183.3. At 16K input, the same three configurations achieved 232.6, 220.3, and 258.9 tokens per second per die respectively. The 64A16F layout exceeded the EP64 baseline at both input lengths, but the 48A16F layout fell below it, illustrating that AFD introduces a configuration sensitivity: the wrong attention-to-FFN ratio does not merely leave performance on the table but adds a network hop that makes the system slower than the monolithic baseline.

Khawaja Shams, presenting at QCon AI on October 27, offered a distributed-systems perspective on the same trajectory: KV cache memory cliffs, tiered storage, tail latency, and disaggregated prefill/decode as fleet-level optimizations3. His framing reinforces the point that disaggregation is a fleet-economics problem, not merely a model-architecture problem, and that the KV cache, which Vector Wire analyzed as programmable infrastructure in October[5], becomes harder to manage with every new pipeline stage.

The models that vLLM's disaggregation work targets are themselves moving targets. DeepSeek V4.1 Flash, which now has metadata-only EPD support on vLLM main, scored 46.34 on independently verified FrontierMath Tier 4, up from DeepSeek V4 Flash's 24.39 on the same benchmark. It scored 71.23 on FrontierMath Tiers 1–3 v2, up from DeepSeek V4 Flash's 57.54. Both independently verified as of November 22, 2026. MiMo V2 Omni, which now declares the embedding fields needed for image EPD, scored 1227.90 on Arena Vision, independently verified as of November 22, 2026. Qwen3-Omni, whose tested checkpoint produces visual outputs of width 8192 and audio outputs of width 2048, scored 61.30 on OCRBench v2, independently verified as of November 22, 2026. The research pipeline feeding these models is active: ScaleVLA scaled a vision-language-action model to 552 billion parameters on DeepSeek-V4.1-Flash, and XM-Sparse explored sparse-attention serving for DeepSeek-V3.2. Each capability jump increases per-request encoder compute and widens the gap between encoding and decode resource profiles. As these models grow more capable across modalities, the encoding stage absorbs more compute per request, strengthening the case for isolating it. But the EPD benchmarks to date cover a narrow hardware and model set, and the strongest gains appear at high media concurrency. Whether EPD yields comparable improvements on commodity GPU fleets with lower media density remains undemonstrated. The architectural logic is sound; the generality of the performance claims is not yet established.

Vector Wire reported in early November that a cache-metadata CVE exposed the trust gap that disaggregation keeps widening[1]. EPD extends that surface: encoder outputs now travel between instances over NIXL or Mooncake, and metadata-only forwarding means downstream nodes trust that the embeddings they receive are faithful representations of the original media. Every new stage in the pipeline is a new trust boundary. A compromised or buggy encoder can inject adversarial embeddings that prefill and decode nodes will process without verification.

The orchestration layer is fragmenting in parallel. llm-d provides a Kubernetes-native deployment path for EPD. SMG supports E/P/D deployments over gRPC, including metadata-only forwarding and Mooncake and NIXL backends. vLLM Router supports HTTP E+PD and E/P/D deployments with multi-image distribution across encoders. Three orchestration surfaces for one disaggregation pattern means the community has not converged on a control plane. Operators adopting EPD are making an orchestration bet alongside a serving-architecture bet. That fragmentation undercuts the framing of EPD as a unified architectural absorption: the compute-plane disaggregation is real, but the management plane remains balkanized.

ANALYSIS Whether the four-stage pipeline becomes the production norm depends on questions no benchmark has yet answered. The AFD plugin's large-scale testing gap is acknowledged by its authors. The interaction between encoder disaggregation and attention-FFN disaggregation in a single deployment has not been described. The security model for multi-stage embedding transfer, which Vector Wire flagged in November[1], remains unspecified. And the orchestration layer must converge before operators can treat disaggregation as infrastructure rather than bespoke integration.

The three-stage split is the defensible bet. EPD is in vLLM main, benchmarked on shipping models, and supported by three orchestration paths. The fourth stage remains a research signal, not a production commitment: its benchmarks show configuration sensitivity, its testing is incomplete, and its interaction with EPD is unexplored. The llm-d team's Kubernetes-native EPD path is the nearest convergence marker. If production operators adopt it for multimodal workloads at scale, the three-stage pipeline becomes infrastructure. The fourth stage will follow only if automated ratio tuning eliminates the configuration sensitivity that makes it an expert exercise.

Was this useful?

The day’s vector, in your inbox.

Vector Wire Daily: one dispatch each publishing day.