vLLM consolidates as default LLM serving layer across chips, clouds, and consumer GPUs

ANALYSIS In a single week, vLLM surfaced as the serving runtime in contexts ranging from a single RTX 5090 running a 27-billion-parameter model to an eight-GPU Blackwell Ultra cluster hosting a 2.4-trillion-parameter open-weights model, and as the integration target for a chipmaker that does not build GPUs at all. The convergence is not coincidental: vLLM is becoming the assumed runtime layer for LLM inference across scales, hardware vendors, and deployment models.

Why it matters

The choice of inference runtime determines what hardware can be used, how many users a single GPU can serve, and how quickly new models reach production. ◆ When one runtime accumulates integrations from cloud providers, chip startups, and solo practitioners simultaneously, it begins to function less like a project and more like infrastructure, reshaping the competitive dynamics around it.

The big picture

At the enterprise end, AWS published a deployment guide for Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod using vLLM on a ml.p6-b300 instance with eight NVIDIA B300 Blackwell Ultra GPUs⁴. The guide covers NVFP4 quantization, built-in reasoning, tool calling, and native Multi-Token Prediction speculative decoding. AMD, meanwhile, detailed optimizations to vLLM-ATOM tuned for "high-interactivity inference" on its MI355 accelerators, framing responsiveness as "a first-class product requirement" for coding agents and multi-step tool-calling loops³.

Tenstorrent, whose accelerators "do not look much like a GPU," shipped a vLLM plugin that registers its hardware through vLLM's out-of-tree platform mechanism, preserving the same OpenAI-compatible API⁵. The plugin supports model families including Llama 3.1 through 3.3, Qwen 2.5 through Qwen 3.6, Mistral 3, and Gemma 4. The Tenstorrent team noted that vLLM's plugin interfaces "turned out to be general enough that we could express those differences — a phase-constrained scheduler, a different data-parallel topology, a sampling path that partly lives on device — entirely outside vLLM core".

On the orchestration layer above the engine, Anyscale and NVIDIA engineers introduced token-load-aware routing for Ray Serve LLM, designed to distribute requests across a fleet of vLLM replicas². The post argues that optimizing solely for KV cache reuse is a "misconception" and that KVAwareRouter, which considers both KV cache overlap and token load, achieves better time-to-first-token, time-per-output-token, and throughput. Red Hat published a guide on FastMTP heads for vLLM speculative decoding, building on the multi-token prediction training objective used by DeepSeek and Qwen model families¹.

At the consumer scale, practitioners are converging on vLLM for workloads that once belonged to llama.cpp. One user reported running Qwen3.8-27B NVFP4 on a single RTX 5090 with vision, FP8 KV cache, and the full 262,144-token context window, achieving 77.2 tok/s after a 1K prompt and 64.7 tok/s with 128K tokens already resident¹⁰. Another user on the same GPU reported 120 tok/s average with a 451K global KV-cache using NVFP4 quantization for both model and cache⁹. A third ran a head-to-head comparison of llama.cpp, vLLM, and NInfer on Qwen3.8-27B across six production evaluation tiers and concluded that NInfer NVFP4 replaced llama.cpp as their production engine, though vLLM handled 192K context without issue⁷. A fourth user, running Qwen3.8-Flash-Next on asymmetric GPUs, asked directly: "For someone who actually wants to use 100K–260K contexts for coding/agent workloads, is vLLM basically the only practical answer right now?"⁸. One developer built a serverless LoRA hosting platform, Lorivo, around vLLM's batching, memory management, and fused LoRA kernels, sharing a single GPU server across many adapters⁶.

ANALYSIS The Tenstorrent plugin is perhaps the most telling data point. A non-GPU accelerator vendor chose to build into vLLM rather than ship a standalone serving stack, treating vLLM's API surface as the contract that application developers already expect. AMD's vLLM-ATOM work and AWS's SageMaker HyperPod guide make the same implicit bet: the serving API is settled, and differentiation happens below it in hardware and quantization.

The consumer-GPU threads reveal a different pressure. Users are not choosing vLLM for its ease of setup; one described it as "not user-friendly at first". They are choosing it because llama.cpp's single-request parallelism and its handling of long-context sparse attention fall short for production-style concurrent serving at extended token contexts. llama.cpp is limited to parallel=1 for the workloads tested, and its sparse-attention implementation "still appears to do top-k selection followed by effectively dense/full-KV work" at scale.

What's next

Anyscale's Ray Serve LLM team plans to extend KVAwareRouter support to prefill-decode disaggregated deployments, data-parallel deployments, and multimodal workloads. Tenstorrent's plugin already registers architectures for Gemma 4 and Qwen 3.6, models still in early adoption. AWS's SageMaker HyperPod guide is the second in a series on trillion-parameter open-weights deployments, with the first covering Kimi K3. Each extension treats vLLM as the fixed layer and everything else as the variable.