FreeToken claims to serve GLM-5.2 753B on an RTX PRO 6000 workstation at 15 tok/s2. Liquid AI's QAD checkpoints recover roughly 97% of BF16 accuracy at 4-bit precision1. Practitioners are running Qwen3.8-27B at 262K context on single consumer cards6. Taken together, the evidence points to a rapid compression of the gap between cloud and local inference.
ANALYSIS The cost structure of AI inference has been defined by cloud GPU rental; if practitioners can serve 100B+ models on hardware they already own, the downstream effects ripple through API pricing, data-sovereignty debates, and the competitive moat of cloud-only model providers. The evidence is not theoretical — it comes from reproducible benchmarks posted by practitioners running real workloads on shipping hardware.
The big picture
FreeToken, a new edge-native MoE serving engine, claims to run GLM-5.2 753B on an RTX PRO 6000 workstation at 15 tok/s, DeepSeek-V4-Flash 284B on an RTX 5090 desktop at 22–25 tok/s, and Qwen3.6 35B on an 8GB RTX 4060 laptop at 39 tok/s. It achieves this through "bandwidth-adaptive CPU–GPU execution + semantic-aware caching across agent turns," claiming 3–4× faster decode and 6–30× faster prefill compared with Ollama. Critically, FreeToken uses official checkpoints "without extreme quantization".
At the 27B scale, practitioners are extracting striking throughput from single cards. One user reports Qwen3.8-27B NVFP4 with vision running on a single RTX 5090 at a conservative average of 120 tok/s, with a 451K-token global KV-cache in NVFP4 and 196K context per session4. A separate setup on the same GPU achieves 77.2 tok/s after a 1K prompt and 64.7 tok/s with 128K tokens already resident, fitting the full 262,144-token window alongside vision, FP8 KV, prefix caching, and tool calling. Even a budget pairing of an RTX 5060 Ti 16GB with an Intel N100 runs Qwen3.8-27B at 73K context for agentic coding, processing over 1M total tokens in a single project with only three prompts9.
On the quantization-quality front, Liquid AI released QAD (Quantization-Aware Distillation) 4-bit checkpoints for LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B, reporting that all four land at roughly 97% of their BF16 averages across reasoning, instruction-following, tool use, and agentic benchmarks. Separately, Atomic Dynamic GGUF quants of Qwen 3.8 27B on an RTX PRO 6000 show AD-Q4_K_M at 95.6% top-1 vs BF16 at 67 tok/s, while AD-Q6_K reaches 98.7% at 49 tok/s3. Detailed KLD analysis of Qwen3.8-27B quants finds the best 4-bit score at 0.00835 and the worst 8-bit score at 0.00071, confirming a large gap between the two precision tiers — but also that group-size 32 checkpoints outperformed group-size 128 variants that kept some modules in BF165.
Between the lines
FreeToken attacks the bandwidth wall with adaptive CPU-GPU scheduling, letting MoE models spill experts to system RAM without collapsing throughput. NVFP4 quantization paired with vLLM's KV-cache compression lets a single RTX 5090 hold a 451K-token global KV-cache. Liquid AI's QAD approach distills a high-precision teacher model into a quantized student model, recovering accuracy that post-training quantization leaves on the table. ANALYSIS These are three distinct technical strategies — bandwidth-adaptive serving, format-level cache compression, and quantization-aware training — converging on the same outcome: making frontier-scale inference viable on consumer hardware.
A hybrid-hardware experiment underscores how far creative placement can go: a Strix Halo APU paired with an RTX 3090 Ti via eGPU runs Qwen3.8-27B at 262K context, reaching 153.32 tok/s at 32K context and 87.74 tok/s at 200K context7. The same setup scored 159/164 on HumanEval in 29.7 minutes, beating a dual-3090 vLLM TP2 server that scored 157/164 in 42.4 minutes. ◆ That a consumer laptop APU plus an older desktop GPU outperforms a dual-GPU server on a standard coding benchmark signals that software-level optimizations — chat template tuning, KV-cache format selection, layer placement — now rival raw hardware upgrades in practical impact.
At the extreme edge, Ling-3.0-tiny (7.9B total, approximately 1.4B active MoE) runs on an NVIDIA Orin Nano Super 8GB at its full 128K native context with an IQ4_NL quant, decoding at 33 tok/s and fitting in 7.4 GiB of unified RAM8. ◆ A board running a 128K-context model at readable decode speed illustrates that "edge" inference now means always-on agent hosting, not just toy demos.
Even a 2.45-trillion-parameter model is no longer cloud-exclusive: Qwen3.8-2.4T-A95B ran locally on an RTX 5090 plus RTX 5060 Ti with 128 GB DDR5 at 0.80 tok/s using a Q1_0 quantization, with MTP speculative decoding achieving 90.48% acceptance10. ◆ The speed is not interactive, but the fact that a consumer desktop can load and generate from a 2.4T-parameter model at all marks a threshold.
What's next
Liquid AI's QAD checkpoints are already on Hugging Face, and FreeToken's claimed performance numbers will face community replication pressure in the weeks ahead. ◆ The Qwen3.8-27B ecosystem is rapidly becoming the de facto local-inference benchmark target. If quantization-aware training and bandwidth-adaptive serving continue to mature at this pace, the economic case for cloud-only inference narrows further — not because local hardware matches cloud throughput, but because it reaches the interactive-speed threshold where most agentic and coding workflows actually operate.