Qwen3.8-2.4T-A95B, a model with 512 routed experts and 10 active per token, was run locally on a rig built around an NVIDIA RTX 5090 (32 GB VRAM) and an NVIDIA RTX 5060 Ti (16 GB VRAM), paired with 128 GB of DDR5 RAM and 350 GB of swap1. The experimenter used the Unsloth Q1_0 GGUF quantization, which compresses the full model to roughly 397 GiB. The system ran on Arch Linux using Unsloth build 10360 of llama.cpp. The final llama-server command used MODEL=/home/marcin/Pobrane/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf and listened on host 127.0.0.1 port 8200. In a controlled 32-token generation test, the setup achieved a generation speed of approximately 0.8033 tokens per second with MTP speculative decoding enabled, and 0.7750 tokens per second without it. The MTP configuration — n_max of 3, p_min of 0.5, with block 92 expert tensors forced to CPU to avoid exhausting VRAM — yielded a 90.48% draft-token acceptance rate on 21 draft tokens.
ANALYSIS Those numbers are orders of magnitude below what any API provider would consider usable. But the point, as the LocalLLaMA community makes plain, is not production throughput — it is proving that the model runs at all on hardware costing a fraction of a datacenter node.
Separately, Tim Dettmers, creator of the bitsandbytes quantization library, is teasing a new quantization method that reportedly ran GLM 5.3 on a single DGX Spark at 7 tokens per second2. Another demonstration showed DS4 Pro on a single B300 with 288 GB VRAM. One Reddit commenter cautioned readers not to "get too hyped" given "an endless amount of quantization schemes with big promises that never really became a thing," while acknowledging that "Tim Dettmers is a pretty well known researcher".
Meanwhile, the community is already looking ahead. A thread titled "How do you plan to run Qwen3.8-2.4T-A95B locally?" catalogues the escalating ambitions of hobbyist runners who have previously wrestled with LLaMA-70B, Mistral Large, and DeepSeek V2/V3, and are now eyeing models like Kimi K33.
ANALYSIS Three dynamics stand out across these strands. First, quantization aggression is intensifying. The experimenter used Q1_0 — a single-bit quantization level — to compress a 2.45-trillion-parameter model into roughly 397 GiB. That the model produces output at that compression level points to meaningful progress in quantization tooling and format design, not just brute compression.
Second, speculative decoding is becoming a standard lever even in hobbyist setups. The Qwen3.8 experimenter methodically tuned MTP parameters, finding that n_max of 2 was too conservative, n_max of 3 was best, and n_max of 4 was much slower. Similarly, p_min of 0.0 produced too many bad speculative tokens, p_min of 0.5 was best, and p_min of 0.7 was too aggressive — reaching 100% draft acceptance but generating too few useful speculative tokens and ending up slower than p_min of 0.5. ◆ This kind of systematic parameter search on consumer hardware mirrors the optimization work that cloud inference providers perform at scale.
Third, VRAM management has become a first-class engineering problem. The RTX 5090 was nearly saturated at roughly 29.6 GB used out of 32 GB, and the RTX 5060 Ti at roughly 12.3 GB used out of 16 GB. Block 92 expert tensors had to remain on CPU because offloading them to GPU would have caused MTP to exceed available VRAM. ◆ The practitioner is effectively hand-scheduling tensor placement across a heterogeneous compute graph — a task that, in datacenter settings, is handled by orchestration software.
The final MTP configuration gave roughly +3.64% generation throughput and -2.74% total wall time compared to running without MTP. With MTP enabled, decode time was 39.84 seconds and total wall time was 59.14 seconds; without MTP, decode time was 41.29 seconds and total wall time was 60.81 seconds.
Broader local-inference experiments
At the other end of the parameter spectrum, a project running a 28.9M parameter LLM on an $8 microcontroller drew substantial attention on Hacker News, accumulating 279 points and 70 comments6. Separately, a project demonstrated a tiny LLM running at 21,000 tokens per second on a $250 FPGA5. Muse Glimmer 30B was shown running locally in-browser with custom WebGPU kernels at approximately 25 tokens per second on an M4 Max, described as the same speed as llama.cpp4.
ANALYSIS The range of hardware targets — from an $8 microcontroller to a $250 FPGA to Apple silicon to a multi-GPU desktop — illustrates that local inference is being pursued across the entire compute spectrum, not just at the high end.
The "How do you plan to run Qwen3.8 locally?" thread already references DeepSeek-V4-Pro, GLM-5.2, MiMo V2.5 Pro, and Kimi K3 as upcoming targets. ◆ Each new model release now triggers a parallel race — not just to evaluate it on benchmarks, but to see who can run it on the least hardware. That race is producing real engineering insights about quantization, memory scheduling, and speculative decoding that feed back into the tools the broader ecosystem depends on.