Skip to content
VECTOR WIREAI INTELLIGENCE
⌘K
UTC
Models832Deals30d260Policy30d80, 18 pendingSources1,170
Home
Thursday, September 17, 20261 min readIN THE DAILY← AI WIRE

Ollama Fixes MLX Memory Leak in Speculative Decoding

Ollama v0.34.2-rc2 patches an MLX runner memory leak during speculative decoding, cutting peak usage from over 90 GB to 30 GB at 98k-token context.

By Ed. U. Ahmad1 min readPublished
Single source · via github.comAdd to Preferred Sources
BY THE PRO DESKED. U. AHMAD
ALERTIndustry ProSINGLE SOURCE · via github.com

Ollama's v0.34.2-rc2 release candidate patches a memory leak in its MLX runner that caused unbounded memory growth during speculative decoding1. The bug stemmed from freed KV-cache buffers never being released: the pool-release check fired only on exact 256-token multiples, but speculative decoding emits multiple tokens per round, causing most rounds to skip the boundary. With qwen3.8:27b-mlx at a 98k-token context on a 128 GB machine, the fix reduces peak memory from over 90 GB to a flat 30 GB.

Was this useful?

The day’s vector, in your inbox.

Vector Wire Daily: one dispatch each publishing day.

More from Pro Wire
Industry ProSep 24KV cache becomes programmable infrastructure across the inference stackIndustry ProSep 16Ollama Ships v0.34.2 Release Candidate With llama.cpp UpdatesIndustry ProSep 15Ollama v0.34.1 Patches MLX Memory, Raises Token Repeat LimitIndustry ProSep 19Ollama v0.34.3 Adds Thinking Controls API, Nemotron H on Apple Silicon