ANALYSIS Token costs for hosted frontier models depend not just on per-token rates but on the scaffolding that wraps each call, and a parallel wave of narrow fine-tuning on sub-billion-parameter models is compressing the cost floor from the other direction.
A June benchmark compared 12 configurations across two models on the same 12 Python tasks and reported tokens per solved task ranging from roughly 3,500 for Aider in architect mode to 292,000 for OpenClaw1,6. Token use varied 70-fold. Meanwhile, a developer fine-tuned a 0.8B model that matched a hosted frontier model on a narrow dictation task2. Together, these two lines of evidence reframe the economics of AI deployment.
Why it matters
Harness benchmarks reveal that identical models produce wildly different token bills depending on the scaffolding around them. Individual practitioners are demonstrating that fine-tuned models with fewer than one billion parameters can match hosted frontier performance on well-scoped tasks. ANALYSIS Together, these findings pressure the assumption that bigger models accessed via API are the default path to production.
The big picture
The ordering of tokens per solved task barely moved between two unrelated models, suggesting the harness, not the model, drove the spread. Composio's August benchmark reinforced the pattern: across 30 enterprise workflows on DeepSeek V4 Flash, cost per successful task ranged from $0.028 for Pi Agent to $0.195 for Claude Code. DeepAgents matched Claude Code's pass rate exactly while costing a quarter as much per success. Cache behavior compounded the divergence. Composio disclosed that Claude Code drew only 1.5% of its input tokens from cache, against roughly 70% for Codex and 57% for OMP. Since fresh input costs about five times as much as cached input, the effective invoice gap between harnesses widened beyond what raw token counts alone would suggest.
ANALYSIS When the scaffolding layer alone can swing costs by multiples, the marginal expense of calling a frontier model is no longer just the per-token rate; it is the per-token rate multiplied by whatever the harness decides to send. That arithmetic changes the breakeven calculus for fine-tuning a small, local model.
The fine-tuning case studies in this packet illustrate the alternative. A developer fine-tuned Qwen3.5-0.8B for dictation cleanup, producing SpeakoFlow Mini, which scored 70.7% on a specialized English-only benchmark; GPT-5.6 Luna scored 65.0% under the same fixed short prompt with reasoning disabled. The creator was careful to frame this as a statistical tie, noting the 95% confidence interval was [-1.5, +12.9]. But the controlled comparison against the untuned base told a clearer story: fine-tuning moved the score from 47.3% to 70.7%, a gain of 23.4 points with a 95% interval of [+16.3, +30.3]. "Under one fixed short prompt with no reasoning budget, a 0.8B model running locally matched a hosted frontier model on this task," the developer wrote. The Q8_0 build is 833 MB and runs fully offline.
Meanwhile, a separate effort produced Scaffold CoT, a roughly 4-million-example, 3-billion-token chain-of-thought dataset designed specifically for models under 5 billion parameters4. Its creator observed that "freeform CoT does not really add much" for small models and "usually results in more confusing, poorly structured and inaccurate responses". The dataset enforces a rigid three-section format (Inventory, Interaction, Execution) with a maximum example size of 2,048 tokens, so "the model spends its capacity on the actual content instead of re-deciding how to lay out a thought every time". ANALYSIS Purpose-built training data like Scaffold CoT lowers the barrier for practitioners who want to fine-tune small models but lack the expertise to curate reasoning examples from scratch.
At the architectural frontier, a 150.13-million-parameter non-Transformer model called WarpState completed pretraining on roughly 300 million tokens, reaching a final sampled validation perplexity of 30.90, trained locally on a laptop GPU with a peak VRAM draw of roughly 4.52 GB5. The long-range memory size does not grow linearly with context in the same way as a conventional full KV cache. The author considers the run primarily a proof that the architecture can train rather than a fully trained 150M language model. ◆ WarpState is an experimental proof of concept, but it demonstrates that novel architectures can be explored entirely on consumer hardware, further compressing the resource floor for small-model work.
Another practitioner built a full coding harness using Qwen 3.x 27B, reporting that more than 90% of the harness was built locally under direct supervision3. The tool includes just-in-time code review before tool calls, sub-agent threads, and voice dictation, effectively replicating hosted agent functionality without API dependency.
ANALYSIS The harness benchmarks and the fine-tuning projects converge on a single insight: the layers surrounding a model (scaffolding, prompt design, training data curation) now exert as much influence on cost and quality as the model itself. The startup tax multiplied by the turn count predicted tokens per solved task with an R-squared of 0.99 across both models in the June benchmark. That near-perfect fit implies harness overhead is not noise; it is the dominant variable. A fine-tuned 0.8B model running locally, meanwhile, spends zero API tokens.
What's next
Artificial Analysis continuously tracks harness-model pairings in its coding-agent index, covering 326 tasks with pass rates averaged across three attempts each. Composio reported 240 runs across eight harnesses. As these benchmarks accumulate data, they will make the cost of harness overhead visible in a way model leaderboards never did. Scaffold CoT's 4-million-example dataset targets models under 5 billion parameters. ◆ If structured training resources like it proliferate, the practical ceiling for small-model fine-tuning rises, and the case for defaulting to a frontier API on every task weakens further.