Research #Open models Oct 22, 2026 · Originally published Oct 21

Dharma AI rebuilds a fine-tuned Qwen3-4B's vocabulary around token cost, cutting a production essay grader's measured inference cost 77%

The tokenizer is usually treated as fixed by the base model; this argues it can be a specialized small model's biggest cost lever. Where the savings come from, and the mistake along the way.

Primary source Hugging Face Blog · Dharma AI · huggingface.co Read the original ↗

// Key points

The Portuguese essay grader is built on Qwen3-4B and scored a 151,669-entry vocabulary at every generated token; VocabX rebuilt the vocabulary while keeping the same base model, data and training recipe.
On the same 500 requests on an RTX 5090 with vLLM, the average response fell from 350 to 101 tokens, cost fell 77.0%, and the bench score rose from 0.703 to 0.810.
The first version fused each grade with the surrounding JSON into one macro token: a 73.3% cost cut, but the bench score fell to 0.689; giving the grade its own token cost 2 extra tokens per response and came out better on both quality and cost.
The numbers come from a paper still under review and may change before publication.

Builder's takeAI Interview's scoring report is also fixed-format JSON, so the setup is nearly identical, but this only applies to self-hosted fine-tuned small models; if you call an API you can't change the vocabulary. The lesson worth keeping is the mistake: compress all you like, but don't fuse the value that carries the decision into the same token as the formatting, or you save cost and lose accuracy.

// Background · from #Open models

Full timeline →