Specialized models, obtained by fine-tuning general-purpose open-source models like Qwen or Gemma, inherit a vocabulary designed for many domains. According to Dharma AI, a method called VocabX, which rebuilds this vocabulary based on the cost of each token, reduced the inference cost measured on a production essay grader by 77%, without modifying the base model.
- VocabX rebuilds the tokenizer vocabulary according to the cost of each token.
- It cuts inference costs by 77% on a Portuguese essay grader, without modifying the base model.
- Generated tokens cost 7.4 times more than read tokens, so the method weights savings toward the output.
- The grader’s accuracy score rose from 0.703 to 0.810, and VocabX beat the best reference method in all fifteen tests.
This automatic grader, built on Qwen3-4B, grades essays in Portuguese. It receives the essay, the topic, and supporting texts, then returns an evaluation in JSON format, with a grade and a comment for each skill. The model, which has 4 billion parameters, had already been trained for this task, but it retained the original tokenizer, i.e., the tool that splits text into tokens, the units the model reads and writes. This tokenizer was intended for many languages and many types of texts.
For such a narrow application, this choice generated two types of extra costs. On the one hand, recurring elements, like the JSON wrapper or the skill names, were split into multiple tokens where a single one would have sufficed. On the other hand, for each generated token, the model calculated a score for the 151,669 entries in the vocabulary, many of which are almost never used for the task.
Writing costs more than reading
During inference, the generation phase is the most costly. In the measured case, a generated token costs 7.4 times more than a read token, but this ratio varies depending on hardware, the problem, batch sizes, and context length. Once the vocabulary was reduced, the grader’s output layer went from about 152,000 to about 65,000 lines.
VocabX does not select tokens based on their frequency, as several reference methods do, including tokenizer expansion in place. Each removed token is weighted by the price of the field where it appears: system prompt, user message, or model response. In the case of the grader, a token saved in the response is therefore worth 7.4 times a token saved in the input.
A sample that doesn’t see everything
The method also relies on five repeated sequence search tools, each specialized in a type of repetition: long words, entire lines, expressions, complete sentences, or punctuated structures like JSON. According to the paper, without these five tools, the token reduction on the grader would drop by 55.6 percentage points.
Evaluating each candidate on the entire corpus would be too expensive. The authors therefore measure on a sample, but an expression appearing 37 times in 50,000 documents only appears in about 26% of samples of 400 documents. For candidates absent from the sample, VocabX uses a statistical estimate rather than a zero value. On a corpus of contracts, this single change increased token reduction from 44.7% to 52.8%, with the same sample and the same budget.
Results measured on the grader
To measure the actual effect, seven versions of the grader were trained with the same model, the same data, and the same settings, only the tokenizer changing. The 500 test requests were executed on an RTX 5090 card with the vLLM inference server. The average response went from 350 to 101 tokens, the cost decreased by 77.0%, and the reference score, which combines the accuracy of grades and the filtering of off-topic or non-compliant texts, rose from 0.703 to 0.810.
An initial version, which merged the grade for each skill with the surrounding JSON, was more compact but less accurate, with a score of 0.689. Giving the grade its own token added 2 tokens per response, while improving the score and reducing the cost by 77.0%, compared to 73.3% for the merged version. The authors draw a lesson: compression has a price when it encompasses the very decision the model must make.
Across five public domains and three model families, VocabX reduced the number of tokens more than the best reference method in all fifteen combinations tested, with a margin of 1.2 to 9.3 percentage points. The figures presented come from the current version of the paper, still under review, and could change before publication. The full paper is due to be published soon.
