Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models
arXiv:2610.22428 [cs.CL] | Submitted 15 Oct 2026 | Computation and Language
Author: Sai Kiran Bathula
Overview
In a small language model, the vocabulary is the single largest component. In Gemma 3 270M, the embedding table accounts for 168M of 268M parameters, and because quantization methods typically leave embeddings in 16-bit precision, the table also dominates the model file once the rest of the network is compressed. This raises a basic question: what does that table actually buy?
This work investigates that question by replacing the learned embedding table with a single frozen table of 512-bit codes โ approximately 16 MB โ with no trainable parameters. The table is derived once from a pretrained embedding via principal-component projection followed by sign binarization. The author then trains the same Gemma-shaped body from scratch on the same 2.05B tokens, both with and without the frozen table, to isolate its contribution.
Key Findings
Validation loss vs. downstream performance. The learned table buys 0.20 nats of validation loss at the cost of 2.6 times the trainable parameters. However, on LAMBADA, HellaSwag and PIQA, the frozen-table model is either better than or tied with the learned-table model; only ARC-Easy favors the learned table.
Ternary integration. When paired with a ternary BitNet b1.58 body, the entire model becomes a 46 MB file. This configuration still ties the float model on LAMBADA and PIQA, and on a laptop CPU it generates 615 to 740 tokens per second within 62 MB of memory โ five to six times the speed of the float model's 8-bit GGUF under this http URL, at one-seventh of the memory.
Rare words and extensibility. The frozen table outperforms by 1.7 to 2.3 nats on rare words and accepts 10,000 new words after training at a 3% cost, whereas the learned table collapses by 43% under the same conditions.
Availability
All of the work is released with the weights, the data and every evaluation answer.
Metadata
- Comments: 9 pages, 2 figures, 6 tables. Code, data, checkpoints and per-question results: this https URL. Preprint DOI: https://doi.org/10.5281/zenodo.23382634
- Subjects: Computation and Language (cs.CL)
- Cite as: arXiv:2610.22428 [cs.CL]
- Version: arXiv:2610.22428v1 [cs.CL]
- arXiv DOI: https://doi.org/10.48550/arXiv.2610.22428
via ArXiv CL+LG