#vocabulary compression

Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for SmallNEW⭐8

A frozen 16 MB embedding table for small and ternary language models matches learned embeddings on LAMBADA, HellaSwag, and PIQA while cutting parameters.