Abstract
Mixture-of-Experts (MoE) architectures have become a cornerstone of large-scale deep learning in 2026, yet the underlying logic of their routing mechanisms remains largely opaque. In this paper, we uncover a fundamental governing principle: MoE routing is not merely a selection processβit is a manifestation of Huffman Coding at the neural level. We introduce the Frequency-Diversity Law, demonstrating that state-of-the-art models, including Phi-3.5-MoE and Gemma-4-27B-A4B, spontaneously act as information-theoretic engines. These systems allocate sparse expert resources to common tokens, while assembling high-diversity expert committees for rare, complex tasks encountered in chain-of-thought trajectories.
However, we also identify a critical redundancy trap in Qwen3.5-35B-A3B: when effective sparsity (k/Eeff_) is sufficiently low, load-balancing constraints inadvertently impose functional redundancy, masking the model's underlying Huffman efficiency signal. To address this, we propose Subset Difference Pruningβa surgical strategy that eliminates functional duplicates without degrading reasoning performance. Our results show that pruning does not impair cognitive capability; rather, it unleashes latent Huffman efficiency, forcing the routing logic to collapse into streamlined, high-density information paths.
These findings suggest that the next generation of MoE architectures should move beyond forced load-balancing toward Minimum Description Length (MDL) optimality. Under this paradigm, shorter expert-routing codes are assigned to high-frequency information, while longer, more diverse codes handle low-frequency information. This transforms routing from a heuristic design choice into a principled compression engine.
Paper details: 20 pages, 20 figures. Submitted on 8 May 2026.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Theory (cs.IT)
Authors: Ching-Chieh Tsao, Zhuoyi Lin, Wenya Wang
Cite as: arXiv:2607.20427 [cs.CL]
via ArXiv CL