In this article, we explore how Mixture-of-Experts (MoE) models have evolved from a handful of experts to nearly 900 per layer, and the compression and stability mechanisms that keep such sparse designs both trainable and cost-effective.
Open-weight MoE models have grown at a remarkable pace: Mixtral boasted about 47 billion total parameters, DeepSeek-V3 reached 671 billion, and by 2026, Kimi K3 has crossed into the trillions. The surprising aspect is not just the sheer size, but how little of each model is activated for any single token. Kimi K3, for instance, has 2.8 trillion parameters, yet it uses only about 104 billion for each token. In nearly every layer, a small router selects 16 of 896 specialized feed-forward networks—known as experts—while two shared experts process every token.
This article focuses on the width-side design: how a model can offer a vast pool of processing capacity without deploying all of it for every token. This is distinct from sequence memory, which deals with how the model stores and retrieves information from earlier tokens. To understand how K3 arrived at this architecture, we trace the evolution of MoE through four key designs.
- Mixtral – A clear open-weight example of the basic MoE pattern: each token is routed to a few full-size experts.
- DeepSeekMoE – Divided the work among finer-grained and shared experts, improving efficiency and specialization.
- LatentMoE – Compressed the routed path so experts could operate in a lower-dimensional space, reducing computational overhead.
- Kimi K3 – Adopted this as Stable LatentMoE, adding mechanisms for numerical stability and balanced routing across 896 experts per layer.
Throughout this journey, you'll learn how to interpret an MoE model's expert counts and active-parameter numbers, and what they imply for computation and data movement. Activating only a small subset of experts is what makes MoE attractive, but it also introduces new bottlenecks: the selected experts' weights must still be loaded from GPU memory, and token representations may need to travel between GPUs, adding communication overhead.
By 2026, these challenges have driven innovations in compression, routing algorithms, and hardware-aware design, making trillion-parameter sparse models not only feasible but increasingly practical for real-world deployment. This evolution highlights a fundamental shift in AI: scaling intelligence is no longer just about adding parameters—it's about using them wisely.
via FreeCodeCamp