Prime Intellect Releases Prime Mega MoE: A Trainable Fused MoE Kernel With a 3.11x Faster Forward Pass

Prime Intellect has released Prime Mega MoE, a training-oriented extension of DeepSeek's fused Mega MoE kernel. DeepSeek's original kernel implemented only the forward pass; Prime Intellect added a fast, overlapping backward pass, making the fused kernel usable for full training rather than inference alone.

In its prime-rl trainer, complete training steps on Qwen3 MoE models run up to 34% faster. Compared with TorchTitan, the gain reaches 55%. For teams training mixture-of-experts (MoE) models, that translates directly into fewer GPU hours—a meaningful advantage in 2026, as MoE architectures continue to dominate large-scale model development and GPU capacity remains a primary cost constraint.

TL;DR

• What it is: A GPU kernel, not a model. Tested on Qwen3-30B-A3B and Qwen3-235B-A22B (128 experts, top-8 routing).
• Hardware support: NVIDIA Hopper (blockwise FP8) and Blackwell (MXFP8) GPUs, with BF16 supported on both. Published benchmarks use 8×H200 nodes across 32 to 128 GPUs.
• Performance: 32% to 34% faster full training steps than the prime-rl baseline, and 48% to 55% faster than TorchTitan.
• Best case: A 3.11x faster forward pass on a Qwen3-30B-A3B MoE layer (12.0 ms down to 3.9 ms).
• Worst case: 1.24x...

via MarkTechPost