LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes

Authors: Jiaming Hu, Zhida Jiang, Zhaolong Xing, Yu Li, Zhen Chen, Ke Zhang

arXiv: 2610.37720 [cs.AI] — Submitted 27 October 2026

Subjects: Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)

DOI: https://doi.org/10.48550/arXiv.2610.37720

Abstract

Sparse Mixture-of-Experts (MoE) models offer scalable capacity for web applications, but uneven routing creates computation and communication bottlenecks. Existing placement and replication methods redistribute this workload, yet the additional expert copies they introduce compete with token-related buffers for limited GPU memory. We present LRMoE, a memory-aware MoE training system that jointly re-calibrates backward demand and balances the retained workload.

During forward execution, LRMoE plans retention quotas and expert placement under resource constraints while preserving all routed computations. During backpropagation, a load re-calibration metric estimates each token–expert pair's contribution to the expert update and selects pairs within these quotas. A capacity-aware routing algorithm then distributes the retained pairs among eligible instances of their original experts. Physical compaction forms smaller communication payloads and variable-size grouped GEMMs, reducing backward traffic, computation, and execution-buffer demand.

Experiments on Qwen3-30B-A3B and GLM4.5-106B-A12B show that LRMoE reduces cumulative MoE-layer training time to fixed-loss targets by 21.2–44.8% relative to SYMI.

Key Contributions

• Memory-aware MoE training system that co-designs expert placement with backward workload reduction, rather than relying on replication alone.
• Demand-side load re-calibration that scores each token–expert pair's contribution to expert updates during backpropagation, enabling principled pruning within planned retention quotas.
• Capacity-aware routing that redistributes retained pairs across eligible instances of the original experts.
• Physical compaction that yields smaller communication payloads and variable-size grouped GEMMs, lowering backward traffic, computation, and execution-buffer pressure.
• Empirical gains of 21.2–44.8% reduction in cumulative MoE-layer training time to fixed-loss targets versus SYMI on Qwen3-30B-A3B and GLM4.5-106B-A12B.

Context

As sparse MoE architectures such as Qwen3 and GLM4.5 continue to scale in 2026, load imbalance and memory pressure on commodity GPU nodes remain central obstacles to efficient training. LRMoE targets exactly this regime: memory-constrained deployments where expert replication alone is not viable, and where recomputing or discarding backward demand must be done carefully to preserve model quality.

Citation

@article{hu2026lrmoe,
  title   = {LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes},
  author  = {Hu, Jiaming and Jiang, Zhida and Xing, Zhaolong and Li, Yu and Chen, Zhen and Zhang, Ke},
  journal = {arXiv preprint arXiv:2610.37720},
  year    = {2026},
  doi     = {10.48550/arXiv.2610.37720}
}

via ArXiv AI