The Active Ingredient in Muon's Grokking⭐8

Ablation study reveals Muon's grokking speedup comes from orthogonalization, not spectral constraints. Newton-Schulz iteration reduces spectral norm by 3x. More...