Overview
Reinforcement Learning from Human Feedback (RLHF) has become the standard approach for aligning large language models (LLMs) with human preferences. However, traditional RLHF methods rely on static, task-agnostic reward models, which often produce sparse learning signals and result in suboptimal alignment, particularly for complex or nuanced tasks.
Introducing MeRLa: Meta-Learned Reward Shaping
To address these limitations, we introduce MeRLa (Meta-Learned Reward Shaping), a principled framework that dynamically shapes reward signals by meta-learning a task-aware shaping function, denoted as $\Phi(x,y;\phi)$, across auxiliary tasks prior to RLHF training. The learned shaping function generates a composite reward that preserves the optimality of the original policy while providing denser, task-specific learning signals.
Key Components of the Meta-Objective
The MeRLa meta-objective integrates three core terms:
- Task discrimination: Ensures the reward model distinguishes between tasks effectively.
- Entropy regularization: Maintains exploration diversity during training.
- Potential-based conservation: Leverages potential-based reward shaping to guarantee stable convergence and policy invariance.
Theoretical and Practical Guarantees
We provide formal theoretical guarantees for:
- Policy invariance: Demonstrating that the optimal policy under the shaped reward remains unchanged.
- Representation drift sensitivity: Analyzing how shifts in learned representations affect performance.
- Incentive misalignment: Formally addressing the misalignment that can arise from entropy maximization in the reward structure.
Experimental Results
We evaluate MeRLa using LLaMA-3-8B across four standard benchmarks. Compared to established RLHF methods — including PPO, DPO, GRPO, and DAPO — MeRLa achieves:
- 90.8% length-controlled win rate on AlpacaEval 2.0
- 9.14 score on MT-Bench
- 41% reduction in training instability (measured by variance in reward and policy updates)
Additionally, MeRLa retains its performance improvements when combined with process-based or rubric-based enhanced reward signals, demonstrating robustness and flexibility.
Conclusion
MeRLa offers a scalable and theoretically grounded solution to the reward sparsity problem in RLHF. By meta-learning task-aware reward shaping, it significantly improves alignment quality, training stability, and overall model performance without sacrificing policy optimality.
via ArXiv LG