Abstract
Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding responses in external knowledge, yet multi-hop reasoning remains susceptible to error propagation, where early retrieval failures cascade into subsequent incorrect steps. Standard outcome-based optimization, which rewards only the final answer, overlooks intermediate retrieval and reasoning errors. Existing process-based methods introduce step-level signals but still score each step against the final answer, inadvertently rewarding spurious successes where flawed retrieval coincidentally yields a correct answer. Effective step-level supervision in RAG demands evaluating both logical validity and evidential grounding at every step.
We introduce PRO-STEP, a novel framework that addresses these limitations. We train a generative Process Reward Model (PRM) that evaluates both logical and evidential dimensions simultaneously. Using PRM-guided value tree search, we construct preference pairs that contrast valid steps against flawed ones, and subsequently optimize the policy via step-level Direct Preference Optimization (DPO). Evaluations on single-hop and multi-hop question-answering datasets demonstrate that PRO-STEP achieves the best average Exact Match (EM) and F1 scores across five benchmarks. Code, models, and training data are publicly available at this GitHub repository.
1. Introduction
Retrieval-Augmented Generation has become a cornerstone technique for grounding LLM outputs in up-to-date, external documents, thereby mitigating hallucination and improving factual accuracy. However, as RAG applications increasingly target complex, multi-hop queries, the underlying reasoning process becomes more susceptible to error propagation—a single mis-retrieval early on can derail all subsequent reasoning steps, yielding incorrect or incomplete answers.
Traditionally, RAG systems have been trained using outcome-based rewards, which optimize for the correctness of the final answer only. This approach fails to provide intermediate feedback, leaving retrieval and reasoning errors undetected during training. While recent process-based methods attempt to introduce step-level supervision, they typically score each step against the final answer. This creates a critical blind spot: a step that retrieves irrelevant documents but nonetheless contributes to a correct final answer (by chance) is rewarded, thereby reinforcing flawed behavior.
In this paper, we argue that step-level supervision in RAG must assess two orthogonal dimensions simultaneously:
- Logical validity—is the reasoning step internally coherent and progressive toward an answer?
- Evidential grounding—is the step supported by relevant, trustworthy retrieved evidence?
To this end, we propose PRO-STEP (Process Reward Optimization for Step-level Evaluation of RAG). Our approach trains a generative process reward model that evaluates both dimensions at each step, constructs rich preference pairs via PRM-guided value tree search, and optimizes the policy using step-level Direct Preference Optimization. This method provides fine-grained, reliable supervision that distinguishes genuine progress from spurious successes.
2. Related Work
2.1 Outcome-based Optimization in RAG
Most prior work in RAG training relies on outcome-based rewards, typically derived from final answer accuracy (e.g., EM/F1). While simple and effective for single-hop tasks, this paradigm struggles with multi-hop reasoning due to the lack of intermediate feedback.
2.2 Process Reward Models
Process reward models (PRMs) have been introduced to offer step-level feedback, particularly in mathematical reasoning. PRMs score intermediate reasoning steps, allowing for more precise credit assignment. However, existing PRMs often conflate logical progress with evidential support, especially in retrieval-augmented settings.
2.3 Preference Optimization for LLMs
Direct Preference Optimization (DPO) has emerged as a stable and efficient alternative to reinforcement learning from human feedback (RLHF). By directly aligning the policy with pairwise preferences, DPO avoids the complexity of reward modeling and policy iteration.
3. Method
3.1 Generative Process Reward Model (PRM)
We train a generative PRM that, given a question, a set of retrieved documents, and a partial reasoning chain, outputs a score reflecting both logical validity and evidential grounding. Unlike discriminative PRMs that output a scalar, our generative approach allows for richer, explainable evaluation.
3.2 PRM-guided Value Tree Search
To generate high-quality preference pairs, we employ a value tree search guided by the PRM. Starting from an initial state, we expand multiple reasoning branches, pruning unpromising paths based on PRM scores. This structured exploration yields contrasting step pairs—valid compared to flawed—that serve as training signals.
3.3 Step-level Direct Preference Optimization
We optimize the RAG policy using step-level DPO, where each preference pair corresponds to a single reasoning step. The policy learns to increase the likelihood of valid steps while decreasing that of flawed ones, thereby internalizing the nuances of effective retrieval and reasoning.
4. Experiments
4.1 Setup
We evaluated PRO-STEP on five question-answering benchmarks covering both single-hop and multi-hop tasks (e.g., HotpotQA, 2WikiMultiHopQA). We compared against baseline RAG systems and existing process-based methods.
4.2 Results
PRO-STEP consistently outperformed all baselines on both Exact Match (EM) and F1 metrics across all benchmarks. Notably, the improvement was most pronounced on multi-hop datasets, highlighting the importance of correct step-level supervision.
5. Conclusion
PRO-STEP addresses a critical gap in RAG training by providing process-level supervision that respects both logical and evidential dimensions. Its generative PRM, combined with value tree search and step-level DPO, enables fine-grained credit assignment, leading to more robust multi-hop reasoning. This work lays a foundation for future advances in process-aware optimization for retrieval-augmented language models.
Availability
Code, models, and training data are available at: https://github.com/keemminnke/PRO-Step.
Accepted to EMNLP 2026. 22 pages, 7 figures, 23 tables.
via ArXiv CL+LG