Li²SAO: Lifting Light Single-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping
Authors: Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu, Jinguo Sun, Muye Li, Shaoqiang Zheng, Xuefei Diao, Zhuwei Zeng, Nan Duan, Peng Cao
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2610.40726 [cs.CL] — Submitted 29 October 2026
Code: https://github.com/jd-cc-coder/Li2SAO
Abstract
Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training, because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. However, SAO relies on a closed-source value model obtained from scaling value pretraining.
In this paper, we analyze GRPO and LiSAO—our lightweight reproduction of SAO without the pretrained value model—on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained, LiSAO lags behind GRPO substantially in early training steps.
To address this issue, we propose Li²SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic; after which training switches to single-rollout GAE. Experiments show that Li²SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling.
Key Contributions
- Empirical analysis of GRPO vs. LiSAO: Demonstrates that GRPO retains an advantage on single-turn reasoning tasks, whereas single-rollout asynchronous optimization (LiSAO) performs better on multi-turn agentic tasks.
- Diagnosis of the cold-critic problem: Identifies that LiSAO's under-trained value model causes it to lag substantially behind GRPO during early training steps.
- Li²SAO method: Introduces group-relative actor-critic bootstrapping—using a group-relative advantage to train the actor and a group-centered return to train the critic while the critic is cold, then switching to single-rollout GAE once the critic warms up.
- Stronger performance ceiling: Shows that Li²SAO matches or exceeds GRPO in early training and raises the later performance ceiling.
Background and Motivation
Asynchronous RL has emerged as a mainstream paradigm for LLM post-training because it updates the model as rollouts arrive, improving training throughput. However, GRPO's group-wise sampling is poorly suited to asynchronous settings: rollout lengths follow a long-tailed distribution, which destabilizes group-based advantage estimation and worsens off-policy behavior—particularly on agentic tasks.
Single-rollout Asynchronous Optimization (SAO) was proposed to address these issues. Yet SAO depends on a closed-source value model produced through scaled value pretraining, limiting reproducibility and practicality. The authors introduce LiSAO, a lightweight reproduction of SAO without the pretrained value model, and use it to isolate where the approach helps and where it falls short.
Method: Li²SAO
Li²SAO combines the strengths of group-relative and single-rollout optimization through a two-phase training scheme:
- Cold-critic bootstrapping phase: While the critic is under-trained, the actor is trained with a group-relative advantage, and the critic is trained with a group-centered return.
- Single-rollout GAE phase: Once the critic has warmed up, training switches to single-rollout generalized advantage estimation (GAE).
- Catches up with or surpasses GRPO during early training, overcoming LiSAO's cold-critic deficit;
- Raises the later performance ceiling compared with both GRPO and LiSAO.
This design preserves the stability benefits of group-relative methods during early training while retaining the efficiency and off-policy robustness of single-rollout asynchronous optimization later on.
Results
Experiments on single-turn reasoning and multi-turn agentic tasks show that Li²SAO:
Takeaways
Li²SAO offers a practical path to asynchronous RL for LLM post-training without relying on a closed-source pretrained value model. By bootstrapping the critic with group-relative signals early in training, it addresses the cold-start weakness of single-rollout methods while preserving their advantages on agentic tasks. The released code aims to make these gains reproducible for the broader research community.
via ArXiv CL+LG