📋 摘要
⭐ 聚焦RL训练系统级加速，与SE for AI/可信AI/测试方向关联较弱，仅属通用ML系统优化。
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
中文
本文针对 frontier 语言模型 RL post-training 中 autoregressive rollout 生成成为主要瓶颈的问题，提出将 speculative decoding 作为一种无损（lossless）加速原语引入 RL rollout 阶段，以在保留 target model 输出分布的前提下提升吞吐。与已有通过改变 rollout 或优化范式（如 off-policy 执行、replay、低精度生成）来提升效率的方法不同，作者方案不改变训练语义。具体实现上，作者在 NeMo-RL 中基于 vLLM 后端集成了 speculative decoding，同时支持 synchronous 与 asynchronous 流水线，并使其可在 RL rollout 过程中启用；该框架兼容多种 speculation 机制，包括预训练的 MTP head、小型外部 draft model 以及通常仅在 RL 之后才使用的 Eagle3 等技术，从而为 SOTA speculative decoding 在 RL 训练内部的部署提供了一条路径。实验在 8B 规模 reasoning post-training、synchronous RL 设置下，rollout 吞吐提升约 1.8x；并通过高保真性能模拟器外推，speculative decoding 与 asynchronous RL 结合在 235B 规模下可带来最高 2.5x 的端到端训练加速。
English abstract
RL post-training of frontier language models is increasingly bottlenecked by autoregressive rollout generation, making rollout acceleration a central systems challenge. Many existing efficiency methods improve throughput by changing the rollout or optimization regime, for example, through off-policy execution, replay, or lower-precision generation. We study speculative decoding as a lossless acceleration primitive for RL rollouts that preserves the target model's output distribution. We implement speculative decoding in NeMo-RL with a vLLM backend, supporting both synchronous and asynchronous pipelines and enabling speculation during RL rollouts. This benefit is realizable across speculation mechanisms, such as pretrained MTP heads, small external draft models or even techniques such as Eagle3, which are traditionally applied after RL phase. This yields a deployment path for state-of-the-art speculative decoding inside RL training. In a reasoning post-training workload at 8B scale under synchronous RL, speculative decoding improves rollout throughput by 1.8x. Using a high-fidelity performance simulator, we project that combining speculative decoding with asynchronous RL yields up to 2.5x end-to-end training speedup at 235B scale.
加载中…
点文件 → 加为 tab；按 Esc 关闭