论文

MiMo-V2.6：扩展强化学习以实现自我提升

MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement

Xiaomi LLM-Core Team, :, Zongming Qiao, Ziyue Hua, Zirui Ou, Zihao Yue, Zihan Jiang, Zhuo Huang, Zhiyang Chen, Zhixian Zheng, Zhipeng Xu, Zhengrui Ma, Yuyang Hu, Yuhang Dong, Yuechen Zhang, Yudong Wang, Yuanxin Liu, Yixin Yang, Yishuo Cai, Yikai Zhao, Yihan Yan, Yifan Zhang, Yifan Song, Xiyu Wei, Xing Zhang, Xin Zhang, Xiaoqian Liu, Xiaodong Ji, Xiangwei Deng, Xueyu Guo, Wenhan Ma, Weimin Xiong, Weikun Wang, Weiji Zhuang, Shuo Liu, Shuhuai Ren, Shuhao Gu, Shimao Chen, Shijie Cao, Shihua Yu, Shicheng Li, Shengjie Zhou, Shaolei Zhang, Rang Li, Qiying Wang, Qingkai Fang, Qianli Chen, Minzheng Wang, Liwen Wang, Linli Yao, Linghao Zhang, Liangyu Cheng, Liang Zhao, Lei Li, Jinhao Dong, Jinyu Xiang, Jianyu Wei, Jiangshan Duo, Huaqiu Liu, Huanjie Fan, Hongyi Guan, Hongshen Xu, Hao Tian, Hanyu Li, Hailin Zhang, Gang Wang, Fuli Luo, Feng Wei, Dong Zhang, Dawei Zhu, Chiheng Lou, Chenhong He, Chenhao He, Chenghua Liu, Bowen Ye, Bowen Shen, Boshen Xu, Bo Yang, Bingquan Xia, Bangjun Xiao, Baixuan Xu, Zhouxiang Mao, Zhiyang Zhang, Zhixiang Xu, Zhenru Lin, Zhengju Tang, Zhaojun Huang, Yuzhe Weng, Yuxing Xiang, Yuxiao Li, Yuheng Yang, Yuhang Wang, Yuchen Liu, Yuanyuan Tian, Yuanliang Dong, Yu Cheng, Yongzhe He, Yongshun Liang, Yong Wang, Yiyan Wang, Yitian Gong, Yijie Zhang, Yanshu Xin, Xun Zhang, Xingjian Zhao, Wenyu Yang, Wenshan Huang, Wenhao Li, Tingwei Huang, Tianyu Yu, Tianyang Lu, Taoyu Yang, Sinan Du, Shutong Tian, Shulin Du, Shengfan Wang, Shanchuan Fang, Qihao Zhang, Qibin Yang, Qian Yu, Qian Tu, Pengrong Xie, Peipei Wang, Peidian Li, Minkun Guo, Mingchen Shao, Luohan Gao, Lijie Wang, Liang Shi, Kaiqi Chen, Kaiming Liu, Kaifei Wang, Kai Yang, Jinlong Xue, Jiechen Zhang, Jiaxuan Liu, Hongxu An, Hao Peng, Hanglong Lü, Guonan Wang, Feiyu Yang, Fanyu Cao, Fangyue Liu, Fan Cui, Cong Wang, Chun Chen, Chenxu Bai, Chengxuan Zhu, Chenghua Wang, Boyi Zeng来源日期：2026.10.09

模型训练 AI 基础设施强化学习奖励建模与过程监督分布式训练基础设施

摘要

强化学习（RL）是推动大型基础模型实现自我改进的核心训练范例。本报告介绍了 MiMo-V2.6 系列，这是一个全模态系列，通过扩展 RL 计算来推动模型智能的前沿。在 RL 之前，我们在广泛的多模态语料库上进行中期训练，以提供充足的探索空间，并在预训练的混合 SWA 架构上构建坚实的基础设施，以支持后续的扩展。我们沿着三个维度扩展 RL 计算：(1) 更大的批次和更高的吞吐量，异步训练在上下文长度高达 1M 的情况下每步消耗 1,568 个样本和 2.7-3.7B 个token； (2) 更加多样化和复杂的环境，跨越代码、通用、视觉和网络领域，混合使用代理工具； (3) 更多的评分器计算，通过分组 agentic 评分，为长期任务产生更准确的奖励信号，并引导模型转向更短、更高效的解决方案。为了保持大规模训练的稳定性，我们冻结了 MoE 路由器并建立了多层防御奖励黑客攻击。我们进一步构建混合任务 agentic RL 的基础设施，包括统一的轨迹表示、高并发多框架部署、解耦控制和数据平面以及训练推理一致性。我们开源训练动态、强化学习环境和强化学习框架，以促进大规模强化学习和模型自我改进的复制和进一步研究。