论文
Kimi K3：开放前沿智能
Kimi K3: Open Frontier Intelligence
Kimi Team; Tongtong Bai; Yifan Bai; Yiping Bao; M. C.; Jianfeng Cai; Xinyuan Cai; Peizhou Cao; Yuxuan Cao; Ziwei Chai; Y. Charles; H. S. Che; Guanduo Chen; Guangyu Chen; Guanzheng Chen; Huarong Chen; Jia Chen; Jianlong Chen; Jun Chen; Kexin Chen; Peng Chen; Ruijue Chen; Wentao Chen; Xin Chen; Yang Chen; Yanru Chen; Yifei Chen; Yingjiang Chen; Yuankun Chen; Yujie Chen; Yutian Chen; Zhirong Chen; Dazhi Cheng; Yean Cheng; Jialei Cui; Jingbing Cui; Anqi Dai; Jiaqi Deng; Hao Ding; Rui Ding; Shaofeng Ding; Mengfan Dong; Mengnan Dong; Yuhao Dong; Yuxin Dong; Angang Du; Chenzhuang Du; Dikang Du; Jusen Du; Yulun Du; Yu Fan; Jing Feng; Qiulin Feng; Yichen Feng; Kelin Fu; Qiang Fu; Fuxuan Gao; Hongcheng Gao; Jingyue Gao; Tong Gao; Weijia Gao; Shangyi Geng; Jie Gong; Linhu Gong; Shengao Gong; Xiaochen Gong; Qizheng Gu; Yicheng Gu; Shuhao Guan; Haiqing Guo; Shiqi Guo; Xiang Guo; Zhengyan Guo; Beixi Hao; Wenxin Hao; Xiaoru Hao; Dailan He; Haotian He; Lehan He; Qi He; Weiran He; Xinran He; Xinyi He; Yibo He; Yunjia He; Chao Hong; Tiange Hong; Hao Hu; Jiaxi Hu; Ruikun Hu; Weiming Hu; Yangyang Hu; Zhenxing Hu; Liang Hua; Jinbin Huang; Ke Huang; Ruiyuan Huang; Siying Huang; Weixiao Huang; Yan Huang; Zhengjie Huang; Zhiqi Huang; Yulong Hui; Chaobo Jia; Yutong Jiang; Zhejun Jiang; Zuoyou Jiang; Wenyi Jin; Xinyi Jin; Yu Jing; Huanjun Kong; Guokun Lai; Aidi Li; Cheng Li; Chengyuan Li; Cong Li; Fang Li; Guanyu Li; Haoyang Li; Jia Li; Junxiong Li; Lei Li; Letian Li; Lincan Li; Weihong Li; Wentao Li; Xintong Li; Yang Li; Yishen Li; Yiwei Li; Yuxiao Li; Zhaowei Li; Zhaoxi Li; Zheming Li; Zhengxiao Li; Zhiyuan Li; Jiawei Lin; Xiaohan Lin; Yibo Lin; Zichao Lin; Ziyan Lin; Bill Liu; Boxiao Liu; Chuan Liu; Liang Liu; Shaowei Liu; Shudong Liu; Shuran Liu; Tianwei Liu; Weizhou Liu; Yangyang Liu; Yanming Liu; Yibo Liu; Yipeng Liu; Zhengying Liu; Zhiheng Liu; Enzhe Lu; Haoyu Lu; Linqiang Lu; Tingzhan Lu; Zhiyuan Lu; Aotian Luo; G. Luo; Junyu Luo; Yifan Luo; B. Lyu; Wenzhou Lyu; Shaoguang Mao; Yuan Mei; Xin Men; Minqing Ni; Yixuan Niu; Siyuan Pan; Shujun Peng; Zhangyang Qi; Ruoyu Qin; ZeChao Qin; Zeyu Qin; Haiquan Qiu; Jianxin Qiu; Jiezhong Qiu; Bowen Qu; Yuhao Qu; Zeyu Shang; Youbo Shao; Han Shen; Jincheng Shi; Juanfeng Shi; Lidong Shi; Shengyuan Shi; Wingchun Siu; Pengwei Song; Xiaoxi Song; Jianlin Su; Yunfeng Su; Zhaochen Su; Lin Sui; Jingsong Sun; Junyao Sun; Shaoning Sun; Shuzhe Sun; Tongyu Sun; Yujun Sun; Yunpeng Tai; Chuning Tang; Heyi Tang; Sirui Tang; Zecheng Tang; Chaoran Tian; Rongpeng Tian; Yu Tian; Wei Tu; Chensi Wang; Chuang Wang; Chunjie Wang; Dinglu Wang; Feng Wang; Hailong Wang; Haiming Wang; Hao Wang; Hao Wang; Huaqing Wang; Hui Wang; Jiayi Wang; Jinglong Wang; Jinhong Wang; Jiuzheng Wang; Linian Wang; Shaobo Wang; Shenzhi Wang; Shuyi Wang; Si Wang; Siyuan Wang; Tianfu Wang; Wenjue Wang; Xingran Wang; Xinmei Wang; Xinyuan Wang; Xusheng Wang; Yalin Wang; Yangkun Wang; Yao Wang; Yaoyu Wang; Yejie Wang; Yiqin Wang; Yucheng Wang; Yuzhi Wang; Zhaoji Wang; Zhaowei Wang; Zhengtao Wang; Zhenhao Wang; Zhongsheng Wang; Zifan Wang; Chu Wei; Ming Wei; Shouxin Wei; Zichen Wen; Fan Wu; Haoning Wu; Rucong Wu; Wenhao Wu; Xiaoxue Wu; Yingcong Wu; Yongqi Wu; Yuxin Wu; Zijian Wu; Xinglang Xian; Chenxuan Xiang; Yuye Xiang; Bocheng Xiao; Chenjun Xiao; Xin Xiao; Jin Xie; Xiaotong Xie; Yifeng Xie; Zhe Xie; Bowei Xing; Yiming Xiong; Baosheng Xu; Boyu Xu; Jiale Xu; Jianfan Xu; Jing Xu; Jinjing Xu; L. H. Xu; Qingtao Xu; Shuyao Xu; Suting Xu; Tiantian Xu; Tianxiang Xu; Weixin Xu; Xinran Xu; Yangchuan Xu; Ye Xu; Yueni Xu; Ziyao Xu; Haonan Xue; Junjie Yan; Yaoyao Yan; Fan Yang; Guangyao Yang; Hao Yang; Junwei Yang; Ruoyu Yang; Wenjie Yang; Xiaofei Yang; Xinyu Yang; Yi Yang; Yiling Yang; Ying Yang; Yuchen Yang; Zhen Yang; Zhilin Yang; Zian Yang; Zuhao Yang; Haotian Yao; Dan Ye; Haoran Ye; Wenjie Ye; Zhanbo Ye; Bohong Yin; Haoxiang Yin; Xietong Yin; Chengzhen Yu; Haozhen Yu; Longhui Yu; Shengnan Yu; Shuying Yu; Tianxiang Yu; Enming Yuan; Mengjie Yuan; Tongtian Yue; Wei Yue; Yang Yue; Dunyuan Zha; Haobing Zhan; B. H. Zhang; Dehao Zhang; Fei Zhang; Hao Zhang; Haoyuan Zhang; Huanyu Zhang; Jiapei Zhang; Jiaxuan Zhang; Jin Zhang; Kaiyi Zhang; Miaozhen Zhang; Puqi Zhang; Qinglei Zhang; Rong Zhang; Rui Zhang; Shaoshuai Zhang; Shiyi Zhang; Xiaobin Zhang; Xiaoyun Zhang; Y. Zhang; Yangkun Zhang; Ye Zhang; Yichi Zhang; Yikun Zhang; Yizhi Zhang; Yongting Zhang; Yu Zhang; Yutao Zhang; Yutong Zhang; Zheng Zhang; Zijing Zhang; Bin Zhao; Chenguang Zhao; Feifan Zhao; Jinglun Zhao; Jinxiang Zhao; Shuai Zhao; Wenshuo Zhao; Xiangyu Zhao; Xuanle Zhao; Yikai Zhao; Zijia Zhao; Haozhi Zheng; Huabin Zheng; Ruihan Zheng; Shaojie Zheng; Tengyang Zheng; Haofeng Zhong; Lei Zhong; Longguang Zhong; M. Zhou; Qiankang Zhou; Runjie Zhou; Ruozhang Zhou; Xinyu Zhou; Yiqiao Zhou; Zaida Zhou; Jinguo Zhu; Liya Zhu; Xinhao Zhu; Yangjunfeng Zhu; Yuxuan Zhu; Zhen Zhu; Chen Zhuang; Weiyu Zhuang; Xinxing Zu·
模型架构Transformer
摘要
我们推出 Kimi K3，一个 2.8T 参数 Mixture-of-Experts 模型，具有 1040 亿个激活参数、原生视觉功能和 100 万个词元上下文窗口。 Kimi K3 基于 Kimi Delta Attention 和 Attention Residuals 构建，可改善序列长度和模型深度上的信息流。与稳定的 LatentMoE（每个词元可有效激活 896 个路由专家中的 16 个）以及改进的训练和数据配方相结合，这些进步使整体扩展效率比 Kimi K2 提高了约 2.5 倍。 后训练 强调跨通用、代理和编码领域以及多个推理努力级别的强化学习，从而实现组合泛化和强大的长期执行。在 2.8T 规模上，Kimi K3 得到了多个领域基础设施进步的支持：KDA 的算法系统协同设计、具有高效内存管理的完美平衡专家并行训练、具有持久执行轨迹和沙箱状态的百万词元代理强化学习以及部署创新。广泛的评估表明，Kimi K3 在长视野编码、代理、知识、推理和视觉任务方面实现了前沿水平的性能。虽然其整体性能仍然落后于最强大的专有模型，即 Claude Fable 5 和 GPT-5.6 Sol，但 Kimi K3 的性能始终优于我们套件中评估的其他开放和专有模型。我们发布了完整的 Kimi K3 模型权重，以促进未来的研究并加速前沿情报的更广泛部署和采用。