论文

DeepSeek-V4：迈向高效的百万词元上下文智能

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

DeepSeek-AI; Anyi Xu; Bangcai Lin; Bing Xue; Bingxuan Wang; Bingzheng Xu; Bochao Wu; Bowei Zhang; Chaofan Lin; Chen Dong; Chenchen Ling; Chengda Lu; Chenggang Zhao; Chengqi Deng; Chengyu Hou; Chenhao Xu; Chenze Shao; Chong Ruan; Conner Sun; Damai Dai; Daya Guo; Dejian Yang; Deli Chen; Donghao Li; Dongjie Ji; Erhang Li; Fang Wei; Fangyun Lin; Fangzhou Yuan; Feiyu Xia; Fucong Dai; Guangbo Hao; Guanting Chen; Guoai Cao; Guolai Meng; Guowei Li; Han Yu; Han Zhang; Hanwei Xu; Hao Li; Haofen Liang; Haoling Zhang; Haoming Luo; Haoran Wei; Haotian Yuan; Haowei Zhang; Haowen Luo; Haoyu Chen; Haozhe Ji; Hengqing Zhang; Honghui Ding; Hongxuan Tang; Huanqi Cao; Huazuo Gao; Hui Qu; Hui Zeng; J Yang; JQ Zhu; Jia Luo; Jia Song; Jia Yu; Jialiang Huang; Jialu Cai; Jian Liang; Jiangting Zhou; Jiasheng Ye; Jiashi Li; Jiaxin Xu; Jiewen Hu; Jieyu Yang; Jin Chen; Jin Yan; Jingchang Chen; Jingli Zhou; Jingting Xiang; Jingyang Yuan; Jingyuan Cheng; Jingzi Zhou; Jinhua Zhu; Jiping Yu; Joseph Sun; Jun Ran; Junguang Jiang; Junjie Qiu; Junlong Li; Junmin Zheng; Junxiao Song; Kai Dong; Kaige Gao; Kang Guan; Kexing Zhou; Kezhao Huang; Kuai Yu; Lean Wang; Lecong Zhang; Lei Wang; Leyi Xia; Li Zhang; Liang Zhao; Lihua Guo; Lingxiao Luo; Linwang Ma; Linyan Zhu; Litong Wang; Liyu Cai; Liyue Zhang; Longhao Chen; MS Di; MY Xu; Max Mei; Miaojun Wang; Mingchuan Zhang; Minghua Zhang; Minghui Tang; Mingming Li; Mingxu Zhou; Minmin Han; Ning Wang; Panpan Huang; Panpan Wang; Peixin Cong; Peiyi Wang; Peng Zhang; Qiancheng Wang; Qihao Zhu; Qingyang Li; Qinyu Chen; Qiushi Du; Qiwei Jiang; Rui Tian; Ruifan Xu; Ruijie Lu; Ruiling Xu; Ruiqi Ge; Ruisong Zhang; Ruizhe Pan; Runji Wang; Runqian Chen; Runqiu Yin; Runxin Xu; Ruomeng Shen; Ruoyu Zhang; Ruyi Chen; SH Liu; Shanghao Lu; Shangmian Sun; Shangyan Zhou; Shanhuang Chen; Shaofei Cai; Shaoheng Nie; Shaoqing Wu; Shaoyuan Chen; Shengding Hu; Shengyu Liu; Shiqiang Hu; Shirong Ma; Shiyu Wang; Shuiping Yu; Shunfeng Zhou; Shuting Pan; Shuying Yu; Songyang Zhou; Tao Ni; Tao Yun; Tian Jin; Tian Pei; Tian Ye; Tianle Lin; Tianran Ji; Tianyi Cui; Tianyuan Yue; Tingting Yu; Tun Wang; W Zhang; WL Xiao; Wangding Zeng; Wei An; Weilin Zhao; Wen Liu; Wenfeng Liang; Wenjie Pang; Wenjing Luo; Wenjing Yao; Wenjun Gao; Wenkai Yang; Wenlve Huang; Wenqing Hou; Wentao Zhang; Wenting Ma; Xi Gao; Xiang He; Xiangwen Wang; Xianzu Wang; Xiao Bi; Xiaodong Liu; Xiaohan Wang; Xiaokang Chen; Xiaokang Zhang; Xiaotao Nie; Xiaowen Sun; Xiaoxiang Wang; Xin Cheng; Xin Liu; Xin Xie; Xingchao Liu; Xingchen Liu; Xingkai Yu; Xingyou Li; Xinyu Yang; Xinyu Zhang; Xu Chen; Xuanyu Wang; Xuecheng Su; Xueyin Chen; Xuheng Lin; Xuwei Fu; YC Yan; YQ Wang; YW Ma; Yanfeng Luo; Yang Zhang; Yanhong Xu; Yanru Ma; Yanwen Huang; Yao Li; Yao Li; Yao Xu; Yao Zhao; Yaofeng Sun; Yaohui Wang; Yi Qian; Yi Shao; Yi Yu; Yichao Zhang; Yifan Ding; Yifan Shi; Yijia Wu; Yiliang Xiong; Yiling Ma; Ying He; Ying Tang; Ying Zhou; Yingjia Luo; Yinmin Zhong; Yishi Piao; Yisong Wang; Yixiang Zhang; Yixiao Chen; Yixuan Tan; Yixuan Wei; Yiyang Ma; Yiyuan Liu; Yonglun Yang; Yongqiang Guo; Yongtong Wu; Yu Wu; YuKun Li; Yuan Cheng; Yuan Ou; Yuanfan Xu; Yuanhao Li; Yuduan Wang; Yuehan Yang; Yuer Xu; Yuhan Wu; Yuhao Meng; Yuheng Zou; Yukun Zha; Yunfan Xiong; Yupeng Chen; Yuping Lin; Yuqian Cao; Yuqian Wang; Yushun Zhang; Yuting Yan; Yutong Lin; Yuxian Gu; Yuxiang Luo; Yuxiang You; Yuxuan Liu; Yuxuan Zhou; Yuyang Zhou; Yuzhen Huang; ZF Wu; Zehao Wang; Zehua Zhao; Zehui Ren; Zekai Zhang; Zhangli Sha; Zhe Fu; Zhe Ju; Zhean Xu; Zhenda Xie; Zhengyan Zhang; Zheren Gao; Zhewen Hao; Zhibin Gou; Zhicheng Ma; Zhigang Yan; Zhihong Shao; Zhixian Huang; Zhixuan Chen; Zhiyu Wu; Zhizhou Ren; Zhongyu Wu; Zhuoshu Li; Zhuping Zhang; Zian Xu; Zihao Wang; Zihua Qu; Zihui Gu; Zijia Zhu; Zilin Li; Zipeng Zhang; Ziwei Xie; Ziyi Gao; Ziyi Wan; Zizheng Pan; Zongqing Yao来源日期：2026.04.26

模型架构Transformer

摘要

我们推出了 DeepSeek-V4 系列的预览版，包括两个强大的 Mixture-of-Experts (MoE) 语言模型 - 具有 1.6T 参数（激活 49B）的 DeepSeek-V4-Pro 和具有 284B 参数（激活 13B）的 DeepSeek-V4-Flash - 均支持一百万个词元的上下文长度。 DeepSeek-V4系列在架构和优化方面进行了多项关键升级：（1）混合注意力架构，结合压缩稀疏注意力（CSA）和重压缩注意力（HCA），提高长上下文效率； (2) 流形约束超连接（mHC），增强传统的残差连接； (3) 和 Muon 优化器可实现更快的收敛和更高的训练稳定性。我们在超过 32T 的多样化和高质量词元上对这两个模型进行预训练，然后通过全面的后训练管道来解锁并进一步增强其功能。 DeepSeek-V4-Pro-Max 是 DeepSeek-V4-Pro 的最大推理工作量模式，重新定义了开放模型的最先进水平，在核心任务上优于其前辈。同时，DeepSeek-V4系列在长上下文场景下效率很高。在百万词元上下文设置下，与 DeepSeek-V3.2 相比，DeepSeek-V4-Pro 仅需要 27% 的单词元推理 FLOP 和 10% 的 KV 缓存。这使我们能够定期支持一百万个词元上下文，从而使长期任务和进一步的测试时间扩展变得更加可行。模型检查点位于 https://huggingface.co/collections/deepseek-ai/deepseek-v4。