论文
DeepSeek-V4.1-Flash：突破 KV 缓存压缩的极限
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek-AI; :; Anyi Xu; B. Li; Bangcai Lin; Bing Xue; BingCheng Xian; Bingzheng Xu; Bochao Wu; Bowei Zhang; Boyi Deng; C. C. Yu; Chao Jin; Chaofan Lin; Chen Dong; Chenbing Wang; Chenfan Feng; Chengda Lu; Chenggang Zhao; Chengqi Deng; Chengyuan Zhang; Chenhao Xu; Chenqi Zhao; Chenze Shao; Chuhao Wang; Chuqi Zhang; Damai Dai; Dejian Yang; Deli Chen; Di Huang; Di Wu; Donghao Li; Erhang Li; Eric Fu; F. Zhou; Fangwei Zhou; Fangyun Lin; Fangzhou Yuan; Feiyu Xia; Fucong Dai; Guangbo Hao; Guanglin Li; Guanting Chen; Guoai Cao; Guofan Fan; Guolai Meng; Guowei Li; Haichuan Zhang; Haiyang Ma; Haiyang Shen; Han Li; Han Yu; Han Zhang; Hangyuan Deng; Hanwei Xu; Hanxiang Xu; Hanxun Zhong; Hao Guo; Hao Jiang; Hao Li; Hao Qin; Haodong Wen; Haofen Liang; Haofeng Huang; Haohua Liu; Haoling Zhang; Haoming Luo; Haoran Yang; Haotian Xu; Haotian Yuan; Haoting Huang; Haowen Luo; Haoyang Cai; Haoyu Chen; Haozhe Ji; Hengran Zhang; Hengrui Wang; Hengxu Wu; Honghui Ding; Hongxuan Tang; Huadong Wang; Huanqi Cao; Huazuo Gao; Hui Qu; Hui Zeng; J. Yang; J. H. Jin; J. H. Zhang; J. X. Zou; Jia Yu; Jiahui Zhou; Jiajun Chen; Jialiang Huang; Jialin Zhao; Jiamin Tang; Jian Zhou; Jianan Tong; Jianwen Li; Jiaqi Zhu; Jiarui Wang; Jiasheng Ye; Jiashi Li; Jiaxin Xu; Jiaying Ding; Jibai Lu; Jiewen Hu; Jin Yan; Jincheng Zhai; Jingchang Chen; Jingcheng Hu; Jingli Zhou; Jingsheng Xu; Jingting Xiang; Jingyan Yun; Jingyang Yuan; Jingyuan Cheng; Jinhua Zhu; Jinpeng Wang; Jinyi Chen; Jinyi Hu; Jiping Yu; Jueliang Guo; Junbo Pei; Junbo Sun; Junguang Jiang; Junjie Qiu; Junkang Zhou; Junqi Liu; Junren Li; Junxian Li; Junxiao Song; Junyi Guo; Kai Dong; Kaifeng Chen; Kaige Gao; Kang Guan; Kangdong Yuan; Ke Hong; Ke Xu; Kefan Zhao; Kexin Ji; Kexin Zhang; Kexing Zhou; Kuai Yu; Lan Zhang; Lean Wang; Lecong Zhang; Lei Wang; Letian Gao; Liang Zhao; Liansheng Xu; Lihua Guo; Lingxiao Luo; Lingyue Fu; Litao Deng; Litong Wang; Liyue Zhang; Longhao Chen; Lu Chen; Luotian Huang; Luyao Ma; Luyao Wang; M. S. Di; Max Mei; Menghao Ye; Miao Cui; Mingchuan Zhang; Minghua Zhang; Minghui Tang; Mingjing Zhang; Mingqi Wei; Mingshu Chen; Mingxing Liu; Mingxu Zhou; Mingyu Xu; Mingyu Yang; Mingze Wang; Muyang Chen; Ni Shentu; Ning Wang; Niufang Ning; Panpan Huang; Peixin Cong; Peiyi Wang; Peiyuan Xin; Pengfei Ren; Pengfei Yan; Pengle Zhang; Qi Kang; Qi Tang; Qiancheng Wang; Qiang Li; Qihao Zhu; Qingyang Li; Qinyu Chen; Qiushi Du; Qizhou Guo; Rongxian Xu; Rui Ding; Rui Hu; Rui Tian; Rui Yu; Ruidong Zhu; Ruifan Xu; Ruihan Yang; Ruihang Xia; Ruijie Lu; Ruilin Geng; Ruipeng Hong; Ruiqi Ge; Ruisong Zhang; Ruize Sun; Ruizhe Pan; Runji Wang; Runqian Chen; Runxin Xu; Ruohong Tian; Ruomeng Shen; Ruoyu Zhang; Ryan X.; S. H. Liu; Shanghao Lu; Shangyan Zhou; Shanhuang Chen; Shaofei Cai; Shaoheng Nie; Shaoyuan Chen; Shengding Hu; Shengkai Lin; Shengwen Ran; Shengyu Liu; Shengyuan Jia; Shi Bai; Shi Feng; Shicheng Xu; Shichun Liu; Shiqiang Hu; Shirong Ma; Shiyu Wang; Shiyuan Feng; Shufan Gong; Shuhan Lin; Shuiping Yu; Shunfeng Zhou; Shuo Yang; Shuomeng Wang; Shuting Guo; Shuting Pan; Shuying Yu; Sinuo Cao; Siyi Lin; Sizhe Chen; Songyang Chen; Songyang Zhou; Tao Ni; Tao Yun; Tian Jin; Tian Pei; Tian Ye; Tianle Lin; Tianran Ji; Tianyi Cui; Tianyuan Yue; Tingting Yu; Tongrui Xiong; Wangding Zeng; Wei Liu; Wei Zhang; Weibin Xu; Weihao Zeng; Weilin Zhao; Wen Liu; Wenfeng Liang; Wenjie Pang; Wenjing Luo; Wenjing Yao; Wenjun Gao; Wenkai Shao; Wenkai Yang; Wenli Zhang; Wenlu Wang; Wenlve Huang; Wenqian Yan; Wentao Zhang; Xi Gao; Xiang He; Xiang Li; Xiangli Li; Xiangwen Wang; Xiangying Zhang; Xiankui Wei; Xiao Bi; Xiaodong Liu; Xiaohan Wang; Xiaojian Qu; Xiaokang Chen; Xiaokang Zhang; Xiaotao Nie; Xiaoyao Zou; Xiaoyuan Li; Xicheng Guo; Xieting Chu; Xin Cheng; Xin Liu; Xin Xie; Xinbo Xu; Xingchao Liu; Xingchen Liu; Xingkai Yu; Xingyou Li; Xintong Yao; Xinyang Chen; Xinyong Jiang; Xinyu Yang; Xinyu Yang; Xu Chen; Xuanyu Wang; Xubei Zhong; Xuecheng Su; Xuejie Liu; Xuheng Lin; Xujie Fan; Xuncheng Zhao; Xuwei Fu; Y. C. Yan; Y. H. Jiang; Y. T. Wu; Y. W. M.; Y. Z. Wang; Yafei Gao; Yang Yang; Yang Zhang; Yanru Ma; Yanwen Huang; Yao Li; Yao Li; Yao Meng; Yao Zhao; Yaofeng Sun; Yaohui Wang; Yaoyang Ye; Yehang Yin; Yexinrui Wu; Yi Qian; Yi Tao; Yi Yu; Yichao Zhang; Yichen Jiang; Yicheng Wang; Yifan Ding; Yifan Shi; Yifeng Peng; Yifeng Zhai; Yijia Wu; Yiliang Xiong; Yilun Wang; Ying He; Ying Zhou; Yingjia Luo; Yinmin Zhong; Yiping Wang; Yisong Wang; Yixiang Zhang; Yixiao Chen; Yixuan Tan; Yixuan Wei; Yiyang Ma; Yiyao Yang; Yiyuan Liu; Yizai Cai; Yizhen Wei; Yizhi Wang; Yonglun Yang; Yongqi Zhuo; Yongqiang Guo; Yongtong Wu; Yu Wu; Yu Zhang; Yuan Bian; Yuan Cheng; Yuan Ou; Yuan Sun; Yuanfan Xu; Yuanhang Sun; Yuanhao Li; Yuchen Liu; Yuchen Yao; Yudong Han; Yuduan Wang; Yuhan Wu; Yuhao Meng; Yuheng Zou; YuKun Li; Yunchuan Wang; Yunfan Xiao; Yunfan Xiong; Yupeng Chen; Yuqian Cao; Yuqian Wang; Yuqing Chen; Yushun Zhang; Yutong Lin; Yuwei Xiao; Yuxian Gu; Yuxiang Chen; Yuxiang Huang; Yuxiang Luo; Yuxiang You; Yuxin Chen; Yuxin Xiang; Yuxuan Liu; Yuxuan Zhou; Yuyang Zhou; Yuzhe Guo; Yuzhen Huang; Yuzhuo Bai; Z. Y. Z.; Zanlin Ni; Zehao Wang; Zehua Zhao; Zehui Ren; Zejun Zhao; Zhangli Sha; Zhanying Wang; Zhaochen Zhang; Zhaoshuai Du; Zhe Fu; Zhean Xu; Zhenda Xie; Zheng Liu; Zhengyan Zhang; Zhenhua Dong; Zhewen Hao; Zhibang Wang; Zhibin Gou; Zhicheng Ma; Zhihao Li; Zhihong Shao; Zhihuan Huang; Zhijie Li; Zhirui Lu; Zhixian Huang; Zhixuan Chen; Zhixuan Chen; Zhixuan Pan; Zhiyu Wu; Zhizhou Ren; Zhu He; Zhuoshu Li; Zhuping Zhang; Zian Xu; Zihao Wang; Zihui Gu; Zijia Zhu; Zili Zhang; Zilin Li; Zilong Hou; Zilong Lyu; Ziqiao Wang; Ziwei Xie; Ziya Zhang; Ziyi Gao; Zizheng Pan; Zonglin Li; Zongqing Yao; Zui Chen; Zuofan Wu; Chenchen Ling; Chengyu Hou; Chong Chen; D. Li; Di Qi; Dongjie Ji; Fang Wei; Fanyi Xia; Fei Xie; Feiyi Tan; Hailong Guo; Haiyan Zhai; Hui Zhou; Huihui Tan; Huijie Li; Jia Luo; Jia Song; Jialu Cai; Jian Liang; Jiangting Zhou; Jiaqi Gao; Jiayi Shao; Jie Chen; Jieyu Yang; Jin Chen; Jingde Zhang; Jingzi Zhou; Jinqian Wang; Jinyang Liu; JinZhao Sun; Junhua Ling; Junmin Zheng; Kaicheng Yang; Ke Xu; Le Su; Leyi Xia; Liangfeng Ding; Lin Zhuo; Linwang Ma; Linyan Zhu; Liyu Cai; Luqi Yao; M. K. Zhang; Meng Li; Miao Lin; Miaojun Wang; Min Zhang; Mingming Li; Mingming Wang; Mingze Yin; Minmin Han; Nan Cao; Ning Wang; Ningxin Ma; Panpan Wang; Peihan Lin; Peng Sun; Peng Zhang; Qian Ying; Qiang Xiang; Qiao Wang; Qingmiao Mao; Qiwei Jiang; Rongli Jin; Ruyi Chen; Sha Tao; Shangmian Sun; Shaoqing Wu; Shichao Zou; Si Lei; Tianyang Zhang; Tianyu Sun; Tingting Yin; W. L. Xiao; Wei An; Wei Li; Wei Wang; Weiwei Lin; Wenqing Hou; X. Lin; Xiangfei Meng; Xianzhu Huang; Xiao Peng; Xiaoqian Li; Xiaoting Zhang; Xiaowen Sun; Xiaoxiang Wang; Xiaoyu Ye; Xinrou Zhang; Xinyu Zhang; Xue Cao; Xueyin Chen; Yanan Zhou; Yanhong Xu; Yao Xia; Yao Xu; Yi Shao; Yihong Zhang; Yiling Ma; Ying Tang; Yining Lou; Yiru Chen; Yishi Piao; Yixuan Chen; Yong Xiong; Yuchen Xuan; Yuehan Yang; Yuer Xu; Yukun Zha; Yunxian Ma; Yuping Lin; Yuting Yan; Yutong Xie; Yuwen Sheng; Yuxuan Zhu; Zekai Zhang; Zhe Ju; Zhenzhen Lin; Zheren Gao; Zheyang Sun; Zhigang Yan; Zhongyu Wu; Zi Wang; Zihua Qu; Ziling Yan; Ziyi Wan·
模型推理KV Cache
摘要
长视野智能体的广泛采用使得模型工作负载的输入量越来越大。尽管之前的工作已经大幅降低了长上下文计算的成本，但预填充的计算成本仍然很高，而且大型 KV 缓存继续对 HBM 和 SSD 容量以及数据传输带宽造成压力。这些计算、存储和带宽需求共同构成了进一步降低部署成本的主要瓶颈。为了应对这一挑战，我们引入了 DeepSeek-V4.1-Flash，这是一种多模式专家混合 (MoE) 模型，具有 552B 主干参数并支持多达 100 万个词元的上下文。凭借其因果编码器-解码器 (CED) 架构，该模型在解码期间为每个词元激活 16B 个参数，但在预填充期间仅激活 8B 个参数，从而大大提高了代理工作负载的成本效率。为了突破 KV 缓存压缩的限制，DeepSeek-V4.1-Flash 将压缩稀疏注意力 2 (CSA2) 中的跨层 KV 缓存重用与 FP4 KV 缓存相结合。这些设计将其全局 KV 缓存占用空间（始终以 HBM 形式）减少到每个词元 890 字节，大约是 DeepSeek-V4-Flash 相应占用空间的 1/4。此外，通过称为 SWA Bounded Replay 的专用部署优化，DeepSeek-V4.1-Flash 将其持久 KV 缓存占用空间（始终在 SSD 上或主机内存中）减少到 DeepSeek-V4-Flash 的大约 1/8。尽管 KV 缓存占用空间小得多，但该模型提供了比基准更好的性能。此外，我们还简化了 DeepSeek-V4 架构，并引入了多种高效的架构扩展。我们在包含 45T token 的多模态语料库上对 DeepSeek-V4.1-Flash 进行预训练，并进行全面的后训练，在不同的基于文本和多模态代理场景中产生了强大的性能。模型检查点位于 https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash。