论文
Cosmos 3：物理 AI 的全模态世界模型
Cosmos 3: Omnimodal World Models for Physical AI
NVIDIA; :; Aditi; Niket Agarwal; Arslan Ali; Jon Allen; Martin Antolini; Adeline Aubame; Alisson Azzolini; Junjie Bai; Maciej Bala; Yogesh Balaji; Josh Bapst; Aarti Basant; Mukesh Beladiya; Mohammad Qazim Bhat; Zaid Pervaiz Bhat; Dan Blick; Vanni Brighella; Han Cai; Tiffany Cai; Eric Cameracci; Jiaxin Cao; Yulong Cao; Mark Carlson; Carlos Casanova; Ting-Yun Chang; Yan Chang; Yu-Wei Chao; Prithvijit Chattopadhyay; Roshan Chaudhari; Chieh-Yun Chen; Junyu Chen; Ke Chen; Qizhi Chen; Wenkai Chen; Xiaotong Chen; Yu Chen; An-Chieh Cheng; Click Cheng; Xiu Chia; Jeana Choi; Chaeyeon Chung; Wenyan Cong; Yin Cui; Magdalena Dadela; Nalin Dadhich; Wenliang Dai; Joyjit Daw; Alperen Degirmenci; Rodrigo Vieira Del Monte; Robert Denomme; Sameer Dharur; Marco Di Lucca; Ke Ding; Wenhao Ding; Yifan Ding; Yuzhu Dong; Nicole Drumheller; Yilun Du; Aigul Dzhumamuratova; Aleksandr Efitorov; Hamid Eghbalzadeh; Naomi Eigbe; Imad El Hanafi; Hassan Eslami; Benedikt Falk; Jiaojiao Fan; Jim Fan; Amol Fasale; Sergiy Fefilatyev; Liang Feng; Francesco Ferroni; Sanja Fidler; Xiao Fu; Vikram Fugro; Prashant Gaikwad; TJ Galda; Katelyn Gao; Yihuai Gao; Wenhang Ge; Sreyan Ghosh; Arushi Goel; Vivek Goel; Akash Gokul; Rama Govindaraju; Jinwei Gu; Miguel Guerrero; Elfie Guo; Aryaman Gupta; Siddharth Gururani; Hugo Hadfield; Song Han; Ankur Handa; Zekun Hao; Mohammad Harrim; Ali Hassani; Nathan Hayes-Roth; Yufan He; Chris Helvig; Cyrus Hogg; Madison Huang; Michael Huang; Sophia Huang; Yufan Huang; Jacob Huffman; DeLesley Hutchins; Suneel Indupuru; Boris Ivanovic; Arihant Jain; Joel Jang; Ryan Ji; Yanan Jian; Dongfu Jiang; Jingyi Jin; Atharva Joshi; Nikhilesh Joshi; Pranjali Joshi; Andy Ju; Jaehun Jung; Weiwei Kang; Scott Kassekert; Jan Kautz; Ashna Khetan; Julia Kiczka; Slawek Kierat; Gwanghyun Kim; Kuno Kim; Sunny Kim; Kezhi Kong; Xin Kong; Zhifeng Kong; Tomasz Kornuta; Egor Krivov; Hui Kuang; Saurav Kumar; Chia-Wen Kuo; George Kurian; Wojciech Kutak; JF Lafleche; Himangshu Lahkar; Omar Laymoun; Jayjun Lee; Sanggil Lee; Gabriele Leone; Boyi Li; Freya Li; Jiajun Li; Jinfeng Li; Ling Li; Pengcheng Li; Shangru Li; Tingle Li; Xiaolong Li; Xuan Li; Zhaoshuo Li; Zhiqi Li; Hao Liang; Maosheng Liao; Chen-Hsuan Lin; Tsung-Yi Lin; Ming-Yu Liu; Sifei Liu; Zihan Liu; Hai Loc Lu; Xiangyu Lu; Alice Luo; Ruipu Luo; Wenjie Luo; Jiangran Lyu; Martin Ding Ma; Nic Ma; Qianli Ma; Dawid Majchrowski; Louis Marcoux; Miguel Martin; Qing Miao; Ashkan Mirzaei; Shreyas Misra; Kaichun Mo; Durra Mohsin; Hyejin Moon; Pawel Morkisz; Saeid Motiian; Kirill Motkov; Seungjun Nah; Yashraj Narang; Deepak Narayanan; Thabang Ngazimbi; Julian Ouyang; Shubham Pachori; David Page; Yatian Pang; Sehwi Park; Mahesh Patekar; Mostofa Patwary; Marco Pavone; Trung Pham; Wei Ping; Soha Pouya; Shrimai Prabhumoye; Varun Praveen; Delin Qu; Hesam Rabeti; Morteza Ramezanali; Marilyn Reeb; Xuanchi Ren; Kristen Rumley; Wojciech Rymer; Jun Saito; Yeongho Seol; John Shao; Piyush Shekdar; Tianwei Shen; Humphrey Shi; Min Shi; Stella Shi; Kevin Shih; Mohammad Shoeybi; Mateusz Sieniawski; Shuran Song; Alexander Sotelo; Amir Sotoodeh; Sunil Srinivasa; Vignesh Srinivasakumar; Bartosz Stefaniak; Rahul Heinrich Steiger; Shangkun Sun; Jiaxiang Tang; Shitao Tang; Yangyang Tang; Yue Tang; Tolou Tavakkoli; Kayley Ting; Krzysztof Tomala; Wei-Cheng Tseng; Jibin Varghese; Sergei Vasilev; Thomas Volk; Raju Wagwani; Roger Waleffe; Andrew Z. Wang; Boxiang Wang; Haoxiang Wang; Qiao Wang; Shihao Wang; Shijie Wang; Ting-Chun Wang; Yan Wang; Yu Wang; Rohit Watve; David Wehr; Fangyin Wei; Xinshuo Weng; Jay Zhangjie Wu; Kedi Wu; Hongchi Xia; Summer Xiao; Tianjun Xiao; Kevin Xie; Daguang Xu; Jiashu Xu; Mengyao Xu; Ruqing Xu; Xingqian Xu; Yao Xu; Dinghao Yang; Dong Yang; Hans Yang; Xiaodong Yang; Xuning Yang; Yichu Yang; Yurong You; Zhiding Yu; Hao Yuan; Simon Yuen; Xiaohui Zeng; Pengcuo Zeren; Cindy Zha; Haotian Zhang; Jenny Zhang; Jing Zhang; Liangkai Zhang; Paris Zhang; Shun Zhang; Xuanmeng Zhang; Zhizheng Zhang; Ann Zhao; Yilin Zhao; Yuliya Zhautouskaya; Charles Zhou; Fengzhe Zhou; Shilin Zhu; Yuke Zhu; Dima Zhylko; Artur Zolkowski·
模型架构混合架构
摘要
我们推出了 Cosmos 3，这是一系列全模态世界模型，旨在在统一的 Transformer 架构混合中联合处理和生成语言、图像、视频、音频和动作序列。通过支持高度灵活的输入输出配置，Cosmos 3 无缝地统一了物理 AI 的关键模式——有效地将视觉语言模型、视频生成器、世界模拟器和世界动作模型纳入一个框架中。我们的评估表明，Cosmos 3 在一系列不同的理解和生成任务中建立了一种新的最先进技术，展示了全模态世界模型作为实体代理的可扩展、通用的骨干。在撰写技术报告时，我们训练后的 Cosmos 3 模型被 Artificial Analysis 评为最佳开源文本到图像和图像到视频模型，并被 RoboArena 评为最佳策略模型。为了加速物理 AI 的开放研究和部署，我们根据 Linux 基金会的 OpenMDW-1.1 许可证（https://github.com/nvidia/cosmos 和 https://huggingface.co/collections/nvidia/cosmos3）提供代码、模型检查点、精选合成数据集和评估基准。该项目网站位于 https://research.nvidia.com/labs/cosmos-lab/cosmos3。