论文
Nemotron 3 Ultra：用于代理推理的开放、高效的专家混合 Mamba-Transformer 模型
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
NVIDIA; :; Aaron Blakeman; Aaron Thomas; Aastha Jhunjhunwala; Abhibha Gupta; Abhinav Khattar; Adam Rajfer; Adi Renduchintala; Adil Asif; Aditya Vavre; Adriana Flores Miranda; Ahmad Bilal; Aileen Zaman; Ajay Hotchandani; Akanksha Shukla; Akhiad Bercovich; Aleksander Ficek; Alex Gronskiy; Alex Kondratenko; Alex Steiner; Alex Ye; Alexander Bukharin; Alexandre Milesi; Ali Taghibakhshi; Alice Gatti; Alisa Liu; Alok Kumar; Amar Phanishayee; Ameya Sunil Mahabaleshwarkar; Amir Klein; Amit Zuker; Amnon Geifman; Anahita Bhiwandiwalla; Ananth Subramaniam; Andrea Santilli; Andrew Fulks; Andrew McHarg; Andrew Tao; Andrii Skliar; Anjulie Agrusa; Ankur Srivastava; Ankur Verma; Anna Shors; Anna Warno; Antoni-Joan Solergibert I Llaquet; Arham Mehta; Arkadiusz Nowaczynski; Arti Jain; Ashwath Aithal; Ashwin Poojary; Asif Ahamed; Asit Mishra; Asma Kuriparambil Thekkumpate; Atefeh Sohrabizadeh; Avinash Kaur; Avinash Vem; Ayush Dattagupta; Barath Subramaniam Anandan; Bardiya Sadeghi; Ben Lanir; Benedikt Schifferer; Besmira Nushi; Bilal Kartal; Bill Thiede; Bita Darvish Rouhani; Bo Deng; Bob Schatz; Boris Ginsburg; Boxin Wang; Brad Nemire; Brandon Norick; Brian Dang; Brian Westphal; Brian Yu; Brucek Khailany; Bryan Catanzaro; Carlo del Mundo; Caryln Aarish; Chankyu Lee; Chantal Hwang; Charbel Sakr; Charles Wang; Charlie Truong; Chen Cui; Cheng Cheng; Cheng-Ping Hsieh; Chenghao Zhang; Chenhui Deng; Chintan Patel; Chris Alexiuk; Christian Cosgrove; Christian Munley; Christine Harvey; Christopher Parisien; Chunyang Shen; Coco Li; Collin Neale; Cynthia Gao; Cyril Meurillon; Dan Gil; Dan Su; Dan Zhao; Dane Corneil; Daniel Afrimi; Daniel Egert; Daniel Korzekwa; Daniel Lo; Daniel Machlab; Daniel Serebrenik; Daniil Sorokin; Daria Gitman; Daria Levy; Darko Stosic; David Mosallanezhad; David Yu; Davit Karamyan; Deena Donia; Deep Debroy; Deepak Narayanan; Devin O'Kelly; Dheeraj Peri; Dhruv Nathawani; Di; Wu; Dima Rekesh; Divyanshu Kakwani; Donald Plummer; Dong Anh; Dongfeng Yu; Dongfu Jiang; Donnie Kim; Dorrin Poorkay; Duncan Riach; Dusan Stosic; Dustin VanStee; Eavan Meng; Edgar Minasyan; Edward Lin; Eileen Margaret Peters Long; Elad Sarafin; Elad Segal; Elena Lantz; Ellie Evans; Elliott Ning; Eric Chung; Eric Harper; Eric Pham-Hung; Eric Tramel; Eric Yang; Erick Galinkin; Erik Pounds; Erika Goncalves Goncalves; Evan Briones; Evan Wu; Evelina Bakhturina; Evgeny Tsykunov; Ewa Dobrowolska; Faisal Ladhak; Farzan Memarian; Fay Wang; Fei Jia; Felipe Soares; Felipe Vieira Frujeri; Feng Chen; Fengguang Lin; Ferenc Galko; Frank Sun; Frankie Siino; Frida Hou; Gal Hubara Agam; Gal Kaplun; Gantavya Bhatt; Gargi Prasad; Garvit Kulshreshtha; George Armstrong; Gerald Shen; Giulio Borghesi; Gordana Neskovic; Gorkem Batmaz; Grace Lam; Greg Mason; Greg Pauloski; Grigor Nalbandyan; Grzegorz Chlebus; Grzegorz Karch; Guan-Ting Liu; Guoming Zhang; Guyue Huang; Haggai Maron; Haifeng Qian; Haim Elisha; Haoxing Ren; Haran Kumar Shiv Kumar; Haribhau Hud; Harris Nover; Harrison Saturley Hall; Hayate Iso; Helen Ngo; Herbert Hum; Herman Sahota; Hexin Wang; Himanshu Soni; Hovhannes Tamoyan; Hua Li; Huanhuan Chen; Hui Li; Hui Wang; Huy Nguyen; Ian Chiles; Ido Galil; Ido Shahaf; Igor Gitman; Igor Shovkun; Ilya Loshchilov; Ingo Guehring; Itamar Schen; Itay Levy; Itay Neeman; Ivan Moshkov; Izik Golan; Izzy Putterman; Jaemin Choi; Jakub Slowikowski; Jan Kautz; Jane Polak Scowcroft; Jared Casper; Jatin Mitra; Jeffrey Glick; Jenny Chen; Jesse Oliver; Jiacheng Xu; Jiafan Zhu; Jialin Song; Jian Zhang; Jiantao Jiao; Jiaqi Zeng; Jie Lou; Jim King; Jimmy Zhang; Jingquan Wang; Jinhang Choi; Jinju Chu; Joey Conway; Joey Guman; Johan Jatko; Johannes Rausch; John Kamalu; John Roberts; Johnny Greco; Johnny Mensel; Jonah Alben; Jonas Yang; Jonathan Cohen; Jonathan Raiman; Joseph Jennings; Joshua Mabry; Joshua Pierce; Joyjit Daw; Julien Veron Vialard; Junkeun Yi; Jupinder Parmar; Kajal Jain; Kan Zhu; Kari Briski; Katherine Cheung; Katherine Luna; Keith Willowhawk; Keith Wyss; Keshav Santhanam; Kevin Shih; Kezhi Kong; Khanh Nguyen; Khushi Bhardwaj; Kirthi Shankar Sivamani; Konstantinos Krommydas; Krishna C. Puvvada; Krzysztof Pawelec; Kumar Anik; Kyle Keprios; Kylie Day; Lawrence McAfee; Leo Du; Leon Derczynski; Li Ding; Linda Liu; Lingjie Wu; Lior Kadoch; Lizzie Wei; Luis Vega; Luke Robison; Lun Su; Maarten Van Segbroeck; Maciej Jakub Mikulski; Maer Rodrigues de Melo; Magda Sypula; Mahan Fathi; Makesh Narsimhan Sreedhar; Makesh Tarun Chandran; Manoj Kilaru; Maor Ashkenazi; Marc Cuevas; Marc Romeijn; Marcin Chochowski; Mark Cai; Mark Mozolewski; Markus Kliegl; Marta Stepniewska-Dziubinska; Martyna Patelka; Mattei Machczynski; Matvei Novikov; Mauricio Ferrato; Maximilian Golub; Mehrzad Samadi; Melissa Corpuz; Mengru Wang; Mengxi Wu; Meredith Price; Meriem Boubdir; Micah Schaffer; Michael Andersch; Michael Boone; Michael Gschwind; Michael Lightstone; Michael Loh; Michal Bien; Michal Zawalski; Michelle Gill; Miguel Martinez; Mikail Khona; Mike Chrzanowski; Mike Houston; Mingyuan Ma; Minseok Lee; Mohamed Fawzy; Mohammad Dabbah; Mohammad Shoeybi; Mostofa Patwary; Nabin Mulepati; Najeeb Nabwani; Namit Dhameja; Narimane Hennouni; Natalie Hereth; Nathaniel Pinckney; Nave Algarici; Nave Assaf; Netanel Haber; Nicholas Knight; Nick Reamaroon; Nickson Quak; Nidhi Bhatia; Nikhil Desai; Nikolai Ludwig; Nima Tajbakhsh; Ning Xu; Nir Ailon; Nirmal Juluru; Nitin Nitin; Ofri Masad; Oleg Rybakov; Oleksii Hrinchuk; Oleksii Kuchaiev; Olivia Viessmann; Olivier Delalleau; Oluwatobi Olabiyi; Omer Ullman Argov; Omri Puny; Oren Tropp; Pablo Ribalta; Pallab Bhattacharya; Panos Lampropoulos; Parth Mannan; Pasha Shamis; Patrick Legresley; Paul Gibbons; Pavlo Molchanov; Pawel Morkisz; Peter Dykas; Peter Jin; Pierre-Yves Aquilanti; Pinky Xu; Piotr Januszewski; Piotr Laskiewicz; Pooya Jannaty; Prakash Gurumurthy; Pranav Prashant Thombre; Prasoon Varshney; Pritam Gundecha; Przemek Tredak; Puhui Meng; Qiyu Wan; Rabeeh Karimi Mahabadi; Rachel Oberman; Rachit Garg; Radha Sri-Tharan; Rahul Kandu; Rakshit Sanadhya; Ran El-Yaniv; Ran Zilberstein; Rasoul Shafipour; Ray Macalisang; Rayen Tian; Reka Kovacs; Renjie Pi; Rick Izzo; Rima Shahbazyan; Rishabh Garg; Rishi Puri; Rita Fernandes Neves; Ritchie Zhao; Ritika Borkar; Ritu Gala; Riyad Islam; Robert Clark; Robert Hesse; Robert Kirby; Roger Waleffe; Rohit Watve; Roi Koren; Ron Banner; Ruoxi Zhang; Russell J. Hewett; Ryan Prenger; Ryan Stewart; Ryota Egashira; Sadegh Mahdavi; Saee Paliwal; Sagar Singh; Sahil Modi; Salika Dave; Samantha Shinagawa; Samuel Kriman; Sandip Bhaskar; Sangkug Lym; Sanjay Kariyappa; Sanjeev Satheesh; Saran Vikas Murari; Satish Pasumarthi; Saurabh Mishra; Saurav Muralidharan; Scott Hara; Sean Narentharen; Selvaraj Anandaraj; Seonjin Na; Seonmeyong Bak; Seonmyeong Bak; Sepehr Sameni; Seph Mard; Serge Panev; Seth Henneman; Seth Poulos; Shahar Mor; Shantanu Acharya; Shaona Ghosh; Sharath Turuvekere Sreenivas; Sharon Mendelson; Shaun Kotek; Shawn Wang; Shay Aharon; Shaya Gharghabi; Sheng-Chieh Lin; Shi Chen; Shiqing Fan; Shirish Baskaran; Shreya Gopa; Shrimai Prabhumoye; Shubham Pachori; Shubham Toshniwal; Shuoyang Ding; Shwetha Krishnamurthy; Siddharth Singh; Simeng Sun; Sirshak Das; Sivakumar Arayandi Thottakara; Smita Ithape; Somshubra Majumdar; Soumye Singhal; Sri Harsha Singudasu; Sridhar Bhuvanapalli; Srimukh Veccham; Stas Sergienko; Stefania Alborghetti; Stephen Ge; Su Rong; Sugam Dipak Devare; Sukrit Rao; Sumeet Kumar Barua; Sungsoo Ha; Sunny Gai; Suriya Gunasekar; Suseella Panguluri; Suyog Gupta; Sviataslau Hinzburh; Sweta Priyadarshi; Syeda Nahida Akter; Talor Abramovich; Tan Bui; Tanay Varshney; Tatevik Ter-Hovhannisyan; Teodor-Dumitru Ene; Terry Kong; Thanh Do; Tianhe Zhang; Tiffany Moore; Tijmen Blankevoort; Tim Moon; Tiyasa Mitra; Tom Balough; Tomasz Grzegorzek; Tomasz Hliwiak; Tomer Asida; Tomer Bar Natan; Tomer Keren; Tomer Ronen; Tony Salim; Tony Wang; Traian Rebedea; Tugrul Konuk; Twinkle Vashishth; Udi Karpas; Ushnish De; Vahid Noorozi; Venkat Srinivasan; Venmugil Elango; Vibhor Agrawal; Victor Cui; Vijay Korthikanti; Vikas Mehta; Vinay Rao; Virginia Wu; Vitaly Kurin; Vitaly Lavrukhin; Vladimir Anisimov; Vu Pham; Wanli Jiang; Wasi Uddin Ahmad; Wataru Ishihara; Wei Du; Wei Ping; Weiheng Chai; Wenliang Dai; Wesley Helmholz; Will Jennings; Will Zhu; Wojciech Prazuch; Xiaowei Ren; Xiwen Yu; Yan Breek; Yang Chen; Yang Yu; Yangyi Chen; Yaniv Galron; Yashaswi Karnati; Yejin Choi; Yev Meyer; Yi-Fu Wu; Yian Zhang; Ying Lin; Yonatan Geifman; Yonggan Fu; Youngeun Kwon; Yu Yao; Yugi Guvvla; Yuki Huang; Yunsheng Liu; Zach Moshe; Zachary Newell; Zhilin Wang; Zhiyu Li; Zhongbo Zhu; Zhuolin Yang; Zihan Liu; Zijie Yan; Zsolt-Alon Wertheimer·
摘要
我们推出了 Nemotron 3 Ultra，这是一个总量为 5500 亿、活跃参数为 550 亿的 Mixture-of-Experts 混合 Mamba-Attention 语言模型。我们在 20 万亿个文本标记上对 Nemotron 3 Ultra 进行预训练，然后将上下文长度扩展到 1M 个标记，并使用监督 微调 (SFT)、强化学习 (RL) 和多教师 同策略 蒸馏 (MOPD) 进行后训练。 Nemotron 3 Ultra 是我们迄今为止最强大的模型，采用了多项关键技术 - LatentMoE、多词元预测 (MTP)、NVFP4 预训练、多环境 RLVR、MOPD 和推理预算控制。与最先进的公开 LLM 相比，Nemotron 3 Ultra 的推理吞吐量提高了约 6 倍，同时达到了同等的精度。最先进的准确性、高推理吞吐量和 1M 词元上下文长度使 Nemotron 3 Ultra 成为长时间运行的自主代理任务的理想选择。我们开源了基础、后训练和量化检查点，以及 HuggingFace 上的训练数据和配方。