论文
Nemotron 3 Super：面向Agent推理的开放MoE混合Mamba—Transformer模型
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
NVIDIA; :; Aakshita Chandiramani; Aaron Blakeman; Abdullahi Olaoye; Abhibha Gupta; Abhilash Somasamudramath; Abhinav Khattar; Adeola Adesoba; Adi Renduchintala; Adil Asif; Aditya Agrawal; Aditya Vavre; Ahmad Kiswani; Aishwarya Padmakumar; Ajay Hotchandani; Akanksha Shukla; Akhiad Bercovich; Aleksander Ficek; Aleksandr Shaposhnikov; Alex Gronskiy; Alex Kondratenko; Alex Neefus; Alex Steiner; Alex Yang; Alexander Bukharin; Alexander Young; Ali Hatamizadeh; Ali Taghibakhshi; Alina Galiautdinova; Alisa Liu; Alok Kumar; Ameya Sunil Mahabaleshwarkar; Amir Klein; Amit Zuker; Amnon Geifman; Anahita Bhiwandiwalla; Ananth Subramaniam; Andrew Tao; Anjaney Shrivastava; Anjulie Agrusa; Ankur Srivastava; Ankur Verma; Ann Guan; Anna Shors; Annamalai Chockalingam; Anubhav Mandarwal; Aparnaa Ramani; Arham Mehta; Arti Jain; Arun Venkatesan; Asha Anoosheh; Ashwath Aithal; Ashwin Poojary; Asif Ahamed; Asit Mishra; Asli Sabanci Demiroz; Asma Kuriparambil Thekkumpate; Atefeh Sohrabizadeh; Avinash Kaur; Ayush Dattagupta; Barath Subramaniam Anandan; Bardiya Sadeghi; Barnaby Simkin; Ben Lanir; Benedikt Schifferer; Benjamin Chislett; Besmira Nushi; Bilal Kartal; Bill Thiede; Bita Darvish Rouhani; Bobby Chen; Boris Ginsburg; Brandon Norick; Branislav Kisacanin; Brian Yu; Bryan Catanzaro; Buvaneswari Mani; Carlo del Mundo; Chankyu Lee; Chanran Kim; Chantal Hwang; Chao Ni; Charles Wang; Charlie Truong; Cheng-Ping Hsieh; Chenhan Yu; Chenjie Luo; Cherie Wang; Chetan Mungekar; Chintan Patel; Chris Alexiuk; Chris Holguin; Chris Wing; Christian Munley; Christopher Parisien; Chuck Desai; Chunyang Sheng; Collin Neale; Cyril Meurillon; Dakshi Kumar; Dan Gil; Dan Su; Dane Corneil; Daniel Afrimi; Daniel Burkhardt Eliuth Triana; Daniel Egert; Daniel Fatade; Daniel Lo; Daniel Rohrer; Daniel Serebrenik; Daniil Sorokin; Daria Gitman; Daria Levy; Darko Stosic; David Edelsohn; David Messina; David Mosallanezhad; David Tamok; Deena Donia; Deepak Narayanan; Devin O'Kelly; Dheeraj Peri; Dhruv Nathawani; Di Wu; Dima Rekesh; Dina Yared; Divyanshu Kakwani; Dmitry Konyagin Brandon Tuttle; Dong Ahn; Dongfu Jiang; Dorrin Poorkay; Douglas O'Flaherty; Duncan Riach; Dusan Stosic; Dustin Van Stee; Edgar Minasyan; Edward Lin; Eileen Peters Long; Elad Segal; Elena Lantz; Elena Lewis; Ellie Evans; Elliott Ning; Eric Chung; Eric Harper; Eric Pham-Hung; Eric W. Tramel; Erick Galinkin; Erik Pounds; Esti Etrog; Evan Briones; Evan Wu; Evelina Bakhturina; Evgeny Tsykunov; Ewa Dobrowolska; Farshad Saberi Movahed; Farzan Memarian; Fay Wang; Fei Jia; Felipe Soares; Felipe Vieira Frujeri; Feng Chen; Fengguang Lin; Ferenc Galko; Fortuna Zhang; Frankie Siino; Frida Hou; Gantavya Bhatt; Gargi Prasad; Geethapriya Venkataramani; Geetika Gupta; George Armstrong; Gerald Shen; Giulio Borghesi; Gordana Neskovic; Gorkem Batmaz; Grace Lam; Grace Wu; Greg Pauloski; Greyson Davis; Grigor Nalbandyan; Guoming Zhang; Guy Farber; Guyue Huang; Haifeng Qian; Haran Kumar Shiv Kumar; Harry Kim; Harsh Sharma; Hayate Iso; Hayley Ross; Herbert Hum; Herman Sahota; Hexin Wang; Himanshu Soni; Hiren Upadhyay; Huy Nguyen; Iain Cunningham; Ido Galil; Ido Shahaf; Igino Padovani; Igor Gitman; Igor Shovkun; Ikroop Dhillon; Ilya Loshchilov; Ingrid Kelly; Itamar Schen; Itay Levy; Ivan Moshkov; Izik Golan; Izzy Putterman; Jain Tu; Jan Baczek; Jan Kautz; Jane Polak Scowcroft; Janica Rosenberg; Jared Casper; Jarrod Pflum; Jason Grant; Jason Sewall; Jatin Mitra; Jeffrey Glick; Jenny Chen; Jesse Oliver; Jiacheng Xu; Jiafan Zhu; Jialin Song; Jian Zhang; Jiaqi Zeng; Jie Lou; Jill Milton; Jim Chow; Jimmy Zhang; Jinhang Choi; Jining Huang; Jocelyn Huang; Joel Caruso; Joey Conway; Joey Guman; Johan Jatko; John Kamalu; Johnny Greco; Jonathan Cohen; Jonathan Raiman; Joseph Jennings; Joyjit Daw; Juan Yu; Julio Tapia; Junkeun Yi; Jupinder Parmar; Jyothi Achar; Kari Briski; Kartik Mattoo; Katherine Cheung; Katherine Luna; Keith Wyss; Kevin Shih; Kezhi Kong; Khanh Nguyen; Khushi Bhardwaj; Kirill Buryak; Kirthi Shankar Sivamani; Konstantinos Krommydas; Kris Murphy; Krishna C. Puvvada; Krzysztof Pawelec; Kumar Anik; Laikh Tewari; Laya Sleiman; Leo Du; Leon Derczynski; Li Ding; Lilach Ilan; Lingjie Wu; Lizzie Wei; Luis Vega; Lun Su; Maarten Van Segbroeck; Maer Rodrigues de Melo; Magaret Zhang; Mahan Fathi; Makesh Narsimhan Sreedhar; Makesh Sreedhar; Makesh Tarun Chandran; Manuel Reyes Gomez; Maor Ashkenazi; Marc Cuevas; Marc Romeijn; Margaret Zhang; Mark Cai; Mark Gabel; Markus Kliegl; Martyna Patelka; Maryam Moosaei; Matthew Varacalli; Matvei Novikov; Mauricio Ferrato; Mehrzad Samadi; Melissa Corpuz; Meng Xin; Mengdi Wang; Mengru Wang; Meredith Price; Micah Schaffer; Michael Andersch; Michael Boone; Michael Evans; Michael Z Wang; Miguel Martinez; Mikail Khona; Mike Chrzanowski; Mike Hollinger; Mingyuan Ma; Minseok Lee; Mohammad Dabbah; Mohammad Shoeybi; Mostofa Patwary; Nabin Mulepati; Nader Khalil; Najeeb Nabwani; Nancy Agarwal; Nanthini Balasubramaniam; Narimane Hennouni; Narsi Kodukula; Natalie Hereth; Nathaniel Pinckney; Nave Assaf; Negar Habibi; Nestor Qin; Neta Zmora; Netanel Haber; Nick Reamaroon; Nickson Quak; Nidhi Bhatia; Nikhil Jukar; Nikki Pope; Nikolai Ludwig; Nima Tajbakhsh; Nir Ailon; Nirmal Juluru; Nirmalya De; Nowel Pitt; Oleg Rybakov; Oleksii Hrinchuk; Oleksii Kuchaiev; Olivier Delalleau; Oluwatobi Olabiyi; Omer Ullman Argov; Omri Almog; Omri Puny; Oren Tropp; Otavio Padovani; Ouye Xie; Parth Chadha; Pasha Shamis; Paul Gibbons; Pavlo Molchanov; Peter Belcak; Peter Jin; Pinky Xu; Piotr Januszewski; Pooya Jannaty; Prachi Shevate; Pradeep Thalasta; Pranav Prashant Thombre; Prasoon Varshney; Prerana Gambhir; Pritam Gundecha; Przemek Tredak; Qing Miao; Qiyu Wan; Quan Tran Minh; Rabeeh Karimi Mahabadi; Rachel Oberman; Rachit Garg; Rahul Kandu; Raina Zhong; Ran El-Yaniv; Ran Zilberstein; Rasoul Shafipour; Renee Yao; Renjie Pi; Richard Mazzarese; Richard Wang; Rick Izzo; Ridhima Singla; Rima Shahbazyan; Rishabh Garg; Ritika Borkar; Ritu Gala; Riyad Islam; Robert Clark; Robert Hesse; Roger Waleffe; Rohit Varma Kalidindi; Rohit Watve; Roi Koren; Ron Fan; Ruchika Kharwar; Ruisi Cai; Ruoxi Zhang; Russell J. Hewett; Ryan Prenger; Ryan Timbrook; Ryota Egashira; Sadegh Mahdavi; Sagar Singh Ashutosh Joshi; Sahil Modi; Samuel Kriman; Sandeep Pombra; Sanjay Kariyappa; Sanjeev Satheesh; Santiago Pombo; Saori Kaji; Satish Pasumarthi; Saurav Mishra; Saurav Muralidharan; Scott Hara; Sean Narenthiran; Sebastian Rogawski; Seonjin Na; Seonmyeong Bak; Sepehr Sameni; Seth Poulos; Shahar Mor; Shantanu Acharya; Shaona Ghosh Adam Lord; Sharath Turuvekere Sreenivas; Shaun Kotek; Shaya Gharghabi; Shelby Thomas; Sheng-Chieh Lin; Shibani Likhite; Shiqing Fan; Shiyang Chen; Shreya Gopal; Shrimai Prabhumoye; Shubham Pachori; Shubham Toshniwal; Shuo Zhang; Shuoyang Ding; Shyam Renjith; Shyamala Prayaga; Siddhartha Jain; Simeng Sun; Sirisha Rella; Sirshak Das; Smita Ithape; Sneha Harishchandra S; Somshubra Majumdar; Soumye Singhal; Sri Harsha Singudasu; Sriharsha Niverty; Stas Sergienko; Stefana Gloginic; Stefania Alborghetti; Stephen Ge; Stephen McCullough; Sugam Dipak Devare; Suguna Varshini Velury; Sukrit Rao; Sumeet Kumar Barua; Sunny Gai; Suseella Panguluri; Sushil Koundinyan; Swathi Patnam; Sweta Priyadarshi; Swetha Bhendigeri; Syeda Nahida Akter; Sylendran Arunagiri; Tailling Yuan; Talor Abramovich; Tan Bui; Tan Yu; Terry Kong; Thanh Do; Thomas Gburek; Thorgane Marques; Tiffany Moore; Tijmen Blankevoort; Tim Moon; Timothy Ma; Tiyasa Mitra; Tomasz Grzegorzek; Tomer Asida; Tomer Bar Natan; Tomer Keren; Tomer Ronen; Traian Rebedea; Trenton Starkey; Tugrul Konuk; Twinkle Vashishth; Tyler Condensa; Udi Karpas; Ushnish De; Vahid Noorozi; Vahid Noroozi; Vanshil Atul Shah; Veena Vaidyanathan; Venkat Srinivasan; Venmugil Elango; Victor Cui; Vijay Korthikanti; Vikas Mehta; Virginia Adams; Virginia Wu; Vitaly Kurin; Vitaly Lavrukhin; Vladimir Anisimov; Wan Seo; Wanli Jiang; Wasi Uddin Ahmad; Wei Du; Wei Ping; Wei-Ming Chen; Wendy Quan; Wenliang Dai; Wenwen Gao; Will Jennings; William Zhang; Xiaowei Ren; Xiaowen Xin; Xin Li; Yang Yu; Yangyi Chen; Yaniv Galron; Yashaswi Karnati; Yejin Choi; Yev Meyer; Yi-Fu Wu; Yian Zhang; Ying Lin; Yonatan Geifman; Yonggan Fu; Yoshi Suhara; Youngeun Kwon; Yuan Zhang; Yuki Huang; Zach Moshe; Zhilin Wang; Zhiyu Cheng; Zhongbo Zhu; Zhuolin Yang; Zihan Liu; Zijia Chen; Zijie Yan; Zuhair Ahmed·
模型架构MoE
摘要
我们介绍Nemotron 3 Super的预训练、后训练与量化。模型为1200亿总参数、120亿激活参数的Mamba—注意力混合MoE，是Nemotron 3系列中首个采用NVFP4预训练、使用同时优化每FLOP和每参数准确率的LatentMoE、并加入MTP层以原生推测解码加速推理的模型。模型先以25万亿词元预训练，再使用监督微调和强化学习进行后训练。最终支持最长100万词元上下文，在常见基准上达到可比准确率，且相对GPT-OSS-120B与Qwen3.5-122B，推理吞吐最高分别提升2.2倍和7.5倍。数据集及基础、后训练、量化检查点均在HuggingFace开放。