论文
Gemma 4 技术报告
Gemma 4 Technical Report
Gemma Team; Sherif El Abd; Vaibhav Aggarwal; Robin Algayres; Alek Andreev; Olivier Bachem; Ian Ballantyne; Cormac Brick; Victor Cărbune; Michelle Casbon; Mayank Chaturvedi; Aditya Chawla; Victor Cotruta; Alice Coucke; Phil Culliton; Robert Dadashi; Lucas Dixon; Mohamed Elhawaty; Utku Evci; Clément Farabet; Johan Ferret; Filippo Galgani; Sertan Girgin; Jean-Bastien Grill; Maarten Grootendorst; Jiaxian Guo; Cassidy Hardin; Yanzhang He; Steven M. Hernandez; Omri Homburger; Léonard Hussenot; Juyeong Ji; Armand Joulin; Aishwarya Kamath; Parnian Kassraie; Olivier Lacombe; Preethi Lahoti; Gaël Liu; Gus Martins; Luciano Martins; Tatiana Matejovicova; Ramona Merhej; Nikola Momchev; Sneha Mondal; Ryan Mullins; Sindhu Raghuram Panyam; Shreya Pathak; Sarah Perrin; André Susano Pinto; Etienne Pot; Angéline Pouget; Alexandre Ramé; Sabela Ramos; Douglas Reid; David Rim; Morgane Rivière; Karsten Roth; Louis Rouillard; Omar Sanseviero; Pier Giuseppe Sessa; Shane Settle; Danila Sinopalnikov; Sara Smoot; Piotr Stanczyk; Andreas Steiner; Lawrence Stewart; Ilya Tolstikhin; Michael Tschannen; Anton Tsitsulin; Nino Vieillard; Renjie Wu; Pingmei Xu; Haichuan Yang; Edouard Yvinec; Biao Zhang; Li Zhang; Joe Zou; Nicolas Aagnes; Abdelrahman Abdelhamed; Jakub Adamek; Shivani Agrawal; Shubham Agrawal; Ibrahim Alabdulmohsin; Jean Baptiste Alayrac; Uri Alon; Chandramouli Amarnath; Ankesh Anand; Chrysovalantis Anastasiou; Setareh Ariafar; François-Xavier Aubet; Kyriakos Axiotis; Federico Barbero; Joelle Barral; Alexei Bendebury; Urs Bergmann; Stanley Bileschi; Kat Black; Mathieu Blondel; Sebastian Borgeaud; Arthur Bražinskas; Ryan Burnell; Robert Busa-Fekete; Mu Cai; Daniele Calandriello; Glenn Cameron; Charlotte Caucheteux; Rahma Chaabouni; Garima Chadha; Jetha Chan; Blake Jianhang Chen; Jesse Chen; Lin Chen; Xu Chen; Derek Cheng; Tzu-hsiang Chien; Nikolai Chinaev; Yi Chou; Zhaohui Chu; Benjamin Coleman; Pooja Consul; Sam Conway-Rahman; Scott Crowell; Dylan Cutler; Vivek Dani; Samira Daruki; Anil Das; Daniel Deutsch; Nishanth Dikkala; Li Ding; Qiuhan Ding; Shenil Dodhia; Konstantin Donhauser; Tulsee Doshi; Anca Dragan; Alex Druinsky; Sahil Dua; Zoltan Egyed; Danielle Eisenbud; Daniel Eppens; Cindy Fan; Bahare Fatemi; Yassir Fathullah; Vlad Feinberg; Milen Ferev; Sebastian Flennerhag; Takumi Fujimoto; João Gabriel Oliveira; Isaac Galatzer-Levy; João Gante; Simon Geisler; Soham Ghosal; Antonious M. Girgis; Tamara von Glehn; Alec Go; Alhaad Gokhale; Alex Grills; Yiming Gu; Mayank Gupta; Pramod Gupta; Guru Guruganesh; Raia Hadsell; Hamza Harkous; Jitendra Harlalka; Demis Hassabis; Anja Hauth; Joe Heyward; Arian Hosseini; Chih-Yang Hsia; I-Hung Hsu; Xiaopeng Huang; Yangsibo Huang; Kevin Hui; Adrian Hutter; Te I; Fotis Iliopoulos; Advait Jain; Ganesh Jawahar; Ziwei Ji; Qilin Jin; Melvin Johnson; Kandarp Joshi; Arun Kandoor; Wang-Cheng Kang; Koray Kavukcuoglu; Mehran Kazemi; Kathleen Kenealy; Amr Khalifa; Phoebe Kirk; Ivan Korotkov; Suraj Kothawade; Vitaly Kovalev; Neel Kovelamudi; Adam Kraft; Ravin Kumar; Vivek Kumar; Harish Kuppam; Justin Lannin; Chen-Yu Lee; Seungji Lee; Dmitry Lepikhin; Alon Levkovitch; Dongdong Li; Qiujia Li; Valentin Liévin; Ethan Lin; Ziqian Lin; Casper Liu; Tianlin Liu; Tianqi Liu; Xin Liu; Ivan Lobov; Mayank Lunayach; Min Ma; Gagan Madan; Andrii Maksai; Eric Malmi; Michal Matuszak; Daniel McDuff; Gaurav Menghani; Maciej Mikuła; Daniil Mirylenka; Karolis Misiunas; Vedant Misra; Andreea Mitran; Kareem Mohamed; Maksim Mukha; Eric Noland; James O'Donnell; Brendan O'Donoghue; Kate Olszewska; Bernett Orlando; Wanqiong Pan; Rina Panigrahy; Unnati Parekh; Nicolas Perez-Nieves; Chunjong Park; Eric Paskie; Liqian Peng; Bryce Petrini; Slav Petrov; Jonas Pfeiffer; Bilal Piot; Martyna Plomecka; Siim Poder; Octavio Ponce; Arijit Pramanik; David Racz; Anish Rajan; Michelle Ramanovich; Anand Rao; Marvin Ritter; Vitor Rodrigues; Evan Rosen; Mikołaj Rybiński; Noveen Sachdeva; Michaël E. Sander; Rohit Sathyanarayana; Sagar Savla; Samuel Schmidgall; Tal Schuster; George Scrivener; Benoit Seguin; Andrew Sellergren; Aliaksei Severyn; Izhak Shafran; Dhruv Shah; Bobak Shahriari; Yuan Shangguan; Ashish Shenoy; Pradeep Shenoy; Rakesh Shivanna; Pauline Sho; Lucas Spangher; Wojciech Stokowiec; Tim Strother; Yao Su; Yinghao Sun; Mukund Sundararajan; Andrea Tacchetti; Mor Hazan Taege; Pouya Tafti; Jean Tarbouriech; Chetan Tekur; Shantanu Thakoor; Rahul Thapa; Madeleine Traverse; Lenart Treven; Tao Tu; Chien Te Tung; Çağlar Ünlü; Petar Veličković; Malini Pooni Venkat; Sagar Gubbi Venkatesh; Vidya Venkiteswaran; Francesco Visin; Alex Vitvitskyi; Kiran Vodrahalli; Weiyi Wang; Xin Wang; Tris Warkentin; Jan Wassenberg; John Wieting; Cindy Wu; Lechao Xiao; Hao Xu; Yuhui Xu; Fuzhao Xue; Arun Yadav; Jun Yan; Antoine Yang; Lin Yang; Ming-Hsuan Yang; Ziyu Ying; Jae Hyeon Yoo; Morteza Zadimoghaddam; Sajjad Zafar; Fred Zhang; Jiageng Zhang; Jianyi Zhang; Xiaofan Zhang; Chao Zhao; David Zhou; Chen Zou·
模型架构Transformer
摘要
我们推出 Gemma 4，它是 Gemma 模型家族中的新一代开放权重、原生多模态语言模型。 Gemma 4 模型套件旨在提高计算效率和推理能力，具有密集的混合专家架构，参数范围从 2.3B 到 31B。除了针对所有模型尺寸改进视觉和音频编码器之外，我们还为我们的 12B 模型提出了一个统一的、无编码器的架构，该架构摄取原始音频和图像补丁。此外，我们集成了一种思维模式，使 Gemma 模型能够在响应之前生成推理痕迹。我们通过关键的设计选择提高推理速度、内存和计算效率以及长上下文能力。 Gemma 4 在 STEM、多模式和长上下文基准方面实现了性能飞跃，并可在人工评估任务中与更大的前沿开放模型相媲美。