论文
智能体的期末考试
Agents' Last Exam
Yiyou Sun; Xinyang Han; Weichen Zhang; Yuanbo Pang; Tianyu Wang; Yuhan Cao; Yixiao Huang; Chris Duroiu; Haoyun Zhang; Jeffrey Lin; Weishu Zhang; Tyler Zeng; Ying Yan; Bo Liu; Hanson Wen; Mingyang Xu; Xiaoyuan Liu; Zimeng Chen; Weiyan Shi; Amanda Dsouza; Vincent Sunn Chen; Patrick Bryant; Carl Boettiger; Yamini Rangan; Bradley Rothenberg; Kyle Steinfeld; Arvind Rao; Tapio Schneider; Georgios Yannakakis; Laure Zanna; Kaan Ozbay; Ida Sim; Tarek Zohdi; George Em Karniadakis; Jack Gallant; Teresa Head-Gordon; Yushan Li; Wenxi Deng; Tao Sun; Huiqi Wang; Zhun Wang; Justin Xu; Chris Yuhao Liu; Yafei Cheng; Rongwang Hu; Aras Bacho; Shengcao Cao; Zengyi Qin; Yixiong Chen; Hengduan Fan; Hao Liu; Lin Zeng; Shashank Muralidhar Bharadwaj; Litian Gong; Yingxuan Yang; Maojia Song; Ruheng Wang; Zongzheng Zhang; Honglin Bao; Shuo Lu; Jianhong Tu; Zhonghua Wang; Zheng Zhang; Zijiao Chen; Yanqiong Jiang; Zhendong Li; Bohan Lyu; Chang Ma; Peiran Xu; Benran Zhang; Shangding Gu; Haoyue Hua; Haoyang Li; Wanzhe Liao; Chengzhi Liu; Junbo Peng; Haoran Sun; Zechen Xu; Bo Chen; Jiayi Cheng; Yi Jiang; Keying Kuang; Yuan Li; Youbang Pan; Ziyan Rao; Alexander Schubert; Yifan Shen; Vincent Siu; Xiatao Sun; Kangqi Zhang; Xiaopan Zhang; Yuchen Zhu; Ishaan Singh Chandok; Lei Ding; Jingxuan Fan; Andrew Glover; Jiaming Hu; Yiran Hu; Wenbo Huang; Zixin Jiang; Haoran Jin; Lukas Kim; Ming Liu; Yang Liu; Alireza Rafiei; Xuhuan Shen; Kunyang Sun; Sophia Sun; Ting Sun; Eric Wang; Yixin Wang; Hanwen Xing; Sihan Xu; Yuzheng Xu; Zhongxing Xu; Zhiling Yan; Boqin Yuan; Ruiqi Zhang; Yifan Zhang; Zibo Zhao; Liana; Santanu Bosu Antu; Haoyue Bai; Carlo Bosio; Joseph Cavanagh; Patricia Cavazos-Rehg; Tianxing Chen; Xuewen Chen; Yipu Chen; Chenyu Zhu; Chen Dai; Stefano De Castro; Yunfu Deng; Kaustubh Dhole; Jiayuan Ding; Chenchen Du; Zhehang Du; Hao Fan; Run-Ze Fan; Hengyu Fu; Shi Gu; Yifan Gu; Charlie Guo; Baihe Huang; Baixiang Huang; Rimika Jaiswal; Zhihan Jiang; Ran Jin; Erin Kasson; Xin Lan; Joseph Lee; Deren Lei; Chenyu Li; Daofeng Li; Haitao Li; Hongwei Li; Jingyan Li; Xiao Li; Yi Li; Yinsheng Li; Yuangang Li; Zhixu Li; Wenyu Liang; Longtai Liao; Kevin Qinghong Lin; Andy Zeyi Liu; Che Liu; Jiaming Liu; Kaiyuan Liu; Xuan Liu; Pan Lu; Wenbo Lv; Yicheng Lyu; Qiuyang Mang; Kyle Montgomery; Yuzhou Nie; Ruoxi Ning; Jorin Overwiening; Xu Pan; Layna Paraboschi; Core Francisco Park; Justin Purnomo; Swati Rajwal; Scott Rankin; Bixuan Ren; Yiren Rong; HaoYang Shang; Ventus Shaw; Fiona Shen; Jiawei Shen; Minqi Shi; Shi Qiu; Huaxiu Yao; Tianneng Shi; Jonah So; Vladislav Susoy; Hannah Szlyk; Haocheng Wang; Jialu Wang; Wei Wang; Xinyu Wang; Zehao Wang; Dowling Wong; Angela Wu; Dehao Wu; Fangyu Wu; Mengyuan "Millie" Wu; Yu Wu; Yuchen Wu; Yuhao Wu; Qingpo Wuwu; Weihang Xiao; Yongyi Xiong; Fan Xu; Ruiling Xu; Mingxuan Yan; Benjamin Yang; Jirong Yang; Sen Yang; Xiaoli Yang; Yushi Yang; Haoran Ye; Xiaohu Yu; Zhengming Yu; Chenlong Zhang; Chi Zhang; Hanning Zhang; Hanwen Zhang; Junge Zhang; Kunpeng Zhang; Song Zhang; Wenjin Zhang; Wenshuo Zhang; Ying Zhang; Yizhi Zhang; Brian Zhao; Qijian Zhao; Yimin Zhao; Yuhaohua Zheng; Liwei Zhou; Tianyue Zhou; Sichen Zhu; Siqi Zhu; Yan Zhu; Yishu Zhu; Jierui Zuo; Chonghao Cai; Helena Casademunt; Wenjia Chen; Cheng Cheng; Nawen Deng; Rao Fu; Tianfu Fu; Yifan Han; He Ren; Zhenyu He; Qiao Jin; Langlang Li; Yuetai Li; Sylvia Liu; Lu Lu; Luqing Zhou; Subhabrata Mukherjee; Yunqi Ouyang; Yin Ren; Dawei Shi; Haoran Wu; Zhiyue Wu; Hannah Yao; Zhuoran Yi; Jenny Yu; Rhea Zhan; Hang Zhou; Blake Zhu; Junfan Zhu; Alan Yuille; Yang Liu; Russell Alan Poldrack; Jiachen Li; Zhenglu Li; Molei Tao; Jing Huang; Wenqi Shi; Costas Spanos; Lichao Sun; Chenguang Wang; Orson Xu; Zhen Dong; Hector Gomez; Aylin Caliskan; Ali Emami; Haimin Hu; Zhi Li; Lihui Liu; Murphy Niu; Yi Shao; Jianxin Sun; Mikko Tolonen; Ting Wang; Sanjiv Das; Yanjun Gao; Wenbo Guo; Erika J Schneider; Zhiyong Lu; Yian Ma; Mark Mueller; Radha Poovendran; Somayeh Sojoudi; Yinglun Zhu; Dawn Song·
智能体系统Agent任务评测
摘要
最近的人工智能系统在广泛的基准测试中取得了强劲的成果，但这些成果并没有转化为在许多专业领域具有经济意义的部署。我们认为，这种差距很大程度上是一个评估问题：广泛使用的基准缺乏对真实且具有经济价值的工作流程的持续性能衡量。本文介绍了智能体最后考试 (ALE)，这是一个基准，旨在评估人工智能智能体的长期、经济价值、现实世界任务以及可验证的结果。 ALE 与 250 多名行业专家合作开发，涵盖参考 O*NET / SOC 2018（美国联邦职业分类法）定义的非实体行业。它围绕任务分类法进行组织，包含 55 个子领域，分为 13 个行业集群，涵盖 1K+ 任务。目前的结果表明，最难的层仍远未饱和：在主流Harness和主干配置中，平均完全通过率低于 1%。 ALE 被设计为一个活生生的基准：随着新的工作流程和行业的加入，它的任务池不断增长。更广泛地说，ALE 不仅旨在作为另一个排行榜，而且作为缩小基准成功与 GDP 相关影响之间差距的工具。
