A benchmark of expert-level academic questions to assess AI capabilities

Sakal, Taom; Gatti, Alice; Li, Nathaniel; Khoja, Adam; Kim, Ryan; Ren, Richard; Hausenloy, Jason; Zhang, Oliver; Mazeika, Mantas; Hendrycks, Dan; AI, Scale; Han, Ziwen; Hu, Jing; Zhang, Hugh; Zhang, Chen; Shaaban, Mohamed; Ling, John; X. Shi, Sean; Choi, Michael; Agrawal, A.; Chopra, Arnav; Nattanmai, Aakaash; McKellips, Gordon; Cheraku, Anish; Suhail, Asim; Luo, Ethan; Deng, Marvin; Luo, Jason; Zhang, Ashley; Jindel, Kavin; Paek, Jay; Halevy, Kasper; Baranov, Allen; Liu, Michael; Avadhanam, Advaith; Zhang, David; Cheng, Vincent; Ma, Brad; Fu, Evan; Do, Liam; Laß, Joshua; Yang, Hubert; Sunkari, Surya; Bharath, Vishruth; Ai, Violet; Leung, James; Agrawal, Rishit; Zhou, Alan; Chen, Kevin; Kalpathi, Tejas; Xu, Ziqi; Wang, Gavin; Xiao, Tyler; Maung, Erik; Lee, Sam; Yang, Ryan; Yue, Roy; Zhao, Ben; Yoon, Julia; Sun, Xiangwan; Singh, Aryan; Peng, Clark; Osbey, Tyler; Wang, Taozhi; Echeazu, Daryl; Wu, Timothy; Patel, Spandan; Kulkarni, Vidhi; Sundarapandiyan, Vijaykaarti; Le, Andrew; Nasim, Zafir; Yalam, Srikar; Kasamsetty, Ritesh; Samal, Soham; Sun, David; Shah, Nihar; Saha, Abhijeet; Zhang, Alex; Nguyen, Leon; Nagumalli, Laasya; Wang, Kaixin; Wu, Aidan; Telluri, Anwith; Yue, Summer; Wang, Alexandr; Dodonov, Dmitry; Nguyen, Tung; Lee, J; Anderson, Daron; Doroshenko, Mikhail; Cennyth Stokes, Alun; Mahmood, Mobeen; Pokutnyi, Oleksandr; Z. Iskra, O.; Wang, Yi-Xiang; Levin, J; Kazakov, Mstyslav; Feng, Fiona; Y. Feng, Steven; Zhao, Haoran; Yu, Michael; Gangal, Varun; Zou, Chelsea; Wang, Zihan; Popov, Serguei; Gerbicz, Robert; Galgon, Geoff; Schmitt, Johannes; Yeadon, Will; Lee, Yong-Ki; Sauers, Scott; Sanchez, Alvaro; Giska, Fabian; Roth, Marc; Riis, Søren; Utpala, Saiteja; Burns, Noah; M. Goshu, Gashaw; Maheshbhai Naiya, Mohinder; Agu, Chidozie; Giboney, Zachary; Cheatom, Antrell; Fournier-Facio, Francesco; Crowson, Sarah-Jane; Finke, L; Cheng, Zerui; Zampese, J.A.; G. Hoerr, Ryan; J. Nandor, Mark; Park, Hyunwoo; Gehrunger, Tim; Cai, Jiaqi; Mccarty, Ben; Garretson, Alexis; W. Taylor, Edwin; Sileo, Damien; Ren, Qiuyu; Qazi, Usman; Li, Lianghui; Nam, Jungbae; Wydallis, John; Arkhipov, Pavel; Shi, J; Bacho, Aras; G. Willcocks, Chris; Cao, H.; Motwani, Sumeet; de Oliveira Santos, Théo; Veith, Johannes; Vendrow, Edward; Cojoc, Doru; Zenitani, Kengo; Robinson, Joshua; Tang, Li; Li, Yuqi; Vendrow, Joshua; Wildner Fraga, Natanael; Kuchkin, Vladyslav; Pupasov Maksimov, Andrey; Marion, Pierre; Efremov, Denis; Lynch, Jayson; Liang, Kaiqu; А. Миков, А.; Gritsevskiy, Andrew; Guillod, Julien; Demir, Gözdenur; Martinez, D.; Pageler, Ben; C. Zhou, Kevin; Soori, Saeed; Press, Ori; Y. H. Tang, Henry; Rissone, Paolo; Green, S; Brüssel, Lina; Twayana, Moon; Dieuleveut, Aymeric; Marvin Imperial, Joseph; Prabhu, Ameya; Yang, Jinzhou; Crispino, Nick; Chakrapani Rao, Arun; Zvonkine, Dimitri; Loiseau, Gabriel; С. Калинин, М.; Lukas, Marco; Manolescu, Ciprian; Stambaugh, Nate; Mishra, Subrata; Hogg, Tad; Bosio, Carlo; P. Coppola, Brian; Salazar, Julián; Jin, Jaehyeok; Sayous, Rafael; Ivanov, Stefan; Schwaller, Philippe; Senthilkumar, Shaipranesh; M Bran, Andres; Algaba, Andres; Van Den Houte, K.; Van der Sypt, Lynn; Verbeken, Brecht; Noever, David; Kopylov, Alexei; Myklebust, Benjamin; Li, Baixian; Schut, Lisa; Zheltonozhskii, Evgenii; Yuan, Qiaochu; Lim, Derek; P. Stanley, Richard; Yang, Tong; Maar, John; Wykowski, Julian; Oller, Mart; Sahu, A.; Giulio Ardito, Cesare; Hu, Yuzheng; Ghislain Kemogne Kamdoum, Ariel; Jin, Alvin; Garcia Vilchis, Tobias; Q. Zu, Y.; Lackner, Martin; Koppel, James; Sun, Gongbo; S. Antonenko, Daniil; Chern, Steffi; Zhao, Bingchen; Arsene, Pierrot; M Cavanagh, Joseph; Li, Daofeng; Shen, Jiawei; Crisostomi, Donato; Zhang, Wenjin; Dehghan, Ali; Иванов, С.Л.; Perrella, David; Kaparov, Nurdin; Zang, Allen; Sucholutsky, Ilia; Kharlamova, Arina; Orel, Daniil; Poritski, Vladislav; Ben-David, Shalev; Berger, Zachary; Whitfill, Parker; Foster, Michael; Munro, Daniel; Ho, Linh; Sivarajan, S; Bar Hava, Dan; Kuchkin, Aleksey; Holmes, David; Rodriguez-Romero, Alexandra; Sommerhage, Frank; Zhang, Anji; Moat, Richard; M. Schneider, Keith; Kazibwe, Zakayo; Clarke, Don; Kim, Dae‐Hyun; Meneguitti Dias, Felipe; F. Fish, S.; Elser, Veit; Kreiman, Tobias; Efren Guadarrama Vilchis, Victor; Klose, Immo; Anantheswaran, Ujjwala; Zweiger, Adam; Rawal, Kaivalya; Li, Jeffery; Nguyen, Jeremy; Daans, Nicolas; Heidinger, Haline; Radionov, Maksim; Rozhoň, Václav; Ginis, Vincent; Stump, Christian; Cohen, Niv; Poświata, Rafał; Tkadlec, Josef; Goldfarb, Alan; Wang, Chenguang; Padlewski, Piotr; Barzowski, Stanislaw; Leon Montgomery, K.; Stendall, Ryan; Tucker-Foltz, Jamie; Stade, Jack; Ryan Rogers, T.; Goertzen, Tom; Grabb, Declan; Shukla, Abhishek; Givré, Alan; Arnold Ambay, John; Sen, Archan; Fayez Aziz, Muhammad; H Inlow, Mark; He, Hao; Zhang, Ling; Kaddar, Younesse; Ängquist, Ivar; Chen, Yanxu; Wang, H.; N Ramakrishnan, K; Thornley, Elliott; Terpin, Antonio; Schoelkopf, Hailey; Zheng, Eric; Carmi, Avishy; Brown, Ethan; Zhu, Kelin; Bartolo, Max; John Wheeler, Richard; Stehberger, Martin; Bradshaw, Peter; Heimonen, JP; Sridhar, Kaustubh; Akov, Ido; A. Sandlin, Jennifer; Makarychev, Yury; Tam, Joanna; D. Hoang, Hieu; M. Cunningham, David; N. Goryachev, V.; Patramanis, Demosthenes; Krause, Michael; Redenti, Andrew; Aldous, David; Lai, Jesyin; Coleman, Shannon; Xu, Jiangnan; Lee, Sang-Won; Magoulas, Ilias; Zhao, Sandy; Tang, Ning; K. Cohen, Michael; Paradise, Orr; H. Kirchner, Jan; Ovchynnikov, Maksym; O. Matos, Jason; Shenoy, A.; Zhuo Wang, Michael; Nie, Yuzhou; Sztyber, Anna; Faraboschi, Paolo; Riblet, Robin; Crozier, Jonathan; Shiv Halasyamani, P.; Verma, Shreyas; Joshi, Prashant; Meril, Eli; Ma, Ziqiao; Andréoletti, Jérémy; Singhal, Raghav; Platnick, Jacob; Nevirkovets, Volodymyr; Basler, Luke; Ivanov, Alexander; Khoury, Seri; Gustafsson, Nils; Piccardo, Marco; Mostaghimi, Hamid; Chen, Qijia; Singh, Virendra; Vân Khánh, Trần; Rosu, Paul; S. Szlyk, Hannah; Brown, Zachary; Narayan, Himanshu; Menezes, Aline; Roberts, Jonathan; E. Alley, William; Sun, Kunyang; Patel, Arkil; Lamparth, Max; Reuel, Anka; Xin, Linwei; Xu, Hanmeng; Loader, J; Martín, F.; Wang, Zixuan; Achilleos, Andrea; Preu, Thomas; Korbak, Tomek; Bosio, Ida; Kazemi, F; Chen, Ziye; Bálint, Biró; J. Y. Lo, Eve; Wang, Jiaqi; Inês S. Nunes, Maria; Milbauer, Jeremiah; Saiful Bari, M; Wang, Zihao; Ansarinejad, Behzad; Sun, Yewen; Durand, Stéphane; Elgnainy, Hossam; Douville, Guillaume; Tordera, Daniel; Balabanian, George; Wolff, Hew; Kvistad, Lynna; Milliron, Hsiao-yun; Sakor, Ahmad; Eron, Murat; Favre, Andrew; Shah, Shailesh; Zhou, Xiaoxiang; Kamalov, Firuz; Abdoli, Sherwin; Santens, Tim; Barkan, Shaul; Tee, Allison; Zhang, Robin; Tomasiello, Alessandro; Luca, Gian; Looi, Shi-Zhuo; Van Le, Tu; Kolt, Noam; Pan, Jiayi; Rodman, Emma; Drori, Jacob; Fossum, Carl; Muennighoff, Niklas; Jagota, Milind; Pradeep, Ronak; Fan, Honglu; Eicher, Jonathan; Chen, Michael; Thaman, Kushal; Merrill, William; Firsching, Moritz; Harris, Carter; Ciobâcă, Ştefan; N. Gross, Jason; Pandey, Rohan; Gusev, Ilya; Jones, Adam; Agnihotri, Shashank; Zhelnov, Pavel; Mofayezi, Mohammadreza; Piperski, Alexander; K. Zhang, David; Dobarskyi, Kostiantyn; Leventov, Roman; Soroko, Ignat; Duersch, Joshua; Taamazyan, Vage; Ho, Andrew; Ma, Wenjie; A. Held, William; Xian, Ruicheng; Randy Zebaze, Armel; Mohamed, Marwa; Noah Leser, Julian; X Yuan, Michelle; Yacar, Laila; Lengler, Johannes; Olszewska, Katarzyna; Di Fratta, Claudio; OliveiraJr, Edson; W. Jackson, Joseph; Zou, Andy; Chidambaram, M.; Manik, Timothy; Haffenden, Hector; Stander, Dashiell; Dasouqi, Ali; Shen, Alexander; Golshani, Bita; Stap, David; Kretov, Egor; Uzhou, Mikalai; Borisovna Zhidkovskaya, Alina; Winter, Nick; Orbegozo Rodriguez, Miguel; Lauff, Robert; Wehr, Dustin; Tang, Colin; Hossain, Zakir; Phillips, Shaun; Samuele, Fortuna; Ekström, Fredrik; Hammon, Angela; Patel, Oam; Farhidi, Faraz; H. Medley, George; Mohammadzadeh, Forough; Peñaflor, Madellene; Kassahun, Haile; Friedrich, Alena; Hernandez Perez, Rayner; Pyda, Daniel; Sakal, Taom; Dhamane, Omkar; Khajegili Mirabadi, Ali; Hallman, Eric; Okutsu, Kenchi; A. Battaglia, Mike; Maghsoudimehrabani, Mohammad; Amit, Alon; I. Hulbert, D.; Pereira, Roberto; Weber, Simon; , Handoko; Peristyy, Anton; Malina, Stephen; Mehkary, Mustafa; Aly, Rami; Reidegeld, Frank; C Dick, A; L Friday, Cary; Singh, Mukhwinder; Shapourian, Hassan; J. Kim, W.; Costa, Mariana; Gurdogan, Hubeyb; Kumar, Harsh; Ceconello, Chiara; Zhuang, Chao; Park, H; Carroll, Micah; R. Tawfeek, Andrew; Steinerberger, Stefan; Aggarwal, Daattavya; Kirchhof, Michael; Dai, Linjie; Kim, Evan; Ferret, Johan; Shah, Jainam; Wang, Yuzhou; Yan, Minghao; Burdzy, Krzysztof; Zhang, Lixin; Franca, A; T. Pham, Diana; Yong Loh, Kang; Robinson, Joshua; Jackson, Abram; Giordano, Paolo; Petersen, Philipp; Cosma, Adrian; Colino, Jesús; White, Colin; Votava, Jacob; Vinnikov, Vladimir; Delaney, Ethan; Špelda, Petr; Střítecký, Vít; M. Shahid, Syed; Mourrat, Jean-Christophe; Vetoshkin, Lavr; Sponselee, Koen; Bacho, Renas; Yong, Zheng-Xin; de la Rosa, Florencia; Cho, Nathan; Li, Xiuyu; Malod, Guillaume; Weller, Orion; Albani, Guglielmo; Lang, Leon; Laurendeau, Julien; Kazakov, Dmitry; Adesanya, Fatimah; Portier, J.; Hollom, Lawrence; H. Souza, Victor; Q Zhou, Y; Degorre, Julien; Yaln, Yiğit; Daniel Obikoya, Gbenga; Michael Pokorny, Rai; Bigi, Filippo; C. Boscá, M.; Shumar, Oleg; Bacho, Kaniuar; Recchia, Gabriel; Popescu, Mara; Shulga, Nikita; Mildred Tanwie, Ngefor; Lux, Thomas; K. Rank, B.; Ni, Cheng; Brooks, Matthew; Yakimchyk, Alesia; Liu, Huanxu; Cavalleri, Stefano; Häggström, Olle; Verkama, Emil; Newbould, Joshua; Gundlach, Hans; Brito-Santana, Leonor; Amaro, Brian; Vajipey, Vivek; Grover, Rynaa; Wang, Ting; Kratish, Yosi; Li, Weizhong; Gopi, Sivakanth; Caciolai, Andrea; Schroeder de Witt, Christian; Hernández-Cámara, Pablo; Rodolà, Emanuele; Robins, Jules; Williamson, Dominic; Raynor, Brad; Hao, Qi; Segev, Ben; Fan, Jingxuan; Martinson, Sarah; A Wang, E; Hausknecht, Kaylie; P. Brenner, Michael; Mao, Mao; Demian, Christoph; Hosseinzadeh Kassani, Peyman; Zhang, Xinyu; Avagian, David; Jessica Scipio, Eshawn; Ragoler, Alon; Kok Soon Tan, Justin; Sims, Blake; Plecnik, Rebeka; Kirtland, Aaron; Faruk Bodur, Omer; P. Shinde, D.; Carlos Leyva Labrador, Yan; Adoul, Zahra; Zekry, Mohamed; Karakoc, Ali; C. B. Santos, Tania; Shamseldeen, Samir; Karim, Loukmane; Liakhovitskaia, Anna; Resman, Nate; Farina, Nicholas; Carlos González, Juan; Maayan, Gabe; Anderson, Earth; De Oliveira Pena, Rodrigo; Kelley, Elizabeth; Mariji, Hodjat; Pouriamanesh, Rasoul; Wu, Wentao; Finocchio, Ross; Alarab, Ismail; Cole, Joshua; Ferreira, Danyelle; T. Johnson, Bryan; Safdari, Mohammad; Dai, Liangti; Arthornthurasuk, Siriphan; C. Mcalister, Isaac; J. Moyano, Alejandro; Pronin, Alexey; Fan, Jing; Ramirez-Trinidad, Angel; Malysheva, Yana; Pottmaier, Daphiny; Taheri, Omid; Stepanic, Stanley; J. Perry, S.; Askew, Luke; Adrián Huerta Rodrguez, Raúl; M. R. Minissi, Ali; Lorena, Ricardo; Swaminathan Iyer, K.; Anil Fasiludeen, Arshad; Clark, R.G.; Ducey, Josh; Piza, Matheus; Somrak, Maja; Vergo, Eric; Qin, J.; Borbás, Benjámin; Chu, Eric; Lindsey, Jack; Jallon, Antoine; McInnis, Ian; Chen, Evan; Semler, Avi; Gloor, Luk; S. Shah, Tej; Carauleanu, Marc; Lauer, Pascal; Đuc Huy, Tran; Shahrtash, Hossein; Duc, Emilien; Lewark, Lukas; Brown, Assaf; Albanie, Samuel; Weber, Brian; S. Vaz, Warren; Clavier, Pierre; Fan, Yiyang; Azevedo e Silva, G; Tony Lian, Long; Abramovitch, Marcus; Jiang, Xi; Mendoza, Sandra; M. Islam, M.; Gonzalez, Juan; Mavroudis, Vasilios; Xu, Justin; Kumar, Pawan; Prasad Goswami, Laxman; Bugas, Daniel; Heydari, Nasser; Jeanplong, Ferenc; Jansen, Thorben; Pinto, Antonella; T. Apronti, Archimedes; Galal, Abdallah; Ze-An, Ng; Singh, Ankit; Jiang, Tong; of Arc Xavier, Joan; Priya Agarwal, Kanu; Berkani, Mohammed; Zhang, Gang; Du, Z.; Alves de Oliveira-Júnior, Benedito; Malishev, D.; Rémy, Nicolas; D. Hartman, Taylor; Tarver, Tim; Mensah, Stephen; Abou Loume, Gautier; Morak, Wiktor; Habibi, Fikri; Y. Hoback, S.; Cai, Will; Giménez, Javier; Grace Montecillo, Roselynn; Łucki, Jakub; Campbell, Russell; Sharma, Asankhaya; van der Meer, Klaas; Gul, Shreen; Espinosa Gonzalez, Daniel; Alapont, Xavier; Hoover, Alex; Chhablani, Gunjan; Vargus, Freddie; Agarwal, Arunim; Jiang, Yibo; Y. Patil, D.; Outevsky, David; Scaria, Kevin; Maheshwari, Rajat; Dendane, Abdelkader; Shukla, Priti; Cartwright, Ashley; Bogdanov, Sergei; Mündler, Niels; Möller, S.; Arnaboldi, Luca; Thaman, Kunvar; Rehan Siddiqi, Muhammad; Saxena, Prajvi; Gupta, Himanshu; Fruhauff, Tony; Sherman, Glen; Vincze, Mátyás; Usawasutsakorn, Siranut; Ler, Dylan; Radhakrishnan, Anil; Enyekwe, Innocent; Md Salauddin, Sk; Muzhen, Jiang; Maksapetyan, Aleksandr; Rossbach, Vivien; Harjadi, Chris; Bahaloohoreh, Mohsen; Sparrow, Claire; Sidhu, Jasdeep; Ali, Sam; Bian, Song; Lai, John; A. Singer, Eric; Leon Uro, Justine; Bateman, Greg; Sayed, Mohamed; Menshawy, Ahmed; Duclosel, Darling; Bezzi, Dario; Jain, Yashaswini; Aaron, Ashley; Tiryakioğlu, Murat; Siddh, Sheeshram; Krenek, Keith; Ali Shah, Imad; Jin, Jun; Creighton, Scott; Peskoff, Denis; El-Wasif, Zienab; P, Ragavendran; Richmond, Michael; McGowan, Joseph; Patwardhan, Tejal; Sun, Hao-Yu; Ting, Sun; Zubić, Nikola; Sala, Samuele; Ebert, Stephen; Kaddour, Jean; Schottdorf, Manuel; Wang, Dianzhuo; Petruzella, Gerol; Meiburg, Alex; Medved, Tilen; ElSheikh, Ali; Ashwin Hebbar, S.; Vaquero, Lorenzo; Yang, Xianjun; Poulos, Jason; Zouhar, Vilém; Bogdanik, Sergey; Zhang, Mingfang; Sanz-Ros, Jorge; Anugraha, David; Dai, Yinwei; N. Nhu, Anh; Wang, Xue; Anıl Demirçalı, Ali; Jia, Zhibai; Q Zhou, Y; Wu, Juncheng; He, Mike; Chandok, Nitin; Sinha, Aarush; Luo, Gaoxiang; Le, Long; Noyé, Mickaël; Perełkiewicz, Michał; Pantidis, Ioannis; Qi, Tianbo; Sachin Purohit, Soham; Parcalabescu, Letitia; Nguyen, Thai-Hoa; Indra Winata, Genta; De Ponti, E; Li, Hanchen; Dhole, Kaustubh; Park, Jongee; Abbondanza, Dario; Wang, Yue; Nayak, Anupam; M. Caetano, Diogo; A. W. L. Wong, Antonio; del Rio-Chanona, Maria; Kondor, Dániel; François, Pieter; Chalstrey, Ed; Zsambok, Jakob; Hoyer, Dan; Reddish, Jenny; Hauser, Jakob; Rodrigo-Ginés, Francisco-Javier; Datta, Suchandra; Shepherd, Maxwell; Kamphuis, Thom; Zhang, Qizheng; Kim, Hyunjun; Sun, Ruiji; Yao, Jianzhu; Dernoncourt, Franck; Krishna, Satyapriya; Rismanchian, Sina; Pu, Bonan; Pinto, Francesco; Wang, Yue; Shridhar, Kumar; J. Overholt, Kalon; Briia, Glib; Nguyen, Hieu; Quod Soler Bartomeu, David; CY Pang, Tony; Wecker, Adam; Xiong, Yifan; Li, Fanfei; S. Huber, Lukas; Jaeger, Joshua; P Maddalena, R; Han Lù, Xing; Zhang, Yuhui; Beger, Claas; Tser Jern Kon, Patrick; Li, Sean; Sanker, Vivek; Yin, Ming; Liang, Yihao; Zhang, Xinlu; Agrawal, Ankit; S. Yifei, Li; Zhang, Zechen; Cai, Mu; Sonmez, Yasin; Cozianu, Costin; K Li, C; Slen, Alex; Yu, Shoubin; Kyu Park, Hyun; Sarti, Gabriele; Briański, Marcin; Stolfo, Alessandro; An Nguyen, Truong; Zhang, Mike; Perlitz, Yotam; Hernandez-Orallo, Jose; Li, Runjia; Shabani, Amin; Juefei-Xu, Felix; Dhingra, Shikhar; Zohar, Orr; Chiffon Nguyen, My; Pondaven, Alexander; Yilmaz, Abdurrahim; Zhao, X; Jin, Chuanyang; Jiang, Muyan; Todoran, Stefan; Han, Xinyao; Kreuer, Jules; Rabern, Brian; Plassart, Anna; Maggetti, Martino; Yap, Luther; Geirhos, Robert; Kean, Jonathon; Wang, Dingsu; Mollaei, Sina; Sun, Chenkai; Yin, Yifan; Wang, S; Li, Rui; Chang, Yaowen; Wei, Anjiang; Bizeul, Alice; Wang, Xiaohan; Oliveira Arrais, Alexandre; Mukherjee, Kushin; Chamorro-Padial, Jorge; Liu, Jiachen; Qu, Xingyu; Guan, Junyi; Bouyamourn, Adam; Sc, Wu; Plomecka, Martyna; Chen, Junda; Tang, Mengze; Deng, Jiaqi; Subramanian, Shreyas; Xi, Haocheng; Chen, Haoxuan; Zhang, Wei; Ren, Yinuo; Tu, Haoqin; Kim, Sejong; Chen, Yushun; Vera Marjanović, Sara; Ha, Junwoo; Luczyna, Grzegorz; Ma, Jeff; Shen, Zewen; Song, Dawn; E. Zhang, Cedegao; Wang, Zhun; Gendron, G.; Xiao, Yunze; Smucker, Leo; Weng, Erica; Hao Lee, Kwok; Ye, Zhe; Ermon, Stefano; D. Lopez-Miguel, Ignacio; Knights, Theo; Gitter, Anthony; Park, NamKyu; Wei, Boyi; Chen, Hongzheng; Pai, Kunal; Elkhanany, Ahmed; Lin, Han; D. Siedler, Philipp; Fang, Jichao; Mishra, Ritwik; Zsolnai-Fehér, Károly; Jiang, Xilin; Khan, Shadab; Yuan, Jun; Kumar Jain, Rishab; Lin, Xi; Peterson, Mike; Wang, Zhe; Malusare, Aditya; Tang, Maosen; Gupta, Isha; Fosin, Ivan; Kang, Timothy; Dworakowska, Barbara; Matsumoto, Kazuki; Zheng, Guangyao; Sewuster, Gerben; Pretel Villanueva, Jorge; Rannev, Ivan; Chernyavsky, Igor; Chen, Jiale; Banik, Deepayan; Racz, Ben; Dong, Wenchao; Wang, Yi-Xiang; Bashmal, Laila; Moreira Goncalves, D; Hu, Wei; Bar, Kaushik; Bohdal, Ondrej; Singh Patlan, Atharv; Dhuliawala, Shehzaad; Geirhos, Caroline; Wist, Julien; Kansal, Yuval; Chen, Bingsen; Tire, Kutay; Talay Yücel, Atak; Christof, Brandon; Singla, Veerupaksh; Song, Zijian; Chen, S; Ge, Jiaxin; Ponkshe, Kaustubh; Park, I.; Shi, Tianneng; Q. Ma, Martin; Mak, Joshua; Lai, Sherwin; Moulin, Antoine; Cheng, Zhuo; Zhu, Zhanda; Zhang, Ziyi; Patil, Vaidehi; Jha, Ketan; Qq, Men; Wu, Jiaxuan; Zhang, Tianchi; Vieira, B; Fikri Aji, Alham; Chung, Jae-Won; Mahfoud, Mohammed; Thi Hoang, Ha; Sperzel, Marc; Hao, Wei; Meding, Kristof; Xu, Sihan; Kostakos, Vassilis; Manini, Davide; Y, Liu; Toukmaji, Christopher; Yu, Eunmi; Engin Demirçali, Arif; Sun, Zhiyi; Dewerpe, Ivan; Qin, Hongsen; Pflugfelder, Roman; Bailey, James; Morris, Johnathan; Heilala, Ville; Rosset, Sybille; Yu, Zishun; E. Chen, Peter; Yeo, Woongyeong; Jain, Eeshaan; Chigurupati, Sreekar; Chernyavsky, Julia; Prajwal Reddy, Sai; Venugopalan, Subhashini; Batra, Hunar; Francisco Park, Core; Tran, Hieu; Maximiano, Guilherme; Zhang, Genghan; Liang, Yizhuo; Shiyu, Hu; Xu, Rongwu; Pan, Rui; Suresh, Siddharth; Liu, Ziqi; Gulati, Samaksh; Zhang, Songyang; Turchin, Peter; W. Bartlett, Christopher; R. Scotese, Christopher; Cao, Phuong; Wu, Ben; Karwowski, Jacek; Scaramuzza, Davide

doi:10.1038/s41586-025-09962-4

Public

A benchmark of expert-level academic questions to assess AI capabilities

Published in Nature • Jan 28, 2026

Authors:

Taom Sakal

Alice Gatti

Nathaniel Li

Abstract

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1, limiting i...

Subject

Grading (engineering)

Pace

Benchmark (surveying)

Research Assistant

AI chat, annotations, notes & similar papers

Finding related papers...

Discussions

(0)

No comments yet

Be the first to share your thoughts!