Ai2 Newsletter

The volume of scientific papers continues to grow, particularly in domains such as computer science. To this year’s upcoming NeurIPS conference alone, researchers submitted a record 25,000 studies, up from 17,000 in 2024.

AI can help sift through the noise, but it’s not always clear which models are best suited to the task. To illuminate this, we just launched SciArena, a platform designed to measure how well models can obtain and synthesize information from scientific literature across healthcare, engineering, and other fields.

SciArena allows users to compare how dozens of different models answer scientific questions such as “What are the recent developments in quantum computing's impact on cryptography?”

"We launched SciArena as an open, evolving, and collaborative platform that invites the scientific community to directly evaluate LLMs on real, literature-based tasks," Arman Cohan, research scientist at Ai2 and assistant professor of computer science at Yale University, says. "Our motivation has been to create a rigorous, dynamic, and community-driven evaluation platform that reflects the needs and standards of scientists and helps measure progress, while making all data and code openly available for others to build on."

SciArena queries models, directing them to retrieve relevant papers from Semantic Scholar, our AI-powered search and discovery tool for scientific literature. Anyone can use SciArena and vote on their preferred model. Rankings feed into a leaderboard showing which models perform best overall. 

"The leaderboard uses the pairwise votes to rank the models using an Elo rating system, similar to what's used in chess rankings," Cohan says. "The models that consistently win head-to-head comparisons move up the rankings. So, the leaderboard aggregates real-world human judgments."

Alongside the platform, we open-sourced SciArena-Eval, a meta-evaluation benchmark, as well as the code and data used to develop SciArena. The goal is to continuously add new models to the platform to ensure that the leaderboard remains a trustworthy source for evaluating models’ ability to parse the vast–and growing–scientific literature.

According to Arman, the SciArena-Eval results in particular highlight the challenges for scientific task evaluation techniques. The top-performing model on SciArena-Eval, o3, achieved only 65.1% accuracy–a significant gap compared to general-purpose benchmarks.

"Open benchmarks such as SciArena's leaderboard put models through identical, expert-vetted tests and publicly show the results in real-time, so strengths and failure modes are visible to researchers, developers, and end-users alike," Cohan says. "Our hope is that these resources help drive both model improvement and more reliable, human-aligned AI evaluation in science."