SciArena: A new platform for evaluating foundation models in scientific literature tasks

Update 7/15: SciArena sunsetting—and learnings

The interactive SciArena demo is being retired on July 15, 2026. This post will remain online as the record of the project, and sciarena.allen.ai will point here once the demo goes offline. The paper, code, and dataset remain available below—thanks to everyone who submitted questions and voted while the platform was live.

What we learned

SciArena was built to help study how well today's foundation models handle real scientific-literature tasks – like answering an open research question with a cited synthesis of the relevant literature – as judged by the researchers who do that work. For each question, the platform showed two models' answers side by side with their names concealed, and a researcher chose the better answer, called it a tie, or judged both to be insufficient. They could add a comment, and the models were revealed only after the vote. 

By July 15, 1,693 users had cast 3,919 votes and left 985 written explanations of why they voted as they did. Every vote fed into the SciArena leaderboard.

Notably, OpenAI's o3 topped the final leaderboard, ahead of flagship systems like Anthropic's Claude Opus 4.1, Google's Gemini 3 Pro Preview, and OpenAI's own GPT-5, plus open-weights models such as DeepSeek-R1 and Qwen3—the same top spot o3 held in our June 2025 SciArena technical report. Users found o3’s answers more detailed and more to the point than the alternatives.

Perhaps the most useful finding, though, was what researchers valued when they chose one model's answer over another's: citation quality came up in the comments more often than any other single factor. Grouped by theme, here are the top reasons researchers gave for their model preferences (each percentage is how frequently that reason came up in researchers' explanations; less common reasons aren't shown, so the numbers don't add up to 100%):

• Citation and reference quality (23.4%)—whether the papers a model cites actually exist, address the question being asked, point to original research rather than secondhand summaries, and can be looked up and verified.
• Completeness and depth (19.1%)—broader coverage, deeper explanation of mechanisms, and more technical specificity.
• Directly answering the question (16.3%)—an answer to the specific question asked, not a generic overview of the literature.
• Structure and readability (13.2%)—clear organization, synthesis, and flow.
• Factual accuracy (5.0%)—cases where a model invented references, made wrong claims, or misattributed work.
• Recency (4.8%)—a preference for newer work, often from 2023–2025, over older references.

The through-line is that in scientific literature tasks, fluent prose is not enough. Researchers reward answers they can trace and verify, and they notice when a model hallucinates a reference or leans on outdated work.

We believe that for anyone building models to work with scientific literature, these are signals worth taking from SciArena.

Original post follows.

Scientific literature is expanding at an unprecedented rate, making it challenging for researchers to stay updated and synthesize new knowledge. Foundation models are increasingly being used to help with this, but evaluating their capabilities in open-ended scientific tasks remains a significant challenge. Traditional benchmarks are often not suitable for nuanced evaluations in scientific tasks as they are static, limited in scale, and quickly becoming outdated.

To address these limitations, we present SciArena, an open and collaborative platform that directly engages the scientific research community in evaluating foundation models for scientific literature tasks. This crowdsourced, head-to-head evaluation approach for LLMs has been successfully pioneered in the general domain by platforms such as Chatbot Arena.

Results

As of June 30, 2025, SciArena hosts 23 frontier foundation models, selected for their representation of current state-of-the-art capabilities. Among them, the o3 model consistently delivers top performance across all scientific domains. Moreover, we find that o3 provides a more detailed elaboration of cited scientific papers, and its output tends to be more technical in Engineering disciplines. Performance among the remaining models varies by discipline, for instance, Claude-4-Opus excels in Healthcare, while DeepSeek-R1-0528 performs well in Natural Science.

SciArena-Eval, the meta-evaluation benchmark, has also highlighted significant challenges for model-based evaluators. Even the top-performing model in our experiments, o3, achieves only 65.1% accuracy in predicting human preferences. This marks a notable gap compared to general-purpose benchmarks like AlpacaEval and WildChat, where pairwise evaluation protocols surpass 70% accuracy. These findings highlight the need for more robust and reliable automated evaluation methods in scientific reasoning tasks.

We will be continuously adding new models to the SciArena platform to ensure ongoing evaluation of the latest advancements. 

What is SciArena?

SciArena is an open evaluation platform where researchers can compare and vote on the performance of different foundation models in tasks related to scientific literature. It's built on a community voting approach, similar to Chatbot Arena, but specifically tailored for the complex and open-ended nature of scientific inquiry.

The platform has three main components:

• SciArena Platform: This is where human researchers submit questions, view side-by-side responses from different foundation models, and cast their votes for the preferred output.
• Leaderboard: Based on community votes, an Elo rating system ranks the models, providing a dynamic and up-to-date assessment of their performance.
• SciArena-Eval: This is a meta-evaluation benchmark built on the collected human preference data, designed to assess the accuracy of model-based evaluation systems.

How SciArena Works: A Glimpse Under the Hood

Unlike general-domain queries, scientific tasks often need to be grounded on scientific literature. When a user submits a question on SciArena, the platform utilizes an advanced multi-stage retrieval pipeline, adapted from Ai2's Scholar QA system, to gather relevant scientific paper contexts. This pipeline includes query decomposition, passage retrieval, and re-ranking to ensure high-quality and relevant information. These retrieved contexts, along with the user's question, are then fed to two randomly selected foundation models. Note that in SciArena, our focus is on evaluating standard and directly comparable foundation models rather than customized agentic or proprietary deep research systems such as Perplexity or OpenAI’s Deep Research, so neither of those agentic systems were included in our comparisons. The models generate long-form, literature-grounded responses, complete with citations. To mitigate potential biases from stylistic elements, responses are post-processed to a standardized, plain-text format with consistent citation styles. Users then evaluate these outputs and vote for the one that best satisfies their information needs.

The Impact and Quality of SciArena Data

The quality of evaluation hinges on the quality of the data, and SciArena has put a strong emphasis on this. Over the first four months of its internal operation, SciArena collected over 13,000 votes from 102 trusted researchers across various scientific fields.

To ensure the reliability and integrity of this human preference data, rigorous quality control measures were applied. This includes a separate annotation pipeline that ensures the quality of pairwise annotations:

• Expert Annotators: The initial data collection involved 102 researchers with at least two peer-reviewed publications and prior experience with AI-assisted literature tools.
• Comprehensive Training: All annotators underwent a one-hour training session to ensure consistency and accuracy in their evaluations.
• Blind rating: In SciArena’s interface, the models that generated each answer are not revealed until after the user submits their vote.
• Inter-Annotator Agreement (IAA) and Self-Consistency: SciArena assesses both IAA and self-consistency to quantify the reliability of the collected data. Results show strong self-consistency (weighted Cohen’s κ as 0.91), meaning individual annotators' judgments remain stable over time, and high IAA (weighted Cohen’s κ as 0.76), indicating that experts tend to reach similar judgments despite the subjective nature of some questions.

This commitment to data quality ensures that SciArena offers a robust and trustworthy assessment of model performance.

Future Challenges

For future work, SciArena welcomes partnerships with model developers that would enable us to evaluate and post new models to our leaderboard. Further, SciArena currently evaluates different models using a fixed approach for other important elements of the retrieval-augmented generation (RAG) pipeline, such as the retrieval index and prompting workflow. These other pipeline elements can have a significant influence on answer quality, and in future work we would like to evaluate other choices for indexing papers and prompting the models.  

Full Results

Join the Arena! 

• Visit SciArena today to compare models and vote. 
• Bookmark the leaderboard page - updated often - to see how the rankings change over time 
• Watch for new model releases - we’ll add them continuously as new models drop
• Download and use the SciArena-Eval set, code and data for your own projects
• Read our paper to learn more about SciArena's design and more detailed results and analysis.
• We are continuously improving the platform, feedback and suggestions are welcome