Allen Institute for AI

OpenBookQA Dataset

Aristo • 2018
OpenBookQA aims to promote research in advanced question-answering, probing a deeper understanding of both the topic (with salient facts summarized as an open book, also provided with the dataset) and the language it is expressed in. In particular, it contains questions that require multi-step reasoning, use of additional common and commonsense knowledge, and rich text comprehension.

OpenBookQA is a new kind of question-answering dataset modeled after open book exams for assessing human understanding of a subject. It consists of 5,957 multiple-choice elementary-level science questions (4,957 train, 500 dev, 500 test), which probe the understanding of a small “book” of 1,326 core science facts and the application of these facts to novel situations. For training, the dataset includes a mapping from each question to the core science fact it was designed to probe. Answering OpenBookQA questions requires additional broad common knowledge, not contained in the book. The questions, by design, are answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. Strong neural baselines achieve around 50% on OpenBookQA, leaving a large gap to the 92% accuracy of crowd-workers.

Additionally, we provide 5,167 crowd-sourced common knowledge facts, and an expanded version of the train/dev/test questions where each question is associated with its originating core fact, a human accuracy score, a clarity score, and an anonymized crowd-worker ID (in the “Additional” folder).

Leaderboard

Top Public Submissions
Details
Created
Accuracy
1
UnifiedQA (T5-11B; finetuned) - with IR
Daniel Khashabi, from AI2
4/24/202087%
2
T5 11B + KB
Pirtoaca George Sebastian from the Polytechnic University of Bucharest
6/12/202085%
3
TTTTT
University of Waterloo and Academia Sinica
4/1/202083%
5/31/202082%
9/30/202081%