Quoref

AllenNLP, AI2 Irvine • 2019
Quoref is a QA dataset which tests the coreferential reasoning capability of reading comprehension systems. In this span-selection benchmark containing 24K questions over 4.7K paragraphs from Wikipedia, a system must resolve hard coreferences before selecting the appropriate span(s) in the paragraphs for answering questions.
License: CC BY

Current Version: 0.2

Clicking Download will provide a link to download the training and development sets of the latest version of the dataset.

Changes from v0.1

We discovered that the start indices for a small number of answers (less than 2%) in the training, development, and test sets of v0.1 of the dataset were slightly off due to unicode processing issues. We fixed those issues in v0.2. Note that the leaderboard results should not be affected by this fix since the metrics are computed over strings, not spans.

If you need v0.1 of the dataset for any reason, you can get it here.

Leaderboard

Top Public Submissions
DetailsCreatedExact Match
1
TASE - CorefRoBERTa
Tsinghua University (Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Maosong Sun, Zhiyuan Liu)
5/15/202081%
2
TASE-CoNLL-joint-qgen
Mingzhu Wu, Nafise Sadat Moosavi, Dan Roth, Iryna Gurevych
12/14/202080%
3
TASE - RoBERTa
Elad Segal, Avia Efrat, Mor Shoham, Amir Globerson, Jonathan Berant
12/8/201980%
4
CorefRoberta-Large
Tsinghua University (Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Maosong Sun, Zhiyuan Liu)
3/25/202076%
5
RoBERTa-MT
WeChatAI
12/9/201973%

Authors

Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, Matt Gardner