IfQA Counterfactual Reasoning Benchmark

Aristo • 2023
Counterfactual reasoning benchmark introduced in the EMNLP-2023 paper titled "IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions".
License: ODC-BY

IfQA is an open-domain question answering dataset where each of the 3,800 questions, annotated by crowdworkers, is based on a counterfactual presupposition via an ``if’’ clause. For example, if Los Angeles was on the east coast of the U.S., what would be the time difference between Los Angeles and Paris?

Such questions require models to go beyond retrieving direct factual knowledge from the Web: they must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters.



Wenhao Wu, Meng Jiang, Peter Clark, Ashish Sabharwal