IfQA Counterfactual Reasoning Benchmark

IfQA is an open-domain question answering dataset where each of the 3,800 questions, annotated by crowdworkers, is based on a counterfactual presupposition via an ``if’’ clause. For example, if Los Angeles was on the east coast of the U.S., what would be the time difference between Los Angeles and Paris?

Such questions require models to go beyond retrieving direct factual knowledge from the Web: they must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters.