Skip to main content ->
Ai2

Research - Papers

Explore a selection of our published work on a variety of key research challenges in AI.

Filter papers

Transformers as Soft Reasoners over Language

Peter ClarkOyvind TafjordKyle Richardson
2020
IJCAI

AI has long pursued the goal of having systems reason over explicitly provided knowledge, but building suitable representations has proved challenging. Here we explore whether transformers can… 

TransOMCS: From Linguistic Graphs to Commonsense Knowledge

Hongming ZhangDaniel KhashabiYangqiu SongDan Roth
2020
IJCAI

Commonsense knowledge acquisition is a key problem for artificial intelligence. Conventional methods of acquiring commonsense knowledge generally require laborious and costly human annotations,… 

CORD-19: The Covid-19 Open Research Dataset

L. Lu WangK. LoY. ChandrasekharS. Kohlmeier
2020
ACL • NLP-COVID

The Covid-19 Open Research Dataset (CORD-19) is a growing 1 resource of scientific papers on Covid-19 and related historical coronavirus research. CORD-19 is designed to facilitate the development… 

SUPP. AI: finding evidence for supplement-drug interactions

Lucy Lu WangOyvind TafjordArman CohanWaleed Ammar
2020
ACL• Demo

Dietary supplements are used by a large portion of the population, but information on their pharmacologic interactions is incomplete. To address this challenge, we present this http URL, an… 

A Formal Hierarchy of RNN Architectures

William. MerrillGail Garfinkel WeissYoav GoldbergEran Yahav
2020
ACL

We develop a formal hierarchy of the expressive capacity of RNN architectures. The hierarchy is based on two formal properties: space complexity, which measures the RNN's memory, and rational… 

A Mixture of h-1 Heads is Better than h Heads

Hao PengRoy SchwartzDianqi LiNoah A. Smith
2020
ACL

Multi-head attentive neural architectures have achieved state-of-the-art results on a variety of natural language processing tasks. Evidence has shown that they are overparameterized; attention… 

A Two-Stage Masked LM Method for Term Set Expansion

Guy KushilevitzShaul MarkovitchYoav Goldberg
2020
ACL

We tackle the task of Term Set Expansion (TSE): given a small seed set of example terms from a semantic class, finding more members of that class. The task is of great practical utility, and also of… 

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

Suchin GururanganAna MarasovićSwabha SwayamdiptaNoah A. Smith
2020
ACL

Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. In light of the success of these broad-coverage models, we investigate whether it is still… 

Improving Transformer Models by Reordering their Sublayers

Ofir PressNoah A. SmithOmer Levy
2020
ACL

Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers. Could ordering the sublayers in a different pattern lead to better performance? We generate randomly… 

Injecting Numerical Reasoning Skills into Language Models

Mor GevaAnkit GuptaJonathan Berant
2020
ACL

Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn…