Papers

Learn more about AI2's Lasting Impact Award
Viewing 1-10 of 94 papers
  • SCROLLS: Standardized CompaRison Over Long Language Sequences

    Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer LevyarXiv2022 NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild. We introduce SCROLLS, a suite of tasks that require reasoning over long texts. We…
  • CommonsenseQA 2.0: Exposing the Limits of AI through Gamification

    Alon Talmor, Ori Yoran, Ronan Le Bras, Chandrasekhar Bhagavatula, Yoav Goldberg, Yejin Choi, Jonathan Berant NeurIPS2021 Constructing benchmarks that test the abilities of modern natural language un1 derstanding models is difficult – pre-trained language models exploit artifacts in 2 benchmarks to achieve human parity, but still fail on adversarial examples and make 3 errors…
  • Achieving Model Robustness through Discrete Adversarial Training

    Maor Ivgi, Jonathan BerantEMNLP2021 Discrete adversarial attacks are symbolic perturbations to a language input that preserve the output label but lead to a prediction error. While such attacks have been extensively explored for the purpose of evaluating model robustness, their utility for…
  • Back to Square One: Bias Detection, Training and Commonsense Disentanglement in the Winograd Schema

    Yanai Elazar, Hongming Zhang, Yoav Goldberg, Dan RothEMNLP2021 The Winograd Schema (WS) has been proposed as a test for measuring commonsense capabilities of models. Recently, pre-trained language model-based approaches have boosted performance on some WS benchmarks but the source of improvement is still not clear. We…
  • Contrastive Explanations for Model Interpretability

    Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, Yoav GoldbergEMNLP2021 Contrastive explanations clarify why an event occurred in contrast to another. They are more inherently intuitive to humans to both produce and comprehend. We propose a methodology to produce contrastive explanations for classification models by modifying the…
  • Parameter Norm Growth During Training of Transformers

    William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, Noah A. Smith EMNLP2021 The capacity of neural networks like the widely adopted transformer is known to be very high. Evidence is emerging that they learn successfully due to inductive bias in the training routine, typically some variant of gradient descent (GD). To better…
  • Transformer Feed-Forward Layers Are Key-Value Memories

    Mor Geva, R. Schuster, Jonathan Berant, Omer LevyEMNLP2021 Feed-forward layers constitute two-thirds of a transformer model’s parameters, yet their role in the network remains underexplored. We show that feed-forward layers in transformer-based language models operate as key-value memories, where each key correlates…
  • Value-aware Approximate Attention

    Ankit Gupta, Jonathan BerantEMNLP2021 Following the success of dot-product attention in Transformers, numerous approximations have been recently proposed to address its quadratic complexity with respect to the input length. However, all approximations thus far have ignored the contribution of the…
  • What's in your Head? Emergent Behaviour in Multi-Task Transformer Models

    Mor Geva, Uri Katz, Aviv Ben-Arie, Jonathan BerantEMNLP2021 The primary paradigm for multi-task training in natural language processing is to represent the input with a shared pre-trained language model, and add a small, thin network (head) per task. Given an input, a target head is the head that is selected for…
  • Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization

    Inbar Oren, Jonathan Herzig, Jonathan BerantEMNLP2021 Modern semantic parsers suffer from two principal limitations. First, training requires expensive collection of utterance-program pairs. Second, semantic parsers fail to generalize at test time to new compositions/structures that have not been observed during…