Skip to main content ->
Ai2

Research - Papers

Explore a selection of our published work on a variety of key research challenges in AI.

Filter papers

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

Marius MosbachTiago PimentelShauli RavfogelYanai Elazar
2023
Findings of ACL 2023

Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity over fine-tuning… 

FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning

Qinyuan YeIz BeltagyMatthew E. PetersHannaneh Hajishirzi
2023
ACL

Large pre-trained models are capable of few-shot in-context learning (ICL), i.e., performing a new task by prepending a few demonstrations before the test input. However, the concatenated… 

HINT: Hypernetwork Instruction Tuning for Efficient Zero-Shot Generalisation

Hamish IvisonAkshita BhagiaYizhong WangMatthew E. Peters
2023
ACL

Recent NLP models have the great ability to generalise ‘zero-shot’ to new tasks using only an instruction as guidance. However, these approaches usually repeat their instructions with every input,… 

NarrowBERT: Accelerating Masked Language Model Pretraining and Inference

Haoxin LiPhillip KeungDaniel ChengNoah A. Smith
2023
ACL • Proceedings

Large-scale language model pretraining is a very successful form of self-supervised learning in natural language processing, but it is increasingly expensive to perform as the models and pretraining… 

Nonparametric Masked Language Modeling

Sewon MinWeijia ShiM. LewisLuke Zettlemoyer
2023
ACL • Findings

Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked… 

One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Hongjin SuWeijia ShiJungo KasaiTao Yu
2023
ACL • Findings

We introduce INSTRUCTOR, a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain… 

PuMer: Pruning and Merging Tokens for Efficient Vision Language Models

Qingqing CaoBhargavi ParanjapeHanna Hajishirzi
2023
ACL

Large-scale vision language (VL) models use Transformers to perform cross-modal interactions between the input text and image. These cross-modal interactions are computationally expensive and… 

Risks and NLP Design: A Case Study on Procedural Document QA

Nikita HaduongAlice GaoNoah A. Smith
2023
ACL • Findings

As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at… 

Riveter: Measuring Power and Social Dynamics Between Entities

Maria AntoniakAnjalie FieldJimin MunMaarten Sap
2023
ACL

Riveter provides a complete easy-to-use pipeline for analyzing verb connotations associated with entities in text corpora. We prepopulate the package with connotation frames of sentiment, power, and… 

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Afra Feyza AkyurekEkin AkyürekAman MadaanNiket Tandon
2023
Annual Meeting of the Association for Computational Linguistics

Despite their unprecedented success, even the largest language models make mistakes.Similar to how humans learn and improve using feedback, previous work proposed providing language models with…