January 6, 2020
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this talk, I will discuss our recent paper where we explored the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compared pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieved state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. I will wrap up by discussing some of our ongoing and future work on transfer learning for NLP.Less More
Towards AI Complete Question Answering: Combining Text-based, Unanswerable and World Knowledge QuestionsDecember 11, 2019
The recent explosion in question answering research produced a wealth of both reading comprehension and commonsense reasoning datasets. Combining them presents a different kind of challenge: deciding not simply whether information is present in the text, but also whether a confident guess could be made for the missing information. We present QuAIL, the first RC dataset to combine text-based, world knowledge and unanswerable questions, and to provide question type annotation that would enable diagnostics of the reasoning strategies by a given QA system. QuAIL contains 15K multi-choice questions for 800 texts in 4 domains. Crucially, it offers both general and context-specific questions, the answers for which are unlikely to be found in pretraining data of large models like BERT. We show that QuAIL poses substantial challenges to the current state-of-the-art systems, with a 30% drop in accuracy compared to the most similar existing dataset, and we discuss methodological issues in creating such datasets.Less More
November 20, 2019
Research has shown that neural models implicitly encode linguistic features, but there has been little work exploring how these encodings arise as the models are trained. I will be presenting work on the learning dynamics of neural language models from a variety of angles. Using Singular Vector Canonical Correlation Analysis to probe the evolution of syntactic, semantic, and topic representations, we find that part-of-speech is learned earlier than topic; that recurrent layers become more similar to those of a tagger during training; and embedding layers less similar. I will also discuss how these results connect to the compositionality of LSTM learning dynamics, with synthetic experimental evidence that language models rely on short memorized segments in learning general long-range relations between words. I will explore links between syntactic relations and mathematical properties of the interactions between words.Less More
November 20, 2019
The web is a collection of massive and mostly unstructured knowledge data. My recent research has focused on organizing every piece of information on a web-scale corpus and making it readily accessible so that it can be efficiently utilized for creating a language understanding system that requires world knowledge (e.g. question answering). I approach the problem by creating an extremely large neural memory where the entire text corpus is discretized into billions of atomic information and hashed with key vectors. This allows us to random-access specific word-level information in the corpus very fast and accurately. In the first part of my talk, I will highlight the direct usage of the neural memory in factual question answering (e.g. SQuAD, Natural Questions), where I show that the “memorification” of the corpus leads to at least 100x faster inference with better accuracy. In the second part, I will discuss a more advanced usage as a future research direction that utilizes an interactive memory controller, which could hint how we can approach language understanding tasks that need to jointly consider several different pieces of information spread over the web.Less More
Medical Question Answering: Dealing with the complexity and specificity of consumer health questions and visual questionsNovember 12, 2019
Consumer health questions pose specific challenges to automated answering. Two of the salient aspects are the higher linguistic and semantic complexity when compared to open domain questions, and the more pronounced need for reliable information. In this talk I will present two main approaches to deal with the increased complexity by recognizing question entailment and by question summarization, recently published respectively in BMC Bioinformatics and ACL 2019. In particular, our question entailment approach to question answering (QA) showed that restricting the answer sources to only reliable resources led to an improvement of the QA performance and our summarization experiments showed the relevance of data augmentation methods for abstractive question summarization. I’ll also talk about the MEDIQA shared task on question entailment, textual inference and medical question answering that we recently organized at ACL-BioNLP. In the second part of the talk, I will address more specifically questions about medications and present our last study and dataset on medication QA. Finally, I’ll describe our recent endeavors in visual question answering (VQA) from radiology images and the medical VQA challenge (VQA-Med) editions for 2019 and 2020 that we organize in the scope of ImageCLEF.Less More
August 7, 2019
The explosion of available idea repositories -- scientific papers, patents, product descriptions -- represents an unprecedented opportunity to accelerate innovation and lead to a wealth of discoveries. Given the scale of the problem and its ever-expanding nature, there is a need for intelligent automation to assist in the process of discovery. In this talk, I will present our work toward addressing this challenging problem.
We developed an approach for boosting people’s creativity by helping them discover analogies -- abstract structural connections between ideas. We learn to decompose innovation texts into functional models that describe the components and goals of inventions, and use them to build a search engine supporting expressive inspiration queries. In ideation studies, our inspirations helped people generate better ideas with significant improvement over standard search. We also construct a commonsense ontology of purposes and mechanisms of products, mapping the landscape of ideas.
I will also describe a novel machine learning framework we developed in order to identify innovation in patents, where labels are extremely hard to obtain. In our setting, called Ballpark Learning, we are only given groups of instances with coarse constraints over label averages. We demonstrate encouraging results in classification and regression tasks across several domains.Less More
July 23, 2019
Many promising cancer immunotherapy treatment protocols rely on efficient and increasingly extensive methods for manipulating human immune cells. T cells are a frequent target of the laboratory and clinical research driving the development of such protocols as they are most often the effector of the cytotoxic activity that makes these treatments so potent. However, the cytokine signaling network that drives the differentiation and function of such cells is complex and difficult to replicate on a large scale in model biological systems. Abridged versions of these networks have been established over decades of research but it remains challenging to define their global structure as the classification of T cell subtypes operating in these networks, the mechanics of their formation, and the purpose of the signaling molecules they excrete are all controversial, with a slowly expanding understanding emerging in literature over time.
To aid in the quantification of this understanding, we are developing a methodology for identifying references to well known cytokines, transcription factors, and T cell types in literature as well as classifying the relationships between the three in an attempt to determine what cytokines initiate the transcription programs that lead to various cell states in addition to the secretion profiles associated with those states. Entity recognition for this task is performed using SciSpacy and classification of the relations between these entities is based on an LSTM trained using Snorkel, where weak supervision is established through a variety of classification heuristics and distant supervision is provided via previously published immunology databases.Less More
May 6, 2019
Consider the difference between the two sentences “Pat didn’t remember to water the plants” and “Pat didn’t remember that she had watered the plants.” Fluent English speakers recognize that the former sentence implies that Pat did not water the plants, while the latter sentence implies she did. This distinction is crucial to understanding the meaning of these sentences, yet it is one that automated natural language processing (NLP) systems struggle to make. In this talk, I will discuss my work on developing state-of-the-art NLP models that make essential inferences about events (e.g., a “watering” event) and participants (e.g., “Pat” and “the plants”) in natural language sentences. In particular, I will focus on two supervised NLP tasks that serve as core tests of language understanding: Event Factuality Prediction and Semantic Proto-Role Labeling. I will also discuss my work on unsupervised acquisition of common-sense knowledge from large natural language text corpora, and the concomitant challenge of detecting problematic social biases in NLP models trained on such data.Less More
May 2, 2019
We present two results: 1) Analysis techniques for state-of-the-art question-answering models on images, tables and passages of text. We show how these networks often ignore important question terms. Leveraging such non-robust behavior, we present a variety of adversarial examples derived by perturbing the questions. Our strongest attacks drop the accuracy of a visual question answering model from 61.1% to 19%, and that of a tabular question answering model from 33.5% to 3.3%. We demonstrate that attributions can augment standard measures of accuracy and empower investigation of model performance. When a model is accurate but for the wrong reasons, attributions can surface erroneous logic in the model that indicates inadequacies in the data. 2) Parameter-efficient transfer learning: We present a novel method for re-purposing pretrained neural networks to new tasks while maintaining most of the weights intact. The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor). Our approach allows both simultaneous (multi-task) as well as sequential transfer learning. In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match single-task performance.Less More
May 1, 2019
A longstanding goal of artificial intelligence (AI) is to develop agents that can assist or augment humans. Such agents have the potential to transform society. While AI agents can excel at well-defined tasks like games, much more limited progress has been made solving real-world problems like interacting with humans, where data collection is costly, objectives are ill-defined, and safety is critical.
In this talk, I will discuss how we can design agents to improve the efficiency and success of collective human work ("crowdsourcing"), by leveraging techniques from AI, reinforcement learning, and optimization, together with structured contributions from human workers and task designers. This approach improves on current methods for designing such agents, which typically require large amounts of manual experimentation and costly data collection to get right. I will demonstrate the effectiveness of this approach on several crowdsourcing management problems, and also share recent work on how agents can make shared decisions with humans to achieve better outcomes.Less More
April 24, 2019
From robots to cars, virtual assistants and voice-controlled drones, computing devices are increasingly expected to communicate naturally with people and to understand the visual context in which they operate. In this talk, I will present our latest work on generating and comprehending visually-grounded language. First, we will discuss the challenging task of describing an image (image captioning). I will introduce captioning models that leverage multiple data sources, including object detection datasets and unaligned text corpora, in order to learn about the long-tail of visual concepts found in the real world. To support and encourage further efforts in this area, I will present the 'nocaps' benchmark for novel object captioning. In the second part of the talk, I will describe our recent work on developing agents that follow natural language instructions in reconstructed 3D environments using the R2R dataset for vision-and-language navigation.Less More
April 12, 2019
The daily actions and decisions of people are increasingly shaped by recommendation systems, from e-commerce and content platforms to education and wellness applications. These systems selectively suggest and present information items based on their characterization of user preferences. However, existing preference modeling methods are limited due to the incomplete and biased nature of the behavioral data that inform the models. As a result, recommendations can be narrow, skewed, homogeneous, and divergent from users’ aspirations.
In this talk, I will introduce user-centric recommendation models and systems that address the incompleteness and bias of existing methods and increase systems’ utility for individuals. Specifically, I will present my work addressing two key research challenges: (1) inferring debiased preferences from biased behavioral data using counterfactual reasoning, and (2) eliciting unobservable current and aspirational preferences from users through interactive machine learning. I will conclude with discussion of field experiments that demonstrate how user-centric systems can promote healthier diets and better content choices.Less More
April 8, 2019
As the availability of data for language learning grows, the role of linguistic structure is under scrutiny. At the same time, it is imperative to closely inspect patterns in data which might present loopholes for models to obtain high performance on benchmarks. In a two-part talk, I will address each of these challenges.
First, I will introduce the paradigm of scaffolded learning. Scaffolds enable us to leverage inductive biases from one structural source for prediction of a different, but related structure, using only as much supervision as is necessary. We show that the resulting representations achieve improved performance across a range of tasks, indicating that linguistic structure remains beneficial even with powerful deep learning architectures.
In the second part of the talk, I will showcase some of the properties exhibited by NLP models in large data regimes. Even as these models report excellent performance, sometimes claimed to beat humans, a closer look reveals that predictions are not a result of complex reasoning, and the task is not being completed in a generalizable way. Instead, this success can be largely attributed to exploitation of some artifacts of annotation in the datasets. I will discuss some questions our finding raises, as well as directions for future work.Less More
April 3, 2019
Extracting information from text entails deriving a structured, and typically domain-specific, representation of entities and relations from unstructured text. The information thus extracted can potentially facilitate applications such as question answering, information retrieval, conversational dialogue and opinion analysis. However, extracting information from text in a structured form is difficult: it requires understanding words and the relations that exist between them in the context of both the current sentence and the document as a whole.
In this talk, I will present my research on neural models that learn structured output representations comprised of textual mentions of entities and relations within a sentence. In particular, I will propose the use of novel output representations that allow the neural models to learn better dependencies in the output structure and achieve state-of-the-art performance on both tasks as well as on nested variations. I will also describe our recent work on expanding the input context beyond sentences by incorporating coreference resolution to learn entity-level rather than mention-level representations and show that these representations can capture the information regarding the saliency of entities in the document.Less More
March 29, 2019
Can we solve language understanding tasks without relying on task-specific annotated data? This could be important in scenarios where the inputs range across various domains and it is expensive to create annotated data.
I discuss two different language understanding problems (Question Answering and Entity Typing) which have traditionally relied on on direct supervision. For these problems, I present two recent works where exploiting properties of the underlying representations and indirect signals help us move beyond traditional paradigms. And as a result, we observe better generalization across domains.Less More
March 11, 2019
Humans are arguably one of the most important entities that AI systems would need to understand to be useful and ubiquitous. From autonomous cars observing pedestrians, to assistive robots helping the elderly, a large part of this understanding is focused on recognizing human actions, and potentially, their intentions. Humans themselves are quite good at this task: we can look at a person and explain in great detail every action they are doing. Moreover, we can reason over those actions over time, and even predict what potential actions they may intend do in the future. Computer vision algorithms, on the other hand, have lagged far behind on this task. In my research, I’ve explored techniques to improve human action understanding from a visual input, with the key insight being that human actions are dependent on the state of their environment (parameterized by the scene and the objects in it) apart from their own state (parameterized by their pose). In this talk, I will talk about three key ways I exploit this dependence: (1) Learning to aggregate this contextual information to recognize human actions; (2) Predicting a prior on human actions by learning about the affordances of the scenes and objects they interact with; and finally, (3) Moving towards longer term temporal reasoning through a new dataset and benchmark tasks.Less More
March 7, 2019
"An Ethical Crisis in Computing?" Moshe Vardi | Karen Ostrum George Distinguished Professor, Computational Engineering, Rice University
"Algorithmic Accountability: Designing for Safety" Ben Shneiderman | Distinguished Professor, Department of Computer Science, University of Maryland, College Park
"AI Policy: What to Do Now, Soon, and One Day" Ryan Calo | Lane Powell & D. Wayne Gittinger Associate Professor of Law, University of Washington
"Less Talk, More Do: Applied Ethics in AI" Tracy Kosa | Adjunct Professor, Faculty of Law and Albers School of Business, Seattle University
Panel Q&A Oren Etzioni and speakersLess More
March 1, 2019
Can we program computers in our native tongue? This idea, termed natural language programming (NLPRO), has attracted attention almost since the inception of computers themselves.
From the point of view of software engineering (SE), efforts to program in natural language (NL) have relied thus far on controlled natural languages (CNL) -- small unambiguous fragments of English with restricted grammars and limited expressivity. Is it possible to replace these CNLs with truly natural, human language? From the point of view of natural language processing (NLP), current technology successfully extracts information from NL texts. However, the level of NL understanding required for programming in NL goes far beyond such information extraction. Is it possible to endow computers with a dynamic kind of NL understanding? In this talk I argue that the solutions to these seemingly separate challenges are actually closely intertwined, and that one community's challenge is the other community's stepping stone for a huge leap and vice versa.
Specifically, in this talk I propose to view executable programs in SE as semantic structures in NLP, as the basis for broad-coverage semantic parsing. I present a feasibility study on the semantic parsing of requirements documents into executable scenarios, where the requirements are written in a restricted yet highly ambiguous fragment of English, and the target representation employs live sequence charts (LSC), a multi-modal executable programming language. The parsing architecture I propose jointly models sentence-level and discourse-level processing in a generative probabilistic framework. I empirically show that the discourse-based model consistently outperforms the sentence-based model, constructing a system that reflects both the static (entities, properties) and dynamic (behavioral scenarios) requirements in the input document.Less More
February 5, 2019
The social sciences are at a crossroads The great challenges of our time are human in nature - terrorism, climate change, the use of natural resources, and the nature of work - and require robust social science to understand the sources and consequences. Yet the lack of reproducibility and replicability evident in many fields is even more acute in the study of human behavior both because of the difficulty of sharing confidential data and because of the lack of scientific infrastructure. Much of the core infrastructure is manual and ad-hoc in nature, threatening the legitimacy and utility of social science research.
A major challenge is search and discovery. The vast majority of social science data and outputs cannot be easily discovered by other researchers even when nominally deposited in the public domain. A new generation of automated search tools could help researchers discover how data are being used, in what research fields, with what methods, with what code and with what findings. And automation can be used to reward researchers who validate the results and contribute additional information about use, fields, methods, code, and findings. In sum, the use of data depends critically on knowing how it has been produced and used before: the required elements what do the data measure, what research has been done by what researchers, with what code, and with what results.
In this presentation I describe the work that we are doing to build and develop automated tools to create the equivalent of an Amazon.com or TripAdvisor for the access and use of confidential microdata.Less More
January 25, 2019
Time is an important dimension when we describe the world because the world is evolving over time and many facts are time-sensitive. Understanding time is thus an important aspect of natural language understanding and many applications may rely on it, e.g., information retrieval, summarization, causality, and question answering.
In this talk, I will mainly focus on a key component of it, temporal relation extraction. The task has long been challenging because the actual timestamps of those events are rarely expressed explicitly and their temporal order has to be inferred, from lexical cues, between the lines, and often based on strong background knowledge. Additionally, collecting enough and high-quality annotations to facilitate machine learning algorithms for this task is also difficult, which makes it even more challenging to investigate the task. I tackled this task in three perspectives, structured learning, common sense, and data collection, and have improved the state-of-the-art by approximately 20% in absolute F1. My current system, CogCompTime, is available at this online demo: http://groupspaceuiuc.com/temporal/. In the future, I expect to expand my research in these directions to other core problems in AI such as incidental supervision, semantic parsing, and knowledge representation.Less More