Scruples: Subreddit Corpus Requiring Understanding Principles in Life-like Ethical Situations

Mosaic • 2021
Scruples is a corpus and benchmark for studying descriptive machine ethics, or machines' ability to understand people's ethical judgments. Scruples offers two datasets: the Anecdotes and the Dilemmas. The Anecdotes collect real-life experiences with ethical judgments about them, while the Dilemmas present pairs of simpler actions with crowdsourced judgments on which is less ethical.
License: See Repo

As AI systems become an increasing part of people’s everyday lives, it becomes increasingly important that they understand people’s ethical norms. Norm understanding involves both prescriptive ethics (how should one act?) and descriptive ethics (what do people believe?).

Scruples facilitates research on descriptive machine ethics, or machines’ ability to understand people’s ethical judgments. Scruples builds upon real-world examples of ethical situations. Similarly, it offers many annotations for each example in order to capture the inherent diversity of ethical judgments often found in communities.

Scruples divides into two parts: the Anecdotes and the Dilemmas. The Anecdotes collect more than 625,000 ethical judgments over 32,000 real-life anecdotes. Each recounts a complex ethical situation, often posing moral dilemmas, and comes with a distribution of judgments contributed by community members. In contrast, the Dilemmas present pairs of simpler actions ranked in terms of which is less ethical by crowdworkers.

See our AAAI 2021 paper: Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-life Anecdotes for more information on the datasets, baselines, and new methodological tools for tackling Scruples.

Leaderboard

Top Public Submissions
DetailsCreatedCross Entropy
1
BERT + Dirichlet-Multinomial
Nicholas Lourie, Ronan Le Bras, Yejin Choi
8/27/2022106%
2
RoBERTa + Dirichlet-Multinomial
Nicholas Lourie, Ronan Le Bras, Yejin Choi
8/27/2022103%

Authors

Nicholas Lourie, Ronan Le Bras, Yejin Choi