Tracing distinctive language in AI-written text

When readers flagged a short story that had won a Commonwealth Foundation prize as machine-written – an episode soon dubbed GrantaGate – it crystallized a larger question about AI-generated text: when a model writes, where do its words come from? Are they new, or do they echo language the model encountered during training?

An AI-writing detector can estimate whether a passage was machine-generated, but it can’t show which expressions also appear in existing sources—or where they appear. Answering that requires searching the vast corpora a model may have been trained on.

Ai2’s infini-gram engine makes that possible. It indexes massive text datasets and can count how often a phrase of any length appears across them. Researchers can use it to examine AI-generated prose phrase by phrase and measure how much of its language appears elsewhere.

Tuhin Chakrabarty, an assistant professor of computer science at Stony Brook University, has been doing just that. Chakrabarty and his co-authors recently published a paper on arXiv examining the prevalence and commercial reach of AI-generated books on Amazon, as well as their overlap with distinctive language from previously published works. The broader phenomenon – and Chakrabarty’s research into it – has also drawn coverage this month from The Atlantic and, on July 28, The New York Times, which examined the spread of AI-generated books and the copyright concerns surrounding them. 

After analyzing the prize-winning story at the center of GrantaGate, Chakrabarty’s group used Google Books and infini-gram to compare top-selling self-published books on Amazon with and without substantial detected AI text. Among the 200 highest-revenue books in each group, those with substantial detected AI text contained 4.4 percentage points more rare expressions from existing books. The gap widened to 22.5 points when compared with award-winning or nominated literature, suggesting that AI-heavy bestsellers draw more heavily on distinctive language from earlier works.

From detection score to textual evidence

Chakrabarty found his way to infini-gram through two Ai2 projects: OlmoTrace, which is powered by infini-gram and links output from our fully open Olmo models back to matching text in their training data, and the Creativity Index, a measure of a text's originality against a reference corpus. Both provided ways to analyze how machine-generated text overlaps with source corpora. 

"Attribution is a very hard problem in machine learning, especially with LLMs," says Chakrabarty.

GrantaGate showed the limits of a detection score. A detector only returns an estimate—a score for how likely a passage is to be synthetic. “It doesn't point to anything beyond,” says Chakrabarty. “Once you know if something is AI, it's helpful to dissect it using tools like infini-gram.” 

When one of Chakrabarty’s students, Xinyue Liu, ran the story at the center of GrantaGate through infini-gram, distinctive fragments turned up in the public corpora infini-gram indexes—the story's "sour tang of fermenting," for instance, echoes the phrasing of a line on a fan-fiction website, providing evidence that the expression appeared elsewhere.

A generated passage may contain short expressions that also appear across many different sources rather than reproducing one source at length. Post-training largely stops a model from repeating a long passage word for word, but the shorter phrasing it picked up in pretraining occasionally still comes through.

“[Finding those distinctive phrases in other sources] acts as circumstantial evidence that supports an AI detector score,” says Chakrabarty.

How much of a book overlaps with distinctive language?

In a new study, Chakrabarty's group is using infini-gram to measure how much of a book overlaps with rare expressions found in previously published works—and whether that overlap is associated with commercial success.

To distinguish phrases concentrated in a small number of books from those in common use – stock lines like “her heart skipped a beat” – the team cross-referenced Google Books with infini-gram. They focused on multi-word expressions that met two conditions: appearing in no more than five Google Books volumes and not appearing at all in infini-gram’s web snapshot.

Take a line like “the warmth of his touch seeped through the thin fabric of her dress.” If a phrase within that sentence appears in only a few books and nowhere in the web corpus, that doesn’t prove where the phrase originated or who wrote it. But it does show that the phrase is associated more closely with published books than with language widely available on the web. When many such phrases appear in text that an AI detector has already flagged, their combined presence provides additional evidence that the text draws heavily on book-derived language.

“In short, if an expression count is zero in infini-gram and the expression occurs in very few books in Google Books, one can say with some confidence that it was taken from books,” explains Tuhin. “This also helps debunk some hackneyed arguments that liken human reading of books to AI being trained on books.”

The study found that, among top-selling self-published genre books on Amazon, books with substantial detected AI text contained more rare expressions from existing books than books with no detected AI text. 

Among the 100 highest-revenue books with substantial detected AI text and the 100 highest-revenue books with no detected AI text, rare expressions from existing books covered 43.2% and 37.6% of the text, respectively. The gap narrowed when the analysis expanded to the top 200 books in each category, but remained statistically significant—suggesting that the higher rate of rare-expression use in books with substantial detected AI text was not simply a product of common genre language, but reflected greater use of distinctive phrasing associated with existing books.

The team also compared the two sets of top-selling self-published genre books – those with substantial detected AI text and those with no detected AI text – with a separate reference set of award-winning or nominated literature. Rare expressions from existing books covered an average of 19.1% of the text in the literary reference set, compared with 37.2% in the top-selling books with no detected AI text and 41.6% in those with substantial detected AI text. 

The award-winning or nominated books were included as a reference point—not as a market-matched control. Still, their much lower rate of rare-expression use helps put the study’s main finding into context: this level of overlap is not an inevitable feature of accomplished writing.

Why it takes an open tool

None of these analyses would've been possible without a way to search a vast collection of human writing at scale. That's what infini-gram –and Olmo’s open training datasets like Dolma – provides: a researcher can take any phrase and see how often it appears across large public corpora that a model may have been trained on.

Chakrabarty sees another promising application in copyright research. His group is experimenting with infini-gram to trace copyrighted books to where they surfaced on the web—a way to investigate whether a model reproduced snippets of a book because it trained on a pirated copy or on a manuscript that had leaked online.