How Goodfire used Ai2’s open post-training stack to trace unwanted model behavior

Preference training is a key step in developing LLMs. It uses examples of better and worse responses to shape a model’s behavior—for instance, how helpful, safe, concise, or compliant the model is. 

But steering a model toward behaviors developers want can also have unintended effects. Training that improves a model overall might weaken a safeguard in a particular context or strengthen a behavior no one thought to test for. As the entire AI ecosystem wrestles with how to develop models that remain in alignment with human values, understanding these relationships and getting ahead of unintended consequences is critical.

When unintended behaviors do appear, model builders face a difficult debugging problem: What changed, which training examples caused it, and can the unwanted behavior be corrected without undoing improvements elsewhere?

Goodfire, an interpretability research company, used Ai2’s fully open post-training stack to show what becomes possible when researchers can answer those questions across the entire training pipeline. They predicted behavioral changes before a full training run, traced an observed safety regression back to individual preference examples, and tested targeted changes designed to reduce the regression without sacrificing the model’s broader capability gains.

That level of debugging is difficult partly because of how preference training works. A preference dataset contains prompts alongside responses labeled as preferred or rejected. Across hundreds of thousands of examples, those individual choices collectively become a signal telling the model which behaviors to strengthen and which to suppress. 

“A preference dataset is effectively ‘programming’ the model,” wrote the Goodfire team, “but the instructions implied by a preference dataset cannot be naively inspected, understood, and debugged.”

Why preference training is hard to debug

To investigate those instructions, Goodfire needed access to much more than a finished model. Researchers needed transparency into how the model was built: the underlying preference data,  intermediate checkpoints, training recipes, and evaluations that could show how the model’s capabilities and behavior changed after preference training.

Ai2 makes each of those layers public. Dolci, our open preference dataset for post-training, includes the individual preferred and rejected responses used to train Olmo 3. Olmo is released with intermediate checkpoints and reproducible training recipes. And OLMES, our standardized evaluation suite for language models, measures changes in a model’s capabilities across a consistent set of benchmarks.

Together, those artifacts let Goodfire intervene across the training pipeline rather than infer what happened from the finished model alone.

“The Olmo pipeline is end-to-end reproducible,” says Leon Bergen, a researcher at Goodfire and associate professor at UC San Diego who was one of the primary authors of the paper. “This meant that we didn’t have to guess which parts of the pipeline contributed to a particular effect—we were able to intervene on a component and measure the effect against Ai2’s official Olmo checkpoints.”

That end-to-end access matters. When researchers or developers adapt a model for a particular purpose, they need to know not only whether its behavior changed, but which choices in the training curriculum produced that change. Fully open systems provide a realistic environment where those relationships can be experimentally isolated rather than guessed at after the fact. Goodfire describes such systems as necessary “model organisms” for alignment and AI safety research.

From prediction to root cause

Using Olmo’s open training pipeline, Goodfire developed predictive data debugging—an approach for predicting which behaviors preference training will strengthen or suppress in a model before a full training run. They then tested whether predictive data debugging could help with tracing a known regression to its source and surfacing behavioral shifts no one had thought to look for.

One set of Goodfire’s experiments showed the first problem in practice. Preference training improved Olmo’s general capabilities, but also made the model more likely to comply with harmful requests, as measured by a refusal benchmark suite—for example, requests for dangerous instructions presented as part of a fictional story or hypothetical scenario.

Goodfire traced part of the regression to Dolci examples in which the preferred response encouraged the model to comply with harmful requests while the rejected response discouraged it from answering. Because Ai2 publishes those individual preferred and rejected responses, the researchers could identify individual training examples associated with the increased compliance, test targeted changes, and measure whether those changes reduced the regression without sacrificing Olmo’s broader gains.

The pipeline also identified behaviors that few researchers would think to evaluate in advance. One unusually specific cluster involved fan-fiction prompts about characters relaxing in a pond, passing gas, and causing nearby fish to die. Preference training made Olmo more willing to produce stories of that kind.

The example was strange but instructive. The harmful-request regression was a behavior researchers knew to evaluate; “fart fishing” was not. Predictive data debugging revealed the shift without anyone first having to define it as a behavior to monitor.

“Dolci’s openness in particular gave us access to individual chosen and rejected responses,” says Ekdeep Singh Lubana, another co-author of the paper, “allowing us to predict behavioral regressions and trace back to the individual data points what was causing that.”

For researchers and developers who want to make sure models stay in alignment with human values, that’s the larger value of a fully open training stack: it makes model behavior something that can be connected back to concrete decisions about data and training. Rather than simply observing that a model has strayed from its intended behavior, researchers can investigate why—and test how to steer it back without sacrificing gains elsewhere.