Ai2 Newsletter

Today we're releasing Olmo Hybrid, a new 7B-parameter fully open model family that combines transformer attention with linear recurrent layers—and the results make a strong case for hybrid architectures. On MMLU, a widely used benchmark for general knowledge and reasoning, Olmo Hybrid reaches the same accuracy as Olmo 3 using 49% fewer tokens—delivering roughly 2× the data efficiency in controlled pretraining studies. That means you can train a model to the same capability with half the data, or train on the same data and get a meaningfully better model.

Hybrid language models have been gaining momentum across the field, with recent efforts from projects like Samba, Nemotron-H, Qwen3-Next, Kimi Linear, and Qwen 3.5. By combining transformers' ability to recall precise details from earlier in a sequence with recurrent layers' efficiency at tracking evolving state, hybrids promise to be both more capable and cheaper to run at long context lengths. But the community has lacked consensus on whether the purported benefits justify the cost of scaling them up. Olmo Hybrid provides compelling evidence that they do.

Pretrained on 6 trillion tokens using an improved data mix, Olmo Hybrid matches Olmo 3 in training throughput—meaning the efficiency gains come from the architecture itself, not from trading speed for performance. After mid-training, Olmo Hybrid outperforms Olmo 3 across all primary evaluation domains. And scaling-law analysis predicts the token-savings factor grows with model size.

The benefits extend beyond efficiency. Our theoretical and empirical results suggest hybrid architectures offer an expressivity advantage—they can learn patterns that neither pure transformers nor pure linear RNNs capture well on their own, which translates to more efficient scaling as models get larger.

We're releasing base, supervised fine-tuning (SFT), and direct preference optimization (DPO) stages, along with all weights, intermediate checkpoints, and training code—plus a technical report covering the empirical results and scaling-law analysis. We think hybrid models represent a promising direction for the field, and we want the community to be able to build on what we've found.