When a model reads a drug's class from its name—not its knowledge

People increasingly bring health and medication questions to AI models, often naming specific drugs. But a model can sound knowledgeable without knowing much about the drug itself. Many drug names encode their class through affixes – often suffixes like -pril, -olol, and -azoline – giving a model a shortcut: infer the answer from the name rather than from learned facts about the medication. In a new study of Olmo 3, researchers found that 51–59% of the drugs they tested elicited little sign of drug-specific knowledge, while another 12–18% appeared affix-driven. By tracing those behaviors back to the model’s open training data, the team showed not only that the shortcut exists, but where it comes from—and why that matters when models answer health questions.

Kaijie Mo at the University of Texas at Austin, working with collaborators at Northeastern University and MD Anderson Cancer Center, led the study. The team set out to determine when a model was relying on the morphology of a drug’s name rather than drug-specific knowledge—and to trace that behavior back to the data the model was trained on. They chose Olmo 3 because the analysis required a fully open model with public weights and training corpora, documentation spanning pretraining through post-training, and released intermediate checkpoints.

"Olmo 3 was particularly valuable because of its openness and because its training corpus includes substantial medical content," says Mo, "allowing us to connect behavior, mechanisms, and training data. Open models make it possible to move from observing a behavior to understanding its cause."

On Olmo 3 7B Instruct, the team built a diagnostic that isolates three things a model might draw on when it answers drug-related health questions: the affix of a drug’s name (the shared ending, like -pril), the stem (the part of the name unique to a drug), and real knowledge of a specific drug. They swapped the stem, the affix, or both for invented, meaningless ("nonce") strings and measured how the model's answers shifted.

The diagnostic found that Olmo 3 knows less about drugs than it lets on. Depending on the evaluation format, about 12–18% of drugs appeared affix-driven. For a much larger 51–59%, the model showed no real sign of knowing a particular drug—its answer to a real name was barely different from its answer to a made-up one.

To connect that behavior to the training data, the team searched Olmo’s open corpus with infini-gram, our engine for exact phrase lookup across massive text datasets, to see how often each drug appeared and in what contexts. They found that the rarer a drug was in training, the more its answers appeared to depend on the drug’s affix rather than drug-specific knowledge.

That reliance isn’t necessarily a problem on its own—drug affixes often encode real information about pharmacological class. The concern is when that useful generalization substitutes for knowledge about the specific medication.

For Mo, Olmo 3 and infini-gram’s value lies in being able to connect model outputs to their likely cause. "The openness made it possible to identify where morphological shortcuts arise and study them beyond what would be possible in a black-box model,” she says. “For high-stakes domains such as medicine, auditing model outputs alone is often not enough. Access to model internals enables more rigorous diagnosis, validation, and ultimately more trustworthy systems.”