Story perspectives

Study Reveals AI Models' Vulnerability to Harmful Drug Prompts

12/23/2025

Summary

Story summary

Researchers warn that large language models (LLMs) are vulnerable to malicious prompt injections in medical contexts.
A recent study tested GPT-4o mini and Claude 3 Haiku in scenarios seeking harmful drug recommendations.
Manipulated prompts produced dangerous recommendations in 94% of cases, including thalidomide, despite the models performing well without injections.
Experts urge improvements so LLMs refuse harmful suggestions, reducing AI-driven medical risk and maintaining trust.

History

Historical Context

The thalidomide tragedy began in 1961 when the drug, marketed as a sedative, caused severe birth defects in thousands of infants, leading to its withdrawal in Europe. This disaster prompted the U.S. FDA to enact the Kefauver-Harris Amendment in 1962, requiring proof of drug efficacy and safety. Today, similar caution is necessary with large language models, as harmful recommendations were found in 94% of manipulated scenarios.

1 of 2

Story summary

Researchers warn that large language models (LLMs) are vulnerable to malicious prompt injections in medical contexts.
A recent study tested GPT-4o mini and Claude 3 Haiku in scenarios seeking harmful drug recommendations.
Manipulated prompts produced dangerous recommendations in 94% of cases, including thalidomide, despite the models performing well without injections.
Experts urge improvements so LLMs refuse harmful suggestions, reducing AI-driven medical risk and maintaining trust.

1 / 2