Research led by Mount Sinai Health System shows artificial intelligence (AI)-based large language models (LLMs) remain susceptible to medical misinformation, particularly if it is phrased in language that implies it comes from a medical expert.
“AI has the potential to be a real help for clinicians and patients, offering faster insights and support,” says co-lead investigator Girish Nadkarni, MD, a professor at the Icahn School of Medicine at Mount Sinai, and chief AI officer of the Mount Sinai Health System, in a press statement.
“But it needs built-in safeguards that check medical claims before they are presented as fact. Our study shows where these systems can still pass on false information, and points to ways we can strengthen them before they are embedded in care.”
As reported in The Lancet Digital Health, the researchers tested over 3.4 million LLM prompts using 20 different models. The investigators aimed to assess how these systems respond to medical misinformation in a variety of different formats including formats such as those seen in public forums and on social media, hospital discharge notes with errors inserted, and many different made up case stories guided by physicians.
The team tested the medical claims or arguments written in a neutral manner and also in 10 different styles to see if this changed the response of the LLM. For example, ‘everyone says this works’ versus ‘a senior doctor says this works,’ among other versions. They measured how often the models agreed with different claims and how often they flagged the style as problematic or wrong.
The claims written in neutral wording were believed 32% of the time. This went up to about 46% for edited hospital discharge notes and fell to around 9% for social‑media style posts.
Most of the emotional argument styles actually reduced the susceptibility of the LLMs to believe misinformation. But, two particular argument styles increased the susceptibility of the LLMs to believing the claims, namely, if phrased as though the information came from a senior doctor, e.g., ‘a senior doctor says this,’ or framed as ‘if you don’t do this, bad things will happen step by step,’ which the researchers called the ‘slippery slope’ style. These two argument styles were believed 35% and 34% of the time, respectively.
The researchers note that some LLM models appeared to be more susceptible to misinformation than others. For example, GPT‑based models were among the least likely to believe the false statements and were the most accurate at spotting the trick argument styles, whereas some others, such as Gemma‑3‑4B‑it, accepted the misinformation in up to about 64% of cases.
“These results emphasize the need for model evaluation frameworks that go beyond accuracy testing to include reasoning style and linguistic framing,” conclude the authors.
“Open release of this benchmark will allow continued testing of emerging models and help develop alignment and fact- grounding strategies tailored to medical and public-health use.”
