Loaded wording in medical notes changed AI triage rankings

A phrase implying that a patient attends the emergency department unnecessarily was enough to change an AI triage ranking. In a controlled test using records from two academic emergency departments, loaded descriptions sometimes pushed a patient who would deteriorate behind a stable patient with the same initial severity score.[1]

Researchers tested Gemma, Qwen and DeepSeek by inserting neutral or stigmatising details into paired records. For frequent-attendance wording, harmful reprioritisation reached 9.4% in the worst model-and-dataset combination. That figure describes altered rankings in a retrospective experiment, not an observed rate of harm in a live clinic.[1]

The experiment makes the danger visible

The researchers paired patients matched on Emergency Severity Index, but with different subsequent outcomes: one deteriorated within six hours, the other did not. They evaluated the models' ordering before and after inserting demographic, social and stigma-related attributes. Eighteen experimental conditions distinguished neutral descriptions from negative or loaded versions of the same underlying topic. The study reports 221,556 model comparisons across these conditions.[1]

For stigmatizing wording about frequent emergency-department use, the harmful change was significant in all six model-by-dataset combinations, reaching 9.4% in the worst reported cell. Compared with neutral formulations, it added between 1.4 and 7.1 percentage points of harmful reprioritisation within the studied cells. Psychiatric history also produced shifts, up to 9.5%, although a consistent advantage of stigmatizing versus neutral wording was established in fewer configurations.[1]

The comparison with neutral wording is the revealing part. The underlying topic—previous use of emergency services—remained similar, while the judgement embedded in the phrase changed. The added harmful reprioritisation ranged from 1.4 to 7.1 percentage points across the studied cells.[1]

The information was not necessarily irrelevant

Past service use and clinical history can sometimes be genuinely informative. That makes the challenge harder than simply deleting every mention of attendance, mental illness or substance use. A useful triage system should be able to distinguish clinical facts from an unsupported judgment about a person's character. Saying that someone has attended ten times in six months is different from implying that they habitually misuse the service, even if both phrases refer to the same broad topic.[1]

The paired-record design supplies a concrete way to test the problem: hold severity and subsequent outcome fixed, change the description, and measure whether the system moves a deteriorating patient down the queue. It does not identify a universal remedy, but it makes an otherwise elusive documentation effect measurable.[1]

Clinical notes do two jobs at once: they record observations and communicate a writer’s interpretation. A triage model must work with both. This experiment shows how easily an unsupported judgement in the second layer can interfere with the first.[1]

Report an error or suggest a correction