Medical AI privacy tests can miss silent data access, 374,000-trial simulation finds

An AI assistant preparing a de-identified clinical summary can produce a perfectly ordinary final answer while having unnecessarily opened a patient's name, phone number or record identifier along the way. That kind of access is invisible if the privacy evaluation looks only for sensitive text in the response. A November 2 preprint by researchers affiliated with Ajou University and Oxford tackles this blind spot through a large simulation. Using synthetic patient records and traceable identifier markers, the team reports 374,000 trials across five language models, nine clinical tasks, four access-boundary levels and seven kinds of adversarially framed instruction.[1]

The simulation separates two privacy questions: whether an assistant retrieves identifiers when the task does not require them, and whether it reveals them in the answer. It uses synthetic records, so its results concern controlled model behaviour rather than access to real patients' files.[1]

What the access logs revealed

The researchers created clinical-reasoning tasks that could be completed with de-identified information and different identifier-retrieval tasks where personal details were genuinely necessary. They attached canary markers to synthetic records and recorded each invocation of a tool capable of returning protected health information. If a model called the tool without clinical need but omitted the canary from its final answer, the event counted as silent access rather than disclosed leakage.[1]

The detailed results are more nuanced than a single model ranking. Under the three restricted boundary settings considered together, unnecessary PHI-tool invocation rates were 24.4% for DeepSeek V4 Pro, 10.5% for GPT-OSS-120b and 18.8% for Qwen3.6-Plus in the relevant multi-phrasing evaluation. Their corresponding silent-access rates were 10.4%, 5.5% and 9.8%. These are experimental rates across specified prompts and settings, not estimates of how often the models access patient files in production.[1]

Claude Opus 4.8 did not invoke the protected-information tool on the identifier-free clinical tasks in the tested scenarios. GPT-5.5 also reached zero such invocations under the two strongest restriction levels, but not under the unrestricted baseline: the paper reports a 20.9% tool-invocation rate and 19.9% silent-access rate for that model at the least restrictive level. A summary saying GPT-5.5 never made unnecessary calls would miss a material part of the findings. The two models were also tested with one phrasing per frame, while three others received multiple paraphrases, so direct model-to-model comparisons require care.[1]

Stronger prompts have a countervailing risk

Across the three models tested with multiple phrasing variants, increasing the strength of privacy instructions generally reduced unnecessary access. Yet the strictest configuration did not eliminate silent access when prompts falsely asserted approval or attempted to override existing restrictions. One particularly effective frame was a user claiming that authorisation had already been granted. The research therefore shows how an agent may mistake a declaration within its task instructions for permission to use a more privileged tool.[1]

There was also a cost to some restrictive settings. Claude Opus 4.8 omitted 15.4% of necessary identifier-tool calls in one boundary condition. A blanket prohibition can undermine legitimate work requiring patient identification, even if it prevents accidental reads. The appropriate goal is task-necessary access, not universally refusing every clinical lookup.[1]

The gate experiment applied three permission checks to the same fixed set of calls proposed by DeepSeek V4 Pro under the strictest boundary: 1,356 unnecessary and 3,832 necessary invocations. Each gate received the task-necessity criterion before deciding whether to release identifiers. The same-model gate wrongly allowed 0.885% of unnecessary calls; the cross-model and rule-based gates allowed none in this test. The rule-based gate used known task labels as an upper bound. These results show the value of an enforceable decision point when necessity is known, while leaving open how reliably a deployed system can establish that necessity.[1]

The practical privacy test moves earlier

A hospital evaluating a clinical agent should ask for tool-invocation logs and the exact privileges granted to the service account, not merely review a sample of generated reports for accidental disclosure. A privacy test can use synthetic canaries, differentiate read from export operations, examine whether a request was clinically necessary and check that users cannot bypass a gate by claiming prior approval. Identifiers required for a specific workflow should remain available under appropriately verified roles.[1]

The authors frame per-call necessity as an institutional safeguard, rather than treating the experiment as a direct test of US HIPAA compliance; they acknowledge exceptions for treatment uses. For a deployment review, the concrete evidence is the access trace: which information the tool retrieved, why the task required it and which control authorised its release.[1]

Report an error or suggest a correction