Giving a language model the guideline used to grade its answers improved performance in a dental prompting study. Researchers tested ChatGPT, Claude and DeepSeek on 18 fictional scenarios involving medication-related osteonecrosis of the jaw, then supplied the complete 2022 American Association of Oral and Maxillofacial Surgeons position paper.[1]
The Scientific Reports study, published on 26 October, found that Claude’s mean score rose from 4.29 to 4.68 on a zero-to-five scale. ChatGPT’s baseline mean was already 4.68. The result measures agreement with the supplied clinical document; because that document also shaped the scenarios and reference answers, the study does not independently validate clinical judgement.[1]
What the scoring captured
Selected overall guideline-concordance scores
| Model and condition | Mean score, 0–5 |
|---|---|
| ChatGPT, baseline | 4.68 |
| Claude, baseline | 4.29 |
| Claude, guideline supplied | 4.68 |
Medication-related osteonecrosis of the jaw is a serious complication associated with particular medications, notably certain antiresorptive treatments. Dental clinicians must weigh preserving a tooth against the risks of invasive treatment in susceptible patients. A generic answer about routine endodontics can be unsafe if it ignores drug exposure, underlying disease, symptoms and consultation needs.[1]
Standardised fictional cases make the comparison repeatable, but they simplify clinical ambiguity. A model can earn a high score by applying the expected document to a well-specified case. That is a different challenge from resolving incomplete medication histories or competing diagnoses in practice.[1]
Two endodontic specialists independently rated anonymised responses across five decision domains on an ordinal scale from zero to five. The overall measure averaged those domain scores, with disagreements resolved by consensus. This makes the scoring standard explicit, while also explaining why access to that standard can improve the result.[1]
The improvement varied by model and domain
Claude gained 0.39 points. The significant baseline gap between ChatGPT and Claude was no longer observed after the guideline was supplied. Across the tested systems, the study found an overall phase effect and an interaction between model and phase, indicating that document access did not affect every model identically.[1]
The largest model-specific improvement concerned follow-up recommendations, particularly for Claude. The authors also checked the ordinal domain scores with a separate statistical model, which did not confirm some apparent gains in consultation and medication management. Improvement in the overall mean therefore did not translate uniformly across decisions.[1]
A useful test of access to evidence
The study supports a specific design choice: putting the relevant reference material in front of a model can improve its adherence to that material. A stronger clinical evaluation would separate case construction from the supplied document and examine consequential errors in unfamiliar cases. The present result is useful evidence about prompting, with a clear boundary around what was tested.[1]
Sources & further reading
- Effect of guideline-assisted prompting on large language model performance in medication-related osteonecrosis of the jaw–related endodontic decision-making Scientific Reports · 26 Oct 2026 · Accessed 26 Oct 2026