A mammogram report can resemble an expert’s wording while missing the assessment that determines what happens next. A Scientific Reports study published on 23 October demonstrates that gap: strong text-generation scores coexisted with weak agreement on BI-RADS, the breast-imaging assessment system.[1]
The researchers trained MedGemma-4B using a synthetic-to-real pipeline. They generated 400 reports from structured annotations, retained 390 and then fine-tuned on 407 real reports. Evaluation used 49 held-out cases. Reported scores included ROUGE-L of 0.656, METEOR of 0.666 and BERTScore of 0.917, while BI-RADS agreement never exceeded a kappa of 0.14.[1]
Two changes had been credited to one training stage
The two-stage model initially appeared to outperform an alternative, but the comparison also changed the amount of trainable adaptation capacity. When capacity was held constant, the apparent synthetic-pretraining advantage disappeared. Across 60 runs varying corpus size, capacity and initialisation, the matched comparisons did not establish a meaningful benefit from the added synthetic stage.[1]
This is a confounding problem. If an experiment changes both the training material and how much of a model can adapt, the resulting gain cannot be assigned to the material alone. Matching capacity asks a cleaner question: does the extra training stage help when the competing models have the same opportunity to learn?[1]
Synthetic reports derived from labels also inherit the limits of those labels. Generating more sentences can supply useful language examples, but it cannot automatically recover an image finding that the annotation never recorded. The study gives a concrete reason to test what the synthetic stage contributes instead of assuming more generated text means more clinical information.[1]
Better wording did not mean better assessment
Kappa measures agreement after accounting for chance; 0.14 is a weak result for the tested assessments. In one comparison, a 0.058 increase in ROUGE-L coincided with worse BI-RADS agreement and sensitivity. A wording score and a clinical classification score were therefore moving in opposite directions.[1]
The authors also separated fabrication from omission. Under their assertion-level criterion, the models did not invent abnormal findings on normal studies; omissions were the predominant problem. That matters because a report can sound restrained and still leave out the finding a clinician needs.[1]
The 49-case test set and specific model configuration limit the breadth of the conclusion. The study does not settle whether synthetic data can help other medical systems. It does show why report evaluation needs separate checks for clinically significant findings and omissions: textual resemblance alone cannot tell a reviewer whether a report supports the right next decision.[1]
Sources & further reading
- Two-stage synthetic-to-real transfer learning for automated mammography report generation using vision-language models Scientific Reports · 23 Oct 2026 · Accessed 26 Oct 2026