A vision model can predict the right class and produce a persuasive heat map that highlights the wrong part of the image. Scaling the network does not reliably fix that problem, according to a study comparing eleven architectures, five post-hoc explanation methods and datasets with ground-truth object or lesion masks.[1]
The researchers asked a narrow but important question: as ResNets, DenseNets and Vision Transformers become deeper or larger, do their explanations align better with the pixels experts say are relevant? In most comparisons, the answer was no.[1]
The scale result
| Measure | Value | Context |
|---|---|---|
| Models evaluated | 11 | Architectures from the ResNet, DenseNet and Vision Transformer families.Source 1 |
| Explanation methods | Five | Saliency-style and class-activation approaches were tested.Source 1 |
| Scale comparisons | 60 | Tests across datasets, model families and explanation methods.Source 1 |
| Significant differences | 21 of 60 | Thirty-five percent remained significant after correction.Source 1 |
| Largest model ranked best | 3 of 60 | Five percent of all scale comparisons.Source 1 |
Classification and localisation answer different questions
Classification asks whether the model assigned the correct label. Localisation asks whether the explanation overlaps the annotated region associated with that label. A network can exploit background, acquisition artefacts or correlated anatomy and still classify accurately. A saliency map can also fail to reveal the true causal features even when it looks plausible.
Three properties that should not be merged
| Property | Typical measure | What it cannot prove |
|---|---|---|
| Predictive performance | Accuracy, area under the curve or F1 | That the model used the medically or semantically correct regionSource 1 |
| Explanation localisation | Overlap or attribution precision against a mask | That highlighted pixels caused the prediction rather than merely correlating with itSource 1 |
| Model scale | Depth or parameter count | That prediction or explanation improves in the deployment settingSource 1 |
Pretraining helped prediction more consistently than explanation
Pretraining generally improved predictive performance and made explanations depend more on learned weights than on random initialization. It did not consistently improve localisation. That distinction matters when a team chooses a larger pretrained model and assumes the accompanying heat map has automatically become more trustworthy.[1]
The authors also found settings with reasonable classification performance and localisation precision close to zero. In a safety-sensitive application, that combination should trigger investigation rather than reassurance.
Ground-truth masks are useful but not final truth
A segmentation mask represents an expert annotation of the relevant region, not a complete account of every feature a valid model may use. Some context outside the mask can be legitimate. The study therefore evaluates spatial agreement, not whether a model has a human-like causal explanation.
The practical conclusion is not that small models are always more interpretable. It is that interpretability is an empirical property. If an explanation is part of the safety case, it must be tested directly rather than inherited from the model’s parameter count.
Sources & further reading
- Scaling vision models does not consistently improve localisation-based explanation quality Scientific Reports · 14 Sep 2026 · Accessed 16 Oct 2026