A longer explanation cannot repair an image detail that an AI system has already misread. PRISM-BENCH examines that problem by evaluating not just final answers, but the visual evidence and explanations behind them. Across 25 multimodal systems, the best reported accuracy on its 1,000 curated questions was 66%.[1]
The preprint, introduced on 27 October, covers five categories, with 200 questions in each. An annotator writes a reference reasoning trace and another checks it. These traces provide a consistent explanation to compare against a model’s answer; they do not reveal its hidden internal computation.[1]
The human comparison requires a separate denominator. People were evaluated on four categories, while the overall model score includes all five. Over the four shared categories, the strongest system reached 65.5%, against a human average of 83.9%: an 18.4-percentage-point gap. Comparing the five-category 66% score directly with the human figure would mix different tests.[1]
What the perception probe actually measured
The authors examined navigation and coreference—the task of working out which references point to the same thing—using GPT-5 and Claude Opus 4.6. Each model first described the image with the question in view, then answered from that description without the image. In a separate text-only context, the same model classified its baseline failures as perceptual or inferential.[1]
Those classifications attributed roughly 85% of failures to perception. Accuracy also fell when the systems relied on their own descriptions, indicating that the descriptions had lost useful information. The result is a diagnostic for two models and two categories, partly dependent on model-generated judgments. It does not establish the cause of 85% of errors across all vision-language tasks.[1]
A second measure compares the content of model explanations with the reference traces. Alignment predicted correctness with a macro AUROC of 0.66, remaining about 0.665 after controlling for response length. On this scale, 0.5 is chance discrimination and 1.0 is perfect separation. The result makes alignment a potentially useful review signal, while leaving substantial uncertainty about individual answers.[1]
The reference requirement also defines where that signal works. A curated evaluation set can supply a checked explanation for each question. An unfamiliar real-world image generally cannot. Trace alignment is therefore most immediately useful for examining a known test set, rather than certifying an arbitrary answer in a deployed assistant.[1]
The ranking changes when cost counts
The authors’ CORE metric adds efficiency to the evaluation. Among 20 open-weight models with exact token counts, it changed the ordering of 46 of 190 model pairs compared with accuracy alone: 24.2%, rounded. Closed systems that conceal thinking-token counts were excluded from that comparison. The result shows why an efficiency leaderboard needs its own coverage and measurement rules.[1]
For teams building visual applications, the benchmark offers a useful way to locate wasted effort. Check whether the model extracts the necessary visual information before spending more inference on an elaborate explanation. Then evaluate whether that explanation follows the evidence and whether the gain justifies the cost. PRISM-BENCH makes those three questions separately measurable within one test.[1]
Sources & further reading
- PRISM-BENCH: Diagnosing Grounded and Faithful Multimodal Reasoning Beyond Final-Answer Accuracy arXiv — Bhatia and colleagues · 27 Oct 2026 · Accessed 30 Oct 2026