Olympiad proof graders improve—but adversarial reasoning still exposes the gap

Language models can now produce impressive solutions to competition mathematics, but verifying a long natural-language proof is a different task. A solver needs one path to an answer. A grader must inspect every consequential claim, identify where an argument stops being valid and assign a score that is consistent with expert judgement.[1][2]

A new study, “Can LLMs Reliably Grade Olympiad Proofs?”, builds a dedicated evaluation around candidate solutions to the 2025 USA Mathematical Olympiad. The dataset is human-curated and expert-graded. The researchers evaluate 20 models across seven metrics, two prompt styles, three inference methods and three prompt templates. They also introduce two tools: TROJAN, which generates difficult adversarial proofs, and MABGrader, which treats grading as a structured choice over possible scores.[1][2]

Why proof grading is harder than answer checking

Olympiad proofs are not multiple-choice responses. A solution can reach the right conclusion through an invalid lemma, omit a case, use circular reasoning or contain a local gap that affects only part of the score. Different errors deserve different deductions. A verifier also has to distinguish a terse but valid argument from a fluent explanation that hides a false step.[1][2]

That creates two linked problems: error detection and calibrated scoring. A model may correctly sense that “something is wrong” yet assign an inconsistent grade. Conversely, it may produce a plausible rubric-based score without locating the decisive flaw.[1][2]

What MABGrader changes

MABGrader reframes the task as selection among candidate scores. Instead of asking a model to produce a final mark in one pass, the system can gather evidence for different score “arms” and allocate more evaluation effort where uncertainty remains. The paper reports a 26.4% improvement in quadratic weighted kappa over the previous ProofGrader method and a 14% reduction in grading error.[1][2]

Quadratic weighted kappa rewards agreement while penalising larger score differences more heavily. That makes it more suitable than simple accuracy for a rubric in which confusing a nearly complete proof with a zero is worse than being one point off. The gain is meaningful, but it is relative to a baseline—not evidence of parity with expert graders.[1][2]

Adversarial proofs change the test

TROJAN uses iterative self-critique to generate proofs that remain coherent while embedding defects. This addresses a common benchmark weakness: models can perform well when invalid solutions are obviously nonsensical. Real grading is difficult because strong students and strong models produce arguments that are mostly right.[1][2]

An adversarial generator also creates a risk of overfitting to its own error patterns. The most convincing evaluation would therefore include independently written flawed proofs, errors from actual competitions and fresh problems that were not used to tune the generator or grader.[1][2]

What reliable verification would require

A trustworthy grader should do more than output a number. It should identify the first unsupported step, explain whether the gap is repairable, map the defect to a rubric and remain consistent when irrelevant wording changes. It should also know when the reference solution or rubric is ambiguous and defer to a human rather than forcing a precise score.[1][2]

Formal proof assistants provide a stronger notion of correctness, but olympiad solutions are written in natural language and often rely on accepted mathematical conventions. A practical system may need to combine natural-language analysis, symbolic checks, retrieval of known theorems and human adjudication.[1][2]

Progress without declaring the problem solved

The study’s broad evaluation and expert-graded data are valuable because they move beyond demonstrations of models solving individual problems. MABGrader shows that the grading procedure itself can be improved, while TROJAN makes the test more demanding.[1][2]

The result is not that AI can replace olympiad juries. It is that verification can be treated as a separate research problem with its own data, adversaries and calibration metrics. A model that writes a convincing proof has not thereby shown that it can recognise one. The gap between generation and judgement remains one of the most important tests of mathematical AI.[1][2]

Report an error or suggest a correction