AI reached the authors’ conclusion on 34.6% of 254 recent theory problems

A new MIT FutureTech working paper reports that GPT-5.5 reached the same conclusion as the human authors on 34.6% of 254 recent theory problems drawn from mathematics, physics and economics. The earliest model in the comparison, o1, did so on 2.8%.[1]

The measured outcome is agreement with the human paper’s conclusion. That is a useful research signal, while reproducing and checking a proof or derivation is a separate task. The authors say the benchmark used answers unavailable to the models when they attempted the questions.

Questions with later human answers

The dataset drew on 254 recent arXiv papers across seven domains spanning mathematics, physics and economics. It compared model releases from December 2024 to April 2026. The FutureTech publication summary describes the later human answers as the reference for judging the model conclusions.[1]

Using answers unavailable to the model helps address simple recall of a known result. The public summary does not give enough detail to independently assess every knowledge-cutoff check or the grading of complex proofs. Those method questions matter when interpreting conclusion agreement as research capability.

The gain is both capability and price

The endpoint comparison is a rise of 31.8 percentage points, from 2.8% to 34.6%. The authors report an approximate inference cost of US$5 per GPT-5.5 attempt and round the cost per successful same-conclusion result to US$15.[1]

Using the reported averages, our calculation is US$5 ÷ 0.346 = approximately US$14.45 per same-conclusion result across the benchmark. It is an aggregate cost illustration, not a promise that retrying a particular unsolved problem will succeed for that price. It excludes expert formulation, checking and the cost of producing a trustworthy paper.

A cheaper candidate solution can still change research

Suppose a researcher has ten plausible analytical routes and can afford to ask a model to explore all of them. Even if most outputs fail, a useful candidate may redirect expert attention. The economic effect can arrive before full autonomy: more attempted derivations, faster rejection of dead ends and wider access to techniques that previously required a specialist collaborator.

That benefit also creates a verification burden. Cheap attempts can produce a flood of plausible but wrong arguments. If checking one answer takes longer than generating fifty, the bottleneck moves from production to adjudication. Research institutions may need better proof checkers, provenance records and incentives for negative verification work.

What the result says about research costs

The research appears on MIT FutureTech’s publication site without a journal venue in the accompanying citation. The reported rates are best treated as findings for this particular benchmark; the public summary alone cannot establish how closely the returned reasoning met the standards of the original papers.[1]

The economic question is how much expert work a candidate solution saves after it has been checked. Lower inference costs can make more routes worth exploring, but verification determines whether that inexpensive output becomes useful science. This benchmark provides a concrete starting measurement for that calculation.

Report an error or suggest a correction