Only one of 12 models discovered hidden laws in a free-exploration benchmark

LongChang, an arXiv preprint, tests whether language models can decide what to investigate before anyone supplies the research question. Agents manipulate simulated worlds, notice anomalies and design experiments to identify altered physical laws. Across 12 tested models, only GPT-6-Astra produced reports that passed both the correctness and experimental-support checks, with seven successful runs out of 42 on altered-physics scenes, a discovery rate of 16.7%.[1][2]

The benchmark contains 14 altered-physics scenes and a normal control. Examples change relationships that ordinary experience makes easy to take for granted: the way an object falls, restarts, rebounds or responds to another object. Agents can manipulate the scene and propose apparatus, while separate model-based components review tools, score reports and evolve scenes. A discovery counts only when the stated law is correct and the exploration record provides appropriate experimental support.[1][2]

Evidence can be strong while the idea is wrong

Astra's evidence-support score was high even when its stated law was incomplete or incorrect. Claude Opus showed the reverse pattern in the aggregate table: some laws were scored as correct, but the reports lacked supporting experimental evidence, leaving the discovery rate at zero. This distinction is valuable. A model can produce careful observations without finding the governing rule, or guess the rule without establishing it. Scientific capability requires both.[1]

The traces show a recurring failure before calculation begins. Agents notice unexpected motion but continue their original plan instead of treating the anomaly as a new research question. They may run many actions without varying the decisive quantity, fail to compare competing explanations or stop after obtaining a plausible story. The bottleneck is not simply access to a simulator; it is deciding what deserves investigation and designing a discriminating test.[1][2]

More time was not the missing ingredient

The guided condition supplied both a law-discovery task and a scene-specific clue, while warning that physical laws might be altered. Astra’s discovery rate rose from 16.7% to 28.6%, and mean exploration fell from 39.3 to 30.9 turns. This does not isolate the effect of a generic instruction alone. In a separate budget comparison, raising the limit from 100 to 500 turns produced discovery rates of 16.7% and 14.3%; the agent still stopped after about 40 turns.[1]

The normal control tested whether agents invented anomalies under ordinary physics. Across models, 36.1% of control reports were rejected for false altered-law claims or unsupported explanations. This makes the null result part of the task: an agent should be able to conclude that its observations fit ordinary behaviour.[1]

LongChang remains a synthetic benchmark. MuJoCo scenes have clean state, repeatable interventions and none of the measurement noise, equipment failure, institutional constraints or tacit knowledge of a real laboratory. Several parts of the evaluation are themselves performed by models, even though the altered law and interaction logs provide an objective reference. The study demonstrates a controlled failure boundary, not a general measure of scientific ability.[1][3]

LongChang distinguishes three steps that answer-based science tests can conflate: choosing a question, identifying a law and supporting it with an experiment. Its low discovery scores locate a weakness in that combined process, while the guided condition shows that supplying a useful clue can change the result.[1]

Report an error or suggest a correction