ScientistTwo puts autonomous research on an experimental trail

ScientistTwo is a more interesting proposition than a system that turns a prompt into a scientific-looking PDF. Its authors describe an agent that proposes ideas, implements them, evaluates experiments and revises the research. The preprint was submitted on 17 September 2026; Google included it in its programme for COLM, held from 6 to 9 October. A conference presentation is not itself peer-review acceptance of the agent’s generated papers.[1][3]

The potential value is the speed at which a promising question can be turned into a testable experiment. The potential failure is just as specific: a persuasive manuscript can hide a changed benchmark, a weak comparison or a conclusion that goes beyond the measurements. A useful research agent needs to carry an experiment trail as carefully as it carries its prose.

Read the denominators before the headline

The final row deserves particular attention. A real reference can be irrelevant, support a weaker claim or describe an incompatible experiment. Verifying that a paper exists is useful, but citation quality also requires checking what it says. Likewise, reproducing a number establishes agreement with a procedure; it does not establish that the procedure measures the right thing.

The research loop is more valuable than a writing loop

ScientistTwo’s architecture separates proposing, evaluating and analysing ideas from writing and simulated peer review. The authors describe screening on subsets before larger experiments and using ablations—removing parts of a method—to investigate which components cause the observed improvement. Criticism can trigger new experiments rather than only a rewritten paragraph.[1]

That last distinction is the key scientific opportunity. Suppose an agent discovers that an added preprocessing step improves a classifier. A writing-only system can invent a satisfying explanation. An experimental system can remove the step, change the relevant condition and measure whether the proposed explanation survives. The useful output is not merely a better result; it is evidence about why the result changed.

Consider an illustrative research question: does a new data-cleaning rule improve classification? The rule should be tested against the same train/test split, with any selection criteria fixed before examining held-out results. The team should record which rules were rejected as well as the one eventually chosen. Otherwise the final manuscript conceals how many opportunities the system had to find a lucky result.

An AI reviewer is a measuring instrument

The project compares manuscripts using automated reviewers against human-authored papers accepted at named venues. That makes the scores a comparative evaluation under those reviewers—not a statement that those venues accepted ScientistTwo’s outputs. The project itself distinguishes its results from the stronger ICML Spotlight reference group.[2]

There is a useful historical comparison. The AI Scientist-v2 study submitted three generated manuscripts to an ICLR workshop, with one exceeding the reported review threshold. Sakana’s account explains that accepted AI-generated work would be withdrawn under the experiment’s agreed protocol. That was a real review experiment, but not an AI-authored paper becoming a main-conference publication.[4][5]

These two forms of evidence should coexist rather than substitute for one another. Automated review offers scale and repeatability. Human review can reveal missing context, importance and assumptions the automated evaluator shares with the generator. Neither process, by itself, proves a result. Actual replication asks a different question again.

More trials change what a convincing result requires

Here is an illustrative statistical calculation, not a measurement of ScientistTwo. If twenty independent experiments each have a 5% chance of producing a false positive, the probability of at least one false positive is 1 − 0.95 to the power of 20, approximately 64.2%. Independence and the assumed error rate matter; real experiments often violate both assumptions. The example shows why fast experimentation needs a record of the search, not only the winner.

An effective research agent could help maintain that record automatically. It could preserve unsuccessful configurations, distinguish exploratory from confirmatory tests, and package the exact inputs needed for a separate rerun. Those are design opportunities, not features we independently established for this release. They make the research process more legible rather than merely faster.

A good next test for autonomous research would start with a problem whose evaluation procedure is fixed by someone other than the agent. Give the system an explicit experimental budget, retain every run and have a separate investigator repeat the strongest result. Publication quality should then follow from the work, instead of becoming the score the system optimises.

ScientistTwo’s reported performance makes this a serious line of research. Its deeper importance may be in normalising a different deliverable: a paper accompanied by executable evidence and an intelligible history of how the conclusion was reached. An agent that makes the best next experiment easier to run—and easier for someone else to challenge—would be a valuable scientific instrument.

Report an error or suggest a correction