An AI assistant may know what a convolutional neural network is, explain a telescope’s imaging pipeline and summarize a thousand papers. A scientist needs something harder: advice about which method to use for a particular dataset, with enough justification to avoid wasting the next month of research. A new astronomy benchmark finds that the difference between knowing methods and choosing one is still substantial.[1]
Researchers Xingzhong Fan, Hangfeng He and Fatemeh Nargesian at the University of Rochester developed Astro2Method, posted on 21 October. They screened 1,028 astronomy papers published between January 2024 and June 2026 and constructed 500 expert-reviewed binary questions from 346 of them. Each question asks which of two machine-learning methods was better for a specific astronomical problem.[1]
The questions were divided by how much explanation the source paper offered. There were 376 'Easy' questions with more complete justifications and 124 'Hard' questions with only partial explanations. Across five tested language models, direct prompting achieved 87.50%–92.55% accuracy on the easier group, but only 58.87%–64.52% on the harder one. The same general task changed dramatically when the contextual evidence became thinner.[1]
Why the hard questions resemble real decisions
Astronomy offers a particularly demanding environment for method selection. Measurements can be noisy, instruments have different resolution, labels are scarce and observational biases can determine what a model appears to learn. Two algorithms may both be respectable while only one handles the actual sampling conditions of a survey. The choice cannot be made reliably from the algorithms’ names alone.[1]
The binary-question format also simplifies what working astronomers do. In practice, they may compare many methods and change the preprocessing, feature engineering or validation regime. The benchmark is a controlled window onto one cognitive task, not proof that an assistant can or cannot design an entire research programme.[1]
Performance depends on available experimental context
| Benchmark element | Reported finding |
|---|---|
| Corpus construction | 500 expert-reviewed binary questions from 346 papers; 1,028 screenedSource 1 |
| Easy questions | 376 questions; five models at 87.50%–92.55% accuracySource 1 |
| Hard questions | 124 questions; five models at 58.87%–64.52% accuracySource 1 |
| Extra retrieval and agent steps | No consistent overall improvement; search often helped by locating original studiesSource 1 |
| Masked pipeline description | Improved hard-question accuracy, matching or exceeding web searchSource 1 |
The researchers tried giving models more ways to think and search. Retrieval-augmented generation and agent methods such as ReAct, Self-Ask and CRITIC did not consistently improve accuracy. Web-search agents often improved when they found the very papers on which the questions were based. Finding the original comparison is a useful research skill, but it is different from deriving the right choice independently from the characteristics of a new problem.[1]
The most revealing intervention was less glamorous. Providing a description of the original paper’s experimental pipeline, while masking the method names, improved performance on hard questions and could match or exceed web search. The result suggests that the useful information is frequently buried in what was actually done to the data rather than in an impressive chain of generic reasoning.[1]
A lesson for scientific assistants beyond astronomy
A method is not a free-floating recipe. Success depends on data volume, missingness, instrument calibration, target definition, baseline comparators and what the research team counts as an error. Without those details, advice can sound technically fluent while being ungrounded. This applies to medical prediction, environmental modelling and industrial quality control as much as to images of galaxies.[1]
The benchmark’s design does not prove models have no general methodological reasoning. It also has a possible contamination concern common to questions derived from published papers: a model may already have encountered the source or closely related material. The authors’ examination of searches finding original papers makes that issue visible rather than hiding it.[1]
There is an encouraging practical implication. Scientists might obtain better assistance by supplying short, precise descriptions of their data pipeline and evaluation conditions, rather than asking a broad question such as 'Which model is best for this?' The necessary information includes the comparator and the result one is optimizing for. A model that cannot see those conditions may be guessing among plausible names.[1]
What a strong next test would require
Future evaluation could ask assistants to recommend methods on genuinely unpublished datasets, explain their trade-offs, choose baselines and produce code that survives replication. The scientist would then judge not only which answer won, but whether the process was transparent enough to trust and reproduce.[1]
Astro2Method’s central contribution is to show that scientific expertise lives in context. The best research assistant will not merely retrieve a label for an algorithm. It will ask what the observations are, what would count as success and which uncertainty is large enough to invalidate a seemingly elegant method.[1]
Sources & further reading
- Astro2Method: Benchmarking LLMs on Machine Learning Method Selection in Astronomy Fan, He and Nargesian, arXiv · 21 Oct 2026 · Accessed 2 Nov 2026