Two of 100 AI-generated rain schemes beat a retuned physics baseline

An AI agent can read a scientific paper, rewrite an equation in a modern programming language and return code that passes formal checks. That is already useful to a climate modeller. The stronger claim—that the generated equations actually predict physical processes better than an established model—requires a fair comparison. In a preprint posted November 2, Columbia University researcher Kara D. Lamb tested language-model agents on how cloud droplets combine into rain. Of 100 unattended agent-generated candidates with iterative refinement, two achieved slightly lower rain-mass error than the strongest conventional scheme after that scheme's parameters had been retuned on the same training data.[1]

The experiment separates useful scientific programming from a harder achievement: improving the underlying predictions. Many agents produced code consistent with the required physics, but only a small fraction improved on the retuned reference. Computational cost adds another comparison, because the established scheme was often cheaper to run.[1]

Why the comparison changed when the old model was retuned

Earth-system models divide the atmosphere into cells that are far larger than individual water droplets. They use parameterisations, simplified equations for unresolved physics, to estimate changes such as cloud water becoming rain. In the test case, the reference process was warm-rain collision and coalescence, simulated at higher resolution by a superdroplet model named McSnow. The trial used 719 initial-condition cases for training and validation and 100 held out for testing, with each initial case simulated as a stochastic ensemble.[1]

The research compared published bulk parameterisations, their retuned versions, data-driven models and new closures generated by agents. An old formula used with its original published constants is not necessarily the strongest competitor in a different simulation setting. In the appendix, several conventional models improved sharply when their coefficients were fitted against the new training data. For example, the published Khairoutdinov–Kogan reference had a held-out loss of 6.021 on the specified composite metric, falling to 0.632 after retuning. Those figures are not real-world rainfall percentages; they are values of the paper's model-error measure under its defined test conditions.[1]

Nineteen of the 100 iteratively refined candidates beat the best published, unretuned scheme. Only two beat its retuned version, both generated by Claude Opus 5.5. The large change in the count shows how strongly the claimed advance depends on whether the classical reference has been calibrated to the same data.[1]

Where the agents did useful scientific work

The agent workflows had to produce differentiable JAX code while preserving physical constraints, including conservation of water mass. Some tasks required translating historical literature, including equations spread across multiple papers. The translation success varied by scientific scheme and the completeness of the supplied material. That is a practical limitation: an agent cannot faithfully implement missing equations just because it can produce compiling code.[1]

In unattended design experiments, ten models each generated candidates and received performance feedback. Median rain-mass error fell across refinement rounds for eight of the ten models, but the improvement was statistically significant for only one. The authors also report that for half the models, the best of four independent attempts performed at least as well as the median iteratively refined result. More rounds of feedback did not uniformly transform weak designs into strong ones.[1]

The work did not integrate its new closures into an operational weather forecast and demonstrate better predictions of actual storms. It tested one constrained cloud microphysics problem in a simulated benchmark. Transfer to different climates, numerical time steps, mixed-phase clouds and broader coupled models remains an open requirement. A solution can be accurate and mass-conserving inside this setup yet unstable or poorly generalised elsewhere.[1]

The near-term application is a research assistant that retrieves formulas, translates methods into inspectable code, runs controlled comparisons and proposes alternatives. This experiment makes the acceptance test clear: calibrate the established method fairly, evaluate new conditions and compare computational cost alongside prediction error.[1]

Report an error or suggest a correction