AI researchers are publishing more of the materials that let other teams inspect and repeat their work. A study of five major conferences found that the share of empirical papers reporting both open code and open datasets rose from 11% in 2014 to 64% in 2024. That change happened while annual publication volume grew roughly tenfold, making the improvement in documentation significant for a field producing far more work to scrutinise.[1]
Kevin Coakley and colleagues examined papers from AAAI, ICLR, ICML, IJCAI and NeurIPS over eleven years. Of 56,800 papers, 52,328 were classified as empirical research and assessed for methodological disclosures. The team used an AI-assisted extraction process, with prompt optimisation and manual checks, to identify what authors documented about their experiments. The analysis measures those disclosures, rather than independently rerunning the studies.[1]
More experiments arrive with their working materials
Papers reporting neither open code nor open datasets fell from 29% to 4%. The researchers also tracked seven indicators: code, datasets, dataset splits, pseudocode, hardware, software dependencies and experimental setup. The share documenting at least five rose from 8% to 43%. These details make it easier for another team to locate the materials, reconstruct an environment and understand how a reported result was produced.[1]
A decade of documentation changes
| Measure | 2014 | 2024 |
|---|---|---|
| Both code and data openly shared | 11% | 64% |
| Neither code nor data shared | 29% | 4% |
| At least 5 of 7 method indicators described | 8% | 43% |
| Reproducibility, estimated from earlier rates | 28% | 64% |
The authors separately estimated that reproducibility rose from 28% to 64%. They applied success rates from an earlier study—six reproduced results among seven papers sharing code and open data, and five among fifteen sharing open datasets only—to the newer distribution of papers. That small calibration sample and its assumed applicability across years are important limits on the estimate. It provides a projection of what better sharing might enable, distinct from the directly measured disclosure trend.[1]
Make the repository usable
A code link is a starting point. Another researcher still needs the appropriate data, dependencies, preprocessing decisions and experimental settings. A repository containing only part of the pipeline can leave the hardest step undocumented. The study’s binary measures count whether an item is reported; they do not grade the completeness of every repository. For authors, the practical standard is whether someone else can reconstruct the experiment from the supplied materials.[1]
Documentation was already improving before conference reproducibility checklists arrived. The authors found no systematic acceleration attributable to those introductions. That makes the trend broader than a simple response to mandatory forms: checklists can organise disclosure, while the analysis shows a community shift that had begun earlier.[1]
Openness at a much larger scale
The conferences published 1,206 papers in 2014 and 12,026 in 2024. More open materials across that growing output give researchers a larger base of work to interrogate, reuse and extend. The remaining gap is practical as well as numerical: accessible files must remain available and sufficiently complete to support another team’s attempt.[1]
The direct result is a substantial improvement in the starting conditions for verification. More papers now supply the materials and methodological details that make independent checking possible. For the team trying to reproduce a result, those improvements are most useful when the shared files form a usable path from the documented method to the reported output.[1]
Sources & further reading
- The Shift Toward Open and Reproducible AI Research arXiv / Coakley and colleagues · 15 Jun 2026 · Accessed 25 Oct 2026