AI investment benchmark separates research quality from forecasting performance

Investor's Last Exam tests whether an AI investment agent can turn the evidence available at a historical cutoff into a sound forecast. Its first results show a gap between well-supported research and accurate predictions: the systems that led the overall assessment did not lead every measure of forecasting or simulated investment performance. The benchmark was developed by Prophetlab with investment-industry and Shanghai Jiao Tong University partners.[1]

The benchmark, led by Prophetlab with investment-industry and Shanghai Jiao Tong University partners, contains 86 assignments for 11 evaluated systems. Seventy-eight are company research tasks and eight concern markets. The team assembled a snapshot of 3,430,336 versions of source documents in nine categories, totalling 1.52 terabytes as of 15 October 2026. Reports, earnings calls, analyst estimates, institutional holdings and dated news are among the materials used to reconstruct what a researcher could have known.[1]

Analysis is not the same as predicting

In the headline answer-quality scoring, GPT-6 Astra led with 68.1 points out of 100, followed by Claude Opus 5.5 at 66.9. But the more revealing comparison is within the tasks: Astra's score for Research & Evidence was 78.9, against 52.5 for Forecast & Model. On realised forecast-outcome criteria, DeepSeek V4.1 Flash led at 40.3, ahead of Astra's 37.7. Those numbers are benchmark scores, not percentages of forecasts that were correct, and they concern different evaluation dimensions.[1]

The study gives a particularly clear example involving Tesla. On an assignment using evidence dated 22 June 2026, Astra, Opus 5.5 and Qwen 3.8 Max each obtained full marks for supporting analysis while receiving zero marks for their realised forecasts. The systems underestimated second-quarter deliveries and revenue and overestimated automotive gross margin beyond the specified error thresholds. A coherent chain of reasoning had not prevented incorrect operating assumptions.[1]

Another task considered American Electric Power's cash generation. Operating cash inflow increased by $750 million between the first halves of 2025 and 2026, yet investing cash use increased by $1.255 billion. Subtracting operating inflow from investing use therefore shows the pre-financing funding shortfall widened from $2.676 billion to $3.181 billion, an increase of $505 million. All eleven systems recovered the source inputs, but only five computed both funding gaps. The distinction is ordinary financial analysis: more cash produced by operations does not necessarily mean less outside financing is required.[1]

The team separately simulated trades derived from fundamental forecasts. Across 20 historical opportunities and 11 companies between March and August 2026, Hunyuan 4 Preview produced a reported cumulative return of 8.50%, versus 1.18% for an always-long control. Astra returned 7.18%. These are not annualised returns, and the experiment assigned only one-twentieth of initial capital to each opportunity, without leverage or redeploying unused cash. The assumptions include transaction charges and short-borrowing costs, but they do not reproduce a live fund's entire constraints.[1]

What the locked date cannot prove

The historical cutoff governs what the agent's tools can retrieve; it cannot remove information already learned during model training. The authors describe the work as a preprint. They also disclose that GPT-6 Astra judged semantic criteria, while numerical checks were deterministic; regrading with another model changed some scores and rankings. That makes the choice of judge relevant when reading Astra's overall lead. The published overview and protocol say the full versioned archive and executable evaluation infrastructure are not distributed with the reported results.[1]

The AEP and Tesla examples expose two different weaknesses: failing to complete a calculation from correct inputs, and making a forecast that later proves wrong despite coherent supporting analysis. Reporting those outcomes separately gives investment teams a more useful basis for evaluation than a single score for fluent research writing.[1][2]

Report an error or suggest a correction