AI weather forecasts are getting sharper. The number that matters is whether their probabilities earn your trust

The WeatherNext 3 announcement in September describes hourly forecasts using live satellite observations and more detailed spatial outputs. Those are meaningful engineering changes. But the most useful question for a reader is not whether the map contains smaller pixels: it is whether the forecast supports a better decision.[1]

A map can be visually precise while the timing of a rain band remains uncertain. A probability can be carefully calibrated while offering little distinction between one day and the next. Understanding these differences makes AI forecasting more useful, without requiring the reader to become a meteorologist.

Four meanings of better

The practical advantage of a shorter update cycle is the opportunity to incorporate newer information. That is not a guarantee that each successive forecast improves. A change in the prediction can be a rational response to changing evidence, not evidence that the system is unreliable. The question is how those revisions perform over many cases.

Why a wet day does not disprove a 30% forecast

ECMWF’s guidance distinguishes reliability from sharpness. Reliability asks whether events occur at the frequencies implied by the probabilities. Sharpness concerns how decisively probabilities move away from a background frequency. Confident probabilities are valuable only when their confidence is justified.[2]

Imagine one hundred comparable occasions, each assigned a 30% chance of a specifically defined rain event. If rain occurs thirty times, the aggregate frequency matches the forecast. The seventy dry occasions and thirty wet ones are both compatible with that statement. Judging the forecast as “wrong” whenever the less likely outcome occurs would misunderstand what probability is for.

The event must remain consistent: for example, at least a specified amount of rain in a defined area during a defined period. Changing the area or time window after seeing the outcome changes the question. A precise probability attached to an ambiguous event is not especially informative.

A small calculation separates confidence from skill

The Brier score measures the average squared difference between a forecast probability and an outcome coded as one for an event and zero otherwise. Lower is better. It evaluates probabilistic forecasts without forcing them into yes-or-no predictions.[2]

For our hypothetical hundred occasions, a constant 30% forecast scores 0.49 on each wet occasion and 0.09 on each dry one. The average is (30 × 0.49 + 70 × 0.09) ÷ 100 = 0.21. A constant 90% forecast looks more decisive but scores 0.57 on those same outcomes. These are calculations from invented examples, not WeatherNext evaluation results.

There is still a weakness in the 30% forecast: it never distinguishes one occasion from another. It can match the overall frequency while adding no day-specific information. A stronger forecaster would identify which circumstances justify moving above or below that baseline—and remain calibrated when it does.

ECMWF’s March research discussion makes a related point about traditional scores. Smoothing a forecast can reduce measured error without adding information about the atmosphere. Its proposed decomposition separates information from noise so that a cleaner score is not automatically interpreted as a richer forecast. This is the researchers’ evaluation framework, not a finding that WeatherNext’s improvement is merely smoothing.[3]

Why action can make sense below 50%

Consider a deliberately simple, non-safety-critical example. A temporary cover costs £8 and completely avoids £40 of damage if rain occurs. Assume no other costs, no benefit when dry and a trustworthy probability. Without the cover, expected loss is the rain probability multiplied by £40. Buying the cover becomes cheaper when the probability exceeds 20%: £8 ÷ £40.

At a 30% rain probability, expected uncovered loss is £12, compared with £8 for protection. Rain remains less likely than no rain, yet acting is economically reasonable under the assumptions. Change the potential loss or the effectiveness of the cover and the threshold changes. This example is not advice for hazardous weather, where official warnings and safety procedures take priority.

The wider lesson is that a forecast does not contain the decision by itself. Two people can read the same well-calibrated probability and sensibly act differently because their consequences differ. An AI assistant explaining the forecast should expose that decision structure, not translate every value below 50% into reassurance.

How to compare a new forecasting model fairly

The WMO Lead Centre’s verification guidance warns against ranking systems whose evaluation periods differ and notes that ensemble size can affect some scores. A meaningful comparison should align the forecast cases and the verification setup before attributing a difference to model quality.[4]

For a reader, the essential fields are the variable, location, forecast horizon, observation source, evaluation period and baseline. A large average improvement may conceal smaller gains on the event that matters to a particular user. A comparison over a few memorable storms cannot establish the frequency behaviour of a whole season.

Our assessment is that AI weather progress should be welcomed on two levels: better representations of the atmosphere and better explanations of uncertainty. Finer grids and frequent updates improve the information available. Trust comes from showing what the probabilities mean, how they have performed and which decisions they can support. The most useful forecast is not always the most confident-looking one.

Report an error or suggest a correction