Why the same AI prompt can produce a different answer

A language model can give two different answers to the same prompt because choosing its next piece of text can involve sampling. Even when that choice is made deterministic, the computation supplying the choices can change slightly. Repeatable AI therefore requires control over both the generation rule and the system executing it.[1][3]

The next word has more than one possible route

An autoregressive language model builds an answer token by token; a token can be a word or part of one. Greedy decoding selects the highest-probability next token. Sampling draws from the available probability distribution, allowing other candidates to be chosen. Hugging Face documents both methods. After one choice changes, the continuation receives a different sequence to work from.[1]

The visible prompt also needs to be distinguished from the complete input. Chat templates include messages and their roles, including earlier exchanges and system instructions. Repeating a question after receiving an answer can therefore present a different conversation. This is an input change before any numerical issue arises.[2]

Tiny numerical changes can alter the winning token

Computers retain a limited number of significant digits when working with floating-point numbers. PyTorch documents that changing the order of operations can change the rounded result, and that a batched calculation need not match the same calculation performed individually, bit for bit. Mathematically equivalent work can follow different numerical paths.[3]

Consider an illustrative calculator that rounds every addition to three significant digits. Adding 90.1, then 0.06, then another 0.06 gives 90.2 after the first rounding and 90.3 after the second. Add the two small numbers first and the final sum rounds to 90.2. These are our invented decimal inputs, not a measurement of AI hardware; they show how regrouping can change a result.

Thinking Machines Lab investigated the corresponding problem in model serving. Changing the number of requests processed together can change the calculation strategy. Its solution kept the relevant operations consistent across batch sizes, including attention calculations. The mechanism concerns numerical computation; it does not require one user’s text to enter another user’s answer.[4]

Reproducibility is an engineering choice

Current vLLM documentation provides a practical implementation route: batch invariance, still labelled beta, makes results insensitive to request grouping. Its reproducibility guidance retains a boundary around the same hardware and vLLM version. A fixed random seed controls random-number generators; it is not, by itself, a promise that every layer of execution will remain unchanged.[5][6]

For someone investigating a changed answer, the useful distinction is between new inputs, a different sampling choice and a different execution path. A repeat with the same full input and documented settings narrows the question. Two responses that phrase a sound explanation differently are also a different issue from two responses that give incompatible facts.

Repeatability helps developers reproduce a failure and assess a repair. It does not establish that the answer is correct: an identical mistake remains a mistake. The achievement is being able to identify and control a source of variation, so a changed result can tell us more about what changed in the system.

Report an error or suggest a correction