Synthetic survey responses capture little of real economic preferences across 74 countries

Language models generated 27.8 million answers for a virtual version of a major international preferences survey, but captured little of the variation in the real responses. Against 67,201 people in 74 countries, the simulated answers explained 1.3% of individual variation and 4.2% of country-level variation. A plausible persona was a weak substitute for an observed respondent.[1][2]

The Nature Human Behaviour study, published on 12 October, matched simulated respondents to the Global Preference Survey and compared six economic preferences. Twenty model configurations covered different prompts, languages, ablations and robustness checks. The main individual-correlation analysis used 12 configurations, giving 72 model-by-preference comparisons.[2]

The survey was about choices, not trivia

The underlying Global Preference Survey measures six experimentally validated economic traits: willingness to take risks, patience, altruism, trust, positive reciprocity and negative reciprocity. The last two capture different responses to how someone else treats you. A person willing to reward kindness need not have the same inclination to punish unkindness. Such distinctions make the exercise more demanding than asking an assistant to recite population statistics.[2]

The prompts included age, gender, cognitive ability and country or region, with question language and monetary amounts adapted to the survey setting. This gave the models considerably more context than a country name alone. The test then asked whether that context recovered the preferences of the matched human respondents.[2]

In 63 of those 72 comparisons, the correlation between simulated and real individual preferences was statistically positive. That sounds impressive until the size of the relationship is examined. The authors calculated that model answers explained only 1.3% of the observed variation among individual human respondents. At the country-average level, explained variation was 4.2%. Statistical significance asks whether a pattern is distinguishable from zero; explained variance asks how much of the actual differences the model captures. A tiny but consistent signal can satisfy the first test while failing the practical second.[2]

Broad trends can coexist with weak individual prediction

The simulations recovered the direction of 77.8% of measured relationships involving gender, age and cognitive ability under the authors' confidence-interval criterion. Those groups are familiar from surveys, textbooks and countless public discussions, and a model may have learned broad regularities about them. But only 48.1% of the relationships involving cultural dimensions were recovered. When researchers looked within and between countries, discrepancies were not random scatter that could be dismissed by generating more text.[2]

The errors were often greater in countries with lower economic development and weaker digital connectivity. Prompts containing demographic characteristics contributed more to alignment than country labels alone. The study does not establish a single cause: training-data coverage, how preferences are expressed, model stereotypes and the difficulty of modeling culture could all matter. It does show that adding the correct country name to a prompt is not sufficient to obtain a reliable national survey estimate.[2]

A particularly revealing comparison concerns the models themselves. At country level their simulated answers were more similar to one another than they were to human observations. If several artificial survey agencies agree on the same figure, their consensus might reflect a shared training signal rather than independent corroboration. The risk is especially serious when the target is a poorly represented population: agreement between models may produce unwarranted confidence in an invented average.[2]

Where synthetic respondents can help

Artificial responses may still help pilot wording, test an analysis pipeline or generate hypotheses for a later human survey. In those roles, their limits are visible. They become a different kind of evidence when used to estimate what a population actually thinks or how a policy will affect it.[1]

There are also limitations to the benchmark. The study measured six specific economic preferences against one major cross-national survey. It does not prove that simulated respondents are useless for every research question, nor that all future models will remain this inaccurate. Survey data themselves contain sampling choices, response biases and uncertainty. But the benchmark is unusually valuable precisely because it contains real observed answers: it provides a demanding external reference instead of a subjective judgment that an AI persona sounds convincing.[2]

The study shows why believable answers and representative evidence require different tests. Agreement with a few demographic trends can coexist with weak prediction of individuals and countries. A synthetic survey needs an external human benchmark before its fluent respondents can carry the weight of a real population.[1]

Report an error or suggest a correction