A convincing online voice can still misrepresent the person it is meant to imitate. Researchers comparing AI-generated replies with human conversations found that techniques which made the language seem more natural sometimes moved it further from what people had actually said. The result places a practical demand on social simulation: measure both the realism of the voice and the fidelity of its message.[1]
Nicolò Pagan, Petter Törnberg, Christopher Bail and Christopher Barrie evaluated nine open-weight language models with five calibration strategies, including style prompting, fine-tuning and retrieval of conversational context. Their Conversational Human Authenticity Test, or CHAT, combines detectability, semantic similarity and interpretable linguistic features. The study used interactions from X, Bluesky and Reddit.[1]
Style and meaning can pull in different directions
Suppose a human reply is sceptical of a proposal but expresses the doubt politely. A model that has learned a platform’s aggressive conversational style might make the response sound more locally authentic while turning hesitation into hostility. Conversely, a model trained to be uniformly helpful might smooth an angry response into bland agreement. In either case the surface resembles a familiar genre, but the position conveyed is different. This example is illustrative, not a case study from the dataset. It exposes why semantic fidelity and stylistic realism need to be measured separately when a synthetic population is used to study human beliefs.[1]
In the tested models, emotional tone remained a persistent source of difference from human language. Larger models did not obviously improve human-likeness, and instruction-tuned versions generally performed worse than base models on that objective. Assistant behaviour and faithful simulation of a social-media participant are different targets: fluency or helpfulness alone cannot select the right model for both.[1]
Separate the questions a synthetic conversation must answer
| Evaluation goal | What to compare |
|---|---|
| Style realism | Conversational habits and emotional expression |
| Semantic fidelity | The stance and content of the response |
| Detectability | How distinguishable generated and human samples are |
| Simulation validity | Whether the resulting patterns fit observed human behaviour |
A synthetic population needs an external reference
Consider an analyst modelling how a community might react to a change in platform rules. If every simulated participant becomes more polite, more hostile or more agreeable than the observed users, the resulting discussion can systematically shift. Generating more participants will not remove that mismatch. The comparison with real conversations has to remain part of the experiment as the simulated population grows.[1]
CHAT provides an aggregate validation framework for particular datasets, rather than a guarantee that any individual AI-written comment can be detected. Its practical use is to reveal mismatches that a plausible-looking sample may conceal. A researcher can then choose a calibration method according to the property the simulation needs to preserve.[1]
Evaluate the position behind the prose
A simulation of disagreement may need to preserve the direction and strength of each position; a study of emotional contagion needs the distribution of emotions to remain representative. The same calibration can serve one purpose poorly while making the dialogue sound more natural. Separating those measurements lets researchers choose a method according to the behaviour their experiment actually needs to model.[1]
Sources & further reading
- AI-generated language remains systematically distinguishable from human communication Duke University Scholars · 20 Oct 2026 · Accessed 25 Oct 2026