A convincing fake social-media operation no longer needs one model pretending to be one person. It can generate an entire thread: the opening post, several apparent users, replies, disagreement, humour and the small shifts of tone that make a group feel real. A study calling this the Collective Turing Test found that readers mistook model-generated Reddit discussions for human conversations 39% of the time.[1][2]
The final analysis retained 203 participants after attention checks and contained 1,205 annotations. Each task showed a real Reddit conversation beside an artificial conversation on the same topic, with the same reply structure and approximately matched comment lengths. Participants knew one was AI-generated and had to identify it. Even with that warning, Llama 3 70B conversations were correctly identified as artificial only 56% of the time.[1][2]
How the collective test was constructed
| Measure | Value | Context |
|---|---|---|
| Participants recruited | 251 | English-fluent participants were recruited through Prolific.Source 2 |
| Participants retained | 203 | Final analysis contained 1,205 annotations after exclusions and removal of attention-check items.Source 2 |
| Models | GPT-4o and Llama 3 70B | Artificial threads were generated before the experiment.Source 1Source 2 |
| Topics | Authentic Reddit posts | AI comments followed the real thread structure and target word counts.Source 2 |
| Overall error | 39% | AI conversations were selected as the human conversation at this rate.Source 1 |
| Llama identification | 56% correct | Barely above the 50% chance level in the two-choice task.Source 1 |
A thread has signals that one comment does not
Readers can look for distinct positions, accurate replies, shifts in tone and coherent disagreement. The researchers therefore tested threads containing one to sixteen comments. Longer conversations did not reliably improve recognition: the study found no significant linear effect of length. A separate comparison found better detection at eight comments than at one, but the pattern did not establish that more context generally exposes a generated thread.[2]
The generation method was also relatively simple. Independent model calls produced comments in sequence using the post and comments generated so far. The system did not rely on elaborate persistent personas or a hand-built influence campaign. That makes the result a warning about the baseline capability available before more sophisticated orchestration is added.
Six applications and risks of simulated discussions
| Use | What realism can help with | Evidence still missing |
|---|---|---|
| Recommendation testing | Expose an algorithm to plausible conversational structures | Whether simulated reactions match real behavioural responses. |
| Policy experiments | Explore possible moderation or ranking effects | External validity across communities, cultures and time. |
| Training moderators | Create varied examples of conflict, drift or manipulation | Coverage of rare harms and expert review of labels. |
| Research on group dynamics | Generate controlled counterfactual discussions | Proof that the model reproduces causal social mechanisms, not only surface style. |
| Influence operations | Scale apparent consensus or opposition | Detection, provenance and platform enforcement rather than scientific usefulness. |
| Synthetic user testing | Stress-test conversational products | Whether fake users reproduce real needs, frustration and abandonment. |
Realistic is not representative
A model can imitate the style of Reddit comments while reproducing stereotypes, smoothing genuine minority positions or overusing patterns from its training data. Passing a perceptual test means the discussion looked human enough to readers. It does not mean the simulated population has the same beliefs, incentives, network structure or response to policy as the community being modelled.
This distinction is critical for social science. A simulation can be useful for generating hypotheses or stress-testing a design. It should not be treated as evidence that real people would react the same way unless the behavioural outputs are calibrated and validated against held-out human data.
The same realism can manufacture social proof
People do not only evaluate a claim; they notice whether several apparently independent participants support it. A generated thread can create the impression that a product is popular, an allegation is widely corroborated or a political view is socially normal. The manipulation comes from the apparent plurality, even when no single comment contains a dramatic falsehood.
- Require provenance or platform-level labelling for synthetic multi-user simulations used publicly.
- Detect coordinated generation at the account and conversation level, not only sentence by sentence.
- Preserve creation metadata and the model or campaign identity across reposts where possible.
- Audit systems for simulated consensus, reciprocal replies and artificial disagreement designed to look organic.
- Validate research simulations against real behavioural outcomes before using them to predict policy effects.
- Give users a way to inspect whether a thread’s apparent participants are independent entities.
The Collective Turing Test is important for both simulation and defence. It shows that a model can generate a group interaction readers struggle to distinguish from a real one. The constructive path is to use that ability inside controlled, validated research environments. The defensive path is to stop assuming that conversational plurality proves human plurality. In the age of generated communities, the identity and independence of the speakers become part of the evidence.
Sources & further reading
- The collective turing test: large language models can generate realistic multi-user discussions Scientific Reports · 17 Jul 2026 · Accessed 11 Oct 2026
- The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions Azza Bouleimen and colleagues / arXiv · 29 Oct 2025 · Accessed 11 Oct 2026