A large imitation study separates human likeness from task performance

An answer can be good at its task without resembling the answer a person would give. A Nature Human Behaviour study published on 26 October measures that distinction across six language and vision tasks. It brings together responses from 636 people and 37 AI agents, assessed through 72,191 Turing-like trials involving 1,916 human judges and ten machine judges.[1]

The tasks covered image captioning, word association and conversation, alongside colour estimation, object detection and attention prediction. Judges tried to identify whether a response or behaviour came from a person or a machine. The objective was recognisable human likeness, rather than a test of consciousness or a universal measure of intelligence.[1]

Some machines were difficult to identify

For the three language tasks, average imitation-detectability scores were 0.57 for captioning, 0.53 for word association and 0.57 for conversation. The measure combines correct identification of human and machine responses; 0.5 represents chance-level discrimination and 1.0 perfect discrimination. The scores show how hard the distinction was under these conditions.[1]

The conversational setting also mattered. Blenderbot was labelled human in 67% of judgements on conversations between AI agents, compared with 49% when it conversed with a person. ChatGPT-4o was somewhat easier to detect than Blenderbot in the reported comparison, despite its broader capabilities. Human likeness depended on the interaction being judged, not simply a model’s general reputation.[1]

Across the tested systems, conventional task scores showed little correlation with imitation performance. This is plausible in vision: two systems may correctly identify different objects in an image, yet only one pattern of selection resembles where a person would look. Getting an acceptable answer and matching human behaviour are distinct evaluation targets.[1]

Machine judges had an advantage—with exceptions

Classifiers trained to distinguish response origins often outperformed human judges, including in object detection and attention prediction. Colour estimation was an exception: human judges did better than the machine judges in the reported comparison. Feedback and the visual features supplied to a classifier also affected results.[1]

A trained detector can exploit regularities that people overlook, but its success is tied to the examples and features it learned. A classifier that separates the study’s response sources is not thereby validated as a universal detector for new models, different languages or unfamiliar tasks.[1]

The paper includes systems from different generations, so its publication date should not be mistaken for a ranking of all current frontier models. Its lasting contribution is the evaluation design: measure task quality and recognisable origin separately. An assistant becoming more convincing as a conversational partner does not, by itself, establish that its answers have become more accurate.[1]

Report an error or suggest a correction