AI-personalised lessons show no achievement gain in an 8,647-pupil trial

Software that chooses a different lesson for each pupil has an appealing promise: practice directed at the difficulty that particular child faces. A large field experiment in rural China found no significant achievement gain from the AI-personalised programme it tested.[1]

The study covered 8,647 pupils in 315 primary-school classes. Its abstract, posted by Stanford on 17 October, reports estimates precise enough to rule out non-trivial positive effects under the experimental conditions. The finding held across assessment-item difficulty and pupils’ initial achievement levels.[1]

The comparison included ordinary learning software

Pupils were assigned to AI-personalised computer-assisted learning, non-personalised computer-assisted learning, or educational activities without computer-assisted learning. The middle group is crucial: it allows the adaptive element to be judged against a software alternative, rather than treating all benefits of computer practice as benefits of AI.[1]

An active comparison makes a product claim more demanding. If both groups practise with software, an advantage for the adaptive programme would be evidence that its additional design choices helped. A comparison only with pupils receiving unrelated activities would leave more room for the amount or format of practice to explain a difference.[1]

The abstract also suggests that the AI programme may have been less effective than conventional software for pupils in the middle third of baseline achievement. That qualified subgroup result should not displace the central finding: this intervention did not demonstrate the expected achievement benefit.[1]

Personalisation describes a method, not an outcome

This was structured computer-assisted learning, not a direct trial of a contemporary conversational chatbot. The result applies to the tested programme, pupils and setting. It cannot establish that every adaptive system fails, or that a different tutoring design would produce the same result.[1]

Nor does a disappointing score identify the mechanism. Content selection, curriculum fit and how pupils use assigned practice are possible explanations, but choosing among them requires evidence about the intervention. The reported result establishes the performance gap that needs explaining; it does not settle its cause.[1]

The study’s useful contribution to education purchasing is the comparison itself. Before paying for an adaptive layer, a school needs to know whether it adds learning beyond a simpler alternative. In this experiment, the personalised label did not translate into the measured advantage.[1]

Report an error or suggest a correction