AI help improved practice scores, but participants struggled more after it was removed

AI assistance can improve a practice session while leaving people less successful when they must work alone. Three online experiments found that participants who had access to an assistant performed worse on later unaided problems than control participants. The study also found evidence of reduced persistence, although that result was not equally strong in every experiment.[1][2]

The research, highlighted by UC Berkeley on 9 October, recruited 1,222 US-based adults through Prolific for two fraction-solving experiments and one reading-comprehension experiment. These were online participants, rather than a sample defined as students. The manuscript was first posted in April and revised on 13 October.[1][2]

What changed when the assistant disappeared

Participants were randomly assigned to an AI-assisted or control condition. They completed a short practice phase, after which assistance was removed and both groups answered the same final problems independently. The studies measured correct answers and skipped problems. The AI setup made direct solutions readily available, so it tested a particular style of help rather than a tutor designed to withhold answers.[2]

The first fraction experiment found a large unaided performance gap, but its exclusions could have retained weaker participants in the AI group who had used correct AI answers during practice. The researchers addressed that concern in the second experiment by using an unaided pretest for exclusions and removing a reference sidebar from the control group too, making the interface change more comparable.[2]

Performance and persistence need separate readings

The larger experiment's accuracy result supports an immediate cost to independent performance under the tested conditions. Its persistence result is less decisive: the observed difference in skipping could not be distinguished from chance at the study's significance threshold. The first fraction experiment and the reading experiment did report significant skipping differences.[2]

The researchers also examined how people chose to use the assistant. Those who reported obtaining direct answers showed poorer later performance and persistence than those using hints or clarification. That subgroup comparison is observational within the AI condition: participants were not randomly assigned to request answers rather than hints, so it cannot by itself prove that the choice caused the difference.[2]

After exclusions, the analysed samples were 305,585 and 168. The headline 1,222 therefore describes recruitment, not the total number included in the main analyses. The distinction helps readers understand both the scale and how the researchers dealt with task engagement and baseline ability.[2]

The design question is what practice preserves

The experiments concern sessions of roughly 10–15 minutes and performance immediately after assistance was withdrawn. They do not measure permanent changes in ability or the long-term effects of all AI use. The authors propose that ready access to solutions can displace practice or change expectations about how much effort a problem should require.[1][2]

For learning tools, that creates a useful distinction between helping someone finish an exercise and helping them develop the ability to finish a later exercise alone. A system can be judged on both outcomes. Hints, explanation and gradually reduced support are design possibilities to investigate, with independent performance remaining the measure that reveals whether the learning transferred.[1]

Report an error or suggest a correction