AI-written therapy feedback helps in a trial, but misses the noninferiority test

AI-written feedback accompanied improvements in social-anxiety symptoms during a nine-week online therapy trial, but the study did not establish that it was no worse than human feedback within the researchers’ chosen margin. The 263-person CHAI trial tested a specific clinical workflow: participants completed structured cognitive behavioural therapy exercises, and a psychologist checked every AI reply before it was sent.[3][1]

Researchers randomized adults with social anxiety into six groups. Four received roughly weekly homework feedback, written either by a therapist or an AI model. Participants were also given either correct or deliberately incorrect information about who wrote it, separating the effect of the message from expectations about its author. A fifth group used the programme without feedback; a sixth waited nine weeks for treatment.[1][3]

What improved, and what the comparison established

In his account of the results, study author Per Carlbring reports that participants who completed the final assessment had average Social Phobia Inventory scores fall from 40.4 to 24.2 with therapist feedback and from 39.9 to 28.2 with AI feedback. Lower scores mean fewer symptoms. Those before-and-after figures describe assessment completers; they are distinct from the statistical analysis including all randomized participants.[3][1]

That analysis estimated a 1.15-point difference favouring human feedback, with a confidence interval from −3.6 to 5.9 points. The researchers had prespecified a 3.3-point margin for deciding that AI was not meaningfully worse. Because the interval extended beyond that margin, noninferiority was not established. This is more informative than simply saying the trial detected no difference: its results allow both comparable performance and a difference large enough to matter under the study’s definition.[3][1]

The study also found no measurable effect from the stated identity of the writer on symptoms, reported negative effects or the perceived working relationship. The design addressed expectations directly: some participants read AI-generated replies while believing a therapist had written them, and others received therapist replies labelled as AI. The ethically approved study warned participants that some treatment information could be inaccurate and offered debriefing afterwards.[3][1]

A psychologist-reviewed workflow

The model followed detailed instructions based on the programme’s self-help material and the researchers’ work on therapist feedback. A psychologist reviewed its drafts and reportedly rarely changed them. It ran on a server in Sweden, received homework responses without identifying details or age and gender, and was not trained on participants’ data. These conditions define the intervention actually tested.[1]

The results are published in Internet Interventions. The university disclosed that Carlbring is the journal’s editor-in-chief and that an independent editor handled the manuscript, without his involvement in its editorial process or peer review.[1]

The trial separates two questions that are often combined: who drafts therapy feedback and whom patients believe is replying. It found no measurable effect from the sender’s stated identity, while leaving the central comparison between AI- and human-written feedback unresolved. The tested approach retained a psychologist’s review of every AI response.[3][1]

Report an error or suggest a correction