Five experiments in schools, universities and human tutoring give contrasting answers about educational AI. Some systems improved learning or helped tutors; another course-integrated system was associated with worse final grades. The comparison below separates the learners and interventions before looking for a useful teaching principle.[1][2][3][4][5]
The central distinction is between an answer service and a learning design. A good tutor manages difficulty, elicits an attempt, diagnoses the mistake and gives the smallest useful hint. A weak tutor optimises for immediate satisfaction: it removes the struggle in which understanding is formed.
What recent experiments found
| Study | Setting | Reported result |
|---|---|---|
| Harvard physics RCT | College students; custom tutor designed with research-based pedagogy | Students learned significantly more in less time than in an in-class active-learning condition and reported greater engagement.Source 1 |
| Khanmigo two-year trial | 18 Tennessee middle schools; remedial mathematics sessions | Assignment raised achievement by about 0.06–0.08 standard deviations annually, but students used substantive dialogue infrequently.Source 2 |
| NUMI mastery experiment | More than 6,000 middle-school students | AI slowed progress but improved next-attempt correctness after mistakes; the clearest delayed evidence appeared when AI was embedded in a mastery workflow.Source 3 |
| University of Maryland trial | 2,379 undergraduates and 30 instructors across disciplines | Within matched course sections, tutor access reduced final grades by 0.37 standard deviations and LMS participation by 0.90 standard deviations.Source 4 |
| Tutor CoPilot | More than 700 tutors and 1,000 students | Students were four percentage points more likely to master topics; gains reached nine points with lower-rated tutors, at about $20 per tutor per year.Source 5 |
The best result is not “AI beats a lecturer”
The Harvard study is exciting because a carefully designed tutor produced more learning in less time in that setting. It does not establish that a generic chatbot is superior to every teacher or course. The intervention was built around pedagogical practices, specific material and a defined learning outcome. The lesson is that design can turn a language model into an effective tutor—not that the model’s default conversation automatically does so.[1]
The Maryland result supplies the counterweight. Students offered a course-integrated tutor performed worse and disengaged from the learning-management system in the clearest comparison. One plausible interpretation is displacement: the convenience of the tutor replaced encounters with readings, assignments or other material that carried essential structure. The paper reports the outcome; it does not imply that every AI tutor will cause the same effect.[4]
A tutor should make the student do the cognitive work
Seven design rules for a productive AI tutor
| Measure | Value | Context |
|---|---|---|
| Diagnose first | Ask the learner to explain or solve before teaching | The response reveals the actual misconception. |
| Hint before answer | Offer the smallest step that unlocks progress | Preserves effort and lets the learner own the solution. |
| Require explanation | Ask why an answer is correct and what would change it | Fluency becomes visible understanding. |
| Use retrieval | Return to earlier material without showing the solution | Strengthens recall rather than recognition. |
| Track errors | Keep a ledger of misconception, hint and later retest | Personalisation becomes evidence-based. |
| Ground in the course | Use the syllabus, textbook, lecture notes and grading criteria | Prevents a generic answer from silently replacing the curriculum. |
| Escalate uncertainty | Mark disputed facts and route consequential questions to a teacher | The tutor does not pretend authority it lacks. |
A concrete “college tutor” session
Suppose a student is learning opportunity cost in economics. Instead of asking for an explanation and copying it, the tutor begins with a decision: “You can work a four-hour shift for £48 or spend the time studying for an exam. What information do you need before deciding?” The student answers. The tutor identifies whether the student understands that the cost of studying includes the foregone wage, then changes one assumption and asks the student to recalculate.
The tutor can next generate a near-transfer problem, delay the answer, compare the student’s explanation with the course definition and schedule a brief retest the following day. At the end it produces an error summary for the learner—not a polished essay to submit. The output is evidence of learning activity: attempts, revisions and what still causes difficulty.
Engagement is a feature, not an afterthought
Khanmigo reached almost every assigned student at least once, but the median student messaged it on only a third of practice days and used it after a mistake in only 17% of relevant exercise sessions. Access alone did not create sustained tutoring. The trial’s effects resembled ordinary Khan Academy practice, and the authors identify engagement as the binding constraint.[2]
The NUMI experiment suggests a second constraint: good tutoring may slow visible progress because the system keeps a learner with the mistake longer. Students attempted fewer questions, but were more accurate on the next attempt after an error. A dashboard that rewards questions completed can therefore punish the very behaviour a tutor is meant to create.[3]
AI can also make a human tutor better
Tutor CoPilot offers a different model. Rather than replacing the tutor, it recommends pedagogical moves in real time. The largest gains appeared with lower-rated tutors, and message analysis found more probing questions and less generic praise. This may be one of the most scalable uses of educational AI: distributing expert instructional practice while preserving the human relationship.[5]
Sources & further reading
- AI tutoring outperforms in-class active learning: an RCT Harvard Science Education Research Laboratory / Scientific Reports · 1 Jan 2025 · Accessed 9 Oct 2026
- One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment National Bureau of Economic Research · 1 Aug 2026 · Accessed 9 Oct 2026
- Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice Experiment National Bureau of Economic Research · 1 Aug 2026 · Accessed 9 Oct 2026
- The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement University of Maryland Center for Educational Data Science and Innovation · 1 Oct 2026 · Accessed 9 Oct 2026
- Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise EdWorkingPapers / Annenberg Institute at Brown University · 1 Nov 2025 · Accessed 9 Oct 2026