ChatGPT lifts consultants’ data-science performance without a detected gain in unassisted knowledge

Management consultants given ChatGPT access and training performed better on unfamiliar data-science tasks in a randomized trial involving 487 consultants. On a subsequent test of technical questions without ChatGPT, however, the treated group showed no detected advantage over the control group. The Nature Human Behaviour study separates improved assisted performance from demonstrated knowledge acquisition.[1]

The distinction is economically important. A company may observe that employees can now complete tasks previously assigned to specialists and conclude that its workforce has been reskilled. The trial suggests a narrower interpretation. The combined human–AI system acquired a new capability; the human did not necessarily acquire the underlying knowledge.

The tasks were outside the consultants’ normal expertise

The treatment group received access and training in using ChatGPT for data-science problems beyond their existing skills. The tasks required human–AI collaboration because ChatGPT could not solve them independently. Treated participants scored higher than controls on all three evaluated tasks, although they still fell short of expert data scientists.[1]

This is stronger evidence than a before-and-after workplace survey because the intervention was randomized. It also avoids an easy task design in which the model simply returns a finished answer. Yet the study remains a controlled experiment with a particular professional population, training programme and set of data-science tasks. It does not establish that every role, tool or learning objective will follow the same pattern.

An exoskeleton changes productivity and dependency together

The authors describe AI as an exoskeleton: it can support performance while in use without necessarily building the knowledge needed to do the same work alone. The post-experiment test found no evidence of an unassisted knowledge advantage; it cannot establish that no participant learned anything, or what a longer training programme might achieve.

That dependency is not automatically undesirable. Organisations already rely on calculators, search engines, spreadsheets and specialist software. The mistake would be to treat tool-mediated capability as interchangeable with internal expertise. The two fail differently. A skilled worker may slow down when a system is unavailable; a worker who never learned the fundamentals may be unable to detect a confident but consequential error.

Training needs two separate objectives

For employers, the result separates two evaluation questions. Can staff complete better work with the tool? Can they explain and perform the relevant reasoning when it is unavailable? Assessing both would distinguish a useful assisted workflow from a programme that also develops independent expertise.

The provenance of the study matters

Several authors are or were employees of Boston Consulting Group, the participating organisation, and one author works for OpenAI. The lead author received consulting fees from BCG during the experiment, which used BCG employee time and internal resources. The paper discloses these relationships, and its analysis code is public, while access to participant data is restricted to qualified academic review under an agreement.[1][2]

Report an error or suggest a correction