At Procter & Gamble, one professional working with AI matched the performance of a two-person team without it on product-innovation challenges. In Alibaba after-sales service, AI assistance made chats faster and improved subjective quality on average, yet did not significantly reduce customer retrials and could worsen quality for top performers. In a 70,000-applicant hiring experiment, an AI voice interviewer collected information while humans retained the hiring decision.[1][2][3]
What five studies measured
| Measure | Value | Context |
|---|---|---|
| Product innovation | 776 professionals | Individuals with AI matched teams without AI on the experiment’s performance measure.Source 1 |
| Customer service | Randomised Alibaba field experiment | Faster issue identification and chats; improved ratings; no significant retrial improvement.Source 2 |
| Hiring interviews | 70,000 applicants | AI-interviewed applicants were 12% more likely to receive offers: a relative increase, not 12 percentage points. Humans made hiring decisions.Source 3 |
| Software engineering | 158 then 101 respondents; 95 matched | 82% reported less time writing code; work shifted toward direction, evaluation and correction.Source 4 |
| Econometric coding | Applied benchmark tasks | Constrained agents raised success from 74% to 96% for about eight additional cents per run.Source 5 |
The unit of productivity is a completed, accepted outcome
A system can make the first draft faster while increasing review cost. It can raise customer ratings without changing repeat contacts. It can write more code while moving effort into checking, integration and correction. It can standardise an interview without making the final hiring decision. Productivity therefore belongs to the entire operating path, not the time between prompt and response.
The Alibaba result makes the heterogeneity visible. Lower-performing agents gained most, narrowing the performance gap. Top performers showed little speed improvement and declines in both subjective and objective quality, which the authors connect to greater multitasking and slower return to the focal customer. Deploying the same assistant to everyone changed the workflow differently depending on the worker’s starting method.[2]
Why the same AI can help one workflow and hurt another
| Mechanism | When it creates value | Failure when copied blindly |
|---|---|---|
| Standardisation | Repeated interviews or service steps vary unnecessarily across people | Rigid conversation suppresses useful adaptation or accessibilitySource 3 |
| Expertise transfer | Less experienced workers can reuse patterns embedded in the system | Skilled workers are interrupted or pulled away from an already effective processSource 2 |
| Search and generation | The task has many candidate solutions and a clear external test | The system optimises a plausible draft without access to the real acceptance standardSource 1Source 5 |
| Agency and execution | The agent can run, inspect and revise code against tests | A loop produces more output while silently widening scope or costSource 5 |
| Supervisory work | Human effort moves to direction, review and exception handling | The organisation counts generated volume but ignores cognitive load and correction timeSource 4 |
The best performer is not always the biggest beneficiary
AI frequently compresses gaps because it makes useful patterns available to people who had less prior experience. That is commercially important: an organisation can raise the floor without waiting years for every worker to acquire the same tacit knowledge. It also means a rollout optimised around average gains can overlook the people for whom the tool is redundant or disruptive.
The coding evidence adds a second qualification. Developers reported less time on many production tasks, especially writing code, while their role shifted toward what the authors call supervisory engineering work. Perceived productivity remained high, yet the matched group reporting a worse experience in at least one dimension almost doubled from 14% to 27%. Faster production can coexist with weaker flow and greater cognitive burden.[4]
Measure the complete task
For a local trial, select a recurring job with a visible completion standard. Compare preparation, production, correction and acceptance—not just generation speed. Report results separately for experienced and less experienced staff. The studies above show why those distinctions can change the rollout decision.
Sources & further reading
- The Cybernetic Teammate: A Field Experiment on Generative AI Reshaping Teamwork and Expertise NBER · Accessed 14 Oct 2026
- Generative AI in Action: Field Experimental Evidence from Alibaba’s Customer Service Operations Ni et al.; arXiv · Accessed 14 Oct 2026
- Voice AI in Firms: A Natural Field Experiment on Automated Job Interviews Jabarian and Henkel; arXiv · Accessed 14 Oct 2026
- The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study Vella and Blincoe; arXiv · Accessed 14 Oct 2026
- AI Agents and Prompt Engineering in Econometric Coding NBER · Accessed 14 Oct 2026