Randomized Measurement of AI Productivity Gains
When you need this method
Decisions about AI investment rest on a basis that is not one. The board paper contains vendor benchmark scores, an internal survey, and a percentage from a slide deck. All three measure confidence, not effect. Anyone building headcount plans, pricing assumptions, or a due diligence thesis on that is working with a number nobody has ever observed. What makes it worse is that self-assessment is not merely imprecise but can point the wrong way: people can feel faster while the clock says the opposite.
Approach
- 1Collect real, already existing tasks from the live backlog, not benchmark or demo tasks, and keep each one down to a few hours of work.
- 2Before assignment, have two estimates recorded per task: expected duration with AI and expected duration without. The without-AI estimate doubles as a difficulty measure and enters the later analysis.
- 3Randomize each task individually into the AI-allowed or AI-disallowed condition, and only after it has been described and estimated. Assign first and scope afterwards, and the procedure measures nothing.
- 4Record implementation time and check it against a second trace, such as screen recordings, tool logs, or ticket timestamps, so that compliance and reported time do not rest on self-report alone.
- 5Set the measured difference against the up-front forecast and against the post-hoc assessment. That gap is the real management output, because it tells you what your internal AI reports are worth.
- 6Use only the measured difference in business cases, break the result out by task type and familiarity with the codebase, and repeat the measurement, since tools and experience shift within months.
Typical application
A typical B2B SaaS in the HR space wants to raise development capacity by a third and justifies it with AI tooling. Team leads report roughly 30 percent time savings, and the board already has that figure in the headcount plan. Instead of taking the number, the company runs every backlog ticket under four hours through the procedure for six weeks: two up-front estimates per ticket, then a coin flip for the condition, then time tracking with spot checks against a second trace. What remains after measurement is a clear advantage on new, isolated modules and no advantage in the mature core that two developers have known for years. The headcount plan is built on the measured value rather than on 30 percent, and adoption goes first to where the measurement holds. The gap between reported and measured effect is minuted at board level so the next self-assessment is read differently.
Limits and counter-indications
The original finding is narrow: 16 developers, 246 tasks, mature open-source projects in which participants had an average of five years of prior experience, and tools from the February to June 2025 frontier. The authors explicitly warn against overgeneralizing and consider substantial speedup plausible for small greenfield projects or unfamiliar codebases. Implementation time was also self-reported; the screen recordings served to verify compliance and to label activity, not as the primary clock. The procedure itself has three known weaknesses. First, it needs many comparable tasks, and with small numbers the effect disappears into noise. Second, selection bias bites: for its follow-up round METR reports that 30 to 50 percent of developers withheld tasks they assumed AI would finish quickly, and that participants increasingly refuse the AI-free condition. Third, timekeeping becomes unreliable with agentic tools, because people work on other things while waiting. METR therefore abandoned task-level randomization in 2026. Outside software development the design is untested. And it is not operationally harmless: forbidding an employee a tool by lottery requires their consent, and in codetermined workplaces the involvement of the works council.
How to measure impact
Track two figures side by side: the measured percentage difference in time between the two conditions, broken out by task type, and the gap between that measurement and the participants' own assessment. The second figure tells you how far you can trust your own AI reporting.
Related methods
Tools for this
Sources
- 1.Becker, Rush, Barnes, Rein: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, arXiv:2507.09089 (opens in a new tab) · arXiv (Preprint, nicht begutachtet) · 2025 · academic and scholarly literature · describes the methodCarries the procedure in every step: developers supply lists of real issues from their own repositories, forecast the duration of each issue with and without AI, and only then is each issue assigned to a condition by a simulated coin flip. The paper states explicitly that tasks are defined before treatment assignment. Analysis runs through a log-linear regression that incorporates the estimated difficulty. Result: 24 percent faster expected beforehand, 20 percent faster estimated afterwards, 19 percent slower measured. Limits the source names itself: 16 developers and 246 issues, implementation time is self-reported and screen recordings serve to verify compliance rather than as the primary clock; the authors explicitly warn against overgeneralizing and consider speedup plausible on small greenfield projects. This is a preprint without peer review.
- 2.METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Blogfassung mit Studienaufbau) (opens in a new tab) · METR · 2025-07-10 · practitioner source · provides benchmark figuresSupplies the boundary conditions needed to replicate the setup: 16 developers from large, mature open-source projects averaging over 22,000 stars and more than a million lines of code, 246 real issues of roughly two hours each, compensation of 150 dollars per hour, screen recording during the work, and Cursor Pro with Claude 3.5 and 3.7 Sonnet as the primary tools. Limit: this is the conducting organization's own account rather than an independent review; the figures match the preprint.
- 3.Peng, Kalliamvakou, Cihon, Demirer: The Impact of AI on Developer Productivity, Evidence from GitHub Copilot, arXiv:2302.06590 (opens in a new tab) · arXiv (Preprint, Microsoft Research und GitHub) · 2023 · academic and scholarly literature · provides the contextShows why task selection drives the result and thereby justifies the central rule of this procedure, to use real work rather than test tasks. In the same construction, a randomized allocation of tool versus no tool, but with an artificial standalone task, namely writing an HTTP server in JavaScript, 95 developers recruited through a freelance platform were 55.8 percent faster with the tool, with a 95 percent interval of 21 to 89 percent. Notable for the method: here participants underestimated their gain, averaging 35 percent against a measured 55.8 percent. The direction of the error is therefore not constant, only its existence. Limits: a preprint by the vendors involved, a single synthetic task, recruited freelancers rather than a workplace setting, and no bearing on mature codebases.
- 4.METR: We are Changing our Developer Productivity Experiment Design (opens in a new tab) · METR · 2026-02-24 · practitioner source · limits the methodBounds the procedure in the words of its own authors. METR is moving away from task-level randomization and names the reasons: developers increasingly refuse to participate because they do not want to work without AI; an estimated 30 to 50 percent withheld tasks they assumed AI would finish quickly; cutting compensation from 150 to 50 dollars per hour likely worsened the selection problem; and with agentic tools participants found it hard to report time spent because they worked on other things while waiting. The follow-up round with 57 developers and over 800 tasks across 143 repositories produced an 18 percent slowdown among the original developers and 4 percent among newly recruited ones, with METR explicitly calling its own data only very weak evidence. Limit: a self-report on ongoing, not yet published work.
Origin: METR / Becker, Rush, Barnes, Rein