Reality Check · Ins7ghts Daily, Wednesday 11 November 2026
Your coding agent tells you it is done.
That sentence carries more weight than it should. In most engineering teams it is the closest thing to a productivity measurement anybody holds, and it is a self-report from the party under assessment. Underneath it sits the claim repeated in every renewal conversation this year: the agents make developers measurably faster.
The counter-event. METR ran a randomised trial in 2025 with sixteen experienced open-source developers. With AI tools allowed, they took 19% longer to close their issues, while reporting that they felt about 20% faster. The gap between the stopwatch and the feeling is the finding. The people doing the work could not tell which way the arrow was pointing, and they were not novices.
The mechanism is not mysterious either. Snorkel's write-up on reward hacking describes coding agents peeking at the test instead of asking what the task requires. An agent graded against a check will satisfy the check, which produces a pass rather than a solution, and a green run rather than working software.
The honest caveat, which is the part that earns the rest. METR came back in February 2026 and called its own earlier result "only very weak evidence", allowing that newer tools may genuinely have improved developer speed. Sixteen developers on open-source issues is not an enterprise engineering organisation. And the figure that would settle the argument is the one nobody has: almost no company wrote down its cycle time, its defect rate or its review load in the month before the agents went on. There is no before. A missing loss is not a gain, and a missing gain is not a loss. It is a measurement that was never taken, and that is a different thing from a disappointing one.
The rest of this piece is for subscribers.
Ins7ghts Daily is free. One letter a morning from Yves Mulkers on what moved in data and AI, and why it matters to your side of the table.
Already subscribed? Sign in at the top of the page.