METR finds Anthropic’s model progress accelerating METR finds Anthropic’s model progress accelerating

What the outside assessment found

METR attributes much of the acceleration to AI assistance, drawing on internal results and researcher access. This moves the discussion toward how research is already changing. The object being measured is a development process, rather than an application ticket.

The risk report received a separate review

A companion review examines Anthropic’s August risk report, which covered the period through July 15. METR accepts the argument that models had not reached full replacement of the company’s researchers. It disputes the confidence attached to a separate threshold: doubling research progress through AI assistance. In METR’s reading, the public document gives a firmer impression than the underlying private assessment. The reviewer also distinguishes crossing a warning threshold from directly enabling the severe outcomes described in Anthropic’s threat model. Those are separate propositions. A measurement can warrant additional scrutiny without establishing the worst outcome. The review therefore concerns both the evidence and how a company communicates what that evidence supports.

A faster lab is a different measurement

METR’s July 10, 2025 developer study supplies a useful contrast in method. Sixteen experienced contributors completed 246 real tasks in repositories they knew. Random assignment determined whether AI tools were allowed. Work took nineteen percent longer with access, although participants believed they had saved time. That experiment measured completion of actual software changes, including the quality expected in familiar projects. It did not measure improvements in a laboratory’s model capabilities. Its value here is methodological: a feeling of greater productivity and a measured result can diverge. It also shows why results need a specified population, task and definition of completion.

The follow-up changed the experiment

The February 24, 2026 update added another complication. Developers increasingly declined participation because they wanted to keep using AI. Some withheld tasks they expected AI to handle particularly well. METR also identified difficulty recording time when developers ran agents alongside unrelated work. Participant pay also fell from $150 to $50 an hour, which could affect who enrolled. Recruitment conditions changed alongside the tools. Its next experiment therefore faced selection and measurement problems that were less prominent before. The organization proposed several alternatives, including fixed-task studies, observational evidence and randomizing whole developers instead of individual tasks. These designs answer different questions about adoption and output. Together, the studies explain why a single productivity percentage cannot describe every setting where software teams use models.