Newshot

A(nother) look at METR's time horizon: tasks of the same duration get solved months apart

Summary

METR's 80% time horizon lags the 50% horizon by about eight months . I had been erroneously reading that lag as a task's transition time, the time a task takes to go from a 50% to an 80% success rate. However, in METR's data the median task's transition time is about four months , half the lag, because tasks of the same duration get solved at different times . That spread among tasks of the same duration also matters for other readings of the horizons, for example: For hazard rates, the pooled data can show a falling hazard rate even if every task has a constant one . For labor displacement, if automating a role needs every task done reliably, the timing depends on the hardest tasks of each duration , which are solved later than the horizon suggests. This may be obvious to others, and in hindsight it is to me too, but I needed a closer look at the data to see it. The 80% horizon trails the 50% horizon by about eight months In February 2025, the frontier model Claude 3.7 Sonnet had a 50…

Original text(via LessWrong)
  • METR's 80% time horizon lags the 50% horizon by about eight months. I had been erroneously reading that lag as a task's transition time, the time a task takes to go from a 50% to an 80% success rate.
  • However, in METR's data the median task's transition time is about four months, half the lag, because tasks of the same duration get solved at different times.
  • That spread among tasks of the same duration also matters for other readings of the horizons, for example:For hazard rates, the pooled data can show a falling hazard rate even if every task has a constant one.For labor displacement, if automating a role needs every task done reliably, the timing depends on the hardest tasks of each duration, which are solved later than the horizon suggests.
  • For hazard rates, the pooled data can show a falling hazard rate even if every task has a constant one.
  • For labor displacement, if automating a role needs every task done reliably, the timing depends on the hardest tasks of each duration, which are solved later than the horizon suggests.

This may be obvious to others, and in hindsight it is to me too, but I needed a closer look at the data to see it.

The 80% horizon trails the 50% horizon by about eight months

In February 2025, the frontier model Claude 3.7 Sonnet had a 50% time horizon of one hour. The 80% horizon reached one hour about nine months later. Across the models since GPT-4 that moved the 50% horizon forward, the median gap is about eight months (90% interval 6.4 to 10.8).

Model50% horizonFirst model with an 80% horizon past itEstimated months later
GPT-44 mino1-preview17.7
GPT-4o7 mino16.7
Claude 3.5 Sonnet (old)11 minClaude 3.7 Sonnet7.9
o1-preview20 mino36.4
o139 minGemini 3 Pro8.2
Claude 3.7 Sonnet60 minGPT-5.29.3
o32.0 hClaude Mythos Preview10.8

First model with an 80% horizon past it

Estimated months later

Claude 3.5 Sonnet (old)

Claude 3.7 Sonnet

Gemini 3 Pro

Claude 3.7 Sonnet

Claude Mythos Preview

The frontier 80% horizon is interpolated between model releases.

The horizon is an average over tasks

For each AI model, METR fits one success curve across all tasks, with human completion time as the difficulty. A 50% horizon of two hours says that a randomly chosen two-hour task is completed half the time.[1] In practice, tasks of the same duration differ a lot in how hard they are for models.

A familiar example is children learning to walk. One child in ten walks alone at 10 months and nine in ten at about 14 and a half, a span of four to five months. Yet a single child's transition from crawling to walking takes only a number of weeks (Adolph and Tamis-LeMonda, 2014).

Figure 1. Share of children walking alone, by age

The blue curve is from the WHO Motor Development Study. The gray lines are illustrative children.

METR's time horizon is an average over tasks in the same way. Tasks of the same duration are solved at different times. Among two-hour tasks, a quarter are as easy for models as a typical 40-minute task and a quarter as hard as a typical six-hour task. Each task goes from unsolved to solved quickly, but the group's success rate rises slowly.

The five example tasks in Figure 2 all take a human five to eight hours but behave like typical tasks of 27 minutes to 35 hours. The hardest is still failed on every attempt, and the other four each took about four months to go from a 50% to an 80% success rate.

Figure 2. Five tasks that take a human five to eight hours

Each column is one frontier model at its release date, and each block is one attempt.

Across the whole suite, a frontier model gets mixed results on a task, passing some attempts and failing others, in 16% of cases. If every task followed METR's fitted curve for its duration, mixed results would be expected in about 40% of cases.[2]

Figure 3. Results are mixed in 16% of cases, against 40% predicted by METR's curves

Each case is one frontier model's attempts at one task, typically four.

The transition time of a single task

To estimate how fast a single task is solved, I fitted a model with a difficulty and a steepness for each task. The median transition time is about four months (90% interval 3.7 to 4.3). It is under three months for a quarter of tasks and over five for another quarter.

That is about half the eight-month gap between the horizons, and only about 6% of tasks take as long as the gap. The gap is longer because two random tasks of similar duration are typically solved by the frontier about seven months apart, and a quarter of such pairs more than a year apart.

Figure 4. Single tasks flip quickly. Groups of tasks decline slowly because they flip at different times.

Chance of failing against calendar time, using the ability of the best model released so far. Dots are raw run outcomes for the whole band.

As an illustration, going from a 10% to a 90% success rate (roughly, from the first occasional successes to nearly always solved) takes the median task about a year. On METR's curves, the 90% horizon reaches a given task length a bit over two years after the 10% horizon.

These single-task estimates need a different model from METR's, which takes a task's difficulty to be its human duration. I fitted a hierarchical Bayesian item response model, close to the one in Jonas Moss's post.[3] It differs from METR's in two ways.

  • Each task gets its own difficulty and steepness. The difficulty starts at the task's human duration and moves as far as the results support. Steepness is how sharply the task goes from unsolved to solved as models get more capable.
  • Each AI model gets an adjustment for each task family, because its results on tasks of the same family are correlated beyond what its ability and the tasks' difficulties explain.

The model gives a doubling time of 4.4 months for the frontier, against 4.1 months from METR's horizons.

What an 80% horizon means for automation

Forecasts such as the AI Futures Project's treat a long enough 80% time horizon as the point where AI can replace software engineers. But an 80% success rate on two-hour tasks could come from every task failing one time in five, in which case you just retry. Or it could come from one task in five failing every time, however often you retry.

METR's data are somewhere in between. For each frontier model I counted the tasks between half and twice its 80% horizon, and how many of them it failed in every run. Until Mythos that was 0 to 14% of them. With Mythos it jumped to 26% (8 of 31 tasks).

Frontier modelReleased80% horizonTasks near itFailed in every run
Claude 3.7 SonnetFeb 202512 min375 (14%)
o3Apr 202530 min360
GPT-5Aug 202538 min320
Gemini 3 ProNov 202554 min300
Claude Opus 4.5Nov 202549 min343 (9%)
GPT-5.2Dec 202566 min282 (7%)
Claude Opus 4.6Feb 202670 min291 (3%)
Claude Mythos PreviewApr 2026about 3 h318 (26%)

Frontier model

Tasks near it

Failed in every run

Claude 3.7 Sonnet

Gemini 3 Pro

Claude Opus 4.5

Claude Opus 4.6

Claude Mythos Preview

Hazard rates

The hazard rate is the chance, per minute of a task, that an agent fails it. Ord proposed that it is constant for a given agent, and Hamilton fits METR's results and finds that it falls as tasks get longer (a Weibull shape of about 0.6, where 1 means constant). With a constant rate, success falls off exponentially with task length. With a falling rate, failure gets less likely the further an agent has got into a task, and long tasks are easier than a constant rate predicts.

Both readings take a task's human duration as its difficulty. But tasks of the same duration differ in difficulty. When you pool tasks that each have their own constant hazard rate, the pooled rate looks like it is falling. In a simulation where every task has a constant hazard rate, the pooled fit still gives Hamilton's shape.

ResultsWeibull shape of the pooled fit
METR's results0.59
Simulated: constant hazard rate for every task, with the difficulty spread estimated in this post0.57
Simulated: constant hazard rate, with duration as the only difficulty1.01

Weibull shape of the pooled fit

METR's results

Simulated: constant hazard rate for every task, with the difficulty spread estimated in this post

Simulated: constant hazard rate, with duration as the only difficulty

Median across frontier models.

Appendix

Recap of METR's time-horizon methodology

For each AI model, METR fits a logistic regression of success on the log of the task's human time. In item response theory (IRT) terms, this is a model in which task difficulty is kept fixed and each AI model gets an ability and a slope of its own. The most widely cited time horizon figure reports the ability estimates for frontier models in the original unit, human time.

Why METR fixes task difficulty

METR sets each task's difficulty to its human time. I think that is a sensible choice, for two reasons.

  1. Fixed difficulties stay the same whichever models are evaluated. Fitted difficulties, like the ones in this post, can shift when a new model is added, and the reported horizons would shift with them.
  1. Fitting a difficulty for every task takes much more data. Each model attempts a task only a handful of times, and for a task that every model solves, or that none does, the data can only bound the difficulty.

By letting task difficulties vary, I can see in return how each task behaves on its own, including how quickly it goes from unsolved to solved.

The model

Each AI model m has an ability , and each task i has a difficulty and a steepness . Abilities and difficulties are on the same scale, the log of human time. A model solves a task half the time when its ability equals the task's difficulty, and the steepness sets how quickly the success probability rises as ability passes that point.

The ability of each AI model is estimated from its own results, with a weak prior. The task parameters are partially pooled. A task's difficulty has a prior centered on its human time, and its estimate is shrunk toward that value when the data say little. A task's steepness is drawn toward the typical steepness when its own attempts carry little information. Each AI model also has an adjustment for each task family. Without it, the fitted model's errors on two tasks of the same family have a small but measurable correlation (0.17, against zero across families).

All three models fit , where is the human time of task i and is its family.

METRMossThis post
Task difficulty, fixed, with , , estimated
Steepnessone per AI model,one per task,one per task,
AI model abilityone per AI model, from that model's own regressionfor release date , so models are pulled toward a time trendindependently, a weak prior with no time trend
Task familieseach task weighted bynonea model-by-family term added to the ability, , with
Fittingeach AI model separatelyall models and tasks togetherall models and tasks together
Uncertaintybootstrap over families, tasks and runsposterior intervalsposterior intervals

Task difficulty

, with , , estimated

one per AI model,

one per task,

AI model ability

one per AI model, from that model's own regression

for release date , so models are pulled toward a time trend

independently, a weak prior with no time trend

Task families

each task weighted by

a model-by-family term added to the ability, , with

each AI model separately

all models and tasks together

bootstrap over families, tasks and runs

posterior intervals

Model comparison

I fitted the model with its parts added one at a time and scored each version by five-fold cross-validation over (AI model, task) pairs. About 17% of the variance in single attempts is run-to-run variance, the differences between repeated attempts by the same AI model on the same task. The remaining 83% is the most any of these models can explain.

ModelHeld-out variance explainedGain in held-out log score over the previous rowMedian transition time
METR's model, with human duration as the difficulty53%
+ a difficulty for each task72%+3,228 (se 132)6.0 months
+ a steepness for each task (the simplest model that gives task-specific transition times)73%+349 (se 48)4.6 months
+ an AI model by family term (the model in this post)74%+365 (se 51)4.0 months

Held-out variance explained

Gain in held-out log score over the previous row

Median transition time

METR's model, with human duration as the difficulty

  • a difficulty for each task

+3,228 (se 132)

  • a steepness for each task (the simplest model that gives task-specific transition times)

+349 (se 48)

  • an AI model by family term (the model in this post)

+365 (se 51)

A difficulty for each task accounts for most of the gain. Steepness and the AI model by family term each add about one point of explained variance or less, because most pairs pass or fail on every attempt and these terms only change predictions for the rest. Both improve the held-out log score by about seven standard errors, and the AI model by family term moves the median transition time from 4.6 to 4.0 months.

Transition time by task duration

Figure A1. Transition time of each task, grouped by task duration

Each dot is one task's estimated time to go from a 50% to an 80% success rate. The line is the group's median and the band its middle half. Hover a dot for the task name.

Transition times are similar across task durations, with medians between about three and four and a half months, and slightly shorter for tasks over an hour. Within each group most tasks fall between two and six months. Single-task estimates are uncertain and are pulled toward the typical value, so the spread in the figure is narrower than the true spread.

  • ^METR's FAQ says the answer is somewhere in between a model doing half of all two-hour tasks and doing each of them half the time. On tasks of 90 minutes to three hours, GPT-5 always succeeds on about a third, always fails on about a third and is mixed on the rest.
  • ^Nguyen and Fithian report the same pattern. They find a model's attempts at a task are mixed in 17% of cases, against about 40% predicted by METR's curves. Their model also estimates a difficulty for each task and each family. It assumes every task goes from unsolved to solved at the same pace, and the model here estimates that pace for each task.
  • ^Moss shows that METR's 80% horizons shift a lot once difficulty varies among tasks of the same duration. He ties each AI model's ability to a trend over time and studies where the horizons are heading. This post asks how long a single task takes to be solved.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗