synodic-ai ML RESEARCH
ML Research Desk · Benchmark Design

Human Minutes Are Not Linear Difficulty: What the METR Time-Horizon Reanalysis and BrickBench Say About Measuring Agentic Capability

2026-10-11

Two arXiv preprints posted on 2026-10-08 probe the construct validity of agentic benchmarks from opposite ends: one reanalyzes METR's 50% time-horizon metric with splines and item-response theory and finds that human task time maps onto AI difficulty nonlinearly, while the other introduces BrickBench, a physically constrained design task on which leading agents satisfy verifiable requirements but fall short of human work. Together they argue that a capability number is only as meaningful as the scale it sits on.

The appeal and the assumption behind the 50% time horizon

METR's 50% time horizon measures the human completion time of software tasks that an AI solves with 50% probability. Its attraction is interpretability: instead of a benchmark-specific accuracy, capability is expressed in units everyone understands, such as minutes or hours of human work. The widely circulated plot of time horizons rising over successive model releases has become one of the most cited artifacts in discussions of AI progress.

The metric rests on a modeling assumption that is easy to overlook. In the original estimation, the AI difficulty of a task is taken to depend linearly on the logarithm of human completion time. Drew T. Nguyen and William Fithian, in a preprint posted 2026-10-08, ask what happens to the estimates when that assumption is relaxed.

Relaxing linearity with splines and item-response theory

Working from 228 tasks and 26 AI systems, the authors recompute time horizons using splines and item-response theory so that the relationship between log human time and AI difficulty can take whatever shape the data support. The fitted spline can be read as a function that converts human time into AI difficulty, and its shape is the paper's central finding: it is nearly flat in a region from roughly 2 to 30 minutes and close to linear elsewhere.

The practical reading is that not all tenfold jumps are equal. A time-horizon increase from 3 minutes to 30 minutes crosses the flat region, where tasks barely get harder for an AI as human time grows, and is therefore much easier than an increase from 30 minutes to 5 hours, even though both represent the same 10x multiplier on the horizontal axis. A chart that places those two jumps at equal distance on a log scale is, in difficulty terms, mis-scaled.

The authors also contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules than the original estimates, and a set of diagnostic plots for assessing the construct validity of a time horizon. Their recommendation is that time horizons be interpreted together with those diagnostics, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks, where the shape of the conversion function is least constrained by data.

BrickBench: validity, alignment, and design under physical constraints

BrickBench, from Kulits, Xu, Jones, Schmid, and Wu, approaches the measurement problem by building a task whose success criteria are partly verifiable by construction. Given a text prompt, an agent must produce a LEGO-set assembly that satisfies semantic and design criteria and that can also be physically built. To do so it selects parts from a discrete library and must reason jointly about local constraints, such as how individual bricks connect, and global constraints, such as whether the whole structure stands and matches the prompt.

Rather than one score, the benchmark reports validity, alignment, and design, across three settings that vary in scale and part availability. The authors also provide BrickAgent, an environment in which coding agents can construct, inspect, and validate their designs iteratively, which makes the task agentic rather than single-shot. Their headline result is that leading agents largely satisfy the verifiable physical and semantic requirements but fall short of human designs. The benchmark and environment are released publicly.

That split result is the interesting part. The axes an automated checker can verify, physical validity and semantic alignment, are mostly saturated by current agents. The axis that requires judgment, design quality, is where the human gap remains. A single aggregate would have blended a solved sub-problem with an unsolved one and reported something in between.

Two lessons about construct validity

The METR reanalysis and BrickBench are about different tasks, but they expose the same two weaknesses in how agentic capability is usually reported.

The first is scale. A metric that reports capability in human-interpretable units is only as good as the function that maps those units onto what the AI actually finds hard. Nguyen and Fithian show that for software tasks the function is far from linear, with a wide plateau in the short-task range. Any benchmark that extrapolates along a log-time axis, or that sets milestones at fixed multipliers, is implicitly assuming a shape the data do not support.

The second is aggregation. BrickBench's three-axis scoring, and its finding that agents pass the verifiable axes while failing the judgment axis, shows that the components of a capability score can be at completely different stages of progress. Reporting them separately is not a stylistic choice; it is the only way to see which sub-problem is actually open.

Implications for anyone reporting agent capability

For research groups publishing time-horizon or long-task results, the reanalysis sets a new baseline: report estimates under cross-validated proper scoring rules, publish the diagnostic plots, and state the shape of the fitted conversion function over the task-time range the benchmark covers. As benchmarks grow to include longer tasks, the region beyond the current data is exactly where a linear assumption is both least justified and most consequential, because that is where the next generation of milestones will be placed.

For teams using benchmark numbers in planning or procurement, the two papers argue for skepticism about any single-number capability claim. Ask what scale the number sits on, whether the axes it aggregates are at similar stages of progress, and whether the sub-criteria an automated checker verified are the ones that matter for the intended use. On BrickBench the agents already pass the checkable requirements; the shortfall relative to human designs lives in the part no checker can score. That pattern is likely to recur across agentic tasks, and benchmarks that cannot separate the two will keep reporting progress that users do not experience.

The deeper point, shared by both papers, is that construct validity is an empirical property of a benchmark, not a label attached to it. A time horizon is valid over the range where its conversion function is well behaved and informative; a design benchmark is valid for the axes it can actually discriminate. Measuring those boundaries is as much a research contribution as the capability number itself.

Sources