synodic-ai ML RESEARCH
ML Research Desk · Evaluation Methodology

What the Average Leaves Out: Tool Evidence, Improvement Curves, and Recombined Goals

2026-10-20

Three papers posted to arXiv on 2026-10-19 each find that a single headline number hides the property that decides whether a system works: where its evidence comes from, how quickly it improves, and whether it holds up when familiar pieces are recombined. Each replaces the average with a measurement built to expose that property.

More thinking is not more evidence

Yu Zhou and colleagues begin with a controlled finding. Multimodal models have advanced quickly on visual reasoning, yet many tasks need evidence that is hard to obtain through internal reasoning alone. In their study across 22 benchmarks, simply increasing the base model's inference budget yielded only modest gains. The tasks demand grounded perception, external knowledge and numerical measurements from heterogeneous tools, whose outputs must be made interpretable to the model and kept across reasoning steps.

Their harness, OpenVLHarness, is organized around a capability layer and persistent multimodal memory. The capability layer pairs task-level tool interfaces with evidence rendering, giving consistent access to heterogeneous backends and turning their outputs into observations the model can read. The memory keeps images, structured data and web references as addressable artifacts for the whole query, and a coding interface runs generated programs over the stored data so results from several tools can be combined.

Across 22 benchmarks and eight open and proprietary backbones, the harness raised average scores by 9.7 to 12.6 points over matched no-tool baselines. With one backbone at medium reasoning effort, it beat the same base model and two existing agentic baselines run at maximum effort by 7.7 to 8.5 points, at no more than 37 percent of their model-token cost. For evaluation, the lesson is that a reasoning-budget sweep measures one axis; on evidence-bound tasks, access to and retention of evidence is a separate axis that moved scores further at lower token cost.

Five axes for an agent that learns from failure

Zekai Li and colleagues study physical recursive self-improvement: how quickly and how far a model-driven agent acting in the physical world improves from its own attempts. Their PhysRSI-eval scores five axes, First-Attempt Performance, Efficiency, Success, Budget and Improvement from failure, and evaluates different vision-language models on 18 manipulation tasks under a fixed interaction budget.

The five-axis view separates systems whose public benchmark averages are close, and it shows that high scores need not come with early or efficient gains. From these results the authors propose the Physical Barrel Hypothesis: different capabilities may constrain different parts of improvement. They also study observational scaling from two directions, predicting the relative performance of agents from their public capabilities and forecasting later progress from early physical feedback.

Workspace interventions show what improvement leaves behind. Removing history reduced late-round success, frozen workspaces stayed useful on unseen layouts, and three cross-task cases reached success earlier. The authors read this as evidence that scene-specific adaptation and reusable experience can coexist.

Familiar parts, unfamiliar combinations

Heejeong Nam, David Klindt and colleagues ask whether world models generalize beyond the state-action combinations seen in expert rollouts, which is what planning with a world model assumes. They frame it as compositional generalization: can a model recombine familiar state and action factors into valid dynamics?

Their benchmark, AnyGoal, builds goal-shifted variants of widely used world-model environments and ships with an open-source library. Each task is designed to be within reach of a model that has learned the underlying dynamics: every individual state factor and action stays within the training support, but their combinations are underrepresented in expert trajectories. Across multiple environments and world-model families, current models consistently failed under these recombinations. The authors examine dynamics and representation failures across model families, interaction regimes and evaluation metrics to locate where generalization holds and where it does not.

The construction is what makes the result informative. Because every factor appears in training, a failure cannot be put down to inputs the model has never seen; it indicates that the model did not learn how the factors combine. That is exactly the case a planner meets when it asks a world model to simulate a goal no expert demonstrated, which is why the authors present the benchmark as a basis for measuring progress toward world models that generalize to new planning tasks.

The same correction in three settings

Each paper adds a measurement that the average cannot supply. OpenVLHarness separates gains from evidence access from gains from reasoning budget. PhysRSI-eval separates how fast and how efficiently a system improves from where it ends up. AnyGoal separates in-distribution accuracy from accuracy on recombinations of in-distribution parts. In every case, systems that look similar on a single score differ on the dimension that matters for deployment.

The papers also share a design habit worth copying: each holds something fixed so the comparison means something, whether matched no-tool baselines, a fixed interaction budget, or individual factors kept within training support.

What evaluation owners should change

These findings come from specific benchmarks, robots and environments, so treat them as checks to add rather than rates that will transfer unchanged.

Sources