synodic-ai ML RESEARCH
ML Research Desk · Evaluation Validity

Three Ways an Evaluation Can Mislead: Confounded Protocols, Borrowed Infrastructure, and Models That Hide What They Can Do

2026-10-12

Three arXiv papers from 2026-10-09 each identify a different reason an evaluation can produce a number that does not mean what it appears to mean. One re-runs a crowded literature under a single protocol, one builds a benchmark that does not depend on third-party APIs, and one proves that an adversarial model can hide arbitrary capabilities from the elicitation methods auditors rely on.

When the protocol is the result: tabular regression objectives

Danil Rudenko, Georgii Bulgakov, Nikolay Kartashev, Ivan Rubachev, Daniil Krasylnikov, and Artem Babenko take on a question that has been answered many times and never settled: which objective function should a deep model for tabular regression use. Several lines of research offer alternatives to the standard mean squared error (MSE), but the authors note that differences in datasets, backbones, and evaluation protocols make the practical benefits of those alternatives difficult to compare. The literature is not short of results; it is short of results measured the same way.

The paper performs a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Under that single protocol, many proposed objectives provide limited or inconsistent gains over a properly tuned MSE baseline. One alternative survives the comparison: quantile loss yields consistent aggregate improvements over MSE across the tested backbones.

The authors then identify a limitation that persists even with the stronger objective. Regions of the input space with different levels of target noise can reach their best predictive performance at different training epochs. A single global early-stopping checkpoint therefore imposes a compromise between uncertainty regions, and this is true even when the objective itself can represent heteroskedastic target distributions. Their response is a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing, which improves over quantile loss on several backbones.

When the infrastructure is the confound: the Wikipedia Game as a navigation benchmark

Alexander Hickey, Joanna Knight, and James Bishop address a different validity problem. Language models increasingly support or automate information retrieval from document collections too large to index, a task that becomes difficult when it requires specialist linguistic or scientific knowledge. Evaluating that capability usually means calling a third-party API, which couples the result to infrastructure the evaluator does not control.

The paper uses the Wikipedia Game, in which players must reach a target article using only article links, as an analogous navigation task. The authors compare sentence embedding models and LLMs across five Wikipedia languages and five academic-literature datasets. To remove the API dependency they construct very large local graphs of Wikipedia pages and of paper citations using OpenAlex, and they provide code to build and navigate those graphs, so a model's navigation capability can be studied without relying on third-party services. Human and TF-IDF benchmarks supply the reference points.

The headline result is that no model significantly outperformed humans, although the four best models achieved comparable success. The more useful result for practitioners is that performance and rankings varied across languages and topics. The format supports robust comparisons across techniques, languages, and domains precisely because the environment is local and fixed, and once that is in place the instability of rankings across domains becomes visible.

When the model is the adversary: provably hidden capabilities

Andis Draguns, Sumeet Ramesh Motwani, Raymond Douglas, and Christian Schroeder de Witt attack the assumption that underwrites every pre-deployment capability evaluation: that if a dangerous capability is present, it can be elicited. The paper shows the assumption does not hold under adversarial conditions by constructing backdoor attacks that embed encrypted neural networks within a host model's weights. The hidden network is decrypted and executed only upon receiving a secret trigger.

The construction uses digital lockers, and the authors argue that the resulting hidden circuits remain provably hard to elicit and interpret under cryptographic assumptions, even given full access to the model weights. This extends prior work, which hid fixed strings, to hiding arbitrary computations. The proof-of-concept constructions are implemented in PyTorch.

Elicitation was attempted through fine-tuning, weight noising, gradient-based trigger search, and latent adversarial training, on a small chemical reaction prediction task that the authors use as a proxy for CBRN-relevant capabilities. The hidden capability resisted these methods. Together with the cryptographic guarantees, the authors present this as a limit of elicitation-based capability audits, and they release the models to support the improvement of defenses.

What the three have in common

Each paper is about the distance between a measurement and the quantity it is supposed to measure, and the three describe validity threats of increasing severity.

Practical consequences for evaluation teams

For teams comparing training methods, the tabular regression work is a reminder to tune the baseline before believing the method, and to check whether a global early-stopping checkpoint is quietly trading off performance between low-noise and high-noise regions of the input. Quantile loss is the one alternative that held up across backbones; the heteroskedasticity-aware extension is the candidate for datasets where noise varies with the input.

For teams evaluating retrieval or navigation, the Wikipedia Game approach shows how to remove API dependence by building and releasing a local graph, and the result that rankings vary across languages and topics means variance across domains should be reported alongside the aggregate score. A model that is best on one language or one literature is not thereby best on the one a deployment will face.

For teams running capability audits, the hidden-capabilities result matters most. A failure to elicit a capability is weak evidence of its absence once an adversary with write access to the weights is in the threat model, and the authors show that adversary can hide arbitrary computation, not only fixed outputs. That moves part of the assurance burden from behavioral testing to provenance: knowing who had write access to the weights, and when, becomes a component of the evaluation rather than an operational afterthought. The released models give defenders something concrete to test against.

Sources