What a benchmark cannot tell you

Revision note, October 16, 2026. This essay was substantially revised to add primary references, narrow unsupported claims and distinguish proposals from measured results. The original address and publication date are retained.

A public score describes a model under a particular evaluation protocol. Choosing a model for real work also requires evidence about the tasks, constraints and failures that matter in that setting.

A score has a scope

The GPT-4 technical report records an MMLU result of 86.4% under a five-shot protocol. That number is meaningful alongside its question set, prompting method and scoring rules. It does not establish how reliably the model will complete an unfamiliar workflow. GPT-4 technical report, Table 2.

A benchmark becomes easier to interpret when the comparison preserves those conditions. Different prompts, tools, sampling budgets or model snapshots can change what is being measured. A small difference between two reported scores is a reason to inspect the evaluation, rather than an automatic reason to switch models.

Public tests remain useful. They support shared comparisons, reproducible checks and inspection of failure cases. Their value for a decision depends on how closely the test represents the intended work and whether its results distinguish the candidates with sufficient confidence.

Test the work you intend to delegate

Start by defining an outcome that can be checked. For a document extraction system, that might include correct values, preserved provenance and an explicit response when a required field is missing. For an agent, completing an action safely may matter more than producing a plausible explanation.

Build a task set from realistic inputs and important exceptions. Keep examples of incomplete records, ambiguous instructions and cases that should be escalated. Separate development examples from a held-out set used for the final comparison. A few handpicked demonstrations can help design this process, but cannot establish a dependable ranking by themselves.

Decide how success will be judged before comparing outputs. Record the model version, prompts, available tools, context, sampling settings and resource budget. Where responses vary, repeat the trials and report the variation. Count consequential failures separately instead of allowing frequent easy successes to obscure them in an average.

Privacy reduces one risk, not every risk

A private question set can make direct memorization harder. It can still leak, become familiar through repeated tuning or reward a proxy for the real objective. Expert-written questions also need clear criteria and a process for resolving disagreement.

Public and private evaluation are complementary. The initial Humanity's Last Exam release included public questions and a separate private holdout. This is a useful example of separating public inspection from checks for overfitting, without claiming that either set measures every practical capability. Humanity's Last Exam, sections 1 and 3.

Make the decision inspectable

Compare quality with the resources and consequences attached to it. Record response latency, token and tool usage, reviewer time and the cost of correcting errors under the same workload. There is no universal evaluation price that substitutes for these inputs.

Set a reason to rerun the comparison: a changed model, a changed task distribution, a new tool or a material failure. Keep the previous evaluation record so a claimed improvement can be checked against the same baseline.

The useful result is a bounded statement: this version performed this well on these tasks, with these resources and these observed failures. That is a stronger basis for deployment than treating a leaderboard position as a guarantee.