Build an AI evaluation set that can overturn the favourite model

Your preferred model should be able to lose the evaluation. If the test set contains only the prompts that made its demonstration impressive, it cannot do that job.

For an enterprise AI lead, the dataset is part of the buying decision. It determines which kinds of failure become visible, which remain untested and how much confidence a score deserves. Build it around the work the proposed system will perform, before adjusting prompts or comparing vendors.

Start with a task and the decision it supports

“Answer questions about company documents” is too broad to score consistently. “Find the current approval rule for this purchase, identify the applicable policy and flag missing information” gives reviewers something concrete to judge. Define who asks, which material they may access, what action follows and what an incorrect answer could cause.

The NIST AI Risk Management Framework Playbook’s Measure guidance recommends choosing metrics against the most significant risks, setting acceptable performance limits and documenting what cannot be measured. Use that approach to make the evaluation’s boundaries visible. A fluent answer may still fail the task if it cites an obsolete rule or supplies information outside the user’s authority.

Assemble permitted examples from the workflow rather than from memory alone. Cover common work, ambiguous requests, missing evidence, changed documents and cases where the appropriate result is to ask a question or decline to conclude. Record the sampling period and which business groups are represented. Keep deliberately difficult challenge cases identifiable so they are not mistaken for a representative sample of everyday demand.

For each example, retain the input, its provenance, the relevant source version, the expected behaviour and the reason that behaviour matters. Check that the organisation is authorised to use the data in the evaluation environment. Production traces can contain confidential inputs and outputs; their availability in a log does not make them suitable for unrestricted reuse.

Write expected behaviour before seeing answers

Use a short, task-specific rubric that separates correctness, supporting evidence and consequential mistakes. Decide which criteria are essential and which admit degrees of quality. A serious permission failure should remain visible even when the same response is concise and well written.

These three cases illustrate a hypothetical purchasing-policy assistant. They are examples of test design, not observed model results.

Case	Expected behaviour	Failure that matters
An authorised user asks about a purchase covered by a superseded policy.	Apply the current policy version and identify the supporting passage.	Use the old approval rule with unwarranted confidence.
The request omits information needed to select the applicable rule.	Ask for the missing detail before giving a definitive approval route.	Invent the missing fact and give an actionable answer.
A user asks for policy material outside their authorised scope.	Respect the configured access boundary and offer an appropriate next step.	Reveal restricted material, including through a summary.

Make the rubric usable by another reviewer. Define what counts as a supporting passage, how to treat partially correct answers and when a clarifying question is sufficient. Have reviewers score some of the same examples independently, then investigate disagreements. A disputed expected answer may reveal an ambiguous policy or weak rubric rather than a model defect.

Keep any automated judge’s verdict separate from the evidence being judged. Use knowledgeable reviewers to check consequential or disputed cases. Agreement with a scoring model is not itself proof that the underlying business answer is correct.

Preserve a test the development team has not optimised against

Use a development set for prompt changes, retrieval tuning and failure diagnosis. Reserve a separate held-out set for the comparison you intend to use in the selection decision. Keep closely related examples together when splitting: paraphrases of one question or copies of the same document passage can otherwise make the apparent separation weak.

Once a held-out case has been inspected and used to fix the system, record that history. It can remain valuable as a regression test, but the next claim of independent performance needs fresh held-out evidence. Do not repeatedly tune against a selection set and continue describing it as untouched.

MLflow’s evaluation-dataset documentation supports datasets drawn from existing traces or manually created examples, including inputs and ground-truth expectations. This is useful for keeping a real failure alongside its expected behaviour and rerunning it when the application changes. The documented dataset feature requires a tracking server with a SQL backend; local FileStore tracking does not support it.

That storage requirement matters when deciding how the team will maintain a shared evaluation. The tool can organise examples, but the team still has to establish data permission, coverage and a defensible expected result. A well-maintained dataset of easy questions remains an easy test.

Compare systems under declared conditions

Record the dataset and rubric version alongside the model identifier, prompt, retrieval configuration, document snapshot and relevant tool permissions. If the document corpus changes between runs, investigate the effect before attributing the difference to the model. When the choice concerns a complete application, evaluate that application; when isolating model differences, hold the surrounding conditions as consistent as practicable.

Inspect results by task type and error severity as well as overall score. A high average can coexist with a failure in the case that carries the greatest operational consequence. Repeat cases where variable outputs could change the decision, and record that variation rather than selecting the most persuasive response.

A small convenience sample is evidence about those examples under those conditions. It does not establish a population-wide accuracy rate. Expand coverage where the decision remains sensitive, and identify important tasks still outside the test.

Use the finished set in an AI-platform comparison that makes the shortlist prove the workflow. The strongest next step is a controlled comparison against agreed acceptance criteria, with a genuine possibility that none of the candidates yet meets them.