The evaluation set we build before shipping any AI feature

A demo built around ten hand-picked examples can look flawless and still fail badly once real users start sending it whatever they actually send, which is exactly why we won't ship an AI feature to production without a real evaluation set built from historical data first. This isn't a formality we add for compliance reasons. It's the single piece of infrastructure that tells us, with actual evidence rather than a gut feeling, whether a system is ready to launch and whether a later change made it better or quietly worse.

We build evaluation sets from real historical cases, not synthetic examples someone wrote to be representative, because synthetic examples tend to reflect what a developer assumes users will ask rather than what they actually do. For a support triage system, that means pulling several hundred real historical tickets across a full range of topics, urgency levels, and writing styles, including the genuinely confusing ones that don't fit a clean category, since those are exactly the cases where a model is most likely to fail and least likely to have been represented in a hand-picked demo set.

We deliberately oversample the edge cases and the ambiguous ones relative to their natural frequency, because a model that gets the easy eighty percent right by default doesn't need much evaluation attention there; it's the hard twenty percent that determines whether the system is actually trustworthy in practice.

For most evaluation sets, we need a domain expert on the client's side, not just our own team, to define what a correct answer actually looks like, since we're rarely the ones with the relevant expertise for a client's specific industry or internal policy. This step takes longer than clients expect, because it surfaces disagreements the client's own team didn't realize they had. We've had cases where two senior staff members labeled the exact same historical ticket differently, which is useful information in its own right: if two experienced humans disagree on the correct answer, an AI system can't be expected to hit a target that isn't actually fixed in the first place, and that ambiguity needs resolving before evaluation numbers mean anything.

Raw accuracy against the labeled set is the starting point, not the whole picture. We also track the error pattern, whether mistakes cluster around a specific topic, a specific input format, or a specific edge case, since a model that's ninety percent accurate overall but fails on a concentrated, identifiable pattern is a very different, and often more fixable, problem than one that's ninety percent accurate with errors scattered randomly across every category. We also track confidence calibration: whether the system's own confidence score for a given answer actually correlates with whether that answer was right, since a system that's equally confident about its correct and incorrect answers can't be trusted to know when to escalate to a human.

The evaluation set doesn't get archived once a system ships. Every model update, every prompt change, every new tool the system gets access to runs back against the same set before it goes live, specifically to catch the case where a change that improves one thing quietly regresses something else that was working fine before. We've caught several regressions this way that would otherwise have shipped invisibly, since the change looked like a clear improvement on the specific examples someone happened to test manually. A standing evaluation set is what turns that from a lucky catch into a routine, boring, repeatable check, which is exactly what we want a safety mechanism to be.

An evaluation set isn't a fixed, permanent artifact, and treating it as one is its own quiet failure mode we've had to correct for. On one project, roughly four months after launch, a client's own policy changed in a way that made a meaningful share of our evaluation set's labeled correct answers technically outdated, since the old policy no longer applied to new cases even though the historical tickets themselves were still accurate reflections of what had been correct at the time. We didn't catch this immediately, because the system kept scoring well against a set that was, by that point, quietly testing against the wrong standard.

We now run a lighter recurring review of the evaluation set itself, not just the system being evaluated against it, on a quarterly cadence, checking with the client whether the underlying policies, products, or standards that the labeled answers reflect are still current. This adds a small amount of ongoing maintenance work relative to the system itself, but it closes a specific gap: a system can be performing exactly as well against its evaluation set as it always has, while the evaluation set has quietly drifted out of sync with what actually counts as correct in the client's live business. Nobody notices that kind of drift from monitoring the system alone; it only shows up when someone deliberately checks whether the yardstick itself has moved.

We've also learned to version evaluation sets explicitly rather than editing them in place, keeping a dated snapshot of every version alongside the specific model or prompt configuration that was tested against it. This matters more than it might sound: without versioning, a reported accuracy improvement over time is ambiguous, since it could reflect a genuinely better system or simply an evaluation set that got quietly easier as outdated or ambiguous examples were removed rather than updated. Keeping the history explicit lets us, and the client, tell the difference, and it's saved us from overstating an improvement at least once, when what had actually happened was a well-intentioned but undocumented cleanup of the evaluation set itself rather than any real gain in the underlying model's performance. That distinction, a real improvement versus an easier test, matters enormously to a client deciding whether to expand a system's scope, and it's exactly the kind of question a properly versioned evaluation history can answer with confidence instead of a guess.