Blog
#0062 min readSample

Evals are tests that admit they're fuzzy

Unit tests catch what breaks. Evals tell you whether a change made things better or worse, on examples that look like real life.

Sample post Placeholder content that shows the blog's design. Replace it before launch.

Tests answer a yes-or-no question: did this break? For AI features that’s necessary, and it isn’t enough. A prompt change can pass every test and still make answers a little worse on the cases your users actually send.

Evals cover that gap. They are tests that admit the answer is a score, not a checkmark.

Start with a small, honest set

An eval set doesn’t need to be big to be useful. It needs to be real:

  • examples pulled from actual inputs, not ones invented to look nice,
  • the hard cases you already know about,
  • every failure you have logged since the last round.

Twenty or thirty well-chosen examples will tell you more than a thousand generated ones.

Score what the brief cares about

The brief already ranked the mistakes that matter, so the scoring follows from it. Some checks are exact, others need judgment:

score.py
def score(expected: Contact, got: Contact | None) -> float:
if got is None:
return 0.0 # rejected or malformed
if got.email and got.email != expected.email:
return 0.0 # invented data is the worst outcome
hits = [got.email == expected.email, got.phone == expected.phone]
return sum(hits) / len(hits)

When a check can’t be written as code, like tone or faithfulness to a source, an LLM judge with a tight rubric can score it. Spot-check the judge too; it’s a model like any other.

Run it on every change

The value of evals comes from running them all the time: new prompt, new model, new retrieval step. Keep the scores next to each change, and the conversation shifts from “it feels better” to “it went from 0.71 to 0.78, and here are the three examples that got worse.”

Two variants, one eval set, one scoreboard.

That’s the whole idea: a small set of real examples, scored the way the brief says, run on every change. Boring on purpose, and the reason the feature keeps getting better instead of drifting.