Blog
#0072 min readSample

Inside the lab, where agents get tested

A tour of the experiments every agent goes through before it ships, from the writing test to the night watch.

Sample post Placeholder content that shows the blog's design. Replace it before launch.

An agent that works in a demo and an agent you can hand to customers are different things. The gap between them is testing, and most of it is unglamorous. This is a quick tour of the lab: what gets checked, in what order, and why.

Start with the brief

Every test grades against something written down. Before a single run, the job gets a one-page brief: what goes in, what comes out, and which mistakes cost the most. The writing test grades each line of output against it.

The agent types at a desk while a board grades each line with a check or a cross.
The writing test: every line is graded against the brief, not against a feeling.

Look at the data before you trust a score

A test set full of bad examples produces confident nonsense. So the second stop is the microscope: sample real inputs, flag the ones that are broken or unrepresentative, and keep the rest.

A round sample screen of pixel cells; the bad ones are marked.
The microscope: bad examples are flagged before they can distort an eval.

Put it in a box

New agents start in a sandbox with no network access, no production data and hard limits on every action. Then we push on the walls on purpose.

The containment chamber, live. The agent bumps the glass; the glass holds.

Let something hostile try

A second bot plays the attacker: prompt injections, requests outside the task, attempts to reach data it shouldn’t. Guardrails take those hits so the agent never sees them.

The red team: hostile prompts burst on the guardrail.

Compare, then choose

When there are two candidate prompts or models, both run the same eval set and the scoreboard decides.

compare.py
def compare(variants, examples, score):
results = {}
for name, run in variants.items():
scores = [score(ex.expected, run(ex.input)) for ex in examples]
results[name] = sum(scores) / len(scores)
return max(results, key=results.get), results

Keep watching after launch

Shipping is not the end of testing. Every run in production is logged, scored on a sample, and compared to a baseline. When something drifts, an alarm goes off and the last good version comes back.

The night watch: drift on one model trips the alarm, then it's rolled back.

That’s the lab. None of it is exotic. It is the same discipline good software teams already use, applied to a component that happens to be probabilistic.