Inside the lab, where agents get tested
A tour of the experiments every agent goes through before it ships, from the writing test to the night watch.
Sample post Placeholder content that shows the blog's design. Replace it before launch.
An agent that works in a demo and an agent you can hand to customers are different things. The gap between them is testing, and most of it is unglamorous. This is a quick tour of the lab: what gets checked, in what order, and why.
Start with the brief
Every test grades against something written down. Before a single run, the job gets a one-page brief: what goes in, what comes out, and which mistakes cost the most. The writing test grades each line of output against it.

Look at the data before you trust a score
A test set full of bad examples produces confident nonsense. So the second stop is the microscope: sample real inputs, flag the ones that are broken or unrepresentative, and keep the rest.

Put it in a box
New agents start in a sandbox with no network access, no production data and hard limits on every action. Then we push on the walls on purpose.
Let something hostile try
A second bot plays the attacker: prompt injections, requests outside the task, attempts to reach data it shouldn’t. Guardrails take those hits so the agent never sees them.
Compare, then choose
When there are two candidate prompts or models, both run the same eval set and the scoreboard decides.
def compare(variants, examples, score): results = {} for name, run in variants.items(): scores = [score(ex.expected, run(ex.input)) for ex in examples] results[name] = sum(scores) / len(scores) return max(results, key=results.get), resultsKeep watching after launch
Shipping is not the end of testing. Every run in production is logged, scored on a sample, and compared to a baseline. When something drifts, an alarm goes off and the last good version comes back.
That’s the lab. None of it is exotic. It is the same discipline good software teams already use, applied to a component that happens to be probabilistic.