Blog
#0021 min readSample

Watch it live, or it will drift quietly

Models, data and prompts all change after launch. Monitoring is how you notice before your users do.

Sample post Placeholder content that shows the blog's design. Replace it before launch.

An AI feature can pass every test on launch day and be worse a month later without a single line of your code changing. The model provider ships an update, the inputs shift, or a teammate tweaks a prompt. Nothing breaks loudly. Quality just slides.

Log every run

You can’t monitor what you didn’t keep. For each run, store the input, the output, the model and prompt version, and how long it took.

log-run.ts
await db.insert('runs', {
input,
output,
latencyMs,
model: MODEL_VERSION,
prompt: PROMPT_VERSION,
createdAt: new Date(),
});

Score a sample, every day

Scoring every production run is usually too expensive. A daily sample, scored with the same evals you use before launch, is enough to see a trend.

Terminal window
# nightly: score yesterday's sample against the baseline
npm run evals -- --source production --sample 200 --compare baseline

Decide what trips the alarm

Pick a threshold before you need it: for example, the daily score falling clearly below the baseline two days in a row. When it trips, the first move is to roll back to the last good version, and only then investigate.

Drift trips the alarm on one model; it's rolled back to baseline.

Monitoring isn’t a dashboard nobody opens. It’s a short loop: log, score, compare, roll back.