Watch it live, or it will drift quietly
Models, data and prompts all change after launch. Monitoring is how you notice before your users do.
Sample post Placeholder content that shows the blog's design. Replace it before launch.
An AI feature can pass every test on launch day and be worse a month later without a single line of your code changing. The model provider ships an update, the inputs shift, or a teammate tweaks a prompt. Nothing breaks loudly. Quality just slides.
Log every run
You can’t monitor what you didn’t keep. For each run, store the input, the output, the model and prompt version, and how long it took.
await db.insert('runs', { input, output, latencyMs, model: MODEL_VERSION, prompt: PROMPT_VERSION, createdAt: new Date(),});Score a sample, every day
Scoring every production run is usually too expensive. A daily sample, scored with the same evals you use before launch, is enough to see a trend.
# nightly: score yesterday's sample against the baselinenpm run evals -- --source production --sample 200 --compare baselineDecide what trips the alarm
Pick a threshold before you need it: for example, the daily score falling clearly below the baseline two days in a row. When it trips, the first move is to roll back to the last good version, and only then investigate.
Monitoring isn’t a dashboard nobody opens. It’s a short loop: log, score, compare, roll back.