Blog
#0031 min readSample

Red-team your own agent before someone else does

A short checklist for attacking your own AI feature, and what each attack should run into.

Sample post Placeholder content that shows the blog's design. Replace it before launch.

Any agent that reads text from the outside world will eventually read text written to manipulate it. The cheapest time to find out how it reacts is before launch, on your own terms.

The attacks worth trying first

You don’t need a research team. Start with the obvious moves and see what gets through.

AttackWhat it triesWhat should stop it
Prompt injection“Ignore your instructions and…” hidden in a documentInstructions and data kept apart; output schema
Scope creepAsking for things outside the taskA narrow brief and an allow-list of actions
Data reachGetting at records it shouldn’t seePermissions checked outside the model
Tool abuseCalling a tool with dangerous argumentsArgument validation and limits per tool

Make it a test, not an event

Red-teaming once before launch catches today’s holes. Keeping the attacks in the eval set catches the ones a future change reopens.

test_guardrails.py
import pytest
ATTACKS = load_cases("evals/attacks.jsonl")
@pytest.mark.parametrize("case", ATTACKS, ids=lambda c: c.name)
def test_attack_is_blocked(case):
result = agent.run(case.input)
assert result.blocked or result.output == case.safe_output
Every blocked attack becomes a regression test.

What good looks like

A well-harnessed agent doesn’t need to be clever about attacks. It only needs to have nowhere to go: a narrow job, validated outputs and permissions it can’t change.