Red-team your own agent before someone else does
A short checklist for attacking your own AI feature, and what each attack should run into.
Sample post Placeholder content that shows the blog's design. Replace it before launch.
Any agent that reads text from the outside world will eventually read text written to manipulate it. The cheapest time to find out how it reacts is before launch, on your own terms.
The attacks worth trying first
You don’t need a research team. Start with the obvious moves and see what gets through.
| Attack | What it tries | What should stop it |
|---|---|---|
| Prompt injection | “Ignore your instructions and…” hidden in a document | Instructions and data kept apart; output schema |
| Scope creep | Asking for things outside the task | A narrow brief and an allow-list of actions |
| Data reach | Getting at records it shouldn’t see | Permissions checked outside the model |
| Tool abuse | Calling a tool with dangerous arguments | Argument validation and limits per tool |
Make it a test, not an event
Red-teaming once before launch catches today’s holes. Keeping the attacks in the eval set catches the ones a future change reopens.
import pytest
ATTACKS = load_cases("evals/attacks.jsonl")
@pytest.mark.parametrize("case", ATTACKS, ids=lambda c: c.name)def test_attack_is_blocked(case): result = agent.run(case.input) assert result.blocked or result.output == case.safe_outputWhat good looks like
A well-harnessed agent doesn’t need to be clever about attacks. It only needs to have nowhere to go: a narrow job, validated outputs and permissions it can’t change.