Short answer: test every channel your agent reads (user input, documents, web pages, emails, tool results and memory) with injected instructions that carry a unique canary word, and fail the test if the canary shows up in the output or the agent takes an action it shouldn't. Run the tests in staging with tools stubbed, and re-run them after every model, prompt or tool change.
OWASP's Top 10 for LLM Applications ranks prompt injection first, as LLM01:2025 Prompt Injection, and covers both direct and indirect injection.
Checklist
- List the channels. Write down everything the agent reads that a user or third party could control: chat input, uploaded files, retrieved documents, web pages, inbound email, tool and API responses, and stored memory.
- List the dangerous actions. Which tools can send, post, pay, delete or change records? Those are the actions an attacker wants.
- Write cases per channel and technique. For each channel, try at least: asking the agent to repeat a canary word, to ignore its instructions, to reveal its system prompt or secrets, to call a dangerous tool, to send data to an outside address, and to change its answer.
- Define pass/fail before running. A case fails if the canary appears, a forbidden tool is called, or protected data leaves. Write the rule next to each case.
- Stub the tools. Run in staging, with dangerous tools in dry-run mode, so a failure is logged, not executed.
- Score by severity. Treat a forbidden action or data leak as high severity and a changed answer as lower severity.
- Gate releases. Make any high-severity failure block the release in CI.
- Add every real incident as a test. When something gets through in production, turn it into a regression case.
What testing does not do
Tests show which attacks your agent resists today; they don't prove it is safe. Limit what injected text can achieve as well: least-privilege tools, human approval before sending, paying or deleting, and treating retrieved content as data, not instructions.
The Prompt-Injection Test Pack ($29) has 36 original cases (6 techniques × 6 channels) with canary-based pass/fail rules, a Python harness that runs them against your agent's HTTP endpoint and exits non-zero on high-severity failures, a scoring rubric and a report template.
FAQ
What is prompt injection?
Text that tries to change an AI system's behaviour, either typed directly by a user or hidden in content the system reads, such as a web page or document. OWASP lists it as LLM01:2025 in its Top 10 for LLM Applications.
What is a canary in prompt-injection testing?
A unique code word placed in the injected text. If it appears in the agent's output or actions, the agent obeyed the injected instruction and the test fails.
Should tests run in production?
No. Run them in staging with tools stubbed or in dry-run mode, so a failed test cannot send, delete or pay for anything.