Agent Field Kit · AI Agent Incident Runbooks (template v1.0)

Adapt the severities, contacts and tools to your environment. Every runbook follows: Detect → Contain → Assess → Fix → Recover → Learn. Keep this file next to your on-call docs.

Severity levels

Sev	Definition	Response target
SEV1	Data exposure, an unauthorized external action (message, payment, deletion), or a safety issue	Contain within [15 min]
SEV2	Wrong outputs reaching users at scale, or runaway cost	Contain within [1 h]
SEV3	Degraded quality, isolated errors, partial outage	Within [1 business day]

Kill switch: [where it is, who can flip it]. Incident channel: [ ]. Owner on call: [ ].

RB-1 Prompt injection / agent followed malicious instructions

1. Contain: disable the affected agent or tool permission; revoke any tokens the agent used.
2. Assess: pull the run logs. Find the injected source (document, web page, email) and list every action taken after ingesting it.
3. Fix: treat retrieved content as data only. Add the case to the regression tests. Require approval before the affected action type runs.
4. Recover: reverse actions where possible. Notify the affected parties per your policy.
5. Learn: postmortem within [5] days.

RB-2 Sensitive data exposed in output or logs

1. Contain: stop the agent; restrict access to the logs and outputs involved.
2. Assess: what data, whose, how many records, where it went, and for how long.
3. Escalate: involve legal/privacy immediately; notification duties may have deadlines.
4. Fix: redaction at the input and log layers; least-privilege data access.
5. Recover: purge copies, rotate any exposed credentials, and document everything.

RB-3 Runaway cost / loop

1. Contain: hit the spend cap or kill switch; pause schedules.
2. Assess: which workflow, the trigger, and spend since start.
3. Fix: per-run step limits, per-day budget caps, loop detection, and retries with backoff.
4. Recover: confirm billing; resume at reduced concurrency.

RB-4 Quality regression after a model, prompt or tool change

1. Detect: eval scores drop, or user reports.
2. Contain: roll back to the last pinned version.
3. Assess: run the frozen eval set on both versions and diff the failures.
4. Fix: adjust and re-test; ship only after the eval gate passes.

RB-5 Unauthorized external action (sent / posted / paid / deleted)

1. Contain: revoke the agent's send, post, pay and delete permissions now.
2. Assess: exactly what went out, to whom, and when.
3. Recover: retract or correct where the platform allows; send a human-written correction if needed.
4. Fix: require human approval for that action class; add a test.