Real-Time Agent Monitoring Kit

A working real-time monitoring pipeline for AI agents: an OpenTelemetry Collector config (validated, with content redaction and tail sampling), a streaming monitor for cost burn, runaway loops, token amplification, unapproved high-risk actions and failure rate, Prometheus alert rules, dashboards spec and a runnable demo.

Doc / YAML / Python (ZIP + PDF)A working real-time monitoring pipeline for AI agents: an OpenTelemetry Collector config (validated, with content redaction and tail sampling), a streaming monitor for cost burn, runaway loops, token amplification, unapproved high-risk actions and failure rate, Prometheus alert rules, dashboards spec and a runnable demo.

What's inside

Verify it works

  1. Monitor logic. python3 live_monitor.py --selftest prints selftest ok and lists all five alert types. It replays 150 simulated runs with injected incidents and checks each alert fires on the right agent and nowhere else. It also parses an OTLP JSON span the way the collector writes it.
  2. Live demo. python3 live_monitor.py --demo --seconds 20 --serve 9464 streams the simulation. Watch the alerts print, then open http://127.0.0.1:9464/metrics and /alerts.
  3. Collector config. With otelcol-contrib installed: TRACE_BACKEND_ENDPOINT=http://localhost:4318 TRACE_BACKEND_AUTH=none otelcol-contrib validate --config otel-collector.yaml exits 0. Run it, send one span, and confirm spans.jsonl gets a line with no gen_ai.input.messages (redaction). Then run python3 live_monitor.py --follow spans.jsonl.
  4. Alert rules. promtool check rules alert_rules.yml reports 6 rules. Add prometheus-scrape.yml to your Prometheus config.
  5. Budgets. Edit budgets.json per agent. Re-run step 1 if you change the defaults heavily (the selftest uses the file). The Python needs 3.9+ and no installs. The collector and promtool are the official open-source binaries.

Preview

First part of the main document, shown as-is:

Real-Time Agent Monitoring: Spec

Goal: see an agent misbehave within minutes, not at invoice time or from a customer. This spec covers what to collect, the live signals, the alerts and the dashboards, and it ships with a working pipeline:

agents --OTLP--> OpenTelemetry Collector (otel-collector.yaml)
                  ├─ spanmetrics ──────────> Prometheus :8889   (rates, errors, latency by agent/tool/model)
                  ├─ spans.jsonl ──> live_monitor.py :9464     (cost burn, loops, token amplification, approvals, failure rate)
                  └─ tail sampling ───────> your trace backend (errors, R2/R3 tool calls, failed and slow runs, 10% baseline)
Prometheus + alert_rules.yml ──> Alertmanager / on-call

Field names follow the OpenTelemetry GenAI semantic conventions (gen_ai.*, https://opentelemetry.io/blog/2026/genai-observability/). afk.* fields carry outcome, risk tier and approval, the same as the AFK Agent Observability Starter. The collector config was validated with otelcol-contrib v0.162.0.

1. Instrumentation minimum

Span Must carry
invoke_agent (one per run) gen_ai.agent.name, afk.run.outcome, afk.run.cost_usd, afk.run.steps
chat (each model call) gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens
execute_tool (each tool call) gen_ai.tool.name, afk.tool.risk_tier (R0–R3), afk.tool.status, afk.approval.decision for R2/R3

The collector deletes prompt, response and tool-argument content attributes by default (attributes/redact). Remove a line only if you have a retention policy that covers that content.

2. Live signals and default thresholds (budgets.json)

Signal Definition Default Why
Cost burn spend in the last hour, projected to 24h, ÷ daily budget alert > 1×, page > 3× catches loops and model swaps the same day
Runaway loop steps > max_steps or tool calls > max_tool_calls in one run 25 / 40 stops a 400-call chain that should have been 4
Token amplification run tokens > N × the agent's median of its last 200 runs 5× prompt growth, context stuffing, retrieval gone wide
Unapproved high-risk action R2/R3 tool ran without an approved decision any the one alert that always pages
Failure rate non-completed runs in a rolling 15-min window (min 20 runs) > 10% earliest user-visible breakage

Set per-agent overrides under agents in budgets.json. Retune after two weeks of data.

3. Alerts (alert_rules.yml)

Alert Severity First 15 minutes
AgentUnapprovedHighRiskAction page Disable the tool row or kill switch; revoke the credential; preserve the trace; start the incident
AgentCostBurnSevere (>3×) page Check the latest deploy and model change; lower the agent's rate limit or pause it
AgentCostBurn (>1×, 10 min) ticket Review the top-cost traces in the backend (tail sampling keeps them)
AgentFailureRate page Compare with the last deploy; roll back the prompt or model if it correlates
AgentRunawayOrAmplification ticket Open the trace; add a step cap or fix the retrieval scope
AgentMonitorDown page You're blind; restart live_monitor.py and check that spans.jsonl is still growing

← All kits