Agent Observability Starter

What to log for every agent run and how to alert on it: a trace field spec aligned to OpenTelemetry GenAI attribute names (plus approval/permission fields), three starter alerts with thresholds, and a script that checks exported traces against the spec and computes the alert metrics.

Doc / JSON / Python script (ZIP + PDF)What to log for every agent run and how to alert on it: a trace field spec aligned to OpenTelemetry GenAI attribute names (plus approval/permission fields), three starter alerts with thresholds, and a script that checks exported traces against the spec and computes the alert metrics.

What's inside

Verify it works

  1. python3 check_traces.py --selftest prints selftest ok (Python 3.9+, no installs).
  2. python3 check_traces.py sample_traces.jsonl --baseline-cost 0.08 reports exactly 1 spec problem: trace t007 sent a customer email (R2) without an approval. It also shows the three alerts: failure rate 25%, cost regression, and the unapproved call. Exit code 1.
  3. Export one day of your own spans as JSON Lines (flatten attributes and set kind), then run the checker. Fix the instrumentation until there are 0 spec problems.
  4. Copy the three alerts in 02_Alerts_and_Dashboards.md into your alerting tool, using the metric definitions as written.
  5. Optional: if your field names differ, edit trace_spec.json and re-run the selftest.

Preview

First part of the main document, shown as-is:

Agent Trace Spec (afk-trace-1)

The minimum to record for every agent run, so you can answer four questions afterwards: Did it finish the task? What did it cost? What did it touch? Who approved the risky parts?

Machine-readable version: trace_spec.json. Field names use the OpenTelemetry GenAI semantic conventions (gen_ai.*) where one exists, so the spans work in any OTel backend. The afk.* fields fill gaps the conventions don't yet cover: outcome, risk tier, approvals. References: https://opentelemetry.io/blog/2026/genai-observability/ and https://github.com/open-telemetry/semantic-conventions-genai. Those conventions are still marked Development, so check the names whenever you upgrade instrumentation libraries.

Span tree per run

invoke_agent  (one per run: who, which agent, outcome, total cost)
├── chat           (each model call: model, input/output tokens)
├── execute_tool   (each tool call: tool, risk tier, status, approval)
└── ...

Fields

Span Field Required Why
invoke_agent gen_ai.agent.name yes which agent
invoke_agent afk.user.id yes on whose behalf (pseudonymous id, not an email)
invoke_agent afk.run.outcome yes completed / failed / abandoned / budget_exceeded / blocked
invoke_agent afk.run.cost_usd, afk.run.steps recommended cost-per-task and runaway-loop alerts
chat gen_ai.request.model yes model drift and cost
chat gen_ai.usage.input_tokens / output_tokens yes cost
execute_tool gen_ai.tool.name yes what it touched
execute_tool afk.tool.risk_tier yes R0–R3, same tiers as the AFK Tool-Access Policy
execute_tool afk.tool.status yes ok / error / denied
execute_tool afk.approval.id / approver / decision yes for R2–R3 audit trail for risky actions
execute_tool afk.tool.args_sha256 recommended prove what was approved equals what ran, without storing raw arguments

Privacy defaults

← All kits