How to Prove Your AI Agent Is Actually Worth It
B · Useful, limited A PagerDuty product analyst gives a usable metric set and an open-source evaluation pipeline for agents, but discloses no outcome numbers for the agent she describes.
Summary based on the full transcript·Original: about 23 minutes to watch
In short
A 23-minute talk at AGNTCon + MCPCon North America 2026 by Inés Bolaños, senior product analyst at PagerDuty. Using PagerDuty's internal SRE (incident) agent as the case, she proposes a "HIRE" test for deciding whether to build, keep or drop an agent, a set of conversation metrics, and a three-tier pipeline for labelling agent conversations. The pipeline is released as an open-source repo with synthetic data. The talk offers vocabulary and design, with no evidence of results.
Key numbers
- up to 3 hourstime several engineers needed to fix the consumer-lag incident that prompted the SRE agent
- 40 of 50lines rewritten by a human in the talk's example of an agent draft that adds noise, not value
- eightaverage back-and-forth turns given as a sign the agent's reasoning needs fixing
- zerotarget redline rate: agent actions blocked by the MCP permission check
As reported by the source.
English·Video·opens youtube.com
What it reports
- Origin: a consumer-lag incident took up to 3 hours and multiple engineers because a years-old runbook no longer matched the architecture or logs and its author had left. A prototype fed the alert, logs and outdated runbook to a model, which adapted the steps and suggested a fix. It was later scaled for internal and customer use.
- HIRE test. H: human readiness and data quality. I: impact and intent, including a written job description, responsibilities, boundaries and a performance review for each agent, to prevent "agent washing". R: rules versus responses versus reasoning; predictable tasks get plain automation, retrieval gets a chatbot, only ambiguity and multi-step decisions justify an agent. E: worst-case exposure, citing the July 2025 Replit production database deletion.
- Guardrails sit at the MCP permission boundary rather than in the model: the agent suggests, and anything beyond read-only routes to a human.
- Metrics: user correction rate (how much of a draft the human rewrites), ping-pong rate (turns before resolution), "just do it myself" abandonment rate, instant resolution rate, and redline rate (attempts blocked by permission checks, target zero). Example redline event: a request for master database credentials, which was blocked.
- Pipeline: deterministic rules taken from the agent's own source code label known intents, tracked as "deterministic coverage"; a cheap model classifies the rest with a confidence score, escalating low-confidence items to a stronger model and then a human queue; leftovers are clustered with BERTopic and named by the stronger model. Every model layer is scored against a human-labelled gold set. Runs on a schedule to produce trends.
- Out-of-scope requests are treated as product demand; one example is engineers asking the agent to draft stakeholder updates after an incident.
How they got it
Practitioner experience from one internal agent, presented as a conference talk. The digest is based on cleaned YouTube auto-captions. No adoption, accuracy, time-saved or cost figures for the agent are given; the repo ships with synthetic data.
Limits
- No outcome numbers for PagerDuty's agent; the metrics are proposed, not shown in use with results.
- The example figures (40 of 50 lines, eight turns) are illustrations, not measurements.
- Single company, single agent; the speaker's employer sells incident-management software.
- Transcript from auto-captions; small wording errors are possible.
Why it matters for companies
The job-description-and-performance-review framing gives business owners a plain way to ask what an agent is for before it is built, and to retire it when it does not earn its cost. The metric set (correction, abandonment, redline) is cheap to instrument from conversation logs and measures friction rather than clicks. Placing permissions at the tool boundary, outside the model, is a concrete control that risk teams can audit. Treat the pipeline as a starting template and attach your own baseline.
Summary based on the full transcript. Read the original for the full argument and data.Watch the original on youtube.com