---
id: "lib-2026-11-05-i-broke-a-working-agent-seven-ways-the-metric-i"
type: "library"
title: "I broke a working agent seven ways. The metric I was watching caught two of them."
lang: "EN"
org: "The New Stack (Kayode Ojo)"
source_type: "Practitioner post"
date: "2026-11-05"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-11-05-i-broke-a-working-agent-seven-ways-the-metric-i/"
original_url: "https://thenewstack.io/monitoring-production-ai-agents/"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "A software engineer's controlled fault-injection experiment on one support agent, with thresholds and per-arm results shown; one agent, three baseline runs, sponsored post on The New Stack."
summary_basis: "full-text"
original_minutes: "12"
tags: ["ai technology", "agent monitoring", "observability", "cost"]
---

# I broke a working agent seven ways. The metric I was watching caught two of them.

Tier B (Useful, limited): A software engineer's controlled fault-injection experiment on one support agent, with thresholds and per-arm results shown; one agent, three baseline runs, sponsored post on The New Stack.

Summary based on the full text.

## In short

A New Stack post (5 November 2026, sponsored by Andela) by Kayode Ojo. He injected eight faults into a working customer-support agent running against a mock enterprise API and checked which metrics noticed. Task success caught only the two faults that made tasks impossible. Tool-call counts and scope drift caught four each. Token cost, the metric most teams already track, was too noisy to be useful. The trigger was finding that an unpinned model alias had silently switched from Claude Haiku 4.5 to Haiku 5.5 between two test months.

## Key numbers

- 2 of 7: active regressions caught by the task-success metric
- 18%: token-use variation across three identical baseline runs, forcing a ±37% alert threshold
- 3%: tool-call variation across the same runs, allowing a ±7% threshold
- +70%: tool-call surge when 25% of API calls failed, while tasks still completed 19 of 19
- 161,013: tokens used when documentation was truncated, 73% above baseline

## What it reports

- Set-up: 30 API operations with an append-only audit log, 19 scored tasks plus two unauthorised-refund traps, baseline on Claude Sonnet 5.5 over three runs; thresholds set at twice baseline variance.
- Faults: weaker model, speed pressure, truncated docs (11 of 30 commands), missing permission, stale reads, 25% transient API errors, 1.5 s latency, distractor tool. The distractor was never used, so seven active arms remained.
- Flaky API: all 31 injected failures absorbed; 19 of 19 tasks done with 99 tool calls against a 58-call baseline. Only the tool-call metric fired.
- Stale reads: every quantitative metric stayed green (+22% tokens, +5% calls, drift 0.133 vs a 0.135 threshold); only the agent's final text summary reported that no writes could be verified.
- Truncated docs: the agent brute-forced endpoint names in local shell loops that never hit the audit log; 6 tasks failed, 15 minutes against 5.
- Model swap and flaky API raised tokens by 36% and 26%, both under the 37% threshold.
- The refund traps that separated models in September caught nothing in October.

## How they got it

Deterministic fault injection behind a per-run wrapper, with injection confirmed in logs before scoring detectors. Each fault was one run; the baseline was three runs.

## Limits

- One agent, one task suite, single runs per fault; thresholds depend on only three baselines.
- The September prompt could not be reconstructed exactly, so two variables changed between months.
- Author's own harness; sponsored content.

## Why it matters for companies

Success rate is a poor production monitor for agents because capable agents absorb faults by retrying and working around them, which hides infrastructure problems until cost or latency explodes. Log the resolved model ID on every run, alert on tool calls per verified task, count retries explicitly, and parse the agent's final summary for stated uncertainty. Refresh adversarial tests whenever the model behind an alias changes.

Original: https://thenewstack.io/monitoring-production-ai-agents/