---
id: "lib-2026-10-26-making-failure-diagnosis-reliable-with-pensive-ai"
type: "library"
title: "Making Failure Diagnosis Reliable with Pensive AI"
lang: "EN"
org: "Netflix"
source_type: "Practitioner post"
date: "2026-10-26"
week: "2026-W44"
url: "https://nosov.ai/library/lib-2026-10-26-making-failure-diagnosis-reliable-with-pensive-ai/"
original_url: "https://netflixtechblog.com/making-failure-diagnosis-reliable-with-pensive-ai-6b0811a1c015"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "Netflix's data platform engineers give a detailed first-hand design account of adding model calls to a rule-based failure-diagnosis system, including evals and release gates; the engineering is concrete, but quality scores are not published and the one outcome figure is observational."
summary_basis: "full-text"
original_minutes: "14"
tags: ["ai technology", "future of work", "enablement", "business transformation", "failure diagnosis", "data platform operations", "log extraction"]
---

# Making Failure Diagnosis Reliable with Pensive AI

Tier B (Useful, limited): Netflix's data platform engineers give a detailed first-hand design account of adding model calls to a rule-based failure-diagnosis system, including evals and release gates; the engineering is concrete, but quality scores are not published and the one outcome figure is observational.

Summary based on the full text.

## In short

A Netflix Technology Blog post (October 2026) by six data-platform engineers. Netflix runs millions of data and ML workloads a day; on peak days about 100,000 fail. Its existing system, Pensive, classified failures with regular expressions and hand-written rules and made about half of them self-service, but left the other half as "unknown". Pensive AI adds model-based error extraction for every failure and on-demand diagnosis, while keeping the rules running. The post is mostly about the system around the model: bounded outputs, deadlines, caching, evals and release gates.

## Key numbers

- 100,000: workloads that fail on peak days (roughly)
- seven million: job-failure classifications over three months, about half left as unknown by the rules
- 100%: share of failures now covered by AI error extraction
- 90-second: hard post-processing timeout the whole diagnosis response must meet
- 321%: more manual log-exploration actions on failed steps where nobody used the tool (observational)

## What it reports

- Over three months Pensive processed about seven million failure classifications; about half landed in the unknown bucket.
- Extraction design: logs are numbered line by line and the model returns only the first and last line numbers of the error span. Pensive rebuilds the text from its own copy, which cuts output tokens and stops the model from inventing log text.
- Extraction runs on every failed step within Maestro's hard 90-second timeout; per-log calls share a deadline and late calls are dropped. Rules always run alongside.
- A new data field was added beside the legacy one, so clients migrate at their own pace; some still use the old path.
- Full diagnosis runs only on demand, since most failed steps are never opened. Responses are cached per failed step attempt for 90 days, so everyone investigating one failure sees the same answer and repeat requests cost no inference.
- Evals: legacy rules act as reference labels for inclusion; a second judge scores precision; a script flags lines recurring in more than about 30% of workflows (three lines ended up excluded); latency is recorded per call. Logs are archived to object storage so eval inputs stay fixed after the 7-to-30-day log expiry.
- Every pull request deploys to a preview environment and must pass precision and inclusion thresholds; evals also run after promotion and on a schedule in production.
- Each thumbs-down sends a business-hours alert linked to the exact trace.

## How they got it

Engineering write-up of a production system. The 321% figure compares failed steps where someone used Pensive AI with steps where no one did; the authors state it is observational, not causal.

## Limits

- No eval scores, accuracy, cost or time-to-resolution figures are published.
- The 321% comparison is confounded: people may open the tool on different kinds of failure.
- Debugging ground truth is synthesised by a model from resolved support threads, which is retrospective and model-written.
- Automated classification of retry decisions, the higher-stakes step, is future work.

## Why it matters for companies

The post is a design pattern for putting models into an operational process without losing what already works: keep the deterministic rules as a floor, ask the model for a small bounded answer (two line numbers) instead of free text, enforce a deadline, cache by the unit of work, and gate releases on evals that run on frozen inputs. Most internal support and incident teams could copy this structure. The authors are explicit that reliability came from the system around the model calls.

Original: https://netflixtechblog.com/making-failure-diagnosis-reliable-with-pensive-ai-6b0811a1c015