---
id: "lib-2026-10-31-xla-from-the-trenches-correctness-on-tpu"
type: "library"
title: "XLA from the Trenches: Correctness on TPU Workloads at Scale"
lang: "EN"
org: "Google"
source_type: "Report"
date: "2026-10-31"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-10-31-xla-from-the-trenches-correctness-on-tpu/"
original_url: "https://arxiv.org/abs/2611.00624"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "Google XLA:TPU engineers report a production verification method using fuzzing and agent swarms with a bug taxonomy; detailed practice, but results are internal counts that outsiders cannot check."
summary_basis: "full-text"
original_minutes: "35"
tags: ["ai technology", "coding agents", "compiler correctness"]
---

# XLA from the Trenches: Correctness on TPU Workloads at Scale

Tier B (Useful, limited): Google XLA:TPU engineers report a production verification method using fuzzing and agent swarms with a bug taxonomy; detailed practice, but results are internal counts that outsiders cannot check.

Summary based on the full text.

## In short

An arXiv paper (31 October 2026) by Ignacio Cano and colleagues at Google on keeping the XLA compiler for TPUs correct. Subtle code-generation defects can silently corrupt numbers in training and serving without any error. The team combines synthetic program generation, fuzzing on real production graphs, and swarms of AI agents that write stress tests, reproduce failures and draft fixes. People review every patch.

## Key numbers

- over 142: XLA:TPU compiler bugs whose resolution the framework drove
- 103: additional findings under review or fixes underway
- 43%: share of resolved defects in the lowest-level TensorCore LLO and emitter layer
- 25%: share in high-level HLO simplifiers and rewrites

## What it reports

- Over 142 compiler bugs resolved and 103 more under review; most are silent numerical miscompilations, because the agent swarms were steered away from loud crashes.
- Defects by subsystem: TensorCore LLO and emitters 43%, HLO simplifiers 25%, precision and emulation rewriters 14%, SparseCore lowering 10%, custom kernel lowering 8%.
- Agent swarm design: a lead agent spawns subagents to inspect passes, build tests, verify cross-pass invariants, patch and estimate production severity; a coordinator queues verified patches for human code review, with agents answering simple reviewer questions.
- A hierarchical differential pipeline bisects large graphs down to single operations inside fused kernels, reading intermediate register values from hardware.
- Case: daily fleet fuzzing flagged a fused conv-reduce bug days before a customer escalation in which evaluation metrics dropped 100x after a compiler update.
- Lessons include treating every false positive as a bug in the harness and that reviewer burden grows non-linearly with patch size.

## How they got it

Experience report from deploying the method on Google's production XLA:TPU fleet, with a taxonomy of peer-reviewed fixes and worked examples of representative bugs.

## Limits

- Self-report; no baseline of how many bugs earlier methods found, and no share attributed to agents versus fuzzing.
- No figures on agent false positives, patch acceptance rate or compute cost.
- Specific to one compiler and accelerator family.

## Why it matters for companies

Agents here are used where outputs can be checked mechanically: a reference interpreter decides whether a bug is real, and people approve fixes. That is the pattern that works for agentic engineering at scale. For firms training or serving models, the paper is a reminder that compiler and runtime updates can change numbers silently, so numerical regression checks belong in release gates.

Original: https://arxiv.org/abs/2611.00624