---
id: "lib-2026-10-14-what-coding-agents-leave-unsaid-measuring-task"
type: "library"
title: "What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports"
lang: "EN"
org: "arXiv"
source_type: "Paper"
date: "2026-10-14"
week: "2026-W42"
url: "https://nosov.ai/library/lib-2026-10-14-what-coding-agents-leave-unsaid-measuring-task/"
original_url: "https://arxiv.org/abs/2610.20003"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Tencent researchers measure, with paired probes, bootstrap confidence intervals and an independent replication chain, how much of a coding agent's task state survives into its own reports."
summary_basis: "full-text"
original_minutes: "35"
tags: ["ai technology", "coding agents", "oversight", "manager of agents"]
---

# What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports

Tier A (Strong evidence): Tencent researchers measure, with paired probes, bootstrap confidence intervals and an independent replication chain, how much of a coding agent's task state survives into its own reports.

Summary based on the full text.

## In short

CTXGAP, an arXiv preprint (October 2026) by Guoyang Xu and colleagues at Tencent, asks how much a supervisor can learn from a coding agent's reports compared with its full execution record. Readers of the reports recover 52.0% of task-state facts; readers of the full context recover 92.7%. The gap stays large even after successful repairs and for states rated relevant to supervision.

## Key numbers

- 92.7% vs 52.0%: task-state facts recovered from full execution context vs from the agent's reports alone
- 34.8 points: remaining gap at the final checkpoint of 30 verified fixes
- 73.6%: recovery when readers see full terminal scrollback
- 4,799: valid probes across 37 SWE-bench Verified tasks

## What it reports

- 37 Python SWE-bench Verified tasks, 4,799 valid probes. Agents answer 93.6% of task-state questions when asked directly.
- Report-only recovery 52.0% vs full-context 92.7%: a 40.7-point gap (95% CI 37.3 to 43.8).
- At the final checkpoint of 30 hidden-test-verified fixes, the gap is 34.8 points; positive in 28 of 30 tasks.
- Full terminal scrollback lifts recovery to 73.6%, leaving a 19.1-point gap.
- Human annotators found the largest category of lost information was states mentioned without enough detail (55.4% and 59.2%). Adding mere mentions gave no measurable recovery.
- Rewriting reports with context at matched length improved recovery by about eight points.
- Gaps recur across Claude Code, Pi and DeepSeek harnesses (0.390 to 0.490) and in an independent DeepSeek measurement chain (0.398).

## How they got it

Task-state "atoms" are extracted from recorded trajectories, turned into probes, and the same questions are answered by an observer reading either the accumulated reports, the terminal scrollback, or the full context. Grading is model-based with human audits; confidence intervals use issue-cluster bootstrap.

## Limits

- The main pipeline uses one model (HY4-preview) for extraction, probes, answering and much of grading, which may favour execution-level detail.
- The 37-task cohort is non-random and underrepresents hard issues.
- Supervision is simulated; no study of real human supervisors or code outcomes.
- Full trajectory texts are not yet public.

## Why it matters for companies

Managers who sign off agent work from summaries see about half of what happened, including failed attempts and known risks. Review processes for agent-written code should require access to logs or structured evidence, not only the closing report. Report templates that force specific detail (what failed, what was worked around, what remains open) help more than asking agents to "mention" issues.

Original: https://arxiv.org/abs/2610.20003