What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports
A · Strong evidence Tencent researchers measure, with paired probes, bootstrap confidence intervals and an independent replication chain, how much of a coding agent's task state survives into its own reports.
Summary based on the full text·Original: about 35 minutes to read
In short
CTXGAP, an arXiv preprint (October 2026) by Guoyang Xu and colleagues at Tencent, asks how much a supervisor can learn from a coding agent's reports compared with its full execution record. Readers of the reports recover 52.0% of task-state facts; readers of the full context recover 92.7%. The gap stays large even after successful repairs and for states rated relevant to supervision.
Key numbers
- 92.7% vs 52.0%task-state facts recovered from full execution context vs from the agent's reports alone
- 34.8 pointsremaining gap at the final checkpoint of 30 verified fixes
- 73.6%recovery when readers see full terminal scrollback
- 4,799valid probes across 37 SWE-bench Verified tasks
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 37 Python SWE-bench Verified tasks, 4,799 valid probes. Agents answer 93.6% of task-state questions when asked directly.
- Report-only recovery 52.0% vs full-context 92.7%: a 40.7-point gap (95% CI 37.3 to 43.8).
- At the final checkpoint of 30 hidden-test-verified fixes, the gap is 34.8 points; positive in 28 of 30 tasks.
- Full terminal scrollback lifts recovery to 73.6%, leaving a 19.1-point gap.
- Human annotators found the largest category of lost information was states mentioned without enough detail (55.4% and 59.2%). Adding mere mentions gave no measurable recovery.
- Rewriting reports with context at matched length improved recovery by about eight points.
- Gaps recur across Claude Code, Pi and DeepSeek harnesses (0.390 to 0.490) and in an independent DeepSeek measurement chain (0.398).
How they got it
Task-state "atoms" are extracted from recorded trajectories, turned into probes, and the same questions are answered by an observer reading either the accumulated reports, the terminal scrollback, or the full context. Grading is model-based with human audits; confidence intervals use issue-cluster bootstrap.
Limits
- The main pipeline uses one model (HY4-preview) for extraction, probes, answering and much of grading, which may favour execution-level detail.
- The 37-task cohort is non-random and underrepresents hard issues.
- Supervision is simulated; no study of real human supervisors or code outcomes.
- Full trajectory texts are not yet public.
Why it matters for companies
Managers who sign off agent work from summaries see about half of what happened, including failed attempts and known risks. Review processes for agent-written code should require access to logs or structured evidence, not only the closing report. Report templates that force specific detail (what failed, what was worked around, what remains open) help more than asking agents to "mention" issues.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Coding agents