---
id: "lib-2026-10-19-swe-residual-evaluating-starting-state-robustness"
type: "library"
title: "SWE-Residual: Evaluating Starting-State Robustness of Coding Agents"
lang: "EN"
org: "arXiv (HKU, Tencent)"
source_type: "Paper"
date: "2026-10-19"
week: "2026-W43"
url: "https://nosov.ai/library/lib-2026-10-19-swe-residual-evaluating-starting-state-robustness/"
original_url: "https://arxiv.org/abs/2610.25899"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "University of Hong Kong and Tencent researchers isolate one variable (the starting state) on two established benchmarks, test five frontier agents, publish code and show the failure mechanism with per-test evidence."
summary_basis: "full-text"
original_minutes: "40"
tags: ["ai technology", "coding agents", "benchmarks", "handoffs"]
frames: ["verification-capacity"]
---

# SWE-Residual: Evaluating Starting-State Robustness of Coding Agents

Tier A (Strong evidence): University of Hong Kong and Tencent researchers isolate one variable (the starting state) on two established benchmarks, test five frontier agents, publish code and show the failure mechanism with per-test evidence.

Summary based on the full text.

## In short

A preprint (arXiv, 19 October 2026) from the University of Hong Kong, Tencent and co-authors at Tokyo, North Texas and British Columbia. Coding benchmarks start agents from a clean repository. In real work agents often continue a task that a person or another agent has reported as done but is not. When frontier agents start from such a state, their success falls by 20 to 54 points and the leaderboard order changes. The authors name the pattern acceptance bias: the agent treats the inherited work as the specification of what remains.

## Key numbers

- 198: Residual States from 111 DeepSWE and Terminal-Bench tasks, each left by an agent that wrongly claimed completion
- 20.6–53.5 percentage points: drop in DeepSWE success for five frontier agents when starting from inherited work instead of a clean start
- 72.2% to 20.0%: Gemini-3.8-Flash success on DeepSWE, clean start versus residual start
- 58.9–88.2%: failed DeepSWE continuations that submit with exactly the same tests failing as at the start
- 6.7% to 15.0%: residual-start runs that change the files the predecessor left out, versus 40.5% to 47.9% from a clean start

## What it reports

- Each Residual State is what a predecessor agent (DeepSeek, Qwen or GLM family) left after submitting work as complete while hidden tests still failed. The evaluated agent gets only the original request and that state.
- On DeepSWE the median Residual State already passes 98.4% of hidden tests, so it looks finished.
- Success drops 20.6–53.5 points on DeepSWE and 20.6–34.9 on Terminal-Bench. Gemini-3.8-Flash falls from 72.2% to 20.0% on DeepSWE. DeepSWE rankings reverse.
- In 58.9–88.2% of failed DeepSWE continuations, the same tests fail at the end as at the start, though the agent passes most of them from a clean start.
- Failed continuations keep 86.4% to 98.6% of the predecessor's lines; no continuation discards the inherited changes and starts over.
- Agents open the files the predecessor left out in 37.6% to 64.0% of runs but change them in only 6.7% to 15.0%.
- GPT-6-Sol fails differently: in 11 of 30 sampled failures it names the defect and tries to repair it, against at most 2 of 30 for three other agents.

## How they got it

A pipeline ("Residualization") runs predecessor agents on DeepSWE v1.1 and Terminal-Bench tasks, keeps states where the agent claimed completion but hidden tests failed, and validates them. Five frontier agents then run from both clean and residual starts with identical request, harness, system prompt, verifier and budget, so the starting state is the only difference. Acceptance bias is measured from per-test outcomes, retained lines, files touched and step counts.

## Limits

- 198 states from 111 tasks, two benchmarks and three predecessor families; inherited work from humans is not covered.
- The acceptance-bias analysis uses DeepSWE only, where per-test results and final patches are available.
- Blind continuation is a hard case by design: real handoffs often carry some notes or history.

## Why it matters for companies

Agent work is increasingly handed between sessions, agents and people. This paper shows that a "done" claim from the previous step strongly lowers the chance the next step finds what is missing, and that clean-start benchmark scores do not predict this. Practical consequences: re-verify inherited work against the original request with fresh tests, do not pass a prior agent's completion claim forward as fact, and add handoff scenarios to internal agent evaluations before ranking vendors on public benchmarks.

Original: https://arxiv.org/abs/2610.25899