Continue or Restart? The Limits of Early Failure Prediction in Coding Agents
A · Strong evidence NC State group including Tim Menzies analyses 147,002 public agent traces with a design that separates task difficulty from attempt quality, plus replayed stopping policies; a clear method and a useful negative result, on open-model benchmark traces only.
Summary based on the full text·Original: about 40 minutes to read
In short
A preprint (arXiv, 26 October 2026) by Srinath Srinivasan, Lunxiao Li, Tim Menzies and Wesley Assunção, NC State University. Early stopping is meant to save money by killing coding-agent runs that look doomed and restarting them. The authors show that early signals can tell hard tasks from easy ones, but cannot tell a good attempt from a bad one on the same task. In replay, letting the run continue solved more issues at lower token cost than any restart policy in almost every configuration.
Key numbers
- 147,002coding-agent execution traces analysed across four model-framework settings
- 0.50mean within-task AUC of a fine-tuned monitor at 20k tokens over 32,406 attempts, i.e. chance
- 216 of 224budget-threshold configurations where simply continuing beat restart policies on solve rate and token cost
- 1.98 ptsmaximum gain of learned over random restarting under a 75k-token cap, almost all from task selection
- -0.11 to +0.24 ptscontribution of picking the right attempt within a task to that gain
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Pooled prediction across tasks improves modestly with early execution prefixes, because it mostly picks up task difficulty.
- Within the same task, discrimination is near chance at 5k to 20k tokens: initial-turn embeddings and task-conditional models give AUCs around 0.50; a fine-tuned Qwen3-0.6B monitor scores 0.50 [0.48, 0.51] over 32,406 attempts.
- Under a 75k-token cap, learned restarting beats random restarting by up to 1.98 points, but an exact decomposition shows the gain comes from not stopping runs on easy issues; choosing among attempts adds -0.11 to +0.24 points.
- Uninterrupted continuation wins on solve rate and mean token cost in all four settings at the 75k cap and in 216 of 224 tested budget-threshold configurations.
- Aborting at 20k tokens on a difficulty-only signal is dominated by filtering tasks before execution, at zero token cost.
How they got it
Traces from Nebius SWE-rebench (OpenHands with Qwen3-Coder-480B) and three slices of NVIDIA Open-SWE-Traces (SWE-agent and OpenHands with Qwen3.5-122B, OpenHands with MiniMax-M2.5). Models range from linear count features to a fine-tuned small language model, trained with five-fold task-grouped cross-validation; within-task AUC is computed only on tasks with mixed outcomes. Stopping policies were replayed against matched random restarts and continuation under shared token budgets.
Limits
- Benchmark issues from public repositories with open-weight models; closed frontier agents were not tested.
- Restarts are cold; state-preserving recovery is out of scope.
- Early landmarks studied stop at 20k tokens; later signals may carry more information.
Why it matters for companies
Teams paying for agent runs at scale may be tempted to buy or build "failure monitors" that kill runs early. This paper suggests checking two things before doing so: whether the monitor distinguishes attempts on the same task, not just hard from easy tasks, and whether it saves tokens relative to simply letting runs finish. If the signal is only difficulty, it is cheaper to route or skip hard tasks before starting.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org