Back Up Before You Regret: Measuring Backup Awareness and Automating Backup in Coding Agents
B · Useful, limited Northeastern, Berkeley, Chicago and UIUC researchers ground a benchmark in real user-reported losses and show a clear gap, but the proposed fix is evaluated only on the authors' own benchmark.
Summary based on the full text·Original: about 50 minutes to read
In short
A preprint (arXiv, 20 October 2026) by Xu Li, Simon Yu, Minzhou Pan, Yiyou Sun, Zhaorun Chen, Bo Li and Weiyan Shi (Northeastern, UC Berkeley, Chicago, UIUC). Coding agents change files, environments and remote services, but rarely save the prior state first. RegretBench turns real cases where users regretted not keeping a backup into tasks. Agents finish most tasks but back up before the damaging step in about one in seven. A harness plug-in that backs up before every state-changing call closes the gap cheaply.
Key numbers
- 205executable tasks built from real 'regret moments' where users lost state they later needed
- 86.1%average task completion across six frontier models
- 15.2%tasks in which agents backed up state before the destructive action
- 92.2%average similarity of state restored from the BackUpper plug-in's backups to the original
- $0.014extra cost per task for BackUpper, with 7x fewer tokens than agent self-backup
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Six frontier models complete 86.1% of tasks on average but back up before regret moments in only 15.2%; four models are below 10%.
- GPT 5.6-Sol in Codex backs up in 5.9% of tasks, DeepSeek V4-Pro in Hermes in 2.9%.
- 72.0% of runs complete the task as requested without preserving the state the user later needed.
- Mainstream harnesses back up only the workspace or native edits; none covers state changed through external tools or remote services.
- BackUpper (rules for common operations plus a subagent for complex cases) covered every regret-moment action. Restored state reached 92.2% similarity to the original, against 14.3% without restoration and 33.8% for post-hoc reconstruction.
- Cost: $0.014 per task, 7x fewer tokens than having the main agent back up. Human annotators confirmed 94.0% of its judge-positive backups.
- With user preferences on what to keep, it matched human annotations at 84.5% F1.
How they got it
The authors searched X, Reddit, GitHub, Stack Overflow and tech news for reports of lost state, kept 205 of 557 candidates where the state and the later need were clear, and rebuilt each as a multi-turn task in a real environment across five tools (about 45 actions per task). Backup rate before the regret action and restoration similarity were measured across six models and several harnesses.
Limits
- Source reports include actions by people as well as agents, so not every case is an agent failure.
- BackUpper is designed and tested by the same team on the same benchmark.
- Server-generated fields and third-party replies cannot be restored byte for byte.
- The subagent tier depends on the strength of the model behind it.
Why it matters for companies
Agents with write access to databases, configuration, mailboxes or HR records will delete or overwrite things people later need, and they rarely protect against it themselves. Reversibility belongs in the harness, as a default snapshot before any state-changing tool call, not in the prompt. At about a cent per task the cost case is easy; the harder part is covering external systems that native harness backups ignore.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Coding agents, AI safety