Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash
B · Useful, limited A first-hand DoorDash engineering report with concrete success, time and cost figures per task, but self-reported by the builders on 50 flags processed once, with the manual baseline an internal estimate.
Summary based on the full text·Original: about 15 minutes to read
In short
An industrial experience paper (arXiv, 22 October 2026) by Atharv Chandratre and Jai Datar of DoorDash. Feature flags left in code after rollout become technical debt and an operational risk. Rule-based cleanup tools could not handle DoorDash's dependency-injected flag pattern, so the team built a multi-agent system: a Claude Sonnet 4.6 orchestrator plans each cleanup and up to four Claude Opus 4.6 agents edit code in parallel. Nine in ten cleanups produced usable pull requests at a few dollars each.
Key numbers
- 1,659stale feature flags among more than 56,000 across 623 DoorDash repositories (May 2026)
- 45 (90%)of 50 stale flags for which the system produced engineer-approved removal pull requests
- 44 of 50pull requests that passed CI on the first attempt; all six failures were test-coverage gaps
- $4.79average LLM API cost per flag cleanup (median $3.49)
- 13.8 minutesaverage time per cleanup, against an internal estimate of one to two hours by hand
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- About 2,300 new flags are created each month, so the stale backlog grows faster than teams clear it.
- The orchestrator reads Jira tickets and queries the experimentation platform via MCP for the live rollout state, so the correct value is hardcoded; an engineer confirms target values before edits start.
- Removal agents run in isolated git worktrees with a one-hour timeout, must pass build, tests, 95% patch coverage and the Detekt linter before opening a pull request.
- Of 50 flags: 31 (62%) accepted first time, 14 (28%) after minor revision, 5 (10%) failed and needed an engineer.
- By complexity: simple flags 100% first-shot, medium 94% cleanup rate, complex 85%. Failures came from deep call chains with parameters threaded across interfaces, not from flag type or file count.
- No bugs or regressions were observed in any of the 50 pull requests; the limit was completeness, not correctness.
- Time and cost rose with complexity, from $2.69 and 7.5 minutes for simple flags to $6.20 and 17.7 minutes for complex ones, with outliers at 40 minutes and $19.
How they got it
The authors analysed the 50 most recent stale-flag removals performed by the system (as of 14 April 2026) in Kotlin repositories, classified post hoc as simple (6), medium (18) or complex (26). Each pull request was judged on CI, code correctness and developer review. Engineers' informal feedback is quoted.
Limits
- Each flag was processed once; non-deterministic runs could give different outcomes.
- Complexity labels were assigned after the fact by the authors.
- The one-to-two-hour manual baseline is an internal estimate, not measured.
- One company, one flag access pattern, Kotlin only.
Why it matters for companies
This is one of the few agent case studies with cost and time per unit of work, and the unit is unglamorous maintenance that every engineering organisation postpones. The design choices transfer: pull the true production state from the system of record rather than inferring it from code, put a human checkpoint before edits rather than after, isolate parallel agents, and refuse to open a pull request until deterministic gates pass. Splitting a cheaper planning model from a stronger editing model kept costs down.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Coding agents