When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories
B · Useful, limited Zhejiang University and Alibaba build a careful benchmark from real enterprise development records with a validated evaluator, but on 43 tasks from one company's codebase, so the results describe that environment more than coding agents in general.
Summary based on the full text·Original: about 45 minutes to read
In short
A preprint (arXiv, 19 October 2026) by Tianqi Liu and colleagues from Zhejiang University and Alibaba's Taobao & Tmall Group. PACE-Bench tests whether coding agents fulfil a full Product Requirement Document in a mature company codebase, with a median change of 1,465 lines across 15 files. Agents deliver most individual pieces but rarely the whole request. The best configuration fully satisfies 14% of tasks; incomplete implementation and missing integration are the main gaps.
Key numbers
- 43enterprise development tasks with 750 expert-audited Code Points (acceptance criteria)
- 52.1 and 78.1range of most functional scores across 11 model and scaffold configurations
- 14.0%best complete-delivery rate, reached by GPT-5.6-Sol (unified scaffold and Codex alike)
- 95.2%agreement of the automated evaluator with independent expert consensus
- 59.4 to 63.0mean score across successive harness releases, same model and task suite
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Scores across 11 configurations: GPT-5.6-Sol 78.1, GLM-5.2 68.8, Qwen3.7-Max 66.7, Opus-4.8 61.7, Gemini-3.1-Pro-Preview 61.3, MiniMax-M2.7 52.1; Qwen3.8-27B is an outlier at 12.3.
- Complete-delivery rates are low: 14.0% at best, 2.3% to 9.3% for most others.
- A native scaffold changes which requirements get completed together: QwenCode raised Qwen3.7-Max complete delivery from 7.0% to 11.6% while the mean score fell by about one point.
- Difficulty tracks how much of the requirement must be inferred rather than read: tasks with over 50% implicit criteria score 56.8 versus 75.0 for under 25%.
- Diagnosis from the benchmark fed three successive harness releases that raised the mean from 59.4 to 63.0 with the same model.
- The evaluator agrees with expert consensus 95.2% of the time.
How they got it
Tasks come from real user and agent sessions on Alibaba's internal platform, consolidated into standalone PRDs. Required behaviour is extracted separately from the PRD and from the accepted diff, reconciled, split into atomic Code Points, reviewed by domain experts and frozen before evaluation. Scoring runs structural checks, call-graph evidence gathering and an LLM judge per criterion with deterministic aggregation. Agent trajectories are linked to unmet criteria to diagnose retrieval, editing and verification failures.
Limits
- 43 tasks from one company's repositories and practices; generalisation to other codebases is untested.
- Tasks are drawn only from recorded agent sessions; tasks mined from merge requests might differ.
- Acceptance relies on an LLM judge, though validated against experts.
- Reviewers were company staff working on the task as part of their jobs.
Why it matters for companies
The gap between a decent average score and 14% full delivery is the review problem in one number: agent output looks mostly done, and someone has to find the missing integration. Teams writing requirements for agents should make implicit conditions explicit, since inferred requirements are where scores fall. The harness result is also practical: improving the tooling around a model, measured on a frozen internal task set, gave a measurable gain without changing models.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org