FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests
A · Strong evidence Baidu AI Cloud's FM Agent team and university partners built a benchmark from 49 real customer projects with customer-accepted engineer solutions as references, and report repeated runs for 18 agent configurations.
Summary based on the full text·Original: about 40 minutes to read
In short
An arXiv paper (10 October 2026) by Huaiming Li, Can Huang and colleagues, led by Baidu AI Cloud. FDE-Bench tests whether agents can do the work of a forward-deployed engineer: take a vague business request, ask the customer the right questions, and iterate until the solution meets acceptance criteria. Cases come from real projects in scheduling, routing, packing, inventory and forecasting. The best agent reaches about 55% of the engineer's quality on average and matches it in about 6% of runs.
Key numbers
- 0.5510best mean delivery score across 18 agent configurations (codex + GPT-6-Astra), vs 1.0 for the engineer reference
- 6.12%highest share of runs reaching the customer-accepted reference
- 49real customer cases (29 combinatorial optimisation, 20 machine learning)
- 8 of 49cases where any run of any system reached reference level
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 18 configurations: opencode and openhands scaffolds with seven models (GPT-6-Astra, GLM-5.3, Qwen3.8-Max, Grok-4.6, DeepSeek-V4-Pro, Kimi-K3, Gemini-3.1-Pro), plus four native pairs (codex, deepseek-harness, kimi, gemini).
- Highest mean quality 0.5510 (codex + GPT-6-Astra); highest delivery success rate 6.12%.
- Only 8 of 49 cases had any reference-level run; in 23 the best system averaged below 0.8. Four systems, including all three Gemini-3.1-Pro systems, never reached the reference in 147 runs; no machine-learning run reached it.
- Reliability across repeats is weak: 85.7-95.9% of cases have at least one valid run in three, but all-three-valid trails by 8.2 to 38.8 points.
- Agents whose questions cover more of the annotated requirements deliver better results. Giving agents full business information helps but still leaves every compared system below the reference.
How they got it
Each case pairs the original request and de-identified data, requirements annotated by the engineer who delivered the project, and the customer-accepted solution. Agents query an LLM-simulated customer (up to 30 clarification rounds, 12-hour budget, three runs per case). Case-specific evaluators score final artifacts against acceptance criteria, normalised to the engineer reference. Information ablations compare no answers, required clarification and full information.
Limits
- The simulated customer never changes its requirements; real customers do.
- 49 cases in two task families, from one vendor's customer base.
- Anthropic models are not among those tested.
- The reference is one engineer's accepted solution, not a proven optimum.
Why it matters for companies
Agents remain far from replacing the people who translate vague operational requests into working optimisation or forecasting solutions. The gap is in two places: not asking about unstated business rules, and not iterating to the acceptance bar. Firms using agents for such work should keep a domain expert in the requirements loop and judge results against an agreed acceptance test, not the agent's own report. The weak repeatability means a single good run says little about the next one.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org