FDE-Gym: The Last Mile of AI Delivery
A · Strong evidence A multi-institution benchmark (DeepLethe, Tsinghua, SJTU and others) built from 20 real delivery projects, scored on the customer's own business metric across 13 model configurations, with limits stated.
Summary based on the full text·Original: about 30 minutes to read
In short
FDE-Gym, an arXiv preprint (15 October 2026), tests agents as forward-deployed engineers: people sent by a vendor to deliver AI inside a customer's systems. Its 95 tasks come from 20 real delivery projects in 18 sectors, with scripted stakeholders and production-like systems. Scores measure improvement on the customer's own metric, with a cost for each stakeholder contact. The best configuration scores 0.56 and fully resolves 14 tasks.
Key numbers
- 0.56best mean FDE-Gym score (Claude Opus 5.5) in the realistic setting, where 1 is the reference solution
- 14 of 95tasks the best configuration fully resolves
- 0.73-0.82 to 0.39-0.62leading models' scores when constraints are given vs when they must discover them
- 79 of 95tasks where Claude Haiku 4.5 leaves the customer below the do-nothing baseline
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 13 model configurations from seven providers. Claude Opus 5.5 leads at 0.56; five configurations resolve no task. Picking the best model per task gives only 0.70 and 30 resolved.
- Literal compliance with the sponsor's request scores at most 0.2; a do-nothing baseline scores 0.
- With rules and constraints provided, the top three models score 0.73-0.82; when they must discover them, 0.39-0.62.
- Leading models get 72-85% of the key facts from stakeholders but deliver only 0.60-0.70 of the reference improvement; contact costs cut a further 0.14-0.22.
- On the 23 tasks where the best model collected every key fact, it reached 0.8 business uplift on only 10.
- Weak models can do harm: Claude Haiku 4.5 is below baseline on 79 of 95 tasks, Mistral Small 4 on 51.
How they got it
Industry experts adapted real projects into simulated customers with fictional companies and changed facts, wrote reference solutions and scoring anchors, and validated graders. Each scenario runs at four instruction levels, from fully specified to the sponsor's original request, with the environment fixed. All models use one minimal agent framework.
Limits
- Stakeholders are scripted; real people are less predictable, and contact costs are modelled, not measured.
- No human baseline yet.
- One framework; stronger harnesses may change scores and ranking.
- Weighted toward regulated operations; training-data contamination cannot be ruled out.
Why it matters for companies
The benchmark measures what customers pay for: whether the business metric moves. It shows that agents can follow a brief and still fail because the brief targets the wrong problem, and that weaker models can leave systems worse than before. Firms buying or building AI delivery should judge projects on the operating metric, keep experienced people on requirement discovery, and treat a small model's autonomous changes to live systems as a risk.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org