FDE-Gym: The Last Mile of AI Delivery
13 model configurations from seven providers.
0.56best mean FDE-Gym score (Claude Opus 5.5) in the realistic setting, where 1 is the reference solution
Specification
Week 42 · 12-18 Oct
3 sources filed · EN 3
A A Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents
56% of first instructions in 100 SWE-chat trajectories omit requirements that later emerge; 52% of useful follow-up requirements were inferable from the project.
56%first instructions in 100 SWE-chat trajectories that omit requirements later needed
A FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests
18 configurations: opencode and openhands scaffolds with seven models (GPT-6-Astra, GLM-5.3, Qwen3.8-Max, Grok-4.6, DeepSeek-V4-Pro, Kimi-K3, Gemini-3.1-Pro), plus four native pairs (codex, deepseek-harness, kimi, gemini).
0.5510best mean delivery score across 18 agent configurations (codex + GPT-6-Astra), vs 1.0 for the engineer reference