Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL
B · Useful, limited Oracle AI researchers test 15 models as forward-deployed engineers on a simulated customer grounded in one real text-to-SQL engagement; clear method, but one case and only eight delivery runs.
Summary based on the full text·Original: about 60 minutes to read
In short
An arXiv paper (2 November 2026) by Marianne Menglin Liu and colleagues at Oracle AI. A simulated customer, built from a completed engagement in which a human forward-deployed engineer delivered a text-to-SQL model, opens with an ambiguous request. An agent must discover acceptance criteria, data permissions and hardware limits, then plan and run a post-training recipe. Strong models discover most requirements, but few delivery runs meet every target, and none beats the human.
Key numbers
- over 80%withheld requirements the strongest models discover (open-weight models 40-50%)
- 25%delivery runs (2 of 8) that pass all acceptance criteria
- 79.90%Spider accuracy of the human FDE's shipped system, against 76.11% for the best agent run
- 40%withheld requirements missing or misrepresented after discovery
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Across 15 models, the strongest discover over 80% of withheld requirements; open-weight models 40-50%.
- Discovery depends on targeted clarification, not on asking more questions.
- Only two of eight delivery runs (decoupled GPT-5.6 Sol at 76.11% Spider, autonomous GPT-6.1 Sol at 71.57%) meet the accuracy, production-compilation and latency targets together.
- The human FDE's shipped system reached 79.90% Spider accuracy with six production failures; the best agent run is about four points behind.
- Production compilation is the most often missed target: four of eight runs exceed the failure limit.
- Four bottlenecks: incomplete discovery (40% of requirements missing or misrepresented), unreliable execution, underused compute budgets and missing systematic comparisons, and benchmark accuracy that does not mean production readiness.
How they got it
Customer knowledge annotated from the real engagement's requirement discussions, experiment log and shipped model; discovery runs across 15 models and multiple ambiguity levels; delivery across eight coding-agent configurations, decoupled or autonomous, with fixed budget and evaluator.
Limits
- One engagement, one task type, and the "customer" is itself a model.
- Eight delivery runs is a small sample for the 25% figure.
- Run by a vendor that sells enterprise AI deployment.
Why it matters for companies
Agents can draft the requirements interview, but a person should review what they discovered before delivery starts. Passing a public benchmark said little about passing the production checks that mattered here. The authors' practical asks are cheap: write down the experiment environment (hardware, budget scope, production checks) and monitor the agent's plan while it runs.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- AI enablement