---
id: "lib-2026-11-02-can-llm-agents-be-forward-deployed-engineers"
type: "library"
title: "Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL"
lang: "EN"
org: "Oracle"
source_type: "Other"
date: "2026-11-02"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-11-02-can-llm-agents-be-forward-deployed-engineers/"
original_url: "https://arxiv.org/abs/2611.03551"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "Oracle AI researchers test 15 models as forward-deployed engineers on a simulated customer grounded in one real text-to-SQL engagement; clear method, but one case and only eight delivery runs."
summary_basis: "full-text"
original_minutes: "60"
tags: ["ai technology", "enablement", "requirements"]
---

# Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL

Tier B (Useful, limited): Oracle AI researchers test 15 models as forward-deployed engineers on a simulated customer grounded in one real text-to-SQL engagement; clear method, but one case and only eight delivery runs.

Summary based on the full text.

## In short

An arXiv paper (2 November 2026) by Marianne Menglin Liu and colleagues at Oracle AI. A simulated customer, built from a completed engagement in which a human forward-deployed engineer delivered a text-to-SQL model, opens with an ambiguous request. An agent must discover acceptance criteria, data permissions and hardware limits, then plan and run a post-training recipe. Strong models discover most requirements, but few delivery runs meet every target, and none beats the human.

## Key numbers

- over 80%: withheld requirements the strongest models discover (open-weight models 40-50%)
- 25%: delivery runs (2 of 8) that pass all acceptance criteria
- 79.90%: Spider accuracy of the human FDE's shipped system, against 76.11% for the best agent run
- 40%: withheld requirements missing or misrepresented after discovery

## What it reports

- Across 15 models, the strongest discover over 80% of withheld requirements; open-weight models 40-50%.
- Discovery depends on targeted clarification, not on asking more questions.
- Only two of eight delivery runs (decoupled GPT-5.6 Sol at 76.11% Spider, autonomous GPT-6.1 Sol at 71.57%) meet the accuracy, production-compilation and latency targets together.
- The human FDE's shipped system reached 79.90% Spider accuracy with six production failures; the best agent run is about four points behind.
- Production compilation is the most often missed target: four of eight runs exceed the failure limit.
- Four bottlenecks: incomplete discovery (40% of requirements missing or misrepresented), unreliable execution, underused compute budgets and missing systematic comparisons, and benchmark accuracy that does not mean production readiness.

## How they got it

Customer knowledge annotated from the real engagement's requirement discussions, experiment log and shipped model; discovery runs across 15 models and multiple ambiguity levels; delivery across eight coding-agent configurations, decoupled or autonomous, with fixed budget and evaluator.

## Limits

- One engagement, one task type, and the "customer" is itself a model.
- Eight delivery runs is a small sample for the 25% figure.
- Run by a vendor that sells enterprise AI deployment.

## Why it matters for companies

Agents can draft the requirements interview, but a person should review what they discovered before delivery starts. Passing a public benchmark said little about passing the production checks that mattered here. The authors' practical asks are cheap: write down the experiment environment (hardware, budget scope, production checks) and monitor the agent's plan while it runs.

Original: https://arxiv.org/abs/2611.03551