Iterate on an LLM, Serve on an SLM
A · Strong evidence A single-author CMU workshop paper (NeurIPS 2026 AABA4ET) on a real production workload, with a clear distillation method, a teacher self-agreement noise floor, confidence intervals and a blinded expert evaluation.
Summary based on the full text·Original: about 12 minutes to read
In short
A short paper by Keegan Wang (Carnegie Mellon University, with the company Phoebe), accepted to a NeurIPS 2026 workshop. The method: build an agent's behaviour quickly in prompts on a frontier model with live feedback, then, once it settles, fine-tune a small model on the recorded production decisions. On a home-care scheduling agent that makes one yes/no call per caregiver per shift, the small model matched past decisions more consistently than the original model, was rated more accurate by a blinded scheduler, and was faster and much cheaper.
Key numbers
- 0.913agreement of a LoRA-tuned Qwen3-8B with recorded production decisions
- 72.7%how often the GPT-5.4-nano teacher agreed with itself when re-run on its own prompts
- 48 times vs. 29blinded scheduler siding with the small model vs the production decision on 77 disagreements
- 2.5×lower median latency for the tuned 8B model than the teacher
- 11.6krecorded production decisions used for fine-tuning, across 17 organisations
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Zero-shot small models failed: five off-the-shelf SLMs missed the quality bar, and base Qwen3-8B produced malformed tool calls on 23% of cases.
- Prompt iteration moved fast: one early version approved 2 of 49 candidates because soft profile tags acted as vetoes; a one-line prompt change raised that to 40.
- Training set: 11,621 train and 1,604 validation examples from 549 production batches, replayed byte-exactly through the live prompt builder; rejects without a hard-conflict reason were filtered out.
- Tuned Qwen3-8B: 0.913 agreement (95% CI 0.89-0.93), no malformed outputs, false-reject rate 0.011. Tuned 4B: 0.863 but 12% malformed outputs, so not usable.
- Teacher self-agreement 72.7% (CI 0.65-0.79), treated as the noise floor.
- Blinded human evaluation of 192 cases: on 77 disagreements the scheduler sided with the student 48-29 (p = 0.040). Against human labels, the student is more accurate (0.888 vs 0.866) but slightly more permissive (false-approve 0.273 vs 0.199).
- Cost per 1k decisions: teacher about $1.9-$7.7 depending on cache, tuned 8B about $0.19-$0.38 at full GPU utilisation.
How they got it
Every production decision is stored; an extractor replays each batch through the same pipeline to produce identical contexts paired with the recorded tool call. LoRA fine-tuning of Qwen3-4B and 8B (about 1B training tokens, 64k context). Evaluation replays held-out prompts at temperature 0; human labels are post-stratified to the 880-case validation set.
Limits
- The small model was not yet serving live traffic; all results are offline replays.
- One professional scheduler provided ground truth; correlation within organisations and batches is not modelled.
- One subtask, one domain, one teacher.
- SLM costs are estimates at full utilisation, not billed figures.
Why it matters for companies
For high-volume, narrow agent decisions, paying frontier prices forever is avoidable: iterate in prompts first, log every decision, then distil once behaviour is stable. Logging production decisions in replayable form is the prerequisite most teams skip. The teacher's 72.7% self-agreement is a warning on its own: re-run any LLM decision step on its own inputs before trusting it as a benchmark. Human review of disagreements, and of error direction, should gate any switch.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Cost of AI