---
id: "lib-2026-10-10-iterate-on-an-llm-serve-on-an-slm"
type: "library"
title: "Iterate on an LLM, Serve on an SLM"
lang: "EN"
org: "arXiv"
source_type: "Paper"
date: "2026-10-10"
week: "2026-W42"
url: "https://nosov.ai/library/lib-2026-10-10-iterate-on-an-llm-serve-on-an-slm/"
original_url: "https://arxiv.org/abs/2610.14576"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "A single-author CMU workshop paper (NeurIPS 2026 AABA4ET) on a real production workload, with a clear distillation method, a teacher self-agreement noise floor, confidence intervals and a blinded expert evaluation."
summary_basis: "full-text"
original_minutes: "12"
tags: ["ai technology", "distillation", "production case", "agent cost"]
---

# Iterate on an LLM, Serve on an SLM

Tier A (Strong evidence): A single-author CMU workshop paper (NeurIPS 2026 AABA4ET) on a real production workload, with a clear distillation method, a teacher self-agreement noise floor, confidence intervals and a blinded expert evaluation.

Summary based on the full text.

## In short

A short paper by Keegan Wang (Carnegie Mellon University, with the company Phoebe), accepted to a NeurIPS 2026 workshop. The method: build an agent's behaviour quickly in prompts on a frontier model with live feedback, then, once it settles, fine-tune a small model on the recorded production decisions. On a home-care scheduling agent that makes one yes/no call per caregiver per shift, the small model matched past decisions more consistently than the original model, was rated more accurate by a blinded scheduler, and was faster and much cheaper.

## Key numbers

- 0.913: agreement of a LoRA-tuned Qwen3-8B with recorded production decisions
- 72.7%: how often the GPT-5.4-nano teacher agreed with itself when re-run on its own prompts
- 48 times vs. 29: blinded scheduler siding with the small model vs the production decision on 77 disagreements
- 2.5×: lower median latency for the tuned 8B model than the teacher
- 11.6k: recorded production decisions used for fine-tuning, across 17 organisations

## What it reports

- Zero-shot small models failed: five off-the-shelf SLMs missed the quality bar, and base Qwen3-8B produced malformed tool calls on 23% of cases.
- Prompt iteration moved fast: one early version approved 2 of 49 candidates because soft profile tags acted as vetoes; a one-line prompt change raised that to 40.
- Training set: 11,621 train and 1,604 validation examples from 549 production batches, replayed byte-exactly through the live prompt builder; rejects without a hard-conflict reason were filtered out.
- Tuned Qwen3-8B: 0.913 agreement (95% CI 0.89-0.93), no malformed outputs, false-reject rate 0.011. Tuned 4B: 0.863 but 12% malformed outputs, so not usable.
- Teacher self-agreement 72.7% (CI 0.65-0.79), treated as the noise floor.
- Blinded human evaluation of 192 cases: on 77 disagreements the scheduler sided with the student 48-29 (p = 0.040). Against human labels, the student is more accurate (0.888 vs 0.866) but slightly more permissive (false-approve 0.273 vs 0.199).
- Cost per 1k decisions: teacher about $1.9-$7.7 depending on cache, tuned 8B about $0.19-$0.38 at full GPU utilisation.

## How they got it

Every production decision is stored; an extractor replays each batch through the same pipeline to produce identical contexts paired with the recorded tool call. LoRA fine-tuning of Qwen3-4B and 8B (about 1B training tokens, 64k context). Evaluation replays held-out prompts at temperature 0; human labels are post-stratified to the 880-case validation set.

## Limits

- The small model was not yet serving live traffic; all results are offline replays.
- One professional scheduler provided ground truth; correlation within organisations and batches is not modelled.
- One subtask, one domain, one teacher.
- SLM costs are estimates at full utilisation, not billed figures.

## Why it matters for companies

For high-volume, narrow agent decisions, paying frontier prices forever is avoidable: iterate in prompts first, log every decision, then distil once behaviour is stable. Logging production decisions in replayable form is the prerequisite most teams skip. The teacher's 72.7% self-agreement is a warning on its own: re-run any LLM decision step on its own inputs before trusting it as a benchmark. Human review of disagreements, and of error direction, should gate any switch.

Original: https://arxiv.org/abs/2610.14576