When Can You Ship on Evals Alone? Trial-Level Surrogacy for Offline Evaluation of LLM Systems
A · Strong evidence Rigorous statistics paper by Spotify's experimentation lead Mårten Schultzberg that derives a ship/test/kill rule, validates it in simulation with released code and applies it to 183 real A/B tests; single author, preprint.
Summary based on the full text·Original: about 45 minutes to read
In short
A methods paper by Mårten Schultzberg of Spotify, posted on arXiv on 7 October 2026. It asks when a team can ship a change to an LLM product on offline evals alone, without an A/B test. The answer depends on whether changes in the eval have predicted changes in the online outcome across past comparable interventions, which the author calls trial-level surrogacy. Showing that an LLM judge agrees with humans does not establish this. A public test on 183 headline experiments found almost no link, so the proposed gate shipped nothing.
Key numbers
- +2.0pp to -0.25ppa real eval improvement turned negative by a judge only three points less sensitive on treated outputs
- 0.023pooled corrected R-squared between LLM-judge eval effects and click-through effects on 183 Upworthy interventions
- 134ppfitted shipping threshold on Upworthy, above the 100pp maximum, so every change goes to an A/B test
- 20% to 6%share of corpora where the gate's false-ship rate exceeds target, as corpus size grows from 100 to 400
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Four levels of eval validity: output-level (judge agrees with humans), cross-system (benchmark ranks models like humans), unit-level (Prentice surrogacy) and trial-level. Shipping decisions need the fourth; the paper proves unit-level and trial-level surrogacy are logically independent.
- Noise in historical eval and online deltas flattens the observed relationship; subtracting reported measurement variances recovers it, giving a closed-form ship/test/kill rule.
- A judge equally accurate on both arms only rescales effects. Arm-dependent accuracy can flip the sign: with sensitivity 0.90 and specificity 0.95, a judge 3 points less sensitive on treated outputs turns a +2.0pp latent gain into -0.25pp. More tasks make the wrong answer more precise.
- Once a gate ships on evals, outcomes in the ship and kill regions are no longer observed; randomized audit experiments are needed to detect drift.
- Simulation: shipping threshold falls from 6.28pp at R-squared 0.3 to 2.35pp at 0.9. With estimated parameters at nominal 0.05, realized false-ship rates average 0.033, 0.030 and 0.027 at corpus sizes 100, 200, 400, but 20%, 13% and 6% of corpora exceed the target.
- Upworthy: 183 independent interventions in 10 classes, GPT-4o-mini judge. Reliabilities 0.93 and 0.74, yet pooled corrected R-squared is 0.023; threshold 134pp, so every change goes to A/B testing.
How they got it
Formal model treating each product change as a "trial" with a true eval effect and true online effect, under a bivariate normal working model with moment correction. Simulations (1,000 replications per setting, code on GitHub) test recovery and gate calibration. The full pipeline was run on the exploratory split of the public Upworthy Research Archive after removing 1,050 tests from a known randomization failure.
Limits
- Rests on an exchangeable-corpus assumption and bivariate normality; heterogeneous classes and selected launch logs break the guarantee.
- The Upworthy eval measures informativeness while the outcome is clicks, a deliberate mismatch; per-class samples of 9-27 are too small for class-level conclusions.
- No Spotify production data is reported.
Why it matters for companies
Teams that ship prompt changes, model swaps or retrieval tweaks because "the eval went up" are assuming something they have rarely tested. The paper gives a checkable standard: keep paired logs of eval deltas and online results for comparable changes, and only skip A/B tests where that history supports it. LLM judges that score new and old outputs differently can reverse conclusions. After adopting any eval gate, keep a random share of changes under live testing, or drift goes unseen.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org