Measuring User Success in Randomized Search Experiments: Interpreting Good Abandonment in Generative AI Search
B · Useful, limited A three-page Microsoft production-metrics paper with out-of-sample accuracy on 6,904 labelled queries; useful method, but the headline sensitivity gain is a rounded ratio over confidential counts.
Summary based on the full text·Original: about 10 minutes to read
In short
A short paper (arXiv, 1 November 2026) by Minha Hwang and colleagues at Microsoft and Simplex AI. When a search page answers the question directly, a user who leaves without clicking may be satisfied ("good abandonment"). Click metrics cannot tell. The authors train a Session Success Rate (SSR) metric on human judgments of full session recordings and use it in A/B tests.
Key numbers
- 53.2%abandoned queries that human judges labelled successful
- 6.5%clicked queries labelled unsuccessful
- 0.75out-of-sample balanced accuracy of the query success model (0.68 within abandoned queries)
- about 2.5 timesbinary sensitivity of Session Success Rate vs Sessions per User across about 4,000 production experiments
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- In 6,904 human-labelled queries, 53.2% of abandonments were successful and 6.5% of clicks were unsuccessful. The sample oversamples difficult sessions, so these are not population rates.
- Out-of-sample balanced accuracy: 0.75 for queries, 0.68 within abandoned queries (the prior abandonment rule scored 0.45), 0.68 for sessions.
- In 12 validation experiments, SSR agreed in direction with Sessions per User in 7, was inconclusive in 4 and disagreed in 1; agreement was strongest for ranking changes.
- Across about 4,000 production experiments, SSR detected significant movement about 2.5 times as often as Sessions per User.
- In two sanitised examples SSR turned an apparent click-metric win and an apparent loss into mixed results, leading teams to iterate rather than ship or reject.
- SSR is not used when a treatment changes the telemetry it depends on; coverage and sample-ratio checks gate it.
How they got it
Stratified session sampling, at least two trained judges per item reviewing privacy-protected interaction logs and video, then interpretable models scored online from query events, viewport behaviour, result-page interactions and reformulations. Session SSR is the mean of query predictions.
Limits
- Judges infer success from observed behaviour; users were not asked.
- The 2.5x figure is a rounded ratio of significant-result rates, not an equal-power comparison; raw counts are confidential.
- The workflow benefits (lower traffic per test, faster decisions) are not causally isolated.
- Evidence is strongest for ranking experiments.
Why it matters for companies
Any product that answers questions on the page, including internal assistants and search, will under-count success if it measures clicks. A small set of human-labelled sessions can calibrate a behavioural success metric that is more sensitive in experiments. Keep it alongside existing metrics and gate it on telemetry coverage, as the authors do.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Measurement