---
id: "lib-2026-11-01-measuring-user-success-in-randomized-search"
type: "library"
title: "Measuring User Success in Randomized Search Experiments: Interpreting Good Abandonment in Generative AI Search"
lang: "EN"
org: "Microsoft"
source_type: "Paper"
date: "2026-11-01"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-11-01-measuring-user-success-in-randomized-search/"
original_url: "https://arxiv.org/abs/2611.01582"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "A three-page Microsoft production-metrics paper with out-of-sample accuracy on 6,904 labelled queries; useful method, but the headline sensitivity gain is a rounded ratio over confidential counts."
summary_basis: "full-text"
original_minutes: "10"
tags: ["ai technology", "measurement", "search"]
---

# Measuring User Success in Randomized Search Experiments: Interpreting Good Abandonment in Generative AI Search

Tier B (Useful, limited): A three-page Microsoft production-metrics paper with out-of-sample accuracy on 6,904 labelled queries; useful method, but the headline sensitivity gain is a rounded ratio over confidential counts.

Summary based on the full text.

## In short

A short paper (arXiv, 1 November 2026) by Minha Hwang and colleagues at Microsoft and Simplex AI. When a search page answers the question directly, a user who leaves without clicking may be satisfied ("good abandonment"). Click metrics cannot tell. The authors train a Session Success Rate (SSR) metric on human judgments of full session recordings and use it in A/B tests.

## Key numbers

- 53.2%: abandoned queries that human judges labelled successful
- 6.5%: clicked queries labelled unsuccessful
- 0.75: out-of-sample balanced accuracy of the query success model (0.68 within abandoned queries)
- about 2.5 times: binary sensitivity of Session Success Rate vs Sessions per User across about 4,000 production experiments

## What it reports

- In 6,904 human-labelled queries, 53.2% of abandonments were successful and 6.5% of clicks were unsuccessful. The sample oversamples difficult sessions, so these are not population rates.
- Out-of-sample balanced accuracy: 0.75 for queries, 0.68 within abandoned queries (the prior abandonment rule scored 0.45), 0.68 for sessions.
- In 12 validation experiments, SSR agreed in direction with Sessions per User in 7, was inconclusive in 4 and disagreed in 1; agreement was strongest for ranking changes.
- Across about 4,000 production experiments, SSR detected significant movement about 2.5 times as often as Sessions per User.
- In two sanitised examples SSR turned an apparent click-metric win and an apparent loss into mixed results, leading teams to iterate rather than ship or reject.
- SSR is not used when a treatment changes the telemetry it depends on; coverage and sample-ratio checks gate it.

## How they got it

Stratified session sampling, at least two trained judges per item reviewing privacy-protected interaction logs and video, then interpretable models scored online from query events, viewport behaviour, result-page interactions and reformulations. Session SSR is the mean of query predictions.

## Limits

- Judges infer success from observed behaviour; users were not asked.
- The 2.5x figure is a rounded ratio of significant-result rates, not an equal-power comparison; raw counts are confidential.
- The workflow benefits (lower traffic per test, faster decisions) are not causally isolated.
- Evidence is strongest for ranking experiments.

## Why it matters for companies

Any product that answers questions on the page, including internal assistants and search, will under-count success if it measures clicks. A small set of human-labelled sessions can calibrate a behavioural success metric that is more sensitive in experiments. Keep it alongside existing metrics and gate it on telemetry coverage, as the authors do.

Original: https://arxiv.org/abs/2611.01582