---
id: "lib-2026-10-01-right-order-wrong-scale-auditing-llm-judges-for"
type: "library"
title: "Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement"
lang: "EN"
org: "MIT FutureTech"
source_type: "Paper"
date: "2026-10-01"
week: "2026-W41"
url: "https://nosov.ai/library/lib-2026-10-01-right-order-wrong-scale-auditing-llm-judges-for/"
original_url: "https://arxiv.org/abs/2610.02492"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "MIT FutureTech preprint (Harry Lyu, Neil Thompson) auditing 33 LLM-judge configurations against 45,796 worker ratings, with a prespecified replication on held-out data."
summary_basis: "full-text"
original_minutes: "60"
tags: ["future of work", "ai technology", "task exposure", "llm judges"]
---

# Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement

Tier A (Strong evidence): MIT FutureTech preprint (Harry Lyu, Neil Thompson) auditing 33 LLM-judge configurations against 45,796 worker ratings, with a prespecified replication on held-out data.

Summary based on the full text.

## In short

A preprint posted to arXiv on 1 October 2026 by Harry Lyu and Neil Thompson of MIT. It tests whether LLM judges, now widely used to grade AI outputs on workplace tasks, give the same answers as workers with experience in those occupations. Most judges rank responses in roughly the right order, but they place scores on very different scales. As a result the headline number such studies report, the share of AI outputs good enough to use, depends heavily on which judge is used.

## Key numbers

- 3.0%-97.9%: share of responses LLM judges rated acceptable, versus 61.1% for occupation-matched workers
- 68.4%: occupations where the harshest judge (Opus) disagreed with the worker majority verdict
- 8.5%: maximum share of individual worker-rating variance explained by calibrated judge scores

## What it reports

- O*NET-Bench is built from 45,796 worker ratings of AI responses to O*NET-derived tasks; the test split has 4,501 ratings.
- 33 judge configurations across six model families were audited; 25 reach tie-aware pair accuracy of at least 0.60, but a simple TF-IDF baseline nearly matches the best judge.
- Workers rated 61.1% of test responses acceptable; judges ranged from 3.0% (Opus) to 97.9% (near-constant Llama), median 43.9%.
- Workers rated a majority acceptable in 68.7% of 307 occupations; Opus did so in 0.3% and disagreed with workers in 68.4% of occupations.
- Mean-bias intervals exclude zero for all 33 configurations.
- In one fine-tuned lineage, switching to a few-shot/listwise protocol raised response-level Spearman correlation from 0.214 to 0.263 but cut occupation-level correlation from 0.254 to 0.087; this reversal replicated on held-out data.
- Calibration removed mean bias (to at most 0.003), but the best cross-validated R2 for individual ratings was 0.085.
- Using judge predictions to stretch a small labelled sample gave at most small precision gains.

## How they got it

The ratings come from an existing survey by Mertens et al. (2026): US Prolific workers with self-reported occupational experience rated AI responses on a nine-point scale, with 7 or above counted as acceptable. Train, validation and test splits are disjoint by task and worker. The authors compared stored predictions from 33 judges and six statistical baselines across response, task, occupation and job-family levels.

## Limits

- English-language text tasks for US occupations; the sample is not employment-weighted.
- Each response has one worker rating, so disagreements cannot be attributed to the worker or the judge.
- The strong reversal is shown for one fine-tuned lineage only.

## Why it matters for companies

Vendors and internal teams increasingly use one model to grade another before deciding where AI is "good enough" for a job. This study shows that a judge can agree with experts on which output is better while disagreeing completely on whether any output is usable. Before relying on automated grading to make automation decisions, a company should check the judge against its own staff's ratings on the exact measure it plans to report.

Original: https://arxiv.org/abs/2610.02492