---
id: "lib-2026-11-02-the-shelf-life-of-llm-findings"
type: "library"
title: "The shelf life of LLM findings"
lang: "EN"
org: "Zou group (zou-group.github.io)"
source_type: "Other"
date: "2026-11-02"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-11-02-the-shelf-life-of-llm-findings/"
original_url: "https://zou-group.github.io/llm-claims-shelf-life/"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "James Zou's Stanford group re-tests 50 widely cited LLM findings on 20 models with reproduction checks, a stated effect-size scale, fitted trends and bootstrap intervals; a web report, not yet peer reviewed."
summary_basis: "full-text"
original_minutes: "30"
tags: ["ai technology", "research reliability", "evals"]
frames: ["verification-capacity"]
---

# The shelf life of LLM findings

Tier A (Strong evidence): James Zou's Stanford group re-tests 50 widely cited LLM findings on 20 models with reproduction checks, a stated effect-size scale, fitted trends and bootstrap intervals; a web report, not yet peer reviewed.

Summary based on the full text.

## In short

A 2 November 2026 web report by Owen Queen and James Zou (Stanford). They re-ran 50 widely cited published findings about LLMs, from fake citations and sycophancy to long-context and bias, on 20 models released July 2024 to September 2026 by eight developers. Of the 50 claims, 44% have expired, meaning the effect has fallen to a fifth of the original or less. Capability failures fade fastest; preferences such as flattering users or judges favouring their own answers last or grow.

## Key numbers

- 44%: share of the 50 tested claims expired in current models (34% weakened, 16% persist, 6% strengthened)
- 16% and 39%: estimated share of claims expired one year and two years after publication
- 14.7 months: median time from publication to expiration for the 22 expired claims (mean 15.7)
- 55% to 1% and 0%: fabricated references: GPT-3.5 in the 2023 paper vs Claude Sonnet 5.5 and GPT-6.1 Sol today
- 69%: references fabricated by Qwen3.7-Flash in the same test, an exception to the expired verdict

## What it reports

- Verdicts at 30 September 2026: 22 claims expired (44%), 17 weakened (34%), 8 persist (16%), 3 strengthened (6%).
- Kaplan-Meier estimates: 4% of claims expired six months after publication, 16% after one year, 39% after two years.
- For the 22 expired claims, time to expiration averaged 15.7 months (median 14.7), ranging from about one month to roughly three years.
- By topic: five of nine reasoning claims and six of eleven sycophancy-and-deception claims expired; no reasoning claim persisted.
- Strengthened: judge self-preference (2.36x the reference effect), praise that follows the user's preference (1.85x), nonuniform random numbers (1.70x).
- From GPT-4o mini (July 2024) to GPT-6.1 Sol (September 2026), claims in the expiration zone rose from 5 to 30 of 50. Among July-September 2026 models, expired counts range from 11 (DeepSeek V4 Pro) to 30 (GPT-6.1 Sol).
- Fake citations: reproduction on GPT-3.5 Turbo gave 59% of 210 references fabricated (paper: 55%). Claude Sonnet 5.5 fabricated 1%, GPT-6.1 Sol 0%, Qwen3.7-Flash 69%.
- The authors question ICLR and ICML 2026 desk-rejection policies that use hallucinated references as a marker of AI-written papers.

## How they got it

Each claim's key metric was rescaled to a relative effect (1 = as in the paper, 0 = gone). The team first reproduced the paper's result: 43 of 48 checked claims reproduced within tolerance and 5 showed stronger effects; 25 used the exact original model, the rest substitutes. They then fitted constant, linear, exponential or logistic curves by model release date through the reproduction anchor, with outlier handling and 500 bootstrap resamples. Claude Code and Codex built the analysis pipelines. Models ran on default API settings.

## Limits

- Only 50 claims, so aggregate shares have wide intervals; each claim uses one minimal design.
- Four 2024 models anchor the early cohorts; retired original models needed substitutes, which are weaker reproductions.
- Default reasoning settings differ across models, so some results reflect configurations rather than models.
- Panel-wide verdicts hide model exceptions, and some failures may have been patched specifically in post-training. Support for the blackmail claim's expiry is weak by the authors' own test.

## Why it matters for companies

Internal risk policies, training material and vendor assessments often rest on LLM failure modes measured on 2023 models. Many of those no longer hold, while sycophancy and self-preference in model judges persist. Any control that depends on a known failure, such as screening for fake citations or using one model to grade another, should be re-tested on the models actually in use and dated. The model-level spread also means a panel verdict says little about the specific model a team deploys.

Original: https://zou-group.github.io/llm-claims-shelf-life/