---
id: "lib-2026-10-08-large-language-model-turnover-undermines"
type: "library"
title: "Large Language Model Turnover Undermines Screening for AI-Assisted Scientific Writing"
lang: "EN"
org: "arXiv"
source_type: "Paper"
date: "2026-10-08"
week: "2026-W41"
url: "https://nosov.ai/library/lib-2026-10-08-large-language-model-turnover-undermines/"
original_url: "https://arxiv.org/abs/2610.11599"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Large controlled study (4,000 PNAS abstracts, 23 model versions, 91,953 rewrites) with an audit of a commercial detector used by NeurIPS, from Tokyo Metropolitan University and Japan's National Institute of Informatics."
summary_basis: "full-text"
original_minutes: "40"
tags: ["ai technology", "ai detection", "measurement"]
---

# Large Language Model Turnover Undermines Screening for AI-Assisted Scientific Writing

Tier A (Strong evidence): Large controlled study (4,000 PNAS abstracts, 23 model versions, 91,953 rewrites) with an audit of a commercial detector used by NeurIPS, from Tokyo Metropolitan University and Japan's National Institute of Informatics.

Summary based on the full text.

## In short

Kazuki Nakajima and Takayuki Mizuno test how AI-writing detectors used by journals and conferences hold up as new language model versions keep arriving. They rewrote PNAS abstracts with 23 models from OpenAI, Meta and Alibaba released between June 2023 and August 2026. Detectors work almost perfectly on versions they were trained on, but can collapse at a new model generation, and also miss older versions. A commercial detector shows the same pattern.

## Key numbers

- 99% to 3.8%: share of rewrites caught just before and just after the sharpest model-generation boundary, at 1% false positives
- 11.9%: human-written abstracts falsely flagged when 23 version-specific detectors run side by side
- 79.8%: rewrites from Meta's Muse-Glimmer missed by the commercial detector Pangram
- 178: submissions a leading AI conference rejected without review in 2026 after detector screening (18.4% of a track)

## What it reports

- Detector trained on the version it screens: median 99.9% caught; trained on past and current versions: 99.3%.
- Trained only on earlier versions: detection stays above 85% for 18 of 20 versions but falls to 26.6% for GPT-5 and 3.8% for Muse-Glimmer.
- Frozen at GPT-4, a detector catches at most 0.6% of rewrites from GPT-5 onward; detectors trained on GPT-5.6 Luna, Sol and Terra catch at most 0.2% of GPT-4 rewrites.
- Vocabulary distance between versions predicts transfer failure (Spearman rho = -0.85, p < 0.001).
- Union of 23 detectors flags 11.9% of human abstracts; a single pooled detector flags 0.8% but misses 33.8% of Muse-Glimmer rewrites.
- Pangram 3.3.2 flagged 1 of 5,000 human abstracts (0.02%) but missed between 6.5% (GPT-5) and 79.8% (Muse-Glimmer) of rewrites; misses were 45.2% in Health Sciences versus 26.5% in Social Sciences.
- Four public detectors used off the shelf missed more than two-thirds of rewrites of every version.

## How they got it

From 34,925 PNAS articles published 2009-2018, 4,000 abstracts were rewritten by each model, giving 91,953 retained rewrites. Detectors were RoBERTa-base classifiers fine-tuned under four maintenance scenarios and calibrated to flag 1% of 30,925 never-rewritten abstracts, over 10 random splits. Results held with a word-frequency detector, other thresholds and two alternative prompts.

## Limits

- Only full rewrites of abstracts; lighter editing or multi-turn drafting was not studied.
- One family of trained detectors and one commercial detector at its default setting.
- Three vendors; Anthropic and Google models are not in the sample.

## Why it matters for companies

Companies are starting to screen job applications, supplier proposals and internal reports for AI-written text. This study shows a detector's accuracy is tied to the model versions it was checked against and can drop sharply when a new generation ships. A detector tuned to avoid false accusations will miss much AI text; one tuned to catch it will wrongly flag honest authors. Decisions about people should not rest on a detector score without re-validation after each major model release.

Original: https://arxiv.org/abs/2610.11599