---
id: "lib-2026-10-07-comprehension-audits-to-mitigate-risks-from"
type: "library"
title: "Comprehension Audits to Mitigate Risks from Automated AI Research"
lang: "EN"
org: "arXiv (Hadfield et al.)"
source_type: "Paper"
date: "2026-10-07"
week: "2026-W41"
url: "https://nosov.ai/library/lib-2026-10-07-comprehension-audits-to-mitigate-risks-from/"
original_url: "https://arxiv.org/abs/2610.10064"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Governance proposal co-authored by Gillian Hadfield (Johns Hopkins), backed by an original measurement of review activity across 224 GitHub organizations and 2,135 AI repositories; strong evidence, but an arXiv position paper rather than a field-shaping report."
summary_basis: "full-text"
original_minutes: "75"
tags: ["ai technology", "governance", "code review", "ai safety"]
frames: ["verification-capacity"]
---

# Comprehension Audits to Mitigate Risks from Automated AI Research

Tier A (Strong evidence): Governance proposal co-authored by Gillian Hadfield (Johns Hopkins), backed by an original measurement of review activity across 224 GitHub organizations and 2,135 AI repositories; strong evidence, but an arXiv position paper rather than a field-shaping report.

Summary based on the full text.

## In short

A paper by Ronald J. Bodkin (independent), Bahrad A. Sokhansanj (Institute for Law and AI) and Gillian Hadfield (Johns Hopkins University), posted to arXiv on 7 October 2026. It proposes "comprehension audits": an independent auditor picks a piece of R&D work and the people responsible must explain it without AI help. A failing grade pauses development and any release that depends on it until fixed. The authors support the case with a measurement of open-source AI projects showing code volume rising while human review attention per line falls.

## Key numbers

- 183%: median increase in code submitted by regular contributors to open-source AI repos, Q1 2024 to Q2 2026
- 36%: human inline comments per kLOC on AI-assisted PRs by June 2026, as a share of the Q1 2024 rate
- <30%: human review rate for PRs from high-volume fleet accounts, against about 80% for non-AI PRs
- 17 audits: quarterly sample that detects at least one failure with 99% probability at a 25% failure rate

## What it reports

- Anthropic, OpenAI and Google report AI writing 75% or more of their new code; Anthropic reported that as of August 2026 Claude "leads" 26% of its R&D work.
- Traceable AI use appears in 17% of all pull requests by the end of the period, which the authors call a floor.
- Regular contributors' code output rose by a median of 183% from Q1 2024 to Q2 2026; contributors with evidence of AI use rose by a median of 202%.
- Review rate for non-AI PRs held at about 80%. AI PRs ran a median 8.1 percentage points lower, a gap that disappears with contributor weighting.
- Fleet accounts (42 accounts with 300+ merged PRs in a quarter) went from 4.7% of merged PRs in Q1 2024 to 15.2% in Q2 2026. Their review rate fell below 30%, from about 50% in Q1 2024.
- Human inline comments per kLOC fell roughly 50%: to 36% of the Q1 2024 base on AI PRs and 46% on non-AI PRs by June 2026.
- Only 3-7% of fleet PRs had any inline commentary, against 20-26% of other PRs.
- Sampling table: at a 10% failure rate, 22 audits give 90% detection; at 25%, 17 audits give 99%.

## How they got it

The empirical part analyses public GitHub PRs from 224 organizations and 2,135 repositories selected from five third-party lists of AI developers, Q1 2024 to Q2 2026, with an 8-week cutoff for merges and reviews. AI use is detected from traceable markers such as agent branch prefixes and co-author tags. The audit process itself (selection, no-AI examination, grading, appeal, remediation, escalation) is specified in detail, with worked hypothetical cases.

## Limits

- The empirical data covers open-source projects; the audits are proposed for closed frontier labs, where no comparable data is available.
- AI detection relies on visible traces, so AI use is undercounted, and fewer comments per line is a proxy for attention, not a direct measure of understanding.
- The audit regime is a proposal; no lab has piloted it, and the audit cases in the appendix are illustrative.

## Why it matters for companies

The same pattern applies to any team that ships AI-written code, reports or analyses: output volume grows faster than the capacity to read it, and sign-off stops meaning understanding. The paper offers a cheap control: sample a small number of deliverables each quarter and ask the owners to explain them without the tool. The sampling arithmetic shows a few dozen checks a year are enough to catch a systematic problem. Volume of approved output is no longer evidence that anyone checked it.

Original: https://arxiv.org/abs/2610.10064