Comprehension Audits to Mitigate Risks from Automated AI Research
A · Strong evidence Governance proposal co-authored by Gillian Hadfield (Johns Hopkins), backed by an original measurement of review activity across 224 GitHub organizations and 2,135 AI repositories; strong evidence, but an arXiv position paper rather than a field-shaping report.
Summary based on the full text·Original: about 75 minutes to read
In short
A paper by Ronald J. Bodkin (independent), Bahrad A. Sokhansanj (Institute for Law and AI) and Gillian Hadfield (Johns Hopkins University), posted to arXiv on 7 October 2026. It proposes "comprehension audits": an independent auditor picks a piece of R&D work and the people responsible must explain it without AI help. A failing grade pauses development and any release that depends on it until fixed. The authors support the case with a measurement of open-source AI projects showing code volume rising while human review attention per line falls.
Key numbers
- 183%median increase in code submitted by regular contributors to open-source AI repos, Q1 2024 to Q2 2026
- 36%human inline comments per kLOC on AI-assisted PRs by June 2026, as a share of the Q1 2024 rate
- <30%human review rate for PRs from high-volume fleet accounts, against about 80% for non-AI PRs
- 17 auditsquarterly sample that detects at least one failure with 99% probability at a 25% failure rate
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Anthropic, OpenAI and Google report AI writing 75% or more of their new code; Anthropic reported that as of August 2026 Claude "leads" 26% of its R&D work.
- Traceable AI use appears in 17% of all pull requests by the end of the period, which the authors call a floor.
- Regular contributors' code output rose by a median of 183% from Q1 2024 to Q2 2026; contributors with evidence of AI use rose by a median of 202%.
- Review rate for non-AI PRs held at about 80%. AI PRs ran a median 8.1 percentage points lower, a gap that disappears with contributor weighting.
- Fleet accounts (42 accounts with 300+ merged PRs in a quarter) went from 4.7% of merged PRs in Q1 2024 to 15.2% in Q2 2026. Their review rate fell below 30%, from about 50% in Q1 2024.
- Human inline comments per kLOC fell roughly 50%: to 36% of the Q1 2024 base on AI PRs and 46% on non-AI PRs by June 2026.
- Only 3-7% of fleet PRs had any inline commentary, against 20-26% of other PRs.
- Sampling table: at a 10% failure rate, 22 audits give 90% detection; at 25%, 17 audits give 99%.
How they got it
The empirical part analyses public GitHub PRs from 224 organizations and 2,135 repositories selected from five third-party lists of AI developers, Q1 2024 to Q2 2026, with an 8-week cutoff for merges and reviews. AI use is detected from traceable markers such as agent branch prefixes and co-author tags. The audit process itself (selection, no-AI examination, grading, appeal, remediation, escalation) is specified in detail, with worked hypothetical cases.
Limits
- The empirical data covers open-source projects; the audits are proposed for closed frontier labs, where no comparable data is available.
- AI detection relies on visible traces, so AI use is undercounted, and fewer comments per line is a proxy for attention, not a direct measure of understanding.
- The audit regime is a proposal; no lab has piloted it, and the audit cases in the appendix are illustrative.
Why it matters for companies
The same pattern applies to any team that ships AI-written code, reports or analyses: output volume grows faster than the capacity to read it, and sign-off stops meaning understanding. The paper offers a cheap control: sample a small number of deliverables each quarter and ask the owners to explain them without the tool. The sampling arithmetic shows a few dozen checks a year are enough to catch a systematic problem. Volume of approved output is no longer evidence that anyone checked it.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Governance, Coding agents, AI safety