Plausible Is Not Enough: Trust and Evidence in AI-Authored Pull Request Review
B · Useful, limited University of Saskatchewan software-engineering survey with a careful within-respondent design, but it measures what 239 reviewers say they do, not observed review behaviour.
Summary based on the full text·Original: about 40 minutes to read
In short
A preprint by Md Shamimur Rahman, Khairul Alam, Banani Roy and Chanchal K. Roy (University of Saskatchewan) surveys 239 practitioners from 31 countries on how they review pull requests written mainly by AI compared with those written by people. Reviewers report lower trust, higher evidence bars and more willingness to block. The authors frame AI PR review as an evidence-ownership problem: who can stand behind the change.
Key numbers
- 64.9%respondents rating initial trust in AI-authored PRs lower than human-authored ones
- 61.1%respondents reporting higher evidence requirements for AI-authored PRs
- 19.4% to 43.9%share who would block approval under residual uncertainty, human vs AI-authored PRs
- 29.3% vs 14.6%unclear ownership as a reason to pause review, AI vs human PRs
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 64.9% give AI PRs lower initial trust. Author track record and subsystem familiarity matter less; prompter expertise and small change size matter more.
- At approval, 55.2% accept their own reasoning through critical cases as sufficient for AI PRs vs 28.5% for human PRs; test coverage counts for less.
- 61.1% demand more evidence; blocking under residual uncertainty rises from 19.4% to 43.9%.
- More priority on test inspection, context checks, local execution and partial reimplementation of non-trivial logic.
- Unclear ownership as a reason to pause: 29.3% vs 14.6%. Open answers stress the difficulty of validating plausible code without project context or a responsible human.
How they got it
A mixed-method questionnaire with matched items for human and AI PRs, so each respondent is compared with themselves. Recruitment drew on repositories with agent-authored PR activity, mature open-source projects and professional networks. The instrument was piloted and expert-reviewed; open answers were coded qualitatively.
Limits
- Stated judgments, not observed review decisions.
- Custom questionnaire items, not validated scales.
- Non-probability sample; "AI-authored" is what the respondent perceives, not verified provenance.
Why it matters for companies
As agents produce more code, the scarce input is a reviewer willing to own the change. Reviewers say they want provenance, a named human accountable, and evidence they can check, not passing tests alone. Teams adopting coding agents should tag AI-generated changes, name the human owner on each PR, and attach the evidence (tests run, outputs, risks) the reviewer needs, or expect slower and more blocked reviews.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Coding agents, Trust