Do AI Coding Agents Write Infrastructure as Code Like Humans? An Empirical Study of GitHub Pull Requests
A · Strong evidence An observational study by Barrak, Ksontini and Wehbe (Oakland University, UNC Wilmington) of about 349,000 infrastructure-as-code pull requests that separates bots, silent agents and disclosed agents, uses repository fixed effects and matched pairs, audits its LLM labels, and publishes a replication package.
Summary based on the full text·Original: about 45 minutes to read
In short
An arXiv preprint (25 October 2026) on how AI coding agents change infrastructure-as-code (Terraform, Ansible, CloudFormation, Puppet, Chef) in public GitHub repositories between January 2025 and May 2026. The authors split authors into humans, automation bots, silent agents (agent commits pushed by a developer) and disclosed agents (agent-opened PRs). Metadata misleads: many "human" PRs come from bots, and agent use is often undisclosed. Agent PRs are about four times larger and touch core module files more often. Within the same repository both agent modes are merged less often than humans, the disclosed ones clearly so. On matched Terraform PRs, agents and humans introduce resource-waste patterns at the same rate.
Key numbers
- 12.3%share of nominally human, non-bot IaC pull requests that carry traces of an AI coding agent
- 8.4 pointslower merge probability for disclosed-agent PRs than human PRs, same repository and same change type
- 302 vs. 71median lines changed in a silent-agent PR versus a human PR
- 87.4% vs. 78.9%raw merge rate of silent-agent PRs versus human PRs, before repository controls
- 10.4% vs. 10.3%share of matched agent and human Terraform PRs that introduce a sustainability smell
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Corpus: 349,975 PRs from 49,241 repositories; 349,156 labelled. Agent PRs rose from fewer than 100 a month before May 2025 to 8,057 in May 2026; silent-agent PRs have outnumbered disclosed ones since November 2025.
- Bots account for about a third of ostensibly human PRs (32.9% in the abstract, 35.8% in the contributions list). Of the rest, 12.3% contain agent traces, a share that grows over the period.
- Raw merge rates: silent agents 87.4%, humans 78.9%, disclosed agents 76.3%, bots 51.8%. With repository fixed effects, silent-agent PRs are merged 4.0 points and disclosed-agent PRs 10.4 points less often than human PRs. Repositories that use silent agents merge even human PRs at 86.2%, against 75.6% elsewhere.
- Copilot PRs are merged at 86.6% when a developer pushes the commits and 70.3% when the agent opens the PR; Claude PRs at 88.3% and 89.8%.
- Median PR size: 302 lines in 8 files (silent agent), 247 in 7 (disclosed agent), 71 in 4 (human), 26 in 1 (bot).
- Disclosed agents create new infrastructure more often than humans (34.0% vs. 21.4% of PRs). Even for the same change type in the same repository, they are 8.4 points less likely to be merged. Silent-agent PRs are flagged for broadening permissions in 14.2% of cases against 9.8% for humans.
- In 2,645 matched agent-human Terraform PR pairs on AWS, smell rates are 10.4% and 10.3% (paired odds ratio 1.02, 95% CI 0.83-1.25); the most common are missing remote state and monolithic configurations.
How they got it
PRs collected through the GitHub search API, kept only if they change files of the five IaC tools. Authorship assigned from agent signatures, commit-trace census and bot accounts; forks, insecure-by-design and demo repositories removed, with a cap of 50 PRs per repository and population for the study corpus. Change types and rejection reasons labelled by two LLMs from different vendors and corrected through a stratified human audit. Merge models use linear probability models with repository fixed effects and clustered errors. The smell detector adapts published definitions (precision 0.98).
Limits
- Silent-agent counts are a lower bound: agent use without traces is missed, which biases comparisons toward no difference.
- Observational; merges measure acceptance, not correctness, and smells indicate possible waste rather than measured cost.
- The sustainability comparison covers AWS Terraform only.
- Two internal inconsistencies in the text: the bot share (32.9% vs. 35.8%) and repository count in the conclusion versus the labelled set.
Why it matters for companies
Infrastructure changes open network paths, grant permissions and create billable resources, so who wrote them matters for review. The study shows that PR metadata understates agent involvement and overstates human work, and that agent PRs are larger and lean toward creating new resources and broader permissions. Agent output does not look worse on resource-waste checks, which argues for the authors' conclusion: apply the same automated IaC policy checks to every PR regardless of author, and track agent involvement from commit traces rather than self-reporting.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Coding agents, Disclosure