---
id: "lib-2026-10-23-large-scale-study-of-github-copilot-trajectories"
type: "library"
title: "Large-Scale Study of GitHub Copilot Trajectories"
lang: "EN"
org: "arXiv"
source_type: "Paper"
date: "2026-10-23"
week: "2026-W44"
url: "https://nosov.ai/library/lib-2026-10-23-large-scale-study-of-github-copilot-trajectories/"
original_url: "https://arxiv.org/abs/2610.32313"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "An NYU Abu Dhabi team (Sohail, Kurmantayev, Mahmoud, Nadi) encodes 7,581 real Copilot agent sessions from public repositories, validates the parser against hand-coded ground truth and reports corrected statistical tests; one agent and one platform, but the method is shown and the data is reusable."
summary_basis: "full-text"
original_minutes: "40"
tags: ["ai technology", "coding agents", "github copilot", "human intervention"]
---

# Large-Scale Study of GitHub Copilot Trajectories

Tier A (Strong evidence): An NYU Abu Dhabi team (Sohail, Kurmantayev, Mahmoud, Nadi) encodes 7,581 real Copilot agent sessions from public repositories, validates the parser against hand-coded ground truth and reports corrected statistical tests; one agent and one platform, but the method is shown and the data is reusable.

Summary based on the full text.

## In short

An arXiv preprint (23 October 2026) from New York University Abu Dhabi that studies how developers actually work with a coding agent, instead of scoring it on a benchmark in one autonomous run. The authors scraped the public session logs of GitHub Copilot's cloud agent for 7,581 issue-assigned pull requests and encoded each into a sequence of agent attempts and human interventions. Most PRs get no feedback to the agent at all; when developers do intervene, they do it briefly (median two rounds), and PRs with an intervention merge more often. Most of the agent's work happens in its first attempt; follow-up rounds are short and focused.

## Key numbers

- 7,581: Copilot cloud-agent pull requests with public session data, across 1,602 repositories
- 58%: share of those PRs with no human intervention directed at the agent
- 78% vs. 62%: merge rate of PRs with at least one human intervention versus none
- 41.1 to 20.3: mean iterations before and after the first human intervention
- 44: median total agent iterations per PR, 35 of them in the initial attempt

## What it reports

- 4,376 PRs (58%) had no intervention directed at Copilot; 3,205 (42%) had at least one human-AI loop. Median loops among intervened PRs: 2, maximum 25.
- Merge rate: 78% with intervention against 62% without (p < 0.001, small effect). The number of loops barely differs between merged and unmerged intervened PRs.
- Of non-intervened merged PRs, 38% still contain extra commits by the developer; 62% count as fully autonomous.
- A PR takes a median of 44 agent iterations and 19.2 minutes of agent time; the initial attempt takes 35 iterations. Merged PRs have shorter first attempts (34 vs. 39 iterations), which the authors read as easier tasks being more likely to succeed.
- After the first human intervention, iterations fall from 41.1 to 20.3 and duration drops 8.4 minutes. After later interventions, iterations move from 20.6 to 19.2 with no significant change in duration. The pattern holds for every task type modelled.
- In 20 PRs read by hand, interventions were mostly small requests: suggest an alternative, refine a requirement, flag a bug, correct a misread specification. Some developers coach the agent over several loops; others make the change themselves; some re-assign the same issue to get several PRs and pick one.
- Task mix: 40.0% bug fixes, 33.1% features, 13.6% documentation.

## How they got it

Source: AIDev dataset v4, restricted to repositories with at least 100 stars, closed PRs, explicit issue assignment and sessions created on or after 5 August 2025 (when Copilot began acting only on comments that tag it). Session pages were rendered in a headless browser and parsed into thought, action and result steps. The parser was checked against 14 hand-encoded PRs: 14 had the right structure and 12 were fully correct. Group comparisons use Mann-Whitney U and chi-square tests with Benjamini-Hochberg correction; pre/post intervention effects use linear mixed models with a random intercept per PR.

## Limits

- One agent (Copilot cloud agent) in popular open-source repositories; behaviour in other agents, IDE modes or private code bases is not covered.
- Unmerged is treated as failure, though PRs close for other reasons; iterations and minutes are proxies for effort.
- Pre/post effects are associations; repository practice and issue difficulty are not fully controlled.
- Interventions are counted only when they tag @copilot; human-to-human review is excluded.
- The qualitative part covers 20 PRs.

## Why it matters for companies

Benchmarks measure an agent working alone; this paper measures the workflow that teams actually run, and the result is a useful planning number. Assigning an issue to an agent usually yields one long attempt and then one or two short review rounds; if a PR needs many more, the task was probably too large or badly specified. The higher merge rate with intervention argues for budgeting reviewer time for agent PRs rather than counting on autonomous merges, and the finding that 38% of "autonomous" merges still carry developer commits is a warning against reading agent PR counts as agent output.

Original: https://arxiv.org/abs/2610.32313