"To Boldly Go Where No One Has Gone Before" What Has AI Let Developers Unlock?
B · Useful, limited Oregon State, Microsoft and DX researchers code open-ended survey answers from 331 Microsoft developers with a documented, cross-checked method; useful framing, but self-reported, cross-sectional and from a company that sells the AI tools.
Summary based on the full text·Original: about 30 minutes to read
In short
An arXiv paper (16 October 2026) by Rudrajit Choudhuri and Anita Sarma (Oregon State), Jenna Butler, Christian Bird and Carmen Badea (Microsoft) and Brian Houck (DX). It argues that measuring AI by how much faster developers work misses the point, and asks instead what developers can now do that they could not before. Answers from 331 Microsoft developers fall into three groups: the same work faster, work previously out of reach, and new work directing AI agents. The paper's warning is the "hollow orchestrator": a developer who approves AI output they can no longer judge.
Key numbers
- 331Microsoft software developers whose open-ended answers were analysed
- 45%responses describing familiar work done faster ('Faster')
- 36.9%responses describing work previously beyond their reach ('Further')
- 12.4%responses describing new work built around directing and building for AI agents ('Afresh')
- κ = 0.87agreement between a blind author re-coding of 50 responses and the AI-assisted coding
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Faster (45%): familiar work done faster or at greater volume.
- Further (36.9%): work previously beyond reach, including new skills, broader scope and coordination across projects.
- Afresh (12.4%): new kinds of work, such as setting standards and context for AI, directing agents and building pipelines around them.
- Learning runs both ways: developers learn new skills through AI and write their tacit knowledge down as specifications, memory and success criteria that guide AI's work.
- The authors frame the "skill-judgment nexus": AI can supply skill faster than it can reliably judge its own work, and developers must close that gap.
- Two paths: judgment grows with AI's skill and is codified so the loop continues, or developers become hollow orchestrators, buried in verification debt and losing skills and professional identity. A two-way scaffolding playbook is offered for developers, teams and tool builders.
How they got it
An open-ended survey analysed with reflexive thematic analysis and a "multi-model AI council": three models proposed themes, two researchers approved a codebook, three fresh model instances coded every response (median Krippendorff's α 0.92, 2-of-3 majority), a calibrated decision model checked each code, and one author blind re-coded 50 responses (κ = 0.87). 14 responses were excluded for weak zone support.
Limits
- Stated experience, not observed performance; the authors note self-reported speed can be wrong even in direction.
- One company, which builds and sells AI coding tools, at one point in time, with possible self-selection.
- AI-assisted coding can surface plausible themes that do not match what people meant, though the checks are reasonable.
- The playbook draws on literature and interpretation rather than tested interventions.
Why it matters for companies
The three-way split is a better frame for AI impact reviews than hours saved: ask what share of use opens new work, not only what speeds old work. The hollow-orchestrator risk is the management issue. If staff approve output they cannot assess, quality and skills degrade quietly. Teams should make people write down the standards and acceptance criteria agents work to, and keep some work where people build judgment by doing it themselves.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Work and AI, Tacit knowledge, Survey