When Code Gets Cheap, Evaluation Gets Scarce: An Experience Report on AI-Assisted Agile Software Engineering
B · Useful, limited Candid experience report from University of St.Gallen software-engineering researchers (Weber, Zimmermann, Franceschetti) with a recorded failure log and supplementary artefacts, but a single developer and case.
Summary based on the full text·Original: about 40 minutes to read
In short
An arXiv experience report (22 October 2026) from the University of St.Gallen. Barbara Weber built a compliance-critical, human-in-the-loop exam-grading system with a coding assistant and wrote no code by hand. The first version graded a real cohort after three days. The authors' claim: AI makes code abundant, and reliable evaluation (knowing a result is correct and meets the need) becomes the binding constraint, with engineering judgment a second scarcity. This held even though the developer was also the end user and domain expert, the cheapest possible evaluation setting.
Key numbers
- three working daystime to build the initial grading system, with no hand-written code
- 27 students, 324 answersreal exam graded with the system at the report's cut-off
- 5 of 39code commits carrying a prompt specification on the day the assistant ran past unenforced gates
- 26 of 27code changes carrying a test once the main branch was made unwritable for the assistant
- 492regression tests guarding a multi-session refactor
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Four features passed the full test suite and a headless smoke test but failed in use, all at mocked boundaries: browser iframe security, a token budget truncating model output, a destructive overwrite on empty model responses, and an oversized plot for the real API.
- The most severe defect, one exam's edits surfacing in another, survived several hundred passing tests and an interface harness; a person at the screen found it.
- Deployment from local to server exposed assumptions local tests had missed.
- The assistant advised against a domain model as over-engineering; when a second grading paradigm was added the same afternoon, rules copied across screens drifted and about fifteen screens showed wrong or empty results.
- On the day a branch-and-PR step existed only as a convention, the assistant committed past it: 5 of 39 code commits had a prompt specification and eight UI iterations shipped without a test. After the main branch was made unwritable for the assistant, 26 of 27 changes had a test and 22 a specification.
- A second application (19 user stories, 147 tests) was built in about two working days with the gates in place from the start.
- Five lessons: gated traceable process, specify the knowable and discover the rest on a prototype, validate in the real environment, gate the assistant as well as the work, and record decisions and process state the assistant does not retain between sessions.
How they got it
A first-person account backed by the commit record, a construction cycle with explicit gates (user story, prompt specification, test-first branch, real-data validation, pull request), and a supplementary artefact package. A university pilot (September 2026 to March 2027) and an independent single-expert case are cited as corroboration.
Limits
- One developer, one assistant setup, mostly one application; the authors call the before-and-after gate data anecdotal.
- The "triple union" of developer, user and domain expert makes iteration gains unusually large; they may shrink where evaluation needs other people.
- No controlled comparison with conventional development.
Why it matters for companies
The defect list is a practical checklist: mocked tests confirm wiring, not that the system works, so agent-built software needs validation in the real browser, against the real model API and on the real deployment target. Process checkpoints that exist only as habits will be skipped by an agent acting across many steps; enforce them in tooling. Budget for evaluation and review capacity as code output rises, because that is where the time goes.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org