---
id: "lib-2026-10-24-build-or-adopt-custom-built-vs-general-purpose"
type: "library"
title: "Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing"
lang: "EN"
org: "arXiv (Volvo Group, Chalmers)"
source_type: "Practitioner post"
date: "2026-10-24"
week: "2026-W44"
url: "https://nosov.ai/library/lib-2026-10-24-build-or-adopt-custom-built-vs-general-purpose/"
original_url: "https://arxiv.org/abs/2610.33636"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "A Chalmers and Volvo Group industrial case study that swaps only the agent harness while holding tools and models fixed, reports success, flakiness, latency and cost per task with paired tests, and adds ten practitioner interviews; one company and one task family, but the comparison is clean and the numbers are checkable."
summary_basis: "full-text"
original_minutes: "45"
tags: ["ai technology", "business transformation", "agent harnesses", "build vs buy", "software testing"]
---

# Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing

Tier A (Strong evidence): A Chalmers and Volvo Group industrial case study that swaps only the agent harness while holding tools and models fixed, reports success, flakiness, latency and cost per task with paired tests, and adds ten practitioner interviews; one company and one task family, but the comparison is clean and the numbers are checkable.

Summary based on the full text.

## In short

An arXiv preprint (24 October 2026) by researchers at Chalmers University of Technology and Volvo Group on a build-versus-adopt decision. Volvo runs a custom agent harness in production to test Android-based in-vehicle infotainment screens. The team kept its tools and models and replaced only the harness with Claude Code or OpenCode, connected through an MCP server. On Volvo's stepwise tests the general-purpose harnesses matched the in-house one; on open-ended public-app tasks they did better, mainly because they recovered from wrong actions. They cost several times more per task, but saved enough engineering time to lower total monthly cost. Interviews describe the in-house harness turning into technical debt without any decay of its own, because outside tools caught up.

## Key numbers

- 2.6 to 10.7 times: inference cost per task of general-purpose harnesses versus the in-house one, same model, on Volvo's infotainment tests
- 3-66%: lower total monthly cost (engineering plus inference) for a nightly regression suite after switching
- 0.5 to 0.1: full-time equivalents of engineering effort for the in-house harness versus the general-purpose setup
- 55-61%: task success of all configurations on 66 stepwise in-vehicle infotainment tests
- 78-83% vs. 24-53%: success after an incorrect tool call on Google Maps, general-purpose versus in-house harness

## What it reports

- Volvo benchmark (66 tasks): all configurations succeed in 55-61% of runs; same-model differences of -4.5 to -0.8 points, none significant. General-purpose harnesses gave inconsistent outcomes on 3-6 tasks against 1-2 for the in-house one.
- Google Maps (40 tasks): general-purpose harnesses ahead with every model, by 28 points with GPT-5.4, 17-19 with Sonnet 4.6 and 9-11 with Opus 4.6. Booking.com (16 tasks) is hardest, at 10-42% success; differences not significant.
- General-purpose harnesses do not make fewer wrong tool calls (88-89% correct on Google Maps against 93-98%); they recover more often, which the authors attribute to keeping the full conversation instead of rebuilding a compact prompt each step.
- Cost per task, same model: 1.9 to 10.7 times higher across benchmarks, 2.6 to 10.7 on Volvo's tests. Example: Claude Code with Sonnet 4.6 at $4.97 per task against $0.46. The production configuration (in-house harness, GPT-5.1) is cheapest, at $0.13 per task. Latency is not worse: 3 to 57 seconds lower per task.
- Engineering effort falls from about 0.5 FTE across four developers to 0.1 FTE. At 60 tasks per working night the extra inference equals 0.06-0.39 FTE, below the 0.4 FTE saved, so total monthly cost is 3-66% lower, with OpenCode saving most.
- Interviews: the in-house harness became hard to justify once vendors maintained an equivalent loop for everyone. Adoption moves upkeep to tool descriptions and instructions, and the team loses control over context and compaction. Testers ranked visibility of the agent's steps as the property they value most and want the agent to stop and report when the app fails, rather than work around a defect.

## How they got it

Same tool layer and device access (uiautomator2 over ADB) in both setups; only the orchestration loop changes. Up to four models, each configuration run twice on the Volvo rig and three times on 56 AMEX tasks. Runs scored by an LLM judge checked against a second judge (93.4% agreement) and against manual labels on Booking.com. Paired Wilcoxon tests for success and cost. Semi-structured interviews of 50-60 minutes with four developers, the product owner and five testers, analysed thematically.

## Limits

- One company, one test domain; the total-cost estimate depends on a manager's estimate of effort shares and on a nightly suite of 60 tasks.
- The general-purpose setup had not been released to testers, so their views concern the in-house system.
- The switch from the Volvo rig to public apps also changes task style and device, so the transfer result has more than one cause.
- Few repetitions per task; Booking.com results rest on 16 tasks.

## Why it matters for companies

Many firms built their own agent loops when no capable general harness existed. This paper gives a way to decide whether to keep them: hold tools and models fixed, swap the harness, and compare success, flakiness, cost per run and engineering time together. The result here is a trade of fixed engineering effort for variable inference spend, worth it at Volvo's run volume and not automatically elsewhere. The design advice holds regardless: keep tools, instructions and test intents separate from the orchestration so the loop can be replaced, and require agents in regression testing to report failures instead of routing around them.

Original: https://arxiv.org/abs/2610.33636