---
id: "lib-2026-10-29-an-ai-qa-team-in-claude-code-what-worked-was"
type: "library"
title: "An AI QA team in Claude Code: what worked was discarding findings"
title_original: "Claude Codeで「QAチーム」を組んだら、効いたのは「捨てる仕組み」だった"
lang: "JA"
org: "SRE Holdings (Zenn)"
source_type: "Practitioner post"
date: "2026-10-29"
week: "2026-W44"
url: "https://nosov.ai/library/lib-2026-10-29-an-ai-qa-team-in-claude-code-what-worked-was/"
original_url: "https://zenn.dev/sre_holdings/articles/d4acb079680a79"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "SRE Holdings project manager reports a month of agent-run manual QA with a full funnel of findings, false positives, misses and the agent definitions; small scale, but the method is shown and the counts reconcile."
summary_basis: "full-text"
original_minutes: "12"
tags: ["ai technology", "enablement", "ai testing", "multi agent qa", "false positives"]
---

# An AI QA team in Claude Code: what worked was discarding findings

Original title: Claude Codeで「QAチーム」を組んだら、効いたのは「捨てる仕組み」だった

Tier A (Strong evidence): SRE Holdings project manager reports a month of agent-run manual QA with a full funnel of findings, false positives, misses and the agent definitions; small scale, but the method is shown and the counts reconcile.

Summary based on the full text.

## In short

A Zenn post from SRE Holdings (29 October 2026) by a project manager who built an AI QA team in Claude Code: a leader in the main session, several tester sub-agents each driving a visible Chrome, and a sceptical verifier sub-agent. It replaced hands-on functional and exploratory testing, not test-code generation. Testers raised 68 findings, about 30% of them confident and wrong. A four-step discard process cut them to 9 tickets, 8 of which were defects nobody else had found. Discarding did nothing about false passes: the team missed 4 defects because no one checked PASS results against actual values.

## Key numbers

- 68 to 9: findings raised by AI testers versus tickets filed after four filtering steps
- 20 (about 30%): plausible but wrong findings: 15 rejected by the verifier agent, 5 more stopped by the human
- 8: filed defects that only the AI team found, compared with developers' and external testers' tickets
- 4: defects inside the AI's test scope that it missed, including a value 14.5 times too high marked PASS
- 146: test cases run over 6 rounds, 20 August to 16 September 2026, on two internal SaaS products

## What it reports

- Human role: provide specs, log in to each Chrome, review the leader's test plan, decide what to file. The human does not touch the screen.
- Typical wrong findings: "button does nothing" when a shared click helper (also AI-written) hit a hidden duplicate button at (0,0) and reported success; "no error shown" when the error was a 2-second toast both tester and verifier missed the same way; "counter dropped by 2" when a parallel tester in the same organisation used the shared counter.
- Others: a row reported greyed out because the mouse hovered on it, a conclusion opposite to the agent's own screenshot, and the leader producing false reproductions three times. The internal pitfall list has 60 items.
- Funnel: 68 raised; the leader removed 22 (duplicates, spec questions, known bugs) leaving 46; the verifier rejected 15; the human declined 22 more, 5 of them wrong findings that had passed verification; 9 filed.
- Verifier rules: observe by a different method than the tester; report results as "M of N tries"; never leave unrefuted alternative hypotheses blank. Even in a round with no rejections it corrected 4 symptom descriptions.
- Biggest AI-only find: a statistics aggregation error, fixed 4 days after filing. A search-condition bug was found before release and fixed the next day.
- Biggest miss: a report marked PASS on layout alone while its value was 14.5 times the same figure on another screen. In the rounds with misses, no test case checked an actual value; expected results now state values to match.
- Hard limits in CLAUDE.md: no logins, approval for filing, deleting or changing settings, no direct API checks, no production, no credentials in output, no default destinations. The last came from a script that, called without a target three times, set off staging monitoring alerts three times while returning a plausible "0 results".
- Cost: the kit took 24 working days and 140 commits, mostly by one person; four Chromes mean four logins and sessions expire in days; image-heavy checks use many tokens.

## How they got it

Six rounds on two internal products, with results compared against tickets from developers, spec owners and external manual testers. Sub-agents did the first comparison and the author checked each item.

## Limits

- Small scale, which the author stresses: 146 cases, one month, two products.
- The human final check relied on the author's long project knowledge; a third-party QA would catch less.
- The 15 rejected findings were not specifically re-audited for real defects, as the author says in reply to a comment.
- No time or cost comparison with human testers.

## Why it matters for companies

Agent testers produce a steady stream of convincing false alarms, and tools that report success when they failed. The design here, an adversarial verifier that must use a different observation method plus a human gate, filtered them well. The more important lesson is the gap it exposed: filtering findings does not catch false passes, so test cases must state concrete expected values. The guardrail list is ready to reuse for anyone letting agents drive a browser in staging.

## Key passages in translation

> 効いたのは、誤った指摘をきちんと捨てる仕組みです。

What worked was a mechanism to properly discard wrong findings.

> 共通するのは、間違っていたのが製品ではなく、観測のしかたか、その解釈だったことです。

What these had in common was that the product was not at fault; the way of observing, or its interpretation, was.

> 誤指摘を捨てる仕組みは、偽のPASSには効きません。

A mechanism for discarding wrong findings does nothing against false passes.

Original: https://zenn.dev/sre_holdings/articles/d4acb079680a79