An AI QA team in Claude Code: what worked was discarding findings
Claude Codeで「QAチーム」を組んだら、効いたのは「捨てる仕組み」だった
A · Strong evidence SRE Holdings project manager reports a month of agent-run manual QA with a full funnel of findings, false positives, misses and the agent definitions; small scale, but the method is shown and the counts reconcile.
Summary based on the full text·Original: about 12 minutes to read
In short
A Zenn post from SRE Holdings (29 October 2026) by a project manager who built an AI QA team in Claude Code: a leader in the main session, several tester sub-agents each driving a visible Chrome, and a sceptical verifier sub-agent. It replaced hands-on functional and exploratory testing, not test-code generation. Testers raised 68 findings, about 30% of them confident and wrong. A four-step discard process cut them to 9 tickets, 8 of which were defects nobody else had found. Discarding did nothing about false passes: the team missed 4 defects because no one checked PASS results against actual values.
Key numbers
- 68 to 9findings raised by AI testers versus tickets filed after four filtering steps
- 20 (about 30%)plausible but wrong findings: 15 rejected by the verifier agent, 5 more stopped by the human
- 8filed defects that only the AI team found, compared with developers' and external testers' tickets
- 4defects inside the AI's test scope that it missed, including a value 14.5 times too high marked PASS
- 146test cases run over 6 rounds, 20 August to 16 September 2026, on two internal SaaS products
As reported by the source.
Japanese·Web page·opens zenn.dev
What it reports
- Human role: provide specs, log in to each Chrome, review the leader's test plan, decide what to file. The human does not touch the screen.
- Typical wrong findings: "button does nothing" when a shared click helper (also AI-written) hit a hidden duplicate button at (0,0) and reported success; "no error shown" when the error was a 2-second toast both tester and verifier missed the same way; "counter dropped by 2" when a parallel tester in the same organisation used the shared counter.
- Others: a row reported greyed out because the mouse hovered on it, a conclusion opposite to the agent's own screenshot, and the leader producing false reproductions three times. The internal pitfall list has 60 items.
- Funnel: 68 raised; the leader removed 22 (duplicates, spec questions, known bugs) leaving 46; the verifier rejected 15; the human declined 22 more, 5 of them wrong findings that had passed verification; 9 filed.
- Verifier rules: observe by a different method than the tester; report results as "M of N tries"; never leave unrefuted alternative hypotheses blank. Even in a round with no rejections it corrected 4 symptom descriptions.
- Biggest AI-only find: a statistics aggregation error, fixed 4 days after filing. A search-condition bug was found before release and fixed the next day.
- Biggest miss: a report marked PASS on layout alone while its value was 14.5 times the same figure on another screen. In the rounds with misses, no test case checked an actual value; expected results now state values to match.
- Hard limits in CLAUDE.md: no logins, approval for filing, deleting or changing settings, no direct API checks, no production, no credentials in output, no default destinations. The last came from a script that, called without a target three times, set off staging monitoring alerts three times while returning a plausible "0 results".
- Cost: the kit took 24 working days and 140 commits, mostly by one person; four Chromes mean four logins and sessions expire in days; image-heavy checks use many tokens.
How they got it
Six rounds on two internal products, with results compared against tickets from developers, spec owners and external manual testers. Sub-agents did the first comparison and the author checked each item.
Limits
- Small scale, which the author stresses: 146 cases, one month, two products.
- The human final check relied on the author's long project knowledge; a third-party QA would catch less.
- The 15 rejected findings were not specifically re-audited for real defects, as the author says in reply to a comment.
- No time or cost comparison with human testers.
Why it matters for companies
Agent testers produce a steady stream of convincing false alarms, and tools that report success when they failed. The design here, an adversarial verifier that must use a different observation method plus a human gate, filtered them well. The more important lesson is the gap it exposed: filtering findings does not catch false passes, so test cases must state concrete expected values. The guardrail list is ready to reuse for anyone letting agents drive a browser in staging.
Key passages in translation
Japanese original
効いたのは、誤った指摘をきちんと捨てる仕組みです。
What worked was a mechanism to properly discard wrong findings.
Japanese original
共通するのは、間違っていたのが製品ではなく、観測のしかたか、その解釈だったことです。
What these had in common was that the product was not at fault; the way of observing, or its interpretation, was.
Japanese original
誤指摘を捨てる仕組みは、偽のPASSには効きません。
A mechanism for discarding wrong findings does nothing against false passes.
Summary based on the full text. Read the original for the full argument and data.Read the original on zenn.dev
- Tags
- AI enablement