How AI helps test a large product
Как ИИ помогает в тестировании большого и сложного продукта?
A · Strong evidence Practitioner case study from CloudLinux with measured outcomes from about 1,500 production runs: success rate, cost medians, flaky rate and monthly spend, plus documented failure modes.
Summary based on the full transcript·Original: about 74 minutes to watch
In short
A 74-minute meetup talk on the Hard&Soft Skills channel by Dmitry Vinokurov, a software engineer at CloudLinux, published 9 October 2026. He describes an internal system that finds gaps in test coverage across Imunify, a web-server security product protecting more than 50 million domains, and writes the missing tests with Claude agents. Since about April 2026 it has run roughly 1,500 times, with 70% of runs producing a mergeable change. The main lesson is that the difficulty moves from writing tests to reviewing, controlling cost and setting up test environments.
Key numbers
- 70%pipeline runs that produced a successful merge request, out of about 1,500
- <1%flaky rate of the generated tests, mostly end-to-end
- $1,000-2,000monthly spend on the system, rising with newer model versions
- 120 / 200agent step limits for unit and end-to-end tests, set empirically
As reported by the source.
Russian·Video·opens youtube.com
What it reports
- An AI pre-analysis of about 26 critical components, checked by component owners, produced a database of scenarios and test cases with priorities; the video description cites 28 components in five languages and more than 11,000 test cases, the talk more than 13,000.
- Two pipelines: a weekly component analysis that emits Markdown for people and schema-strict JSON for the database, and a nightly generator that picks the highest-priority uncovered case and the simplest test level.
- An independent review agent checks every merge request; if the two agents disagree beyond an iteration limit, a human takes over.
- Results: about 1,500 runs, 70% successful merge requests, 73.2% successful runs. Median cost per successful test case is a few dollars; the median failed run costs under one dollar.
- Generated tests have a flaky rate below 1%. Real product bugs were found and filed.
- Monthly spend is about $1,000-2,000, and the average cost per test case rose with new model versions.
- The Agent SDK computes cost from local price tables that lagged real prices, so the team hit spending limits mid-month while their own dashboard looked normal.
- Independent work units duplicated helper functions; a consolidation skill fixed it. Opus tended to spawn too many sub-agents. Two people built and run the system.
How they got it
First-hand engineering account of a proprietary system in production since spring 2026, with figures from its own run logs. Deterministic Python handles orchestration; models handle analysis and generation, using routing, parallel work, evaluator-optimizer loops and a weekly self-review of CI runs. The digest uses cleaned auto-captions, where some numbers were garbled; only clearly stated figures are reported.
Limits
- No measure of how many generated tests are useless; the speaker says that metric does not exist yet.
- No baseline against human-written tests on cost or defect detection.
- Figures are self-reported from one company and one product.
Why it matters for companies
The design is a template for any programme where agents work through a backlog of well-defined units: a work queue in a database, strict output formats, step limits, an independent reviewer and escalation to people. The cost lesson is practical - track spend outside the vendor's tooling, because the vendor's own counters can lag. Review becomes the bottleneck, so rules on what gets merged automatically and what goes to a person need to be set before the volume arrives.
Key passages in translation
Russian original
просто создание теста с помощью ИИ совсем не представляет сложности в подавляющем большинстве случаев. А когда это выходит на серьёзный масштаб, то это добавляет сложности и требует специального подхода.
Simply creating a test with AI is not difficult at all in the vast majority of cases. When it reaches serious scale, that adds complexity and calls for a special approach.
Russian original
Процент успешно созданного MR 70%, возможно, не такой высокий, ну и не такой низкий, но существенная доля всё равно успешно дошла до зелёного.
The share of successfully created MRs is 70%, perhaps not that high, but not that low either; a substantial share still made it to green.
Russian original
два агента ушли в бесконечный цикл, не было лимита, и они там несколько десятков тысяч долларов за ночь просадили.
Two agents went into an infinite loop, there was no limit, and they burned several tens of thousands of dollars overnight. (A case he recalls from an earlier talk on the same channel.)
Russian original
агентский SDK для расчёта стоимости учитывает локальные таблицы с ценами, которые могут отставать от реальных расценок... Мы на это напоролись, когда в середине месяца вышли за лимиты.
The agent SDK calculates cost from local price tables, which can lag behind real prices... We ran into this when we exceeded our limits in the middle of the month.
Russian original
прямо построчно проверять не вариант. Полностью не проверяя тоже не вариант. Вот нахождение баланса - это один из краеугольных вопросов.
Checking line by line is not an option. Not checking at all is not an option either. Finding the balance is one of the cornerstone questions.
Summary based on the full transcript. Read the original for the full argument and data.Watch the original on youtube.com