Chaos as a service: how Avito tests resilience in production and adds an LLM to it (Хаос как сервис: как мы в Авито проверяем устойчивость в проде и подключаем к этому LLM)
A · Strong evidence Production report by Ilya Sadykov of Avito's TestOps team with full run counts, score distributions, agreement rates and an explicit model-error case, and an honest note that the headline improvement is correlation, not attribution.
Summary based on the full text·Original: about 20 minutes to read
In short
A Habr post on the AvitoTech blog (3 November 2026) by Ilya Sadykov of the TestOps team in Avito's platform group. Avito built its own chaos-testing orchestrator on LitmusChaos and added an LLM in two places: mining past incident post-mortems for test scenarios, and writing a diagnosis of each production test run with access to the service's code. The system runs in production. The main constraint turned out to be owners switching services on, not technology or risk gates.
Key numbers
- 977incident post-mortems analysed by Llama 3 70B to generate chaos-test hypotheses
- 41% vs 22%share of past incidents Avito's orchestrator (LitmusChaos plus its own scenarios) can reproduce vs stock Litmus
- 1,129finished production chaos runs; 95.6% (1,079) were primary re-election in MongoDB or Redis; 79% scored excellent or good
- 61 of 70 (87%)service-owner verdicts agreeing with the agent's written conclusion on a run
- about 6 vs 44tests per day actually run vs planned; the bottleneck is owners opting services in
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Stock Litmus, which works only with pods, can simulate about 22% of past incidents; with Avito's own database and network scenarios, 41% (400 of 977).
- A daily pipeline asks Llama 3 70B (internal deployment) six fixed questions per post-mortem, including a 0-10 score for how well the failure fits existing checks. 41% scored 10. Incident types: network 41%, resources 31%, application 16%, state and processes 12%.
- A Redis master failover incident (19 critical user journeys down, authentication out for 12 minutes, about 50% of users unable to refresh tokens) led to a new scenario, trigger-primary-re-election, now 95.6% of production auto-runs.
- Model error case: an incident scored 10/10 for HTTP-latency simulation, but the real cause was a third-party config with SSL handshake errors and 502s. The test would have stayed green while checking nothing.
- Per-run agent conclusion: in the past month it ran on 92% of production runs and flagged action needed in 54%: 26% of excellent runs and 80% of the rest. One example pointed to the exact handler lines returning 5xx on any cache error.
- Owner verdicts: 61 agree, 9 disagree (87%); 23 findings taken into work, 30 linked to tickets.
- Run outcomes over 1,129 finished runs: excellent 45%, good 34%, moderate 14%, poor 2%, critical 4%. 89 escalations on 64 services (7.9% of runs); 188 runs were cancelled before fault injection.
- Coverage: at least one test passed on 408 services, 229 of 1,202 critical and high-priority services (about 20%). Error-budget cost: median 0.14% of a service's monthly budget, maximum 17.3%.
- Of 360 services assessed on "survives loss of the database master", 141 ever failed; failures went from 3 a year ago to 69 in March to 8 now, as coverage first exposed defects and teams then fixed them.
How they got it
Production telemetry from Avito's chaos platform: baseline, a 5-minute fault, and a 5-15 minute recovery phase scored on 5xx errors and dependent-service p98 latency, with a separate verifier rechecking recovery before escalation. The LLM pipeline and agent run inside Avito's network. Counts come from the platform's own records.
Limits
- The authors say the drop in failing services is correlation; architecture reviews and launch requirements pushed teams at the same time.
- Almost all runs use one scenario; others are enabled on few services.
- "41% reproducible" is the model's own score, and the post shows a case where a 10/10 score was wrong. Owner agreement covers only 70 verdicts.
Why it matters for companies
Two transferable uses of LLMs here: reading the backlog of incident reports nobody acts on and turning them into tests, and explaining a red result in terms an owner can act on Monday morning. Both keep a person deciding. The finding that opt-in, not technology, limits throughput is a common pattern for internal platforms: budget the next quarter for adoption work, not more scenarios.
Key passages in translation
Russian original
Такой тест жил бы в расписании, стабильно зелёный, и не проверял бы ровно ничего.
Such a test would sit in the schedule, consistently green, and check precisely nothing.
Russian original
Самый узкий участок автоматизации оказался не техническим: каждый сервис входит в программу через явное включение флага владельцем.
The narrowest point of the automation turned out not to be technical: each service joins the programme only when its owner explicitly switches on a flag.
Russian original
Тут нужна честная оговорка, без которой цифра превращается в рекламу: это корреляция, а не атрибуция.
An honest caveat is needed here, without which the figure becomes advertising: this is correlation, not attribution.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com
- Tags
- AI enablement