AIOps: HolmesGPT on three projects
AIOps: наш опыт внедрения HolmesGPT на трёх проектах
A · Strong evidence DevOps team Tuna's detailed month-one report on running the open-source HolmesGPT agent across six Kubernetes clusters, with an open-sourced bridge, a 52-run model comparison and costs reconciled to billing; the headline accuracy figure is openly labelled an impression.
Summary based on the full text·Original: about 20 minutes to read
In short
A Habr post (13 October 2026) from Tuna, a DevOps services team, on putting HolmesGPT, a CNCF Sandbox SRE agent, behind Alertmanager in six Kubernetes clusters: its own and five client clusters across a Russian cloud and a foreign two-cloud project. The agent has read-only access and only writes a hypothesis into Slack or Telegram a few minutes after an alert; it fixes nothing. Because the open-source version has no push integration, the team built and published holmes-alert-bridge, a Go service whose main job is to cap spend.
Key numbers
- 645 to 41alerts delivered to the bridge in Tuna's first month versus those that reached the agent after dedup, allowlist and throttling
- 52test runs across eight models in three waves, including a task with a known answer
- 10-30 centscost of one alert investigation, depending on model; 2-3 minutes median duration
- 90-95%share of hypotheses the team believes are correct, an on-call impression rather than a measurement
- ~90%estimated share of input tokens served from cache, which carries the cost economics
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Median time from alert firing to a hypothesis in chat is about four minutes including Alertmanager grouping delay.
- Bridge controls: allowlist by alert name (empty list blocks start-up), dedup per episode, default cap of two investigations an hour and ten a day, one worker, 15-minute timeout that cuts the stream.
- Model bake-off: deepseek-v4-flash completed 16 of 16 runs and solved the known-answer task 4 of 4; half the candidates failed it the same way, confidently stating that a managed autoscaler did not exist.
- Variance within one model exceeded the price difference between models; single-run comparisons were misleading.
- Production results over 30 days to 6 October 2026: 22 of 22 successful investigations in Tuna, 92.9% in the Russian-cloud project, 15 of 15 abroad on Claude Sonnet 5. Monthly bills were a few dollars per project; Tuna's estimate matched provider billing to the cent.
- A 200k-token window triggered compaction near 126k and made runs slower and costlier; a 400k window stopped it.
- A "have we seen this before" skill searched the wrong log field in two of three setups for a month and always reported no history, including for an episode investigated 19 times.
- Tools not named in a skill were almost never called.
How they got it
Production deployment notes with configuration, logs and billing. The model comparison used 4 to 16 runs per model, priced as an upper bound without cache. The team states it has no clean before-and-after comparison for the model switch, because prompts and the bridge changed at the same time.
Limits
- Hypothesis accuracy has not been measured against real incident causes; the 90-95% is a stated impression.
- No measurement yet of time saved for on-call staff.
- Small samples per model in the bake-off; the Sonnet project had no model comparison.
Why it matters for companies
This is a useful template for putting an agent into operations without giving it write access: read-only RBAC, hypothesis-only output, hard spend caps outside the model, and a monthly limit on the provider key. The data warning applies to any agent that reads logs: whatever it reads goes to the model provider, so tokens and personal data in logs must be fixed in the application first, and zero-retention settings must be verified with live requests. The broken memory skill is the strongest lesson: an agent tool that returns nothing looks the same as one that works, so skills need testing like code.
Key passages in translation
Russian original
По нашему опыту гипотеза оказывается верной в 90–95 % случаев. Это именно оценка по ощущениям от дежурств, а не результат замера
In our experience the hypothesis turns out correct in 90-95% of cases. This is an estimate from how on-call shifts felt, not the result of a measurement.
Russian original
Агент не отличает секрет от обычной строки: содержимое лога попадает в запрос к модели как есть.
The agent does not tell a secret from an ordinary string: the log content goes into the request to the model as is.
Russian original
Мораль шире: skill для агента надо тестировать как код.
The moral is wider: a skill for an agent has to be tested like code.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com