Zero false escalations on synthetic data, 39 of 100 on real traffic: building a local L1 SOC analyst (На синтетике 0 ложных эскалаций, на реальном потоке 39 из 100: как я делал локального L1-аналитика SOC)
A · Strong evidence A solo security engineer reports a fine-tuned local SOC triage model with before-and-after tables on held-out real alerts, open code and weights, and an explicit list of what the numbers do not prove.
Summary based on the full text·Original: about 4 minutes to read
In short
A 6 November 2026 Habr case study on "kezekshi", a local first-line SOC assistant built on a fine-tuned Gemma 4 E2B that runs on a laptop with 8 GB of video memory and keeps logs inside the perimeter. It perfectly triaged a synthetic test set, then needlessly escalated 39 of 100 real Wazuh lab alerts and invented incidents. Training on 215 real benign cases fixed that, and a deterministic allowlist now settles almost all traffic before the model is called. The author keeps it in assist mode because recall on real attacks has not been measured.
Key numbers
- 55 of 55correct verdicts on the synthetic test set, with no false escalations
- 39 of 100 to 0 of 100needless escalations on held-out real lab alerts before and after adding 215 real benign cases to training
- 98.1%share of 3,000 lab alerts settled by the deterministic allowlist before the model is called
- 9/20 to 0/20malformed answer cards on unseen alert types after switching to a per-alert response schema
- 20/20 to 18/20recall on new attack types after a prompt caveat about MITRE candidates; the caveat was removed
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Input is one alert as an evidence pack E1...En; output is a JSON card with a four-class verdict, claims citing evidence numbers, MITRE ATT&CK techniques, gaps and next checks.
- Hard negatives (SCCM PowerShell -enc, vssadmin from a backup, a scheduled Nessus scan) raised the negative set from 8 to 22 with 0 false escalations and no recall loss.
- A severity rubric dropped recall on unfamiliar types from 20/20 to 16/20 (all misses were OAuth consent); one prompt rule restored 20/20 without retraining. Thinking mode of the fine-tuned model hurt.
- On 100 held-out real alerts: v4 (synthetic only) matched 39/100 verdicts and escalated 39/100 needlessly; v5 (plus real noise) matched 96/100 with 0/100 needless escalations.
- On 3,000 lab alerts 98.1% were resolved without the model and 1.9% went to it, about 50 times fewer calls with the same auto-close decisions.
- v6 grew to 3,546 training examples but 9 of 20 cards on unseen types failed validation (the model wrote field name ev1 instead of E1). A per-alert Ollama schema cut that to 0/20 and lifted MITRE recall from 13/36 to 25/36; false escalations rose to 1/34 and 1/100.
- Listing MITRE candidates acted as a hint: the model described normal Outlook reads of Credential Manager as credential harvesting.
- Before publishing weights, 306 attempts produced no lab host names.
How they got it
Fine-tuning and evaluation on synthetic cases, familiar types (76), unseen types (20) and 100 held-out real benign alerts from a closed lab, with tables of before and after each change. Code, schema, ATT&CK index and v6 weights (4.9 GB GGUF) are public; the dataset is not.
Limits
- Recall on real attacks is unmeasured; the real set is noise only, and the 20/20 covers five attack families.
- 30 of 76 familiar and 63 of 100 real cases nearly duplicate training items.
- The real-noise reference labels are one analyst's judgement.
- Metrics cannot be reproduced from the public files.
Why it matters for companies
The gap between 0 false escalations on synthetic data and 39 of 100 on real traffic is a clean demonstration of why pilots need shadow runs on real data before anyone trusts a benchmark. The design choices transfer to any triage task: route deterministic cases around the model, force citations that code can verify, and constrain output with a per-item schema rather than retraining. Small local models are viable where data cannot leave the network, if a person still reviews what reaches the model.
Key passages in translation
Russian original
Она зря подняла на вторую линию 39 из них и к некоторым дописала инцидент, которого не было.
It needlessly escalated 39 of them to the second line, and for some it added an incident that never happened.
Russian original
Список кандидатов работает как намёк на атаку.
The list of candidates acts as a hint that there is an attack.
Russian original
Как автономный решатель эскалаций не проверен.
As an autonomous decision-maker on escalations, it has not been tested.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com