---
id: "lib-2026-11-06-zero-false-escalations-on-synthetic-data-39-of"
type: "library"
title: "Zero false escalations on synthetic data, 39 of 100 on real traffic: building a local L1 SOC analyst (На синтетике 0 ложных эскалаций, на реальном потоке 39 из 100: как я делал локального L1-аналитика SOC)"
lang: "RU"
org: "Habr-Author"
source_type: "Practitioner post"
date: "2026-11-06"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-11-06-zero-false-escalations-on-synthetic-data-39-of/"
original_url: "https://habr.com/ru/articles/1104570/"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "A solo security engineer reports a fine-tuned local SOC triage model with before-and-after tables on held-out real alerts, open code and weights, and an explicit list of what the numbers do not prove."
summary_basis: "full-text"
original_minutes: "4"
tags: ["ai technology", "security operations", "small local models", "fine tuning on real noise", "deterministic pre filtering"]
---

# Zero false escalations on synthetic data, 39 of 100 on real traffic: building a local L1 SOC analyst (На синтетике 0 ложных эскалаций, на реальном потоке 39 из 100: как я делал локального L1-аналитика SOC)

Tier A (Strong evidence): A solo security engineer reports a fine-tuned local SOC triage model with before-and-after tables on held-out real alerts, open code and weights, and an explicit list of what the numbers do not prove.

Summary based on the full text.

## In short

A 6 November 2026 Habr case study on "kezekshi", a local first-line SOC assistant built on a fine-tuned Gemma 4 E2B that runs on a laptop with 8 GB of video memory and keeps logs inside the perimeter. It perfectly triaged a synthetic test set, then needlessly escalated 39 of 100 real Wazuh lab alerts and invented incidents. Training on 215 real benign cases fixed that, and a deterministic allowlist now settles almost all traffic before the model is called. The author keeps it in assist mode because recall on real attacks has not been measured.

## Key numbers

- 55 of 55: correct verdicts on the synthetic test set, with no false escalations
- 39 of 100 to 0 of 100: needless escalations on held-out real lab alerts before and after adding 215 real benign cases to training
- 98.1%: share of 3,000 lab alerts settled by the deterministic allowlist before the model is called
- 9/20 to 0/20: malformed answer cards on unseen alert types after switching to a per-alert response schema
- 20/20 to 18/20: recall on new attack types after a prompt caveat about MITRE candidates; the caveat was removed

## What it reports

- Input is one alert as an evidence pack E1...En; output is a JSON card with a four-class verdict, claims citing evidence numbers, MITRE ATT&CK techniques, gaps and next checks.
- Hard negatives (SCCM PowerShell -enc, vssadmin from a backup, a scheduled Nessus scan) raised the negative set from 8 to 22 with 0 false escalations and no recall loss.
- A severity rubric dropped recall on unfamiliar types from 20/20 to 16/20 (all misses were OAuth consent); one prompt rule restored 20/20 without retraining. Thinking mode of the fine-tuned model hurt.
- On 100 held-out real alerts: v4 (synthetic only) matched 39/100 verdicts and escalated 39/100 needlessly; v5 (plus real noise) matched 96/100 with 0/100 needless escalations.
- On 3,000 lab alerts 98.1% were resolved without the model and 1.9% went to it, about 50 times fewer calls with the same auto-close decisions.
- v6 grew to 3,546 training examples but 9 of 20 cards on unseen types failed validation (the model wrote field name ev1 instead of E1). A per-alert Ollama schema cut that to 0/20 and lifted MITRE recall from 13/36 to 25/36; false escalations rose to 1/34 and 1/100.
- Listing MITRE candidates acted as a hint: the model described normal Outlook reads of Credential Manager as credential harvesting.
- Before publishing weights, 306 attempts produced no lab host names.

## How they got it

Fine-tuning and evaluation on synthetic cases, familiar types (76), unseen types (20) and 100 held-out real benign alerts from a closed lab, with tables of before and after each change. Code, schema, ATT&CK index and v6 weights (4.9 GB GGUF) are public; the dataset is not.

## Limits

- Recall on real attacks is unmeasured; the real set is noise only, and the 20/20 covers five attack families.
- 30 of 76 familiar and 63 of 100 real cases nearly duplicate training items.
- The real-noise reference labels are one analyst's judgement.
- Metrics cannot be reproduced from the public files.

## Why it matters for companies

The gap between 0 false escalations on synthetic data and 39 of 100 on real traffic is a clean demonstration of why pilots need shadow runs on real data before anyone trusts a benchmark. The design choices transfer to any triage task: route deterministic cases around the model, force citations that code can verify, and constrain output with a per-item schema rather than retraining. Small local models are viable where data cannot leave the network, if a person still reviews what reaches the model.

## Key passages in translation

> Она зря подняла на вторую линию 39 из них и к некоторым дописала инцидент, которого не было.

It needlessly escalated 39 of them to the second line, and for some it added an incident that never happened.

> Список кандидатов работает как намёк на атаку.

The list of candidates acts as a hint that there is an attack.

> Как автономный решатель эскалаций не проверен.

As an autonomous decision-maker on escalations, it has not been tested.

Original: https://habr.com/ru/articles/1104570/