How we put LLMs into Avito moderation
Как мы внедряли LLM в модерацию Авито
A · Strong evidence First-hand production write-up from Avito's DS team lead with the full architecture, model comparison, prompt-optimisation method and a production precision figure on a flow of hundreds of thousands of listings a day.
Summary based on the full text·Original: about 8 minutes to read
In short
A Habr case (22 October 2026) by Ruslan Sakaev, data science team lead at Avito, the Russian classifieds marketplace. Avito checks more than 20 million listings a day and had automated over 95% of checks with classic ML; the remainder, hundreds of thousands a day, went to several hundred human moderators. The team moved those checks to self-hosted LLMs. The LLM now handles about 70% of that residual flow at about 94% precision. Most of the post covers what had to be built around the model to hold up in production.
Key numbers
- 20M+listings Avito checks a day; over 95% of checks were already automated before the LLM
- ~70%share of the residual human-review flow now handled automatically by the LLM
- ~94%precision of LLM decisions on that flow
- 76%F1 of the fine-tuned Avision-8B, above the default Qwen3.5-35B-A3B and a 397B model out of the box
- +14%F1 gain from the raw policy text as prompt to agent-run prompt optimisation
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Policies run up to 200 pages. The team compared RAG, full policy in context, and a compact prompt derived from the policy; the compact prompt gave the best metrics and avoided a second engineering system.
- Data constraints (personal data, legal risk) require a closed perimeter, so models are self-hosted.
- Model choice: bigger models score higher, but the fine-tuned Avision-8B reaches F1 76%, above the default Qwen3.5-35B-A3B and a 397B model out of the box. Mass rollout uses the 35B model untuned; the 8B is fine-tuned only where quality drops.
- Prompt work went through five stages: raw policy, human-condensed instruction, LLM-rewritten prompt, automatic prompt search, then agents with MCP servers and skills that analyse errors and run optimisation. F1 rose 14% from the first to the last stage.
- Prompt search uses an adapted GEPA (token probabilities, capped change per step, internal instructions translated to Russian); MiPRO, ProTeGi, SAMMO and others gave no material gain.
- Fine-tuning: LoRA on language layers only; training uses a pool of candidate prompts weighted by quality.
- Production: classic ML flags a suspected violation and picks the prompt; a GuardRails layer sends suspected prompt-injection or jailbreak inputs straight to humans; low-confidence LLM decisions also go to moderators.
- Inference on vLLM, chosen for prefix caching: the long policy prefix is identical across listings. Output is one deterministic token (True/False), temperature 0.
- A prompt admin panel lets non-technical managers version prompts, run experiments and start retraining.
How they got it
Internal evaluation on Avito's own labelled checks, presented as F1 charts by model and by prompt stage, plus production precision on live flow. The post names the methods and tools used. Sample sizes and per-violation results are not given.
Limits
- Self-reported by the company; test-set sizes, recall and per-category results are not disclosed.
- The 70% and 94% figures apply to the residual flow, not to all moderation.
- The remaining 30% of hardest cases is not yet automated.
Why it matters for companies
This is a reusable pattern for any rule-heavy review task, such as compliance checks, claims or content policy: keep classic models for recall, let an LLM decide on the suspicious slice, route uncertain and adversarial cases to people. Two lessons transfer directly. A small fine-tuned model can beat a far larger general one on a narrow task. And prompts should be treated as tunable parameters, optimised against labelled data rather than written by intuition, with business owners able to manage them.
Key passages in translation
Russian original
Наша дообученная Avision-8B показывает F1 76%: это лучше, чем у Qwen3.5-35B-A3B, которую мы используем по умолчанию, и даже лучше, чем у 397B-модели.
Our fine-tuned Avision-8B reaches F1 76%: better than Qwen3.5-35B-A3B, which we use by default, and even better than the 397B model.
Russian original
Сейчас LLM автоматически обрабатывает около 70% объявлений, которые основной пайплайн направляет на дополнительную проверку. Precision на этом потоке около 94%.
The LLM now automatically handles about 70% of listings that the main pipeline sends for additional review. Precision on this flow is about 94%.
Russian original
В продакшене LLM работает как часть системы.
In production the LLM works as one part of a system.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com