How we built and improved an LLM layer for talking to buyers (Как мы строили и улучшали LLM‑слой для общения с покупателями)
A · Strong evidence A marketplace support team reports each step of building an LLM decision layer with comparison tables on the same 500 manager-labelled dialogues, including failed variants and the finding that their first reference labels were wrong.
Summary based on the full text·Original: about 12 minutes to read
In short
A 7 November 2026 Habr case by a support team at a marketplace seller (author Krioneee). Chats opened after low ratings often stall, and managers sent reminders by hand. The team automated reminders, then added an LLM to decide whether to continue, stop or hand over to a manager. The decision layer reached 84% agreement with a manager's labels, up from 74%, with 38% fewer errors at about the same cost. The most useful lessons are about evaluation: a high overall score hid many wrong "continue" decisions, the AI-made reference labels were wrong 40% of the time, and one process signal beat several model swaps. The result is experimental and not yet in production.
Key numbers
- 605 of 655specialists' closing messages that also invited the buyer to write again, which defeated keyword rules
- 120 of 200 (60%)agreement of the early AI-assistant reference labels with the support manager; the model matched the manager in 170
- 74% to 84%action agreement with manager labels, single Qwen versus the chosen two-judge cascade, on 500 dialogues
- 410 of 500 (82%)agreement of a one-line rule (no manager reply yet: continue; manager replied: stop), better than earlier model variants
- about 0.03 roublesaverage cost per decision for the chosen cascade, roughly the same as the single cheap model
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Rollout of real sends went 10%, 30%, 50%, 100%, with dry run as default and sends held when the model layer failed.
- Keywords failed: 605 of 655 closing messages also asked the buyer to write again. First LLM layer (Qwen3-30B-A3B on Cloud.ru) answered in about 0.95 s; quality was not measured then.
- Of 310 buyers who replied after reminders: 155 waited for the seller, 85 were irritated or conflicted, 60 thanked or resolved, 10 refused contact.
- Prompt r3 scored about 92% action accuracy on 1,705 slices while 35 of 145 permitted continuations (24%) were wrong; repeated runs changed some decisions.
- A post-model filter cut wrong continuations to 0-7% but dropped about 40% of valid ones; not shipped. Eight context messages did no better than four. The system prompt was about 1,320 of 1,520 input tokens.
- Manager check on 200 examples: model matched 170, early AI-assistant labels only 120.
- Cascade: Qwen3-30B-A3B and gpt-oss-120b as judges, DeepSeek-V4-Flash as arbiter; judges agreed about 78% of the time with r3, so the arbiter ran in about 21%.
- Adding a "manager already replied: yes/no" flag plus a seven-class Russian prompt gave the chosen cascade: 84% agreement, errors 130 to 80, wrong continuations 85 to 50, lost valid continuations 45 to 30.
How they got it
Offline evaluation on dialogue slices, first against labels made by an AI assistant, then against 500 dialogues labelled by one support manager (345 stop, 155 continue). Each variant is reported with action accuracy, share of wrong continuations and lost valid continuations, plus cost per decision.
Limits
- Experimental; the chosen cascade still needs testing on new dialogues and in production.
- One manager's labels are the reference, and the manager was inconsistent on some borderline cases.
- Run-to-run variance means small differences (such as English versus Russian prompts) are not significant; the authors say so.
- Marketplace and company are not named.
Why it matters for companies
The case is a template for evaluating any LLM decision step. Pick the error that costs most (here, an unwanted message) and track it separately from overall accuracy. Check the reference labels with the people who own the process before tuning prompts against them. Try simple process rules as a baseline: here a one-line rule beat model upgrades. A cascade of cheap judges with an occasional arbiter kept cost flat.
Key passages in translation
Russian original
Хороший общий показатель скрывал заметную долю ошибок в действии, которое нас особенно интересовало.
A good overall score hid a noticeable share of errors in the very action we cared about most.
Russian original
Промпт мог лучше соответствовать меткам ассистента и при этом не становиться полезнее с точки зрения менеджера.
The prompt could fit the assistant's labels better without becoming more useful from the manager's point of view.
Russian original
Одна замена модели не дала наблюдаемого прироста точности действия.
Swapping the model alone gave no observable gain in action accuracy.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com