NER for lawyers without an A100: matching Ministral 14B with a model almost 10x smaller (NER для юристов, у которых нет A100: как мы догнали Ministral 14B моделью почти в 10 раз меньше)
A · Strong evidence Daniil Sheremet of Agentic Lab reports a distillation experiment with a held-out golden set of real documents, a separate judge model, per-class precision and recall, measured latency and a documented failed approach; method shown and numbers checkable.
Summary based on the full text·Original: about 15 minutes to read
In short
A Habr post (3 November 2026) by Daniil Sheremet of Agentic Lab, which builds Agata, an assistant for lawyers. Law firms will not send case files to external APIs, and a bigger chat model had taken the whole A100. The team distilled 9-class entity extraction into Qwen2.5-1.5B with a LoRA adapter and nearly matched Ministral 3 14B on real documents at 2.2 times the speed on a cheaper card. Rebalancing classes with synthetic data did not help. Generating the missing document types did.
Key numbers
- 0.70 vs 0.71macro-F1 of the distilled Qwen2.5-1.5B vs Ministral 3 14B on a 145-document golden set
- 14.8 s vs 32.7 stime per document, student on an A5000 vs Ministral on an A100 (2.2x faster)
- about 5,500 / about 45,000real client documents used, and teacher-labelled document-class training examples
- 93.6%share of training examples with an empty answer for the correspondence class
- 82xgap between the most and least frequent class (26,607 participant entities vs 325 correspondence)
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Nine classes (parties, document details, contracts, object of dispute, performance, payments, breaches, damages, correspondence), each extracted by a separate call returning JSON. Nine calls of 3.6 s on the A100 gave about 33 s per document.
- Reason for on-prem: legal privilege, client trade secrets and personal data; Russian law 152-FZ requires citizens' personal data to be stored in Russia.
- Data: about 5,500 real documents from client cases, used with consent; median length 2,380 characters, with the longest 10% holding 61% of the text. Documents over the 16k-token training window were dropped.
- Distillation: a teacher labelled about 45,000 document-class examples; the student is Qwen2.5-1.5B-Instruct with an adapter of about 80 MB.
- Four classes were empty in 84-94% of examples (correspondence 93.6%), so the first model learned to answer "nothing".
- Synthetic rebalancing (coherent breach-claim-payment stories inserted into real documents) moved precision against recall but left F1 flat on the golden set.
- Splitting golden-set errors by document type showed failures in motions and claim letters, which were nearly absent from training. Generating whole documents of those types added 23 points precision and 22 points recall on performance events and 34 points precision on correspondence.
- Final: macro-F1 0.70 vs 0.71. The student wins on performance events (0.56 vs 0.32), damages and correspondence; it loses on contracts (precision 0.79, recall 0.62), objects and breaches.
- Memory: about 3 GB of weights vs 28 GB; a 4,000-token document needs 112 MB of KV cache, so about a hundred documents fit in 15 GB.
How they got it
Training and validation split by text hash; evaluation on 145 real documents not used in training or synthesis, scored by a separate judge model rather than the teacher, to avoid self-enhancement bias. Latency measured per call, unbatched.
Limits
- A throughput comparison of both models on the same A100 has not been run yet.
- Part of the performance-event gain may come from the student learning the team's own labelling conventions, as the authors note.
- One golden set of 145 documents, scored by an LLM judge rather than lawyers.
Why it matters for companies
For firms with confidential documents, the post shows a small on-prem model can do structured extraction close to a model ten times its size. The general lesson applies to any internal model: validation cut from the training corpus inherits its gaps, so decisions need an outside test set, and errors should be broken down by input type as well as by output class. Synthetic data helped where it created missing kinds of input and did not help where it only reweighted existing ones.
Key passages in translation
Russian original
Балансировать нужно было не классы, а типы документов.
What needed balancing was document types, not classes.
Russian original
Валидация, нарезанная из того же корпуса, что и обучение, наследует все его перекосы.
Validation cut from the same corpus as training inherits all of its skews.
Russian original
Синтетика хорошо создаёт то, чего в данных нет, и плохо исправляет пропорции того, что есть.
Synthetic data is good at creating what is missing from the data and poor at fixing the proportions of what is there.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com
- Tags
- Distillation