Computer vision for customer claims
KARPOV.CONF 2026: Magnit Tech, CV для претензий
B · Useful, limited A first-hand production account from a Magnit Tech engineering lead with a clear pipeline and per-model accuracy figures, but the volume and automation numbers lack a period and baseline and appear to contradict each other.
Summary based on the full transcript·Original: about 23 minutes to watch
In short
A 23-minute talk at KARPOV.CONF 2026 (28 October 2026) by Karen (surname not given in the captions), head of a technology and engineering function at Magnit Tech, the tech arm of the Russian food retailer Magnit. Customers of Magnit's delivery service file claims with photos and text. A separate asynchronous service runs a set of computer vision models that identify the product, read barcodes, check image quality, catch duplicate photos and read expiry dates, then hand a verdict to operators. A vision-language model (VLM) is in pilot for defect detection. The business goal is to reject as many invalid or fraudulent claims as possible.
Key numbers
- ~65,000customer claims processed by the service (period not stated)
- 85-90%accuracy of the three-headed barcode model (barcode, digits, rotation)
- 90%expiry-date recognition accuracy on hidden test sets
- up to 20%claims filtered by barcode checks alone without an operator
- 180,000claim volume discussed for the next rollout
As reported by the source.
Russian·Video·opens youtube.com
What it reports
- Architecture: the CRM is synchronous, so the models run in a separate service; the customer is told the claim is under review while the models process it.
- Product identification: a proof-of-concept classifier covered a dataset of about one thousand items (captions read "1.00"). The business then asked for the full range, which runs to millions of items, so the team moved to metric learning: EfficientNet embeddings trained with ArcFace and nearest-neighbour search by cosine similarity in a vector database. The right item is usually in the top five, and the speaker said top-1 is generally good.
- Barcodes: one three-headed model reads the barcode, reads the digits and classifies rotation; results are cross-checked. Accuracy about 85-90%; on failure the customer is asked to re-shoot.
- Fraud and quality: an open-source image quality assessment model with a threshold, and duplicate detection with DINOv2 embeddings against the full database of claim photos.
- Expiry dates: a modified CRNN with CTC loss that outputs both a mask of the date region and the text; 90% accuracy on hidden test sets. Operators still double-check because errors carry legal and reputational risk.
- Volume: about 65,000 claims processed; "half" automated with AI, and of those 10% rejected automatically by rules. The stated goal is to raise automation to 30-50%. Barcode checks alone filter up to 20% without an operator. A rollout to 180,000 claims is under discussion.
- VLM pilot: takes the photo and the claim text, returns a verdict with a confidence score; running in hybrid mode, with full replacement of the model zoo by one VLM possible if results hold.
- Infrastructure (Q&A): own models on private cloud GPUs. Current models run on A100-class cards; VLMs need H100 or larger, so the cost gap is noticeable. Owning servers is not planned until inference gets cheaper. Nothing is released as open source.
How they got it
A practitioner describing his team's system. Accuracy figures are per model on internal test sets; the test sets, their size and the error mix are not described. Business metrics are owned by the product team, not the ML team, per the Q&A.
Limits
- Self-reported; no test-set sizes, no confusion data, no operator-time or money savings.
- The period behind "about 65,000 claims" is not stated, and "half automated" sits oddly next to a target of 30-50% automation; the talk does not resolve this.
- The library summary reads the 10% as a share of all claims; in the talk it is 10% of the AI-handled half.
- Source is YouTube auto-captions: numbers ("1.00" items), model names and the service name ("CRMS") are partly garbled.
- The host followed the talk with a promotion of a fast-track hiring programme into Magnit Tech for course graduates.
Why it matters for companies
Claims, returns and warranty handling are common candidates for automation in retail and consumer goods. The talk shows a practical shape: keep inference out of the synchronous CRM, split the problem into small checkable models (identity, barcode, quality, duplicates, date), keep an operator on the legally sensitive step, and measure success by invalid claims rejected. The move from a closed classifier to embedding search is the standard answer when the catalogue is too large to label. The A100 versus H100 point is a useful reminder that replacing a set of small models with one VLM can raise inference cost.
Key passages in translation
Russian original
Спойлер AI внутри сирма не живёт, поскольку CRM он синхронный.
Spoiler: the AI does not live inside the CRM, because the CRM is synchronous.
Russian original
На данный момент мы обрабатываем примерно 65.000 претензий. Уже половина автоматизирована AIМ, и из них 10% автоматически отсеивается нашей логикой.
At the moment we process about 65,000 claims. Half are already automated with AI, and of those 10% are filtered out automatically by our logic.
Russian original
Работает с точностью 90% на скрытых тестовых множествах, что достаточно хороший результат, но мы продолжаем его развивать.
It works at 90% accuracy on hidden test sets, which is a fairly good result, but we keep developing it.
Summary based on the full transcript. Read the original for the full argument and data.Watch the original on youtube.com