From operator RAG to a RAG platform for a large bank
От RAG для операторов до RAG-платформы для крупного банка
B · Useful, limited First-hand DevOops conference talk by the Alfa-Bank team that runs the system, with concrete load, latency, cache and hardware figures, but no controlled measure of answer quality or handling time.
Summary based on the full transcript·Original: about 37 minutes to watch
In short
A 36-minute DevOops talk by Mark Kuznetsov (head of an MLOps services centre) and Alexey Fateev (team lead, RAG and vector data platform) at Alfa-Bank, published on 9 October 2026. They describe how a retrieval-augmented answer tab for contact-centre operators grew into a bank-wide platform for RAG pipelines and model serving. The system summarises answers from the internal knowledge base for 12,000 active users and 200,000 queries a day on two GPUs. Most of the talk covers what broke: serving infrastructure, latency targets, a permission-aware cache, power supply and model upgrades.
Key numbers
- 12,000active users, mostly contact-centre operators, out of 75,000+ with access
- 200,000queries a day served on two GPUs at about 7 RPS
- ~27%daily cache hit rate after caching hashes of the source-article IDs
- 5 s to <4 sresponse time after tuning the vLLM batch parameter, with RPS per model up from 3 to almost 8
As reported by the source.
Russian·Video·opens youtube.com
What it reports
- The original hypothesis, about 18 months before the talk: cut average customer handling time by at least 30 seconds without lowering quality.
- Average operator session went from about 5 minutes with manual article search to about 4.5 minutes with the summarised answer.
- The service-level target was 5 seconds per answer. FP8 quantisation gave a speed-up with no quality difference the team could detect.
- Tuning vLLM's maximum-sequences (batch) setting moved throughput from 3 RPS to almost 8 per model and response time to under 4 seconds.
- Caching evolved in three steps: query-only cache (~20% hit rate, public articles only), query plus user ID (hit rate collapsed), then hashes of the source-article IDs behind each answer (almost 30%, about 27% daily). Target is 50% with semantic matching.
- The bank runs more than 800 classical ML models in production; the old Seldon setup built one image per model and Seldon's licence change ruled out commercial open-source use. They moved to KServe with weights stored outside the image.
- Failures: one of two power feeds to a GPU rack was switched off and cards dropped from about 300 W to 130 W; index streaming silently stopped for 2 weeks; upgrading from Gemma 2 to Gemma 3 raised answer length and latency while user likes rose.
- Budget: 8 GPUs planned, 2 needed for the LLM; embedders and rerankers run on GPU slices.
How they got it
Practitioners' account of a production system started in summer 2024, with figures from their own dashboards and load tests using NVIDIA's GenAI-Perf on production data. Model choice relied on public arena rankings checked against internal comparisons. The digest is based on YouTube auto-captions of the talk.
Limits
- No reported measure of answer accuracy, error rate or the effect on customer outcomes; quality evidence is operator likes and internal labelling.
- The video description mentions scaling to 6,000 operators while the speakers cite 12,000 active users; the talk does not reconcile the two.
- Recorded at a 2025 conference and published in October 2026, so model choices (Gemma 3, testing T-Pro 2) are dated.
Why it matters for companies
This is a rare look at the operating cost of an internal knowledge assistant at scale: two GPUs carried 200,000 daily queries once batching and quantisation were tuned. Access rights shaped the architecture as much as the model did, because a shared cache cannot leak answers built from restricted articles. The biggest incidents were plumbing - power, stale indexes, a model upgrade that changed latency - which argues for monitoring before scaling. The 30-second handling-time goal shows the kind of narrow, measurable target that keeps such a project fundable.
Key passages in translation
Russian original
гипотеза примерно полтора года назад была в том, что мы при помощи RAG-платформы сможем оптимизировать работу операторов... чтобы хотя бы на 30 секунд быстрее среднее время обслуживания клиента было.
The hypothesis about a year and a half ago was that with a RAG platform we could optimise the work of operators... so that average customer handling time would be at least 30 seconds shorter.
Russian original
мы не можем допустить такого, чтобы оператор видел ответ на основе статей, к которым у него нет доступа. И поэтому нам здесь пришлось, собственно говоря, изобретать велосипед.
We cannot allow an operator to see an answer based on articles they have no access to. So here we had to reinvent the wheel.
Russian original
Мы теперь кэшируем хэши от айдишников статей, на базе которых был дан ответ. И это действительно у нас сработало.
We now cache hashes of the IDs of the articles the answer was based on. And that actually worked for us.
Russian original
У нас модель новая начала сильно больше думать, сильно больше отвечать. И это понравилось пользователям... но при этом у нас время ответа сильно увеличилось.
Our new model started thinking much more and answering at much greater length. Users liked it... but our response time went up sharply.
Russian original
для 12.000 сотрудников и более чем 200.000 запросов в день нам нужно две карточки... Наши расчёты в Экселе сошлись с реальностью.
For 12,000 employees and more than 200,000 queries a day we need two cards... Our Excel calculations matched reality.
Summary based on the full transcript. Read the original for the full argument and data.Watch the original on youtube.com