LLM adoption without rose-tinted glasses: 500 engineers
Внедрение LLM без розовых очков: честный разбор гипотез, ошибок, метрик на масштабе 500 инженеров
A · Strong evidence First-hand report from Raiffeisenbank's domain agile coach on a year of AI adoption across about 50 teams, with a measured usage funnel by intervention, flow metrics as the goal, and negative results reported plainly.
Summary based on the full transcript·Original: about 40 minutes to watch
In short
A conference talk (uploaded 15 October 2026) by Marat Kinyabulatov, domain agile coach at Raiffeisenbank Russia, on about a year of AI adoption across roughly 50 teams and 500 people in the bank's operational core. His alternative title: three hypotheses about AI were tested and none reduced development time. The value is the funnel: which interventions turned occasional chat use into regular tool use, and the finding that lead time, the goal metric, did not move.
Key numbers
- 50% vs 3%share of people using any LLM chat monthly versus share using the API (agents, autocomplete) after awareness campaigns
- 88%workshop attendees who went on to use the API at least monthly after a 90-minute hands-on session
- 95%six-month retention of people who started using the API
- 35-40%share of a team pulled into API use when a tech lead attended, versus 10-20% for an ordinary engineer
- 86% / ~65%current share using any LLM / using the API
As reported by the source.
Russian·Video·opens youtube.com
What it reports
- Request and token counts were dropped as adoption metrics because agents and autocomplete inflate them. The bank tracks people: monthly use, use on 50%+ of working days (W50) and on about 80% (W80).
- Hypothesis 1, give people LLMs: awareness campaigns took monthly use from about 5% to about 50%, with a self-reported gain of 15-20%. Only 3% used the API; the rest used chat. Lead time did not change.
- Hypothesis 2, coding agents: the main barrier was not knowing how to install and apply the tools. 90-minute hands-on workshops in small groups, on real tickets, converted 88% of attendees to monthly API use, 62% to use on more than half of working days and 35% to daily use. Six-month retention was 95%.
- Spread: an ordinary engineer who attended pulled in 1-2 colleagues (10-20% of a team), a tech lead 35-40%, and a tech lead who ran regular sharing sessions about 55%.
- Flow metrics stayed flat: defect rate and throughput unchanged; lead time fell in some teams and rose in others. His reading: lead time is set by the slowest stage and the hand-offs, so faster coding alone changes nothing end to end.
- Hypothesis 3, AI-first teams on a greenfield service: two developers delivered in a month roughly what five had, but lead time still did not fall. Humans became the review bottleneck; small end-to-end teams carry high cognitive load and burnout risk.
- What did not work: webinars, mailings and documentation (5-10% uptake), mandates. What worked: workshops, hackathons, small groups, scripted environment setup. Current state: 86% use some LLM, about 65% the API; chat use has plateaued.
- The three people from the five-person team moved to another feature team; no one was laid off.
How they got it
Usage measured through corporate logins and tool telemetry, tracked per person over time; flow metrics (lead time, throughput, defect rate) from the delivery system. Change framework: ADKAR. Results are presented as a sequence of hypotheses with outcomes, including the failures.
Limits
- Internal figures from one bank, shown on slides only; no external audit.
- Lead-time effects are not tested statistically, and team-level changes have many causes.
- The AI-first result is a single pilot on a non-critical greenfield service.
- He leaves open whether any of it pays back.
Why it matters for companies
This is one of the few adoption reports that separates chat tourism from working use and shows which intervention moved which number. Hands-on workshops and tech leads did most of the work; broadcasts and mandates did little. The flat lead time is the warning for anyone promising productivity: speeding up one stage does not shorten delivery unless the hand-offs, review and neighbouring teams change too.
Key passages in translation
Russian original
мы проверили три гипотезы про AI, ни одна из них не уменьшила время разработки
We tested three hypotheses about AI, and none of them reduced development time.
Russian original
Чат это как туризм. Просто пришёл, что-то сделал, но системно в свою работу не встроил.
Chat is like tourism. You just came, did something, but did not build it into your work in any systematic way.
Russian original
Никто ощутимо быстрее пока не стал.
Nobody has become noticeably faster yet.
Summary based on the full transcript. Read the original for the full argument and data.Watch the original on youtube.com