45% of our AI meeting reports scored 3 or lower: how we built metrics and what we found (45% наших ИИ-отчетов получали тройку и ниже: как мы считали метрики и что нашли)
A · Strong evidence Atom's AI LAB product team publishes an internal quality audit with sample sizes, a coded root-cause breakdown, a before-and-after test and an explicit list of the test's weaknesses.
Summary based on the full text·Original: about 11 minutes to read
In short
A Habr post (6 November 2026) by Yulia Efimova and Maya Azarova of AI LAB at Atom on building metrics for an internal AI portfolio (a meeting transcriber, a company chat, a Jira monitor and a GitLab code reviewer), then auditing report quality. An LLM judge scored 45% of working-meeting reports at 3 or lower. A manual read of 56 showed four in five bad scores came from the prompt or the wrong report type, not from the judge or the transcript. Tightening "do not invent tasks" rules made the model drop real tasks.
Key numbers
- 168 of 374 (45%)AI reports on working meetings scored 3 or lower out of 5 by the LLM judge, 3 July to 22 September 2026
- 45% / 36% / 12% / 7%main cause in 56 hand-read reports: summarisation prompt / not a working meeting / poor source data / judge error
- 3.4 to 6.8average value actions per active person on the transcriber, April to August 2026
- under 100 rublescost of one AI meeting report; a one-hour meeting is processed in about 15 minutes
- 2.62 to 3.50average score on the 8 prompt-caused meetings after the fix (median 3 to 4)
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Transcriber reach: new people per month rose from 5 in April to almost 30 in June and July, then about 13 in August; close to 100 staff had taken at least one action by end of August.
- Value actions per active person rose from about 3.4 to 6.8; since 24 August over 80% of visitors take at least one action; stickiness (DAU/MAU) 10-25% on most working days.
- Metric layers: universal usage metrics, a cross-service layer, efficiency and reliability (calls, cost per request and per value event, error rate, p50/p95 latency, SLO), and a business layer estimating hours saved from work volume and agreed baselines.
- One report costs under 100 rubles; an hour of meeting takes about 15 minutes to process.
- Judge in Langfuse: 424 reports, 374 from working meetings; 168 scored 3 or lower. Research-meeting reports score 4-5 because their format was designed with researchers.
- 56 read in full (all 3 ones, 22 of 37 twos, 31 of 128 threes): average 2.9 problems per report; main conclusion missing in 28; junk tasks 15; contradictory sections 15; "no decisions" when there were some 10; invented decisions 4.
- Fixes: prompt rewritten defect by defect, task detection by explicit signals, at most five open questions, priority labels removed, separate report types for test drives, demos and lectures, and a judge that first checks whether there was material for a report.
- Re-test on 17 meetings: average 2.53 to 3.35 (median stays 3); 8 of 17 reached 4 or 5.
How they got it
Usage from admin logs and event definitions agreed per product; quality from an LLM-as-judge with per-meeting-type criteria (details under NDA); root causes from a manual read with one main cause recorded per report; reference summaries written by a stronger model.
Limits
- The judge changed together with the prompt, so before and after scores are only roughly comparable.
- Only the weakest reports were re-tested, each run once; 17 meetings is too few for significance.
- Judge scores were not checked against human ratings, and reference summaries were model-written.
Why it matters for companies
Most firms rolling out internal AI tools count logins. This team shows a cheap way to get further: define a value action per product, add counter-metrics for errors, and read a sample of bad outputs by hand before changing anything. The finding that over-strict "do not invent" rules make the model drop real commitments is a general lesson for any summariser that feeds task lists. Different meeting types need different templates.
Key passages in translation
Russian original
Получается, что четыре из пяти плохих оценок дал не придирчивый оценщик и не сломанный транскрипт, а сам промпт и неподходящий тип отчета.
So four out of five bad scores came neither from a fussy judge nor from a broken transcript, but from the prompt itself and an unsuitable report type.
Russian original
Правило «лучше не написать, чем выдумать» звучит разумно, но без противовеса модель уходит в другую крайность.
The rule "better to leave it out than make it up" sounds reasonable, but without a counterweight the model goes to the other extreme.
Russian original
Свободные комментарии полезнее оценок по шкале.
Free-text comments are more useful than ratings on a scale.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com
- Tags
- AI enablement