The model labels, the code scores: how AI grading of the Russian social studies state exam works (Модель размечает, код считает: как устроена ИИ-проверка ЕГЭ по обществознанию)
A · Strong evidence A teacher and state-exam expert documents two years of building an exam-grading bot with code, schemas, failure handling and a per-task cost table from a 393-answer run, so the design and numbers can be checked and reused.
Summary based on the full text·Original: about 10 minutes to read
In short
A 6 November 2026 Habr post by a social studies teacher and exam expert who has spent about two years on a Telegram bot that grades the essay part (tasks 17-25) of Russia's state exam (EGE) in social studies. The central rule: the model does not assign the score. It fills a JSON schema marking which examples are concrete and which judgements are correct; ordinary code recomputes points per official criteria, and rechecks can only lower the score. The post covers the per-task split of work, failure handling, cost and why the author chose code rules over fine-tuning. Accuracy measurement is promised for part two.
Key numbers
- $8.87cost of the full November run: 393 answers, 497 model calls
- $0.023 and 9 saverage cost and model time per graded answer (1.26 calls)
- 65.8% vs 54.5%agreement with the expert on the same 202 answers, Claude versus YandexGPT
- 185 of 185 (was 147)reference plans scoring full marks on criterion K1 after the synonym fix, with synonym headings
- 16 of 110silent zeros on task 25 when the model failed, before failures were flagged
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Each task has its own plugin; there is no universal prompt. Task 17 needs 942 lines of code, tasks 24 and 25 about 6,800-7,200 each: harder criteria mean more code around the model, not longer prompts.
- Code can only lower the model's score on all tasks but one. On task 20 code can raise it, because the model kept zeroing answers where the rule deducts one point.
- Task 19 recheck runs only on explicit text markers and rejects an example only when at least two of four negative flags agree, because a broad first version caused underscoring.
- Task 24 is scored fully by code (lemmatisation with pymorphy2 against 185 reference plans); a model finding can deduct a point only after four filters, including an exact quote of at least 8 characters and an independent verifier run. Two identical runs of one model shared the same false error on reference plans.
- Failures (no slot within 90 seconds, max 6 concurrent requests, network error after three retries, refusal, invalid JSON) set an
ai_unavailableflag and show no score; the November run on 393 cases had no silent zeros. Tasks 21, 23 and the website path of 24 still fall back without a flag. - Cost per answer ranges from $0.010 (task 24, website path) to $0.031 (task 25); prompt cache covers 54% of input overall, 0% on tasks 17, 21 and 22.
- Stack: Python, FastAPI, Redis, SQLite and PostgreSQL, Claude Sonnet 5.5 on one server; about 33,000 lines of core code. Implementation is largely delegated to Claude Code; the author writes criteria, reference markup and specs.
- Switched from YandexGPT in July 2026; YandexGPT also refused ordinary exam questions, which became zero scores. The first systematic measurement in spring 2026 showed the bot overscoring about one answer in three.
How they got it
First-hand engineering account with code excerpts, JSON schemas and a cost and latency table from a November 2026 run on 393 answers. The model comparison uses 202 answers graded by an expert. Student answers are not stored, so accuracy on real users cannot be measured.
Limits
- Single author; accuracy and measurement method are deferred to part two.
- Test sets are hundreds of answers, not thousands, and graded against the author's own expert judgement.
- Bot path cost for task 24 and end-to-end latency in production were not measured.
- No automatic failover to the backup provider.
Why it matters for companies
The pattern transfers to any rule-bound evaluation: compliance checks, claims review, quality scoring. Let the model extract and label, let deterministic code apply the rules, and allow rechecks to move scores in only one direction. The post also shows two traps worth copying into test plans: silent defaults that turn outages into plausible results, and treating two runs of the same model as independent opinions. For rules that change yearly, editing code was cheaper than retraining.
Key passages in translation
Russian original
главное, чему научил меня этот проект: модели нельзя поручать ставить балл.
The main thing this project taught me: you cannot let the model assign the score.
Russian original
Чем сложнее критерии, тем больше кода вокруг модели, а не длиннее промпт.
The more complex the criteria, the more code around the model, rather than a longer prompt.
Russian original
Два одинаковых ответа одной модели — это не два независимых мнения.
Two identical answers from the same model are not two independent opinions.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com