Replacing GPT-4 as judge
KARPOV.CONF 2026: локальный LLM-as-a-judge
B · Useful, limited A conference talk by an executive director on Sber's GigaChat SFT team giving a sensible judge-selection checklist and a first-hand comparison, but the agreement scores for the chosen judges are not shown and most bias figures come from the literature.
Summary based on the full transcript·Original: about 31 minutes to watch
In short
A 30-minute talk at KARPOV.CONF 2026 (28 October 2026) by Daniil Smirnov, executive director at Sber, who runs a supervised fine-tuning team on GigaChat. He argues that GPT-4, still the default judge in public LLM-as-a-judge benchmarks published in 2023-24, is now too expensive, unstable and weak to rank strong 2026 models. His team compared recent open-weight models deployed locally against human labels and found GLM 5.1 and MiniMax M2.7 beat GPT-4 Turbo on agreement with their assessors, at several times lower cost.
Key numbers
- ~$400cost of scoring one model checkpoint on the full public benchmark suite with a GPT-4 judge
- $180cost of the GPT-4 Turbo baseline run in Sber's own judge comparison; the local judges came out several times cheaper
- ~80%GPT-4 agreement with human labels, about the same as human-to-human agreement (literature figure)
- 5-17%gain in judge-human agreement from writing explicit scoring criteria into the prompt (GPT-4 itself gained only 0.3%)
- 13-30%share of pairwise comparisons where a judge picks the first answer when both are acceptable (positional bias, literature)
As reported by the source.
Russian·Video·opens youtube.com
What it reports
- Human assessors on a domain benchmark (medicine, law, science) cost from several thousand to several tens of thousands of dollars per model, so they cannot be used for routine evaluation of every checkpoint.
- Public judge benchmarks hold from a few hundred to a couple of thousand questions each. Scoring one checkpoint on all of them with a GPT-4 judge costs about $400, and a training run saves several to dozens of candidate checkpoints.
- Other GPT-4 problems: OpenAI retires model versions, so old scores cannot be compared with new ones and the whole model pool has to be re-scored; style-shift bias favours models from GPT-4's own era; ranking between two strong 2026 models is unreliable; runs are not reproducible.
- Explicit scoring criteria in the prompt raised judge-human agreement by 5-17% in one cited study (GPT-4 gained only 0.3%). Reference answers must be strong, or every tested model wins and ranking collapses.
- Biases to handle: positional (first answer picked in 13-30% of cases; fix by swapping order and counting only consistent verdicts), verbosity, self-family preference, style shift, and language (a judge without the target language in training data scores Russian answers badly).
- Fine-tuning a custom judge was rejected: prompt changes gave more gain than weight changes, fine-tuning is costly in compute and labelling, a stronger open model is likely within about six months, and it risks catastrophic forgetting of other domains.
- Result: GLM 5.1 and MiniMax M2.7 (April releases) agreed with Sber's human labels better than GPT-4 Turbo. The GPT-4 Turbo baseline cost $180; all local judges cost less.
How they got it
Candidate judges were chosen by the checklist (open weights, deployed locally, large, recent, trained on the target language, strong at instruction following, explicit criteria, different family from the model judged). They were scored by share of agreement with Sber's in-house assessor labels across open and closed benchmarks, with GPT-4 Turbo as baseline. In Q&A the speaker said the cost figures use average cloud GPU rental for the same token volume as the GPT-4 Turbo run, not Sber's own amortised hardware and power costs.
Limits
- The agreement scores for the winning judges and the exact cost ratio are not stated in the talk ("several times" cheaper); only the GPT-4 Turbo baseline cost is given.
- The 80%, 5-17% and 13-30% figures are cited from published studies, not measured by the team.
- Benchmarks are partly closed (internal perimeter), so the comparison cannot be repeated. GigaChat's own position was withheld when asked.
- Source is YouTube auto-captions: model and team names are garbled in places (for example "Minimx M2.7", "sфT и DE").
Why it matters for companies
Any team that evaluates LLM outputs at volume (model choice, prompt changes, agent regressions) faces the same cost and drift problem. The practical points carry over: pin a judge you control so scores stay comparable over time, write explicit criteria, swap answer order, use a judge from a different model family, and check agreement against a small human-labelled set before trusting it. The claim that prompt work beat fine-tuning is a useful default for teams tempted to train their own evaluator.
Key passages in translation
Russian original
Ага, на самом деле правильный ответ около 400 долларов.
Right, the actual answer is about 400 dollars (to score one model on the full benchmark suite with GPT-4).
Russian original
Open AI просто берёт и выключает свои модели.
OpenAI simply switches its models off.
Russian original
И мы поняли, что изменение весов даёт меньший прирост, нежели изменение парамптов.
And we found that changing the weights gives a smaller gain than changing the prompts.
Summary based on the full transcript. Read the original for the full argument and data.Watch the original on youtube.com
- Tags
- LLM as judge