Artificial Analysis v5 and the benchmarker's dilemma
B · Useful, limited An opinion essay by Nathan Lambert (Interconnects), a well-placed open-model researcher, on the new Artificial Analysis index and the incentives of benchmark companies; informed and clearly argued, but commentary with no new data.
Summary based on the full text·Original: about 10 minutes to read
In short
A 28 October 2026 post by Nathan Lambert on Artificial Analysis's Intelligence Index v5, the most cited single-number ranking of frontier models. He describes the new weighting, argues that v5 measures where frontier models are heading (open-ended agentic work, science) rather than where most current economic use sits, and questions the move to private test sets. His wider point is about trust: one or two indices shape a very large market, and the leading index is paid for by the labs it scores.
Key numbers
- 25%weight of each of the four blocks in Artificial Analysis Intelligence Index v5: knowledge work, coding, science, general
- 8closed models scoring above the best open-weight model, Kimi K3.1 Pro, on v5
- 2-3 monthsLambert's expected horizon for the next open-model release cycle to close part of the gap
As reported by the source.
English·Web page·opens interconnects.ai
What it reports
- v5 weights four blocks at 25% each. Knowledge work: AA-Briefcase 15%, AutomationBench-AA 5%, GDP.pdf 5%. Coding: FrontierCode 2.0 15%, Terminal-Bench 4.0 10%. Science: Terminal-Bench-Science 15%, CritPt v2 hard subset 10%. General: AA-Omniscience v1.1 15% and HLE-Diamond 10%.
- On v5, eight closed models beat the best open-weight model, Kimi K3.1 Pro. Lambert thinks the previous index flattered open models, but that their decline is overstated; open models trail most on the science evaluations.
- He expects leading open labs (GLM 6, Kimi 3.5) to gain substantially at the next release cycle in 2-3 months.
- Private benchmarks reduce training to the test, but remove community bug-finding and require full trust in the benchmark builder.
- Artificial Analysis earns money from labs to verify their results; Arena does paid pre-release testing. He trusts non-profits, academics and government evaluators such as CAISI most for fairness, and startups like Artificial Analysis and Arena most for accuracy at the frontier.
- His remedy is more indices with different biases, for example coding-only ones, rather than one dominant number.
How they got it
Commentary based on Artificial Analysis's own announcement and published charts, the author's experience in model training, and his prior writing on open models. No new measurements.
Limits
- Opinion; the forecasts about open-model progress are the author's judgement.
- The author works in open-model research, which he does not hide but which shapes the framing.
- Index details are taken from the vendor's description.
Why it matters for companies
Model selection decisions and vendor pitches often cite a single index score. This post is a useful reminder of what such a number contains: a chosen mix of tasks, weighted by someone with a view and, in this case, a commercial relationship with the labs. For buyers the practical step is to check which components of an index match their own workload, read more than one index, and run their own task-level evaluations, especially when comparing open-weight models for routine, high-volume work where a frontier-weighted index may understate them.
Summary based on the full text. Read the original for the full argument and data.Read the original on interconnects.ai