EBR-bench update
B · Useful, limited Short benchmark maintenance note from Epoch AI researchers with first-hand results, but small-scale experiments and few reported figures; useful practice rather than a flagship Epoch report.
Summary based on the full text·Original: about 8 minutes to read
In short
An update by Benjamin Ou and Avery Yen of Epoch AI, published 7 October 2026, on EBR-bench, a benchmark that tests whether models learn from experience by playing the board game Earthborne Rangers. GPT-6 Astra scored 100% by finding a card combination that grants unlimited turns, so Epoch banned the card. Under the ban Astra still leads by a wide margin. Multi-agent setups led models to try more decks but did not raise scores.
Key numbers
- 100%GPT-6 Astra score on the original EBR-bench, reached in over half its attempts via one card
- 20/21Astra's best result in objectives after the card was banned
- ~50%jump in Astra's average performance over the strongest previous models, under the ban
As reported by the source.
English·Web page·opens epoch.ai
What it reports
- A single human playthrough takes about 2-4 hours; a high score means completing up to 21 objectives within limited turns.
- Astra achieved a perfect score in over half its attempts. All its top scores used one card that, with others, allows indefinite turns. The top human baseline player used the same card; the #2 human scored 100% without it.
- Astra often found and fully exploited the card in under two playthroughs; the top human needed several warm-up playthroughs.
- Under the ban, Astra's best was 20/21 objectives, about a 50% jump in average performance over the strongest previous models, though still slightly weaker than top humans. Its fatigue management showed no improvement over time.
- The ban did not significantly change scores for three recent models. Re-scored models: Claude Fable 5.1, Claude Opus 5, GPT-5.6 Sol and GPT-6 Astra.
- Multi-agent runs (up to four subagents, same 10 playthroughs) increased deck exploration for most models, significantly only for Claude Opus 5, with no consistent effect on scores. Eight subagents made no significant difference.
- Epoch expects the benchmark to be fully saturated within a few months and cut sample size from 10 to 5 runs.
How they got it
Models play the game through Epoch's harness with a fixed number of playthroughs. Multi-agent runs used the UK AI Security Institute's Inspect Deep Agent harness, modified so the lead agent can message subagents mid-run. Scores are compared against human baseline players.
Limits
- Multi-agent experiments were small-scale, and the post gives few exact scores or confidence intervals.
- Changing the rules retroactively breaks comparability with older results; Epoch argues the effect is small but re-ran only three models.
Why it matters for companies
A benchmark built to require exploration and learning was solved by a model that found a shortcut the designers knew existed but had not thought through. The same thing happens with internal KPIs and evaluation sets: a capable model will find the cheapest path to the score. The multi-agent result is a useful check on vendor claims that more agents mean better output; here they changed behaviour but not results.
Summary based on the full text. Read the original for the full argument and data.Read the original on epoch.ai