MirrorCode
A · Strong evidence Epoch AI benchmark paper with a METR co-author that measures multi-week software tasks under controlled, fully autonomous, cheat-proofed conditions across three frontier models; a careful single benchmark rather than an Epoch flagship report.
Summary based on the full text·Original: about 60 minutes to read
In short
MirrorCode is a long-horizon coding benchmark from Epoch AI, with contributors from METR, Prime Intellect, the University of Warwick and Equistamp. AI agents must rebuild an existing command-line program from its behavior alone: they can run the original binary and read its documentation, but never see its source code, and their output must match byte for byte, including on hidden tests. The best model, Claude Opus 4.7, scored 56% across 25 programs and rebuilt software that contributors estimate would take a human engineer weeks. Results depend heavily on having precise tests to work against.
Key numbers
- 56%benchmark score of the best model, Claude Opus 4.7, across 25 target programs
- 14 hours, $251time and cost for Opus 4.7 to reimplement gotree (about 16,000 lines of Go), passing 2,000/2,001 tests
- 2-17 weekscontributors' estimate for a skilled human engineer to do the same gotree task without AI
- about 30%score leading models from eight months earlier would have reached
As reported by the source.
English·PDF·opens epoch.ai
What it reports
- 25 target programs across Unix utilities, data query tools, bioinformatics, interpreters, static analysis, cryptography and compression. Tests per program range from 63 to 2,665, and on average 34% are held out.
- 17/25 targets had at least one perfect run. Four more had a run above 99%. 8/25 were never solved to 100%, and only 11/25 were solved to 100% reliably in every language and repetition.
- Opus 4.7 rebuilt gotree in 14 hours for $251, passing 2,000/2,001 tests. It also rebuilt pkl, a 60,000-line configuration language from Apple, to over 99% of tests.
- The hardest target, the ruff Python linter, peaked at 67% of hidden tests.
- A single attempt on a large task cost $2,600 over 19 days. Typical coding benchmarks spend $1-$10 per task.
- GPT-5.5 was cheaper when it did succeed: Opus 4.7 cost 2.0x more in the median successful run.
- Without visible tests, scores on complex programs fell from 95%+ of withheld tests to below 50%.
- About 40% of Opus 4.7 runs had at least one edge-case failure. In 26% of runs, models submitted failing solutions with over 90% of their token budget unused.
How they got it
Three frontier models (Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro Preview) ran fully autonomously, with no internet access, in a ReAct scaffold. They could work in six languages: Python, C, Rust, Go, OCaml and Ada. Token budgets were 1 billion per task, or 10 billion for large tasks, with three repetitions per program-language pair. Scoring ran in a separate sandbox and required exact string equality with the reference output.
Limits
- The original program serves as a perfect specification. That is rare in real software work, and the authors say the results do not show AI can handle arbitrary implementation tasks.
- A memorization screen found possible contamination for 17 of the 25 targets, although some memorized programs went unsolved and some unmemorized ones were solved.
- There is no human baseline for the long tasks. The 2-17 week figure comes from four contributors' estimates, and the benchmark covers only programs with text output.
Why it matters for companies
Agents can already finish weeks of engineering work when the target behavior is pinned down by tests. The constraint moves from writing code to writing specifications and test suites. Legacy tools with known inputs and outputs are the natural first candidates for rebuilding. Budgets need to plan for agent runs that cost hundreds to thousands of dollars each and still fail on edge cases about 40% of the time.
Summary based on the full text. Read the original for the full argument and data.Open PDF on epoch.ai
- Tags
- AI agents