Beyond Final Scores long-horizon AI RnD
A · Strong evidence A controlled evaluation by Meituan and the University of Chinese Academy of Sciences of seven frontier models over 756 multi-hour research runs, with deterministic process metrics and about $100,000 of inference.
Summary based on the full text·Original: about 75 minutes to read
In short
Yiwei Li, Wanli Yang, Hexiang Tan and colleagues at Meituan and the University of Chinese Academy of Sciences (arXiv, 13 August 2026) tested how well AI agents carry out long-horizon AI research and engineering, such as improving a model, a system or a CUDA kernel over 2-12 hours. Instead of reporting only a final score, they measure how agents frame solutions, execute and respond to feedback. Their conclusion: current agents behave like engineering optimisers, able to improve an artifact with known techniques, but inconsistent across runs and rarely original.
Key numbers
- 3 of 252best-seed solutions judged methodologically novel
- 0.237 vs 0.122gap between strongest and weakest model on average vs best-of-three score
- $89.9 vs $16.5mean cost per task for Claude Opus 4.7 vs GPT-5.5
- 756rollouts: 7 models x 36 tasks x 3 runs
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Claude Opus 4.7 ranks first on average (0.739) and best-of-three (0.790); GPT-5.5, GLM-5.2 and Gemini-3.1-Pro form a second tier within 0.029 of each other on average.
- Reliability separates models more than peak ability: the strongest-to-weakest gap is 0.237 on average score but 0.122 on best-of-three.
- Only 3 of 252 best-seed solutions qualified as novel approaches; the rest adapt or combine established techniques.
- Execution is uniformly high (0.880-0.967), while solution framing (0.473-0.612) and feedback control (0.772-0.928) vary more.
- CUDA work is hardest and most separating, with gaps of 0.403 (average) and 0.414 (best).
- Mean cost per task: Opus 4.7 $89.9, GLM-5.2 $33.0, GPT-5.5 $16.5, DeepSeek-V4-Pro $4.3, LongCat-2.0 $3.9.
- Experience transferred from other tasks raises DeepSeek-V4-Pro's average by 0.093 but lowers Gemini-3.1-Pro's by 0.017, enough to reorder models.
- Native harnesses gave GPT-5.5 and Kimi-K2.7-Code more stable runs than the shared Claude Code harness without changing best scores.
How they got it
36 expert-curated tasks from the AutoLab suite across model development, system optimisation, puzzles and CUDA, each with a weak starting artifact, an expert reference solution and an automated verifier. All seven models ran in Claude Code v2.1.152 as a shared harness, three rollouts each, between June and 10 July 2026. Process scores were computed from verifier logs, not LLM judges; LLM judges were used only for novelty.
Limits
- Tasks are bounded engineering problems with a verifier; open research questions are not tested.
- One author institution develops LongCat, one of the evaluated models.
- Costs use list prices without cache discounts.
Why it matters for companies
For any team delegating multi-hour technical work to agents, average-case reliability matters more than the best demo, and the gap between the two is large. The surrounding harness and stored experience change results enough that buying the top model is only part of the decision. Cheaper models at a fifth of the cost reach competitive peaks, which makes running several attempts and picking the best a practical option.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org