VIALS-AA: different paths to scientific image reasoning
A · Strong evidence Artificial Analysis, an independent model benchmarking firm, runs Handshake AI's VIALS tasks three times per model with a public harness, published traces and per-task cost, so results are checkable.
Summary based on the full text·Original: about 15 minutes to read
In short
Artificial Analysis launched VIALS-AA (6 November 2026), its implementation of Handshake AI's VIALS benchmark on interpreting scientific images such as gels, plasmid maps, flow cytometry and protein structures. Models get code execution and image tools in the open Stirrup harness. Scores count only tasks answered correctly in all three runs. The spread in work done and cost is far wider than the spread in scores.
Key numbers
- 64%pass^3 score of the leader, Gemini 4 Carbon (high): all three repeats correct
- 3.6xtool calls per task of Claude Sonnet 5.5 (60%) vs GPT-6.1 Sol (59%) for a similar score
- 45% vs 63%pass^3 of Kimi K3.1 Pro vs GPT-6 Astra, despite similar pass@3 of about three-quarters
- $0.02 to $1.77cost per task range across models, about 99x
- 161scientific image tasks, each run three times
As reported by the source.
English·Web page·opens artificialanalysis.ai
What it reports
- Top scores (pass^3): Gemini 4 Carbon (high) 64%, GPT-6 Astra (max) 63%, Claude Opus 5.5 (max, with fallback) 61%, Claude Sonnet 5.5 60%, GPT-6.1 Sol (max) 59%.
- Consistency separates open-weight models: Kimi K3.1 Pro, MiMo-V2.6-Pro and MiniMax-M3.1-Flash each solve 114-119 tasks at least once, but pass all three repeats on only 48-73 tasks, against 95-103 for the seven top closed models.
- Cost per task ranges from $0.02 (GPT-6 Luna) to $1.77 (Grok 4.7); GPT-6.1 Sol scores 59% at $0.06, about a thirtieth of Claude Opus 5.5's cost for 61%.
- Output tokens per task range from about 2.21k (GPT-6 Astra) to 94.6k (MiMo-V2.6-Pro); Sonnet 5.5 uses 63.5k, about 25x Sol's.
- Across 9,177 rollouts of 19 models, 38.3% of tool calls viewed images and 47.3% prepared or measured them; image measurement was 5-6% of calls for Astra and Sol vs 25-27% for Opus and Sonnet.
- Example: Grok 4.7 used 27 turns and 20 image views and still read the label instead of the faint protein spot.
How they got it
Each of 161 tasks run three times per model; cost computed from token usage and list prices with representative cache hit rates, excluding judge calls. Tool calls classified into one primary category each. Traces and methodology are published.
Limits
- Launch snapshot; the leaderboard will change.
- One harness; results may differ under vendor-native tools.
- Some models have no public API, and cost excludes grading.
Why it matters for companies
For lab, quality-control or document-image work, pick models on repeatability and cost per correct answer, not peak score: near-identical scores here hide a 30x cost gap and very different reliability. Requiring several consistent runs before accepting an answer is a cheap control worth copying in production. More tool calls and more "visible effort" did not mean better answers.
Summary based on the full text. Read the original for the full argument and data.Read the original on artificialanalysis.ai