6 sources filed · EN 4 · ZH 2
Test set: 100 lesions not safe to discharge (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, 20 actinic keratoses) and 100 safe (50 moles, 50 seborrhoeic keratoses).
200biopsy-proven lesions from the public MILK10k dataset, each with a clinical and a dermoscopic image
Annual 2024 returns after costs: GPT-6 Astra 10.7%, GPT-6 Sol 8.6%, GPT-5.6 Terra 8.4%, Claude Opus 5.5 4.4%; Claude Sonnet 5.5 -12.2%, Gemini 3.8 Flash -2.9%, DeepSeek V4.1 Flash -5.7%, Kimi K3 -8.5%, GLM-5.3 -7.8%.
10.7% vs 17.1%annual return of the best model (GPT-6 Astra) versus the equal-weight market benchmark
Four features passed the full test suite and a headless smoke test but failed in use, all at mocked boundaries: browser iframe security, a token budget truncating model output, a destructive overwrite on empty model responses, and an oversized plot for the real API.
5 of 39code commits carrying a prompt specification on the day the assistant ran past unenforced gates
A "dangerous misreport" is an unhedged success claim that the trace does not support, for example "Done, the duplicates were removed" after the agent looked at an empty list.
43.4 percentage pointsrise in recall of dangerous misreports when the judge does not see the agent's own success claim (Qwen3.5-35B-A3B)
COBOL migration eval: models convert COBOL to C++, Java, Kotlin, Python or Rust; hidden tests require byte-for-byte parity on observable outputs, and a migration passes only if all tests pass.
5.4% to 86.6%OpenAI COBOL migration eval pass rate, GPT-5.6 Sol low to GPT-6.1 Sol low reasoning
Simulation (12 benchmarks): LIBERO-Long++ 100%, LIBERO-Pro 99.3%, SIMPLE 98.8%, LIBERO-Ultra-Long 98.9%, DexJoCo 87.3%, RoboDojo 81.9%.
100%GPT-6 Astra success on LIBERO-Long++ with in-context demonstrations and an extended step budget
3 sources filed · EN 3
A 1-SD higher model score makes publication 44% more likely, or 2.9 points from a 6.5% base rate, controlling for school ranking, publication record and editorial experience.
44%higher likelihood of top-three finance journal publication per 1-SD rise in the model's score, net of author reputation
Small task (tile open windows, score out of 19): 18.3 with comments, 19.0 without; tokens 2.18M vs 1.71M; cost $1.42 vs $1.04.
7.01M to 10.64Mtoken spread across the three no-comment runs of the bigger task
The best method reached a QWK of 0.616 at Cambridge, 0.420 at Nottingham and 0.345 at Manchester Met.
761undergraduate psychology assignments from Cambridge, Manchester Metropolitan and Nottingham
2 sources filed · EN 2
Six simulated traits: knowledge, understanding, reflexivity, memory, verbosity, disclosure; derived from 20 characteristics found in a literature review.
0.86average Pearson correlation between simulated and human-study results across 12 measures
Four levels of eval validity: output-level (judge agrees with humans), cross-system (benchmark ranks models like humans), unit-level (Prentice surrogacy) and trial-level.
+2.0pp to -0.25ppa real eval improvement turned negative by a judge only three points less sensitive on treated outputs
1 source filed · EN 1
Five blockers from two years of builds: brownfield integrations, unreliable enterprise data, lack of evaluations, governance and audit overhead, and operating-model frictions when moving from proof of concept to production.
75%technology leaders who fear 'silent failure', spending heavily on AI without real impact (BCG market research)