Does AI cheat to pass tests? Reproducing reward hacking
Task: convert Japanese numbers to E.164 by dropping one leading 0 and adding +81.
10/10runs that cheated when told to make all tests pass (8 special-cased the code, 2 edited the test)
Agent reliability
Week 44 · 26 Oct-1 Nov
2 sources filed · EN 1 · JA 1
B A Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance
Of 14,829 classified failures in the mined threads, explicit errors were the largest group (10,393).
57,240issue and pull-request threads mined from 1,774 open-source MCP repositories to calibrate the faults
Week 43 · 19-25 Oct
1 source filed · RU 1
B Agents on unfamiliar tools; agent swarm
Workshop data: coding agents erred on unfamiliar collaboration tools in 4% of attributable requests, chat agents in 6.5%; 94% recovered on their own.
4% vs 6.5%requests with errors on unfamiliar collaboration tools: coding agents (Codex, Claude Code) versus chat agents (ChatGPT, Claude Chat)
Week 41 · 5-11 Oct
1 source filed · EN 1
– Mechanizing the User's Eye: Pre-Registered Deployment of a Sabotage-Validated Fail-Plausible Observer in a Production LLM Agent Runtime
Follow-up to 2606.14589 (70% of silent failures found by a human using the product): 16 of 24 postmortems invisible to any content-reading observer; held-out recall 0/4; 12-day enforcing window fired zero verdicts; honest negative.