Beyond Benchmarks: Can Small Open-Weight Models Get Real Coding Work Done?
B · Useful, limited A careful single-practitioner experience report by independent engineer Vaibhav Jain with hidden-check scoring and honest failure counts, but small trial counts, private logs and setups specific to his own machines.
Summary based on the full text·Original: about 45 minutes to read
In short
A retrospective experience report (arXiv, 25 October 2026) by Vaibhav Jain, an independent engineer in India, covering July to October 2026. He tried to get small open-weight models (30B-total, about 3B-active mixture-of-experts checkpoints, 4-bit) to do real coding work through agent clients, first across pooled laptops, then on single cloud nodes and a 32 GB Mac. The main finding is that "served", "coherent text", "tool calls executed" and "code that passes hidden checks" are four different levels, and endpoints that claim the same API behave differently on the last two. In his delegation studies, handing implementation to a local worker under a Claude Opus 5.5 parent did not lower frontier cost per verified result.
Key numbers
- 15/15 vs 12/15 and 14/15verified passes for direct frontier coding against two delegation setups using a quantized Qwen3-Coder 30B-A3B worker
- 45/45 vs 12/45verified passes for direct coding against delegation to a Nemotron Cascade 2 worker on a second 15-task suite
- $0.153 vs $1.313API list-price-equivalent frontier cost per verified success, direct against Nemotron delegation
- 69.4 s vs 2.8 smedian time to first token for cold-prefix requests against high-reuse cached requests (observational)
- 19 of 36worker artifacts rejected only for renamed parameters that passed the hidden checks unchanged
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Evidence base: 4,165 coding-agent gateway turns, 623 debug jobs and three delegation studies scored by hidden behavioural checks.
- Study 1 (45 trials): direct coding passed 15/15; delegation to Qwen3-Coder passed 12/15 and 14/15. Frontier cost per verified success was $0.136 direct against $0.214 and $0.199 delegated.
- Study 2 (15 synthetic TypeScript repositories): direct 45/45, Nemotron Cascade 2 delegation 12/45. Counting 16 sessions stopped by a subscription limit as failures gives 37/45 and 8/45.
- A Qwen3-Coder worker passed 0/45 on the second suite under the same artifact contract.
- Cache state outside the model drove latency: cold-prefix requests had a median first-token time of about 69.4 s against 2.8 s for high-reuse requests.
- A strict interface rule rejected 19 of 36 artifacts that were functionally correct, showing that contract design can discard working code before review.
How they got it
Reconstruction from saved configurations, logs, integration tests and study results, with counts re-checked against saved records rather than re-run. The delegation studies fixed the parent model, hash-recorded inputs and model revisions before scoring, and used hidden checks separate from the agents' own success claims. Costs are API list-price equivalents from recorded usage; the sessions actually ran under a personal subscription.
Limits
- One author's own work; the records are private and cannot be checked by readers.
- Small cells: three trials per task category, and some figures rest on a single success.
- The studies are not pooled and do not rank models or runtimes; results apply to the tested model, quantization, runtime and client combinations.
- Local worker compute and energy were not costed, so "cheaper" covers frontier API cost only.
Why it matters for companies
Plans to cut coding-agent bills by routing work to small self-hosted models should budget for the integration layers this report lists: serving recipes, tool-call parsing, cache continuity and verification. In these tests the frontier model working alone was both more reliable and cheaper per verified result than a frontier parent supervising a small worker. Teams should also test their acceptance rules: a strict artifact contract threw away correct code here.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org