Can a local LLM replace Claude Code? Accuracy, concurrency, cost
【LocalLLM】Claude Codeの代わりになる？精度・同時人数・費用を検証
B · Useful, limited Solvio consultancy test of Qwen3.5-122B on one rented B200 against Claude Code, with pass criteria fixed in advance and a validated load-replay tool; useful and well documented, but six coding tasks, fictitious documents and a closing sales pitch.
Summary based on the full text·Original: about 15 minutes to read
In short
A Zenn post (15 October 2026) by okawa2929 of Solvio, a Japanese generative-AI consultancy, testing whether a self-hosted model can stand in for Claude Code where code and data may not leave the company. The setup was one NVIDIA B200 rented on RunPod, vLLM, the OpenCode agent, and two open-weight models: Qwen3.5-122B and gpt-oss-120b. Qwen came close on small coding fixes, served about 16 people at once, and handled masking, summarising and translation at roughly Claude Haiku 5.5 level, with more errors in summaries.
Key numbers
- 17/18 vs 18/18coding runs passing all tests: Qwen3.5-122B on vLLM versus Claude Opus 5.5 in Claude Code
- ~2.6xQwen's time per task relative to Opus 5.5 (66 s with six tasks in parallel vs 25 s one at a time)
- 16 peoplepeople one B200 can serve at once with Qwen before task time stretches past 3x (8 for gpt-oss-120b)
- ~JPY 870,000 a monthcost of renting one B200 continuously at the tested rate (about JPY 14,000 per person if shared by 64)
- 0.40 vs 0.07hallucinated additions per meeting summary, Qwen versus Opus 5.5 (Haiku 5.5: 0.13)
As reported by the source.
Japanese·Web page·opens zenn.dev
What it reports
- Six small real fixes (three bugs, three features) from six public Python libraries, all made after July 2026, three runs each. Qwen passed 17 of 18, all on the first answer; Opus 5.5 and Haiku 5.5 passed 18 of 18; gpt-oss-120b passed 16, only 11 on the first answer.
- Qwen took about 2.6 times as long per task as Opus 5.5 and gpt-oss about 6 times, partly because the local models solved six tasks in parallel on one GPU.
- Load test with 1 to 64 simulated people: the pre-set pass bar was median time to first token under 30 s and task time stretch under 3x. Qwen passed up to 16 (3.14x at 32), gpt-oss up to 8 (3.13x at 16). Time to first token stayed under 2 s up to 32 people; the limit was GPU compute, not memory (KV cache peaked at 35%).
- Generation speed for Qwen fell from 145 tokens per second for one person to 73 with eight.
- Non-coding tasks on fictitious documents: errors per summary were 0.07 for Opus 5.5, 0.13 for Haiku 5.5 and 0.40 for Qwen; translation errors 0, 0.6 and 1.0.
- Cost: $8.04 an hour; the whole test took 4 hours 52 minutes for about JPY 5,900. Continuous rental would be about JPY 870,000 a month.
- Privacy setup: OpenCode sharing, auto-update, web fetch and search turned off; the author notes OpenCode still fetched its model list and vLLM sends usage stats by default, and gives the environment variables that stop both.
How they got it
Real library commits were used as tasks, graded by the added and existing tests, with up to two follow-up prompts phrased as a human tester would. Load was simulated by replaying recorded OpenCode requests, checked against eight real concurrent OpenCode sessions (within 2% per request for one person, 5-12% for eight). Summaries and translations were scored against checklists by Opus 5.5 with model identities hidden.
Limits
- Six coding tasks is a small set; the author says it cannot show whether the gap widens on harder work.
- Different agents (OpenCode versus Claude Code) and network paths, so time differences are not pure model differences.
- A commenter points out the tasks are public and may be known to the models; the post's answer is only that fixes were recent.
- Non-coding tests used five fictitious documents per task and an LLM judge.
- Ends with a pitch for Solvio's Claude Code adoption services.
Why it matters for companies
For firms that cannot send code or documents to a cloud model, this gives a concrete sizing: one B200 class GPU, about 16 people at once, at a monthly cost that can be set against per-seat licences. It also shows where the gap sits: speed, context length (65,000 tokens configured against Claude Code's 1 million) and summary accuracy, so human review of summaries and translations remains necessary. The recommended first step, renting the hardware for a few hours and testing your own tasks, is cheap.
Key passages in translation
Japanese original
小さめのバグ修正や機能追加の課題を18回解かせたところ、Qwen3.5-122Bは17回でテストをすべて通る修正ができました。
When given small bug-fix and feature tasks 18 times, Qwen3.5-122B produced fixes that passed all tests 17 times.
Japanese original
同時に使える人数の上限を決めていたのは、メモリの量ではなくGPUの計算速度でした。
What capped the number of simultaneous people was GPU compute speed, not the amount of memory.
Japanese original
業務で使える水準ではあるものの、要約や翻訳の結果は人が確認する前提で使うのがよいと思います。
The quality is usable for work, but summaries and translations should be used on the assumption that a person checks them.
Summary based on the full text. Read the original for the full argument and data.Read the original on zenn.dev
- Tags
- Coding agents, Cost