Gave nine popular Claude Code skills a placebo
Дал девяти популярным скиллам для Claude Code плацебо: два лучше пустышки, один хуже
A · Strong evidence Independent practitioner experiment with a pre-registered method, length-matched placebo controls, 450 test-verified runs, bootstrap intervals and Holm correction, with full logs published.
Summary based on the full text·Original: about 10 minutes to read
In short
Simon Sudarushkin tested nine of the most-starred Claude Code skills against a placebo: neutral text about the environment, matched in length to the always-loaded part of each skill. Only ponytail and agent-skills beat their placebo, both on cost by small margins; planning-with-files did worse; six, including the most popular, superpowers, were indistinguishable. No skill was clearly cheaper than running with no skill, because the skill text itself adds tokens to every request.
Key numbers
- 2 of 9popular Claude Code skills that beat a length-matched placebo, both on cost only (12% and 5% cheaper)
- 2-16%cost increase from adding a neutral placebo text with no advice at all
- 80% vs 100%tasks solved with planning-with-files versus its placebo
- +14%cost change with caveman against no skill, despite its README promise of 65% fewer tokens
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Skills chosen by GitHub stars on 28 September 2026, from superpowers (292,189) to compound-engineering (25,299); domain skills, memory systems and orchestrators excluded.
- Setup: Claude Code 2.1.282, claude-opus-5-5 at medium reasoning; 15 tasks from SWE-bench Verified, Terminal-Bench 2.1 and OpenThoughts-TBLite; 30 runs per group, 450 runs in total, each in a clean Docker container.
- Cost ratio against placebo: ponytail 0.88 [0.76, 0.97], agent-skills 0.95 [0.90, 0.99], both with adjusted p = 0.049.
- planning-with-files solved 80% of tasks versus 100% for its placebo, with no cost difference.
- Against no skill, agent-skills cost 9% more (1.09 [1.02, 1.19]); ponytail was 0.99 [0.81, 1.13].
- caveman cut agent output tokens by 2% [-9%, +6%], within noise, while total tokens rose 18% and cost 14%; it adds about 3,900 tokens to each request.
- A Codex pilot (gpt-6-sol, three skills, five tasks, 10 runs per group) found agent-skills and compound-engineering 24% and 42% more expensive than placebo; the author draws no conclusion from it.
How they got it
Three groups per skill: no skill, placebo, skill, installed identically via plugin, hook or CLAUDE.md. The method and verdict rule were committed to METHOD.md before the first run; 19 dated amendments followed, including stopping at 450 of a planned 1,125 runs before seeing results and fixing a scoring bug on publication day. Success was judged by each task's own tests, cost estimated from tokens and public API prices.
Limits
- One model, short tasks; skills meant for long multi-day sessions are not measured.
- Public tasks likely seen in training, so absolute solve rates mean little; solve-rate intervals are about plus or minus 10 points.
- README claims were measured by their authors on different tasks and baselines.
Why it matters for companies
Teams adopting agent tooling often install prompt packs on the strength of stars and README numbers. This test shows most deliver no measurable gain, and the extra context they add raises cost on every call. Any internal prompt library or agent instruction set should be tested against a length-matched control, not just against nothing. The pre-registration discipline is cheap to copy.
Key passages in translation
Russian original
Когда автор скилла сравнивает работу со скиллом и без него, в разнице смешаны две вещи: эффект содержания скилла и эффект того, что в контексте просто стало больше текста.
When a skill's author compares work with and without the skill, two things are mixed in the difference: the effect of the skill's content and the effect of there simply being more text in the context.
Russian original
Лучше плацебо оказались два скилла, ponytail и agent-skills, оба по стоимости: на 12% и 5% дешевле своих пустышек.
Two skills turned out better than placebo, ponytail and agent-skills, both on cost: 12% and 5% cheaper than their dummies.
Russian original
Одно плацебо, нейтральный текст без единого совета, поднимало стоимость на 2-16%. Ни один из девяти скиллов не оказался заметно дешевле работы без скилла.
The placebo alone, a neutral text without a single piece of advice, raised cost by 2-16%. None of the nine skills was noticeably cheaper than working without a skill.
Russian original
Часть того, что вы платите за скилл, это просто его длина.
Part of what you pay for a skill is simply its length.
Russian original
Расхождение не значит, что их цифра ложная. Оно показывает, насколько цифра меняется в другой обстановке.
The discrepancy does not mean their figure is false. It shows how much the figure changes in a different setting.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com