Which syntax is easier for LLMs to write?
LLMにとって書きやすい構文とは何か：`end` と `end func foo` を1,000回以上比べてみた
A · Strong evidence Controlled single-author experiment by nextbeat engineer Mizushima (kmizu), who did doctoral work on parsing, with confound controls, strict two-interpreter grading, no retries and full data published; limited to one model and labelled a pilot.
Summary based on the full text·Original: about 15 minutes to read
In short
A Zenn post (16 October 2026) on the nextbeat Tech Blog by Mizushima (kmizu), nextbeat's technology evangelist, who did doctoral work on parsing (PEG). He asks which syntax an LLM writes most reliably, using a made-up language explained only in the prompt so that prior knowledge of real languages cannot help. Closing each block with its kind and name (end func foo) kept generation correct at deep nesting where a bare end failed completely. His original hypothesis, that shorter dependency distance is what helps, was not cleanly supported.
Key numbers
- 0/24 vs 24/24correct programs at nesting depth 8 with a bare `end` versus `end func foo` closing lines
- 1,800trials in the first pilot
- 62 vs 15 (of 72)correct generations for named closing versus bare `end` at depths 4-16 in the follow-up run
- ~5 of 72variation seen when the same condition was simply measured twice
As reported by the source.
Japanese·Web page·opens zenn.dev
What it reports
- Task: given a shuffled table describing a program's syntax tree, write the full program in a grammar explained in the prompt with eight fixed examples. Only a perfect parse and exact tree match counts.
- First pilot (1,800 trials), generation at depth 8: braces 10/24, bare
end0/24,end func12/24,end func foo24/24, and a control with closing lines of the same length but no information 1/24. Almost all failures were programs that ended with blocks left open. - The padded control shows the gain comes from the information in the closing line, not its length.
- Reading tasks showed small differences (named 66/72 vs bare 60/72, confidence interval crossing zero): labels matter for writing, much less for reading.
- Changing block length at fixed depth did not change scores in the pilot; the number of blocks open at once mattered. With blocks of 128 lines, a small distance effect appeared.
- Follow-up (288 trials, depths 4/8/16): named closing 62/72, Python-style indentation 60/72, depth-number closing (
end 3) 54/72, bareend15/72. Depth numbers broke at depth 16 through miscounting. - Fully qualified imports were written slightly more accurately than short names (55 vs 50 of 72), a trend only.
- Combining indentation with named closing did not beat named closing alone; remaining errors were structural, not syntactic.
How they got it
A tiny scope-only language with three statement kinds; five closing styles for the same programs; a nonce-vocabulary condition replacing keywords with meaningless words; correct answers checked by two independently written interpreters; no retries, best-of-N or per-condition prompt tuning. Model: gpt-5.6-terra at low reasoning effort. Code in Scala 3; requests, responses, scores and failures are published.
Limits
- One model at one reasoning setting; each condition measured once, and re-measuring the same condition moved results by about 5 of 72.
- Artificial language with fixed few-shot examples, not real code.
- Indentation results cannot be separated from pre-training on Python, Markdown and YAML.
- A pre-registered reading experiment stopped at 1,412 of 5,120 trials when API credit ran out.
Why it matters for companies
Teams that have agents write internal DSLs, configuration or code in house conventions can lower error rates by choosing redundant, checkable syntax: named block ends, fully qualified references. The general lesson is that formats humans find verbose may be easier for models to produce correctly. The study is also a model of a cheap, honest evaluation: controls for the obvious confounds, strict scoring and published data.
Key passages in translation
Japanese original
深さ8のネストで end だけだと24回中0回しか正しいプログラムを書けなかったのに、end func foo 形式だと24回中24回正解
At nesting depth 8, a bare end produced a correct program 0 times out of 24, while the end func foo form was correct 24 times out of 24.
Japanese original
効いているのは長さではなく「どのブロックを閉じたか」という情報
What helps is not the length but the information about which block was closed.
Japanese original
LLMにとって書きやすいのは「依存が短い構文」というより、「書いた行をその場で正解と突き合わせられる構文」
What is easy for an LLM to write is less "syntax with short dependencies" than "syntax where each line written can be checked against the right answer on the spot".
Summary based on the full text. Read the original for the full argument and data.Read the original on zenn.dev