A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML Conversion
A · Strong evidence Thomson Reuters engineers run a controlled 3x3 ablation of harnesses and Claude models on 153 gold-standard bills from a production pipeline, with paired significance tests and per-dimension error analysis.
Summary based on the full text·Original: about 35 minutes to read
In short
Palvika Bansal, Rittika Jindal and Ruichong Wang of Thomson Reuters report a pipeline that converts US state legislation into strict XML for a legal publisher, where a dropped citation or mislabelled amendment is an error. They test whether accuracy comes from the model or from the harness around it, crossing three harnesses with three Claude models. The cheapest model in the full pipeline beat the most capable model working alone. Remaining error sits in citation normalisation, which needs a lookup against an external authority rather than a larger model.
Key numbers
- 0.948 vs 0.806overall score of the weakest model (Haiku 4.5) in the full pipeline vs the strongest (Opus) zero-shot, p < 0.001
- 0.677 to 0.948Haiku's overall score zero-shot vs in the decomposed multi-agent pipeline
- 0.232Opus zero-shot citation normalisation score
- 153gold bills from two US state jurisdictions in the evaluation
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Three harnesses: zero-shot (T1), one agent with the full tool set (T2), and a decomposed multi-agent pipeline modelled on production (T3). T2 and T3 share identical tools.
- Opus zero-shot reaches 0.806 overall and only 0.232 on citation normalisation.
- Haiku in T3 scores 0.948, beating Opus zero-shot (p < 0.001); Haiku rises from 0.677 at T1.
- Across four scored dimensions, the model spread of 0.68 to 0.81 at zero-shot narrows to about 0.95 in the full pipeline.
- Handing a single agent the citation lookup tool is not enough for weak models: normalisation 0.88 (Sonnet) and 0.59 (Opus) against 0.10 for Haiku, which reaches 0.80 only with a dedicated citation agent in T3.
- Because T2 and T3 share tools, the gap between them is attributed to orchestration: decomposition, persisted state and per-agent context.
How they got it
153 gold-standard bills from two jurisdictions. Outputs scored on four dimensions: character-level normalised edit distance, tree-edit similarity, exact amendment-span matching, and a citation score split into localisation and normalisation. Paired bootstrap tests compare conditions.
Limits
- One task, two jurisdictions and one model family (Claude Haiku 4.5, Sonnet 4.6, Opus 4.6).
- The production harness was designed for this task, so the comparison favours T3.
- Specific model snapshots; results may shift with model updates, as the authors note.
Why it matters for companies
For exact, rule-bound document work, investment in tooling, decomposition and deterministic lookups pays more than paying for the largest model. A cheaper model inside a well-built pipeline can match or beat a frontier model used directly, cutting run costs. Where correctness depends on external reference data (codes, identifiers, registries), give the system a lookup tool rather than expecting the model to know it.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org