What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion
A · Strong evidence Amazon researchers measure a specific ingestion failure on real policy documents with deterministic scoring (no LLM judge), replicate it across six models from two vendors, localise it with corrected tests and release scripts and data.
Summary based on the full text·Original: about 60 minutes to read
In short
A preprint (arXiv, 21 October 2026) by Yikai Zhao, Saurabh Pandey and Pradeep Kumar Misra of Amazon. Before an agent can follow a policy, a converter turns the Word or PDF file into text. Mainstream converters keep words, tables and structure but drop colour and highlighting, which authors use to mark the requirements that matter. Agents reading the converted text do not abstain; they guess, and become confidently wrong. The fix sits in ingestion and is cheap.
Key numbers
- 0.690 to 0.150micro F1 on recovering author-flagged requirements, formatting-aware versus docling ingestion (0.075 with plain text)
- +0.302share of the loss attributable to highlight in leave-one-out tests (colour +0.262)
- 32.8%of 6,460 real policy documents that carry deliberate colour or highlight
- 10.3%documents with colour or highlight on a sentence expressing an obligation
- +0.085pre-registered gain from telling the agent what a cue means, beyond preserving it
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Task: from 76 real policy documents, recover the requirements the author singled out, scored against the file's own OOXML formatting properties by a deterministic matcher.
- Formatting-blind ingestion drops micro F1 from 0.690 to 0.150 with docling and 0.075 with plain text. The gap holds across five Claude models (Opus, Sonnet, Haiku tiers) and GPT-5.6.
- The failure inflates answers: docling's false-positive rate on requirements the author did not flag is four times that of the formatting-aware representation.
- Leave-one-out tests put the loss on highlight (+0.302) and colour (+0.262); bold, italic, monospace and strikethrough each sit within ±0.003 of zero. The same pattern appears in PDF.
- Prevalence: 32.8% of 6,460 public policy documents carry deliberate colour or highlight, 10.3% on an obligation sentence.
- A pre-registered test shows that labelling what a cue means adds +0.085 beyond preserving it.
- What did not work: progressive disclosure did not cut token cost, and the typed layer gave no benefit on two models that attempt the task only sparsely.
How they got it
The authors scanned a public corpus of real .docx policy documents, detected formatting cues at OOXML run level, and built a task where some requirement-like sentences are author-marked and others are not. Questions were generated and second-opinion judging done by different model families from the answering model. Labels were validated against rendered page images (κ=0.72), and a supervised text-only classifier bounds what wording alone can reach (AUC 0.664). Claims rest on paired per-document tests with Holm correction.
Limits
- Labels are validated against rendered pages, not yet by human annotators.
- PDF support needs a text layer; scanned documents are not covered.
- The formatting-aware representation costs about twice plain text in tokens.
- Absolute F1 varies by about ±0.08 across configurations; the typed-cue gain is modest.
Why it matters for companies
HR handbooks, compliance procedures and policy libraries are typical inputs to internal assistants, and their authors often mark exceptions and obligations in red or yellow. If the ingestion pipeline drops that layer, the assistant answers fluently and wrongly with no error signal. Teams building document assistants should test their converter on formatting cues, preserve colour and highlight explicitly with their meaning, and treat ingestion as part of the evaluation, not plumbing.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- RAG