Pre-processing documents for RAG cut wrong answers from 30% to 13% (RAGに入れる資料を「下ごしらえ」したら、誤答率が30%→13%に下がった)
A · Strong evidence Controlled single-author comparison on Qiita (Freelance Connect) using public National Tax Agency documents, a held-out question set, retrieval metrics and per-question human grading; small sample, but method and data are reproducible.
Summary based on the full text·Original: about 9 minutes to read
In short
A Qiita post (6 November 2026) by an engineer at Freelance Connect compares two ways of loading the same documents into Amazon Bedrock Knowledge Bases: as-is with the default 300-token chunking, and pre-processed into section-based chunks of about 400 characters, each headed with the article number, title, tax type and as-of date. On 757 National Tax Agency "Tax Answer" articles the share of answers with errors fell from 30% to 13%. On well-structured PDFs pre-processing did not help.
Key numbers
- 30% to 13%answers containing errors on 67 tax questions, raw documents versus pre-processed chunks
- 61% to 76%fully correct answers on the same 67 questions
- 7 of 9remaining errors where the model misread the correct document it was given
- 4 vs 7errors on 35 PDF questions, raw versus pre-processed (pre-processing did not win)
- 93% to 97%held-out questions with the correct article in the top 8 retrieved
As reported by the source.
Japanese·Web page·opens qiita.com
What it reports
- Data: 757 Tax Answer articles (about 1.17 million characters); a 250-page invoice-system Q&A PDF (153 questions); a 94-page model work-rules PDF (69 articles).
- 67 tax questions in everyday wording, split into 37 tuning and 30 held-out; 35 PDF questions.
- Raw version: default fixed 300-token chunks with 20% overlap, so the title appears only in the first chunk. Pre-processed: section boundaries, about 400 characters, header on every chunk, plus a synonym list.
- Retrieval (top 8 found): 80%/93% raw vs 91%/97% pre-processed (tuning/held-out); overall MRR 0.666 to 0.742. 900-character chunks did worse than 400.
- The synonym list helped on tuning questions but barely on held-out ones; the author treats it as something to grow from real misses.
- Answers, graded by a person: correct 41 (61%) vs 51 (76%); escalated 6 vs 7; with errors 20 (30%) vs 9 (13%).
- Raw-version errors included outdated deadlines, missing conditions, invented special rules and giving another office's phone number from a headless chunk, which also passes "grounded in the source" checks.
- 7 of the 9 remaining errors were model misreadings of the right document.
- PDFs: raw 26 correct, 5 escalated, 4 errors; pre-processed 27, 1, 7. The extra errors came from the model filling blanks in template rules; about a third of raw PDF answers showed page-number fragments.
- Rewriting queries with the model before search did not help.
How they got it
Bedrock Knowledge Bases (Tokyo region), Aurora PostgreSQL with pgvector, Titan Text Embeddings V2, hybrid search with top 8 passed to Amazon Nova 2 Lite; run in October 2026. Each answer was compared with the source article by a person, not by an LLM.
Limits
- 67 and 35 questions; no confidence intervals.
- One embedding model, one answer model and one platform.
- Questions written by the author, though in everyday phrasing and with a held-out split.
Why it matters for companies
Many internal RAG projects load documents as-is and trust the platform's chunking. This test shows the cheapest fix is to put the document name and section on every chunk, which more than halved error rates on article-style content, and that the effect depends on document type. The phone-number case shows why: a chunk without its context can produce a fluent, source-matching wrong answer that groundedness checks miss.
Key passages in translation
Japanese original
効いた下ごしらえは、資料の構造（節）で細かく分けて、各まとまりの頭に記事名を付けること
The pre-processing that worked: splitting finely along the document's structure (sections) and putting the article name at the head of each chunk.
Japanese original
回答が資料の文と一致しているので、「資料に書いてあることだけで答えているか」のチェックもすり抜けます。
Because the answer matches text in the documents, it also slips past checks of "is it answering only from what the documents say".
Japanese original
残った9件のうち 7件は、正しい記事を渡しているのにモデルが読み違えたものでした。
Of the nine remaining errors, seven were cases where the model misread the correct article it had been given.
Summary based on the full text. Read the original for the full argument and data.Read the original on qiita.com
- Tags
- AI enablement