---
id: "lib-2026-11-06-pre-processing-documents-for-rag-cut-wrong"
type: "library"
title: "Pre-processing documents for RAG cut wrong answers from 30% to 13% (RAGに入れる資料を「下ごしらえ」したら、誤答率が30%→13%に下がった)"
lang: "JA"
org: "Freelance Connect (Qiita)"
source_type: "Practitioner post"
date: "2026-11-06"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-11-06-pre-processing-documents-for-rag-cut-wrong/"
original_url: "https://qiita.com/MURAMASA2470/items/a7c351ba251da81f507e"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Controlled single-author comparison on Qiita (Freelance Connect) using public National Tax Agency documents, a held-out question set, retrieval metrics and per-question human grading; small sample, but method and data are reproducible."
summary_basis: "full-text"
original_minutes: "9"
tags: ["ai technology", "enablement", "internal knowledge rag", "chunking"]
---

# Pre-processing documents for RAG cut wrong answers from 30% to 13% (RAGに入れる資料を「下ごしらえ」したら、誤答率が30%→13%に下がった)

Tier A (Strong evidence): Controlled single-author comparison on Qiita (Freelance Connect) using public National Tax Agency documents, a held-out question set, retrieval metrics and per-question human grading; small sample, but method and data are reproducible.

Summary based on the full text.

## In short

A Qiita post (6 November 2026) by an engineer at Freelance Connect compares two ways of loading the same documents into Amazon Bedrock Knowledge Bases: as-is with the default 300-token chunking, and pre-processed into section-based chunks of about 400 characters, each headed with the article number, title, tax type and as-of date. On 757 National Tax Agency "Tax Answer" articles the share of answers with errors fell from 30% to 13%. On well-structured PDFs pre-processing did not help.

## Key numbers

- 30% to 13%: answers containing errors on 67 tax questions, raw documents versus pre-processed chunks
- 61% to 76%: fully correct answers on the same 67 questions
- 7 of 9: remaining errors where the model misread the correct document it was given
- 4 vs 7: errors on 35 PDF questions, raw versus pre-processed (pre-processing did not win)
- 93% to 97%: held-out questions with the correct article in the top 8 retrieved

## What it reports

- Data: 757 Tax Answer articles (about 1.17 million characters); a 250-page invoice-system Q&A PDF (153 questions); a 94-page model work-rules PDF (69 articles).
- 67 tax questions in everyday wording, split into 37 tuning and 30 held-out; 35 PDF questions.
- Raw version: default fixed 300-token chunks with 20% overlap, so the title appears only in the first chunk. Pre-processed: section boundaries, about 400 characters, header on every chunk, plus a synonym list.
- Retrieval (top 8 found): 80%/93% raw vs 91%/97% pre-processed (tuning/held-out); overall MRR 0.666 to 0.742. 900-character chunks did worse than 400.
- The synonym list helped on tuning questions but barely on held-out ones; the author treats it as something to grow from real misses.
- Answers, graded by a person: correct 41 (61%) vs 51 (76%); escalated 6 vs 7; with errors 20 (30%) vs 9 (13%).
- Raw-version errors included outdated deadlines, missing conditions, invented special rules and giving another office's phone number from a headless chunk, which also passes "grounded in the source" checks.
- 7 of the 9 remaining errors were model misreadings of the right document.
- PDFs: raw 26 correct, 5 escalated, 4 errors; pre-processed 27, 1, 7. The extra errors came from the model filling blanks in template rules; about a third of raw PDF answers showed page-number fragments.
- Rewriting queries with the model before search did not help.

## How they got it

Bedrock Knowledge Bases (Tokyo region), Aurora PostgreSQL with pgvector, Titan Text Embeddings V2, hybrid search with top 8 passed to Amazon Nova 2 Lite; run in October 2026. Each answer was compared with the source article by a person, not by an LLM.

## Limits

- 67 and 35 questions; no confidence intervals.
- One embedding model, one answer model and one platform.
- Questions written by the author, though in everyday phrasing and with a held-out split.

## Why it matters for companies

Many internal RAG projects load documents as-is and trust the platform's chunking. This test shows the cheapest fix is to put the document name and section on every chunk, which more than halved error rates on article-style content, and that the effect depends on document type. The phone-number case shows why: a chunk without its context can produce a fluent, source-matching wrong answer that groundedness checks miss.

## Key passages in translation

> 効いた下ごしらえは、資料の構造（節）で細かく分けて、各まとまりの頭に記事名を付けること

The pre-processing that worked:  splitting finely along the document's structure (sections) and putting the article name at the head of each chunk.

> 回答が資料の文と一致しているので、「資料に書いてあることだけで答えているか」のチェックもすり抜けます。

Because the answer matches text in the documents, it also slips past checks of "is it answering only from what the documents say".

> 残った9件のうち 7件は、正しい記事を渡しているのにモデルが読み違えたものでした。

Of the nine remaining errors, seven were cases where the model misread the correct article it had been given.

Original: https://qiita.com/MURAMASA2470/items/a7c351ba251da81f507e