96.7% on 10k+ internal documents: replacing RAG with filter-search-read
1万件超の社内文書で正答率96.7% ― RAGをやめて「絞る・探す・読み切る」にしたNAYUTAの設計
B · Useful, limited Bocek CTO Yuta Euchi documents an agentic search design on the company's own 10,302 files with a scored question set and per-question cost; credible and specific, but 30 questions, no baseline at that scale, and the design is the firm's product.
Summary based on the full text·Original: about 10 minutes to read
In short
A Zenn post (18 October 2026) by Yuta Euchi, CTO of Bocek, a Japanese AI-transformation consultancy, on NAYUTA, its replacement for flat RAG over company documents. Documents are converted once into Markdown with a few strict metadata fields; an agent then filters by metadata, searches only within that set, and reads whole documents before answering. On 30 questions over 10,302 of Bocek's own files it scored 96.7%. The author's conclusion: accuracy depended on how documents are stored and the order of search more than on the search model or LLM.
Key numbers
- 96.7%score on 30 business questions over the company's own documents (29 correct, 1 wrong)
- 10,302SharePoint files converted to Markdown with metadata for the evaluation
- ~150,000-170,000tokens per question, costing tens of yen each with Gemini 3.5 Flash
- 197categories produced when the LLM was allowed free-text metadata, which made the field useless for filtering
As reported by the source.
Japanese·Web page·opens zenn.dev
What it reports
- Why flat RAG degrades with scale: old and current versions of a rule sit side by side and the old one can rank higher; top chunks fill up with fragments of the same long document. On a 51-document mock set flat RAG worked well (above 90%).
- Conversion: extract text (OCR if needed), have an LLM restructure it into Markdown, and generate four metadata fields by structured output: title, document type (12 fixed choices), company name, and status (current or old). Old status is set only when the text says old, invalid or abolished.
- Agent steps: filter by metadata, search within the filtered set (the tool refuses unfiltered vector search, so flat RAG cannot run), use full-text search for exact terms such as clause numbers or amounts, then read the full document and quote amounts, dates and permissions verbatim with a link to the file.
- Budget in the system instructions: 8 searches and 6 documents per question (10 for questions about changes over time); try three different wordings before saying something cannot be found.
- Built as four MCP tools called by an agent on Google's Agent Development Kit, with no hard-coded call order.
- Evaluation: 30 questions (4 easy, 18 medium, 8 hard) including trick questions with no answer in the rules; scored 1, 0.5 or 0; each run once in a new conversation on Gemini 3.5 Flash. Result 29 correct, 1 wrong; the wrong one read the right parent page but used a figure from an interim report.
- Cost: about 150,000-170,000 tokens per question, tens of yen at Gemini 3.5 Flash prices; slower than flat RAG.
- Free-text metadata failed in practice: 197 categories, 12% of company names still carrying legal-entity suffixes, 1,756 blank dates.
How they got it
In-house evaluation on the company's own SharePoint corpus with expected answers and source files set in advance. The author wrote the questions and graded the answers.
Limits
- 30 questions, one run each; no confidence interval.
- No flat-RAG baseline on the 10,302-file corpus, so the gain at scale is asserted rather than measured.
- Author-written questions and grading on the author's own data; NAYUTA is Bocek's commercial service.
- Table conversion from Excel and some PDFs still causes misreadings.
Why it matters for companies
The design lessons transfer to any internal knowledge assistant: hold documents in a clean, model-readable form; keep metadata to a few fixed-choice fields; mark superseded policies explicitly; and enforce filtering in the tool, not in the prompt. The cost per answer is higher than chunk retrieval, which is acceptable for policy and contract questions where a wrong answer is expensive. The metadata failures are a warning for anyone letting an LLM tag a document estate freely.
Key passages in translation
Japanese original
社内文書の検索精度は検索エンジンやLLMの性能より前に、「文書をどんな形で持つか」と「どの順序で検索するか」で決まります。
The search accuracy of internal documents is decided, before search-engine or LLM performance, by "what form the documents are held in" and "in what order they are searched".
Japanese original
フィルタなしの検索は、プロンプトで禁止するのではなくツール側で拒否するようにしました。
We made the tool reject unfiltered searches instead of forbidding them in the prompt.
Japanese original
NAYUTAはそのための道具であって、置けば精度が出る箱ではありません。
NAYUTA is a tool for that work, not a box that delivers accuracy just by being installed.
Summary based on the full text. Read the original for the full argument and data.Read the original on zenn.dev