---
id: "lib-2026-10-18-96-7-on-10k-internal-documents-replacing-rag-with"
type: "library"
title: "96.7% on 10k+ internal documents: replacing RAG with filter-search-read"
title_original: "1万件超の社内文書で正答率96.7% ― RAGをやめて「絞る・探す・読み切る」にしたNAYUTAの設計"
lang: "JA"
org: "Bocek (Zenn)"
source_type: "Practitioner post"
date: "2026-10-18"
week: "2026-W42"
url: "https://nosov.ai/library/lib-2026-10-18-96-7-on-10k-internal-documents-replacing-rag-with/"
original_url: "https://zenn.dev/bocek/articles/nayuta-agentic-search-design"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "Bocek CTO Yuta Euchi documents an agentic search design on the company's own 10,302 files with a scored question set and per-question cost; credible and specific, but 30 questions, no baseline at that scale, and the design is the firm's product."
summary_basis: "full-text"
original_minutes: "10"
tags: ["ai technology", "agentic search", "enterprise knowledge", "rag alternatives"]
---

# 96.7% on 10k+ internal documents: replacing RAG with filter-search-read

Original title: 1万件超の社内文書で正答率96.7% ― RAGをやめて「絞る・探す・読み切る」にしたNAYUTAの設計

Tier B (Useful, limited): Bocek CTO Yuta Euchi documents an agentic search design on the company's own 10,302 files with a scored question set and per-question cost; credible and specific, but 30 questions, no baseline at that scale, and the design is the firm's product.

Summary based on the full text.

## In short

A Zenn post (18 October 2026) by Yuta Euchi, CTO of Bocek, a Japanese AI-transformation consultancy, on NAYUTA, its replacement for flat RAG over company documents. Documents are converted once into Markdown with a few strict metadata fields; an agent then filters by metadata, searches only within that set, and reads whole documents before answering. On 30 questions over 10,302 of Bocek's own files it scored 96.7%. The author's conclusion: accuracy depended on how documents are stored and the order of search more than on the search model or LLM.

## Key numbers

- 96.7%: score on 30 business questions over the company's own documents (29 correct, 1 wrong)
- 10,302: SharePoint files converted to Markdown with metadata for the evaluation
- ~150,000-170,000: tokens per question, costing tens of yen each with Gemini 3.5 Flash
- 197: categories produced when the LLM was allowed free-text metadata, which made the field useless for filtering

## What it reports

- Why flat RAG degrades with scale: old and current versions of a rule sit side by side and the old one can rank higher; top chunks fill up with fragments of the same long document. On a 51-document mock set flat RAG worked well (above 90%).
- Conversion: extract text (OCR if needed), have an LLM restructure it into Markdown, and generate four metadata fields by structured output: title, document type (12 fixed choices), company name, and status (current or old). Old status is set only when the text says old, invalid or abolished.
- Agent steps: filter by metadata, search within the filtered set (the tool refuses unfiltered vector search, so flat RAG cannot run), use full-text search for exact terms such as clause numbers or amounts, then read the full document and quote amounts, dates and permissions verbatim with a link to the file.
- Budget in the system instructions: 8 searches and 6 documents per question (10 for questions about changes over time); try three different wordings before saying something cannot be found.
- Built as four MCP tools called by an agent on Google's Agent Development Kit, with no hard-coded call order.
- Evaluation: 30 questions (4 easy, 18 medium, 8 hard) including trick questions with no answer in the rules; scored 1, 0.5 or 0; each run once in a new conversation on Gemini 3.5 Flash. Result 29 correct, 1 wrong; the wrong one read the right parent page but used a figure from an interim report.
- Cost: about 150,000-170,000 tokens per question, tens of yen at Gemini 3.5 Flash prices; slower than flat RAG.
- Free-text metadata failed in practice: 197 categories, 12% of company names still carrying legal-entity suffixes, 1,756 blank dates.

## How they got it

In-house evaluation on the company's own SharePoint corpus with expected answers and source files set in advance. The author wrote the questions and graded the answers.

## Limits

- 30 questions, one run each; no confidence interval.
- No flat-RAG baseline on the 10,302-file corpus, so the gain at scale is asserted rather than measured.
- Author-written questions and grading on the author's own data; NAYUTA is Bocek's commercial service.
- Table conversion from Excel and some PDFs still causes misreadings.

## Why it matters for companies

The design lessons transfer to any internal knowledge assistant: hold documents in a clean, model-readable form; keep metadata to a few fixed-choice fields; mark superseded policies explicitly; and enforce filtering in the tool, not in the prompt. The cost per answer is higher than chunk retrieval, which is acceptable for policy and contract questions where a wrong answer is expensive. The metadata failures are a warning for anyone letting an LLM tag a document estate freely.

## Key passages in translation

> 社内文書の検索精度は検索エンジンやLLMの性能より前に、「文書をどんな形で持つか」と「どの順序で検索するか」で決まります。

The search accuracy of internal documents is decided, before search-engine or LLM performance, by "what form the documents are held in" and "in what order they are searched".

> フィルタなしの検索は、プロンプトで禁止するのではなくツール側で拒否するようにしました。

We made the tool reject unfiltered searches instead of forbidding them in the prompt.

> NAYUTAはそのための道具であって、置けば精度が出る箱ではありません。

NAYUTA is a tool for that work, not a box that delivers accuracy just by being installed.

Original: https://zenn.dev/bocek/articles/nayuta-agentic-search-design