---
id: "lib-2026-11-01-agentic-search-7-search-tools-16-models-one"
type: "library"
title: "Agentic search: 7 search tools, 16 models, one company"
title_original: "Агентский поиск на open-source моделях: 7 поисковых инструментов, 16 моделей и одна компания"
lang: "RU"
org: "iseedeadpixels (Habr)"
source_type: "Practitioner post"
date: "2026-11-01"
week: "2026-W44"
url: "https://nosov.ai/library/lib-2026-11-01-agentic-search-7-search-tools-16-models-one/"
original_url: "https://habr.com/ru/articles/1102340/"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "A single-author Habr benchmark by SaaS marketer Mark Grabarev with a hand-built gold standard, a published scoring rule, three runs per pairing, per-run costs and an incident log; the method is shown and the numbers can be rechecked."
summary_basis: "full-text"
original_minutes: "19"
tags: ["ai technology", "agentic search", "open models", "benchmark variance"]
frames: ["verification-capacity"]
---

# Agentic search: 7 search tools, 16 models, one company

Original title: Агентский поиск на open-source моделях: 7 поисковых инструментов, 16 моделей и одна компания

Tier A (Strong evidence): A single-author Habr benchmark by SaaS marketer Mark Grabarev with a hand-built gold standard, a published scoring rule, three runs per pairing, per-run costs and an incident log; the method is shown and the numbers can be rechecked.

Summary based on the full text.

## In short

A Habr case study (1 November 2026) testing how agents built on 16 models and 7 search tools collect public facts about one company, Bitrix24. The same seven-part question (legal entity, revenue, CEO and CMO, products and prices, 2026 updates, recent and upcoming webinars, press mentions) was scored against a hand-checked gold standard on a 14-point scale, with a separate count of "confident fabrications". The author ran six stages, from no search through live search to a code-driven harness where the model only extracts fields. Run-to-run variance for one model (3 to 6 points) exceeded the gap between neighbouring ranks (about 1 point), so single-run leaderboards mislead.

## Key numbers

- 7 to 13: score range (of 14) for one model, Sol, across three runs with the same prompt and search tool
- 12.3: mean score of MiMo v2.6 Pro on live Brave search, with zero confident fabrications, at about $0.10 for three runs
- 9 of 42: Brave runs that silently ran with empty search after the free tier ran out and returned HTTP 402
- 8-30: facts per run sent to human review by the verifying harness
- 2 of about 66: manually checked URLs that were dead; the main error was misreading live pages

## What it reports

- Without search, 8 models scored 1 to 5 of 14; gpt-oss:120b produced six confident fabrications, including webinars and press quotes dated in the future.
- Search matrix (two models, one run per tool): Exa 10.5, DuckDuckGo 10.0, Keenable 7-8, Firecrawl 5-9, Tavily 5-7, an unconfigured SearXNG about 3.
- 8 paid models on Exa averaged 11.1; all found the CMO, whom no model named without search.
- Completeness tracked page reads: leaders made about 22-23 searches and 5-6 page extracts; weak runs made about 7 searches and no extracts.
- Live Brave, three-run means: Qwen 3.8 Max 12.7 (about $0.64 for three runs), MiMo v2.6 Pro 12.3 (about $0.10), Sonnet 5.5 8.0, Inkling 3.7.
- The harness (fixed source snapshot, quote-on-page and year checks, domain allowlist, two independent calls per field) lifted weak live performers (Mistral Large 3 from 4.0 to 10.7) and sank others (Nemotron Super from 9.0 to 3.3). Top models scored 11.3-12.7 on live Exa versus 8.7-12.0 on the DDG snapshot.
- Webinars were the hardest item in all modes (0.77 of 2 on Exa, 0.36 on Brave snapshot): past event pages are deleted within a week and the listing renders in JavaScript.
- A typosquatted helpdesk domain returned HTTP 200 and one agent cited it as confirmed.
- A trap entity (an empty Kaliningrad holding company with the same name) caught models that did not query the company register.

## How they got it

A gold standard compiled by hand before the first run; each item scored 2, 1 or 0; net score is completeness minus verifiable errors. Grading was manual with help from Grok, without an automated LLM judge. Tool-call logs were inspected per run. Snapshot runs used a hashed source file so every run saw the same data.

## Limits

- One company, one question set, Russian-language web; results may not transfer.
- Partly assisted grading by a chatbot; no inter-rater check.
- Providers changed mid-experiment for DeepSeek and GLM, which the author says makes those cells incomparable; one Nemotron Ultra run never finished.
- Some internal inconsistencies: the conclusions give Sol $0.2-0.4 per run while the table shows about $0.15; the text mentions 90 runs while mode counts add up to more.

## Why it matters for companies

For anyone building research or competitor-monitoring agents, the post gives three practical rules: test each configuration several times before choosing, verify the search tool is actually returning results before blaming the model, and measure whether the agent opens pages, not only whether it searches. It also shows that verification harnesses shift work to people (8-30 facts per run for review) and help some models while hurting others, so they should be added for a specific error type, not by default.

## Key passages in translation

> Разница между соседними местами в рейтинге — около 1 балла.

The gap between neighbouring places in the ranking is about 1 point.

> Точнее та модель, которая после поиска идёт читать страницы.

The more accurate model is the one that goes on to read the pages after searching.

> Сбой инструмента не равен слабости модели.

A tool failure is not the same as a weak model.

Original: https://habr.com/ru/articles/1102340/