Agentic search: 7 search tools, 16 models, one company
Агентский поиск на open-source моделях: 7 поисковых инструментов, 16 моделей и одна компания
A · Strong evidence A single-author Habr benchmark by SaaS marketer Mark Grabarev with a hand-built gold standard, a published scoring rule, three runs per pairing, per-run costs and an incident log; the method is shown and the numbers can be rechecked.
Summary based on the full text·Original: about 19 minutes to read
In short
A Habr case study (1 November 2026) testing how agents built on 16 models and 7 search tools collect public facts about one company, Bitrix24. The same seven-part question (legal entity, revenue, CEO and CMO, products and prices, 2026 updates, recent and upcoming webinars, press mentions) was scored against a hand-checked gold standard on a 14-point scale, with a separate count of "confident fabrications". The author ran six stages, from no search through live search to a code-driven harness where the model only extracts fields. Run-to-run variance for one model (3 to 6 points) exceeded the gap between neighbouring ranks (about 1 point), so single-run leaderboards mislead.
Key numbers
- 7 to 13score range (of 14) for one model, Sol, across three runs with the same prompt and search tool
- 12.3mean score of MiMo v2.6 Pro on live Brave search, with zero confident fabrications, at about $0.10 for three runs
- 9 of 42Brave runs that silently ran with empty search after the free tier ran out and returned HTTP 402
- 8-30facts per run sent to human review by the verifying harness
- 2 of about 66manually checked URLs that were dead; the main error was misreading live pages
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Without search, 8 models scored 1 to 5 of 14; gpt-oss:120b produced six confident fabrications, including webinars and press quotes dated in the future.
- Search matrix (two models, one run per tool): Exa 10.5, DuckDuckGo 10.0, Keenable 7-8, Firecrawl 5-9, Tavily 5-7, an unconfigured SearXNG about 3.
- 8 paid models on Exa averaged 11.1; all found the CMO, whom no model named without search.
- Completeness tracked page reads: leaders made about 22-23 searches and 5-6 page extracts; weak runs made about 7 searches and no extracts.
- Live Brave, three-run means: Qwen 3.8 Max 12.7 (about $0.64 for three runs), MiMo v2.6 Pro 12.3 (about $0.10), Sonnet 5.5 8.0, Inkling 3.7.
- The harness (fixed source snapshot, quote-on-page and year checks, domain allowlist, two independent calls per field) lifted weak live performers (Mistral Large 3 from 4.0 to 10.7) and sank others (Nemotron Super from 9.0 to 3.3). Top models scored 11.3-12.7 on live Exa versus 8.7-12.0 on the DDG snapshot.
- Webinars were the hardest item in all modes (0.77 of 2 on Exa, 0.36 on Brave snapshot): past event pages are deleted within a week and the listing renders in JavaScript.
- A typosquatted helpdesk domain returned HTTP 200 and one agent cited it as confirmed.
- A trap entity (an empty Kaliningrad holding company with the same name) caught models that did not query the company register.
How they got it
A gold standard compiled by hand before the first run; each item scored 2, 1 or 0; net score is completeness minus verifiable errors. Grading was manual with help from Grok, without an automated LLM judge. Tool-call logs were inspected per run. Snapshot runs used a hashed source file so every run saw the same data.
Limits
- One company, one question set, Russian-language web; results may not transfer.
- Partly assisted grading by a chatbot; no inter-rater check.
- Providers changed mid-experiment for DeepSeek and GLM, which the author says makes those cells incomparable; one Nemotron Ultra run never finished.
- Some internal inconsistencies: the conclusions give Sol $0.2-0.4 per run while the table shows about $0.15; the text mentions 90 runs while mode counts add up to more.
Why it matters for companies
For anyone building research or competitor-monitoring agents, the post gives three practical rules: test each configuration several times before choosing, verify the search tool is actually returning results before blaming the model, and measure whether the agent opens pages, not only whether it searches. It also shows that verification harnesses shift work to people (8-30 facts per run for review) and help some models while hurting others, so they should be added for a specific error type, not by default.
Key passages in translation
Russian original
Разница между соседними местами в рейтинге — около 1 балла.
The gap between neighbouring places in the ranking is about 1 point.
Russian original
Точнее та модель, которая после поиска идёт читать страницы.
The more accurate model is the one that goes on to read the pages after searching.
Russian original
Сбой инструмента не равен слабости модели.
A tool failure is not the same as a weak model.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com
- Tags
- Open models