NEEDLE: an open benchmark of web-search engines for agents
NEEDLE: открытый бенчмарк движков для веб-поиска
B · Useful, limited Conference talk by Ilya Gusev, a known Russian NLP open-source author, with concrete prices, benchmark construction details and an engine-overlap analysis; the speaker's own start-up tops his own leaderboard.
Summary based on the full transcript·Original: about 30 minutes to watch
In short
A 30-minute Russian-language talk at Yandex's Practical ML Conf, published 7 October 2026, by Ilya Gusev, lead researcher at a web-search-for-agents start-up founded in December 2025 and author of the Saiga open models. He argues that the search API an agent calls has gone largely unmeasured, and that static benchmarks are easy to game because answers leak online, end up in training data or get cached by the engines. His team built NEEDLE, an open benchmark that regenerates queries from fresh feeds. Its overlap analysis shows that many agent-search vendors resell Brave or Google results.
Key numbers
- 300 combinations, ~$30,000cost of one full agent benchmark run across models, harnesses and search engines at ~$100 per combination
- >80%share of GPT-5.6 Luna's search queries that use quotation marks in Artificial Analysis runs
- $7 vs $0.30price per 1,000 queries: Exa vs a Google-wrapper API
- 600 of 6,000AA-Omniscience questions that are public; the other two index components are fully public
As reported by the source.
Russian·Video·opens youtube.com
What it reports
- Agents use two tools: search (query to documents) and fetch (URL to page text). Human-oriented search assumes rank order matters and returns 10 snippets; agents run dozens of queries in parallel and lean on
site:operators and quotes. - Artificial Analysis search index scores: Perplexity 80, Parallel 75, the speaker's engine 67. Components: BrowseComp (1,300 multi-hop questions), DeepSearchQA and AA-Omniscience.
- Failure modes seen: agents downloading published answer sets, recalling BrowseComp answers before any search, and engines able to run any seen query against competitors and cache good results.
- The AA harness is custom, uses GPT-5.6 Luna, and fetches pages with a basic Python request that cannot read PDFs or get past captchas.
- NEEDLE builds queries from news (RSS, Google Trends), finance (Wikidata and similar), US court cases, arXiv and other science feeds, plus filtered open agent logs from Hugging Face. Only that last slice, one in five (20%), is agentic.
- Scoring uses an exact gold document where known, otherwise a 0-4 relevance grade by an LLM judge (GPT-5.6).
- Results: the speaker's engine first, Exa best of the rest; then a Brave cluster (Parallel, You.com, Brave) and a Google-wrapper cluster (Serper, Tavily, TinyFish, Firecrawl). Shared wrong results show You.com reselling Brave, and Firecrawl and TinyFish drawing on Google.
- Prices per 1,000 queries: Exa $7, the speaker's engine $1 for companies and $4 for individuals, a Google wrapper $0.30.
How they got it
A practitioner talk drawing on his team's benchmark work. The overlap measure counts the share of queries where two engines return the same irrelevant document, or more than three shared documents. Source: YouTube auto-captions; some names are garbled and only clearly stated figures are used here.
Limits
- The speaker sells a competing search API and his engine ranks first on his own benchmark, which he acknowledges.
- He concedes NEEDLE is mostly non-agentic and its query mix has drifted from real user queries.
Why it matters for companies
When a company builds an agent that searches the web, the search provider shapes answer quality as much as the model, yet procurement usually looks only at the model. Several "different" vendors return the same underlying index, so paying more may buy only a wrapper. Public leaderboards for search are easy to game. Test candidate engines on your own recent queries before signing.
Key passages in translation
Russian original
Google оптимизировался под людей. Он оптимизировался под то, например, что порядок результатов выдачи важен, что первый результат выдачи важнее, чем, например, пятый. Для агентов это, вообще говоря, неверно.
Google was optimized for people. It was optimized, for example, on the assumption that the order of results matters, that the first result is more important than, say, the fifth. For agents this is, generally speaking, not true.
Russian original
агенту зачастую проще скачать правильный ответы из интернета, вместо того, чтобы искать по каким-то странным сайтам, проходить через много шагов
It is often easier for the agent to download the correct answers from the internet than to search obscure sites and go through many steps.
Russian original
если поисковый движок хоть раз видел запрос, можно в оффлайне запустить этот же запрос по всем конкурентам
If a search engine has seen a query even once, it can run that same query offline against all its competitors.
Russian original
Он должен быть динамичным. То есть запросы должны регулярно пересобираться на основе какой-то новой информации, чтобы их нельзя было запомнить в модели
It has to be dynamic. That is, the queries must be rebuilt regularly from new information so that a model cannot memorize them.
Russian original
вам нужно, наверное, лучше брать тот вариант ... который дешевле, потому что, скорее всего, в остальном они особо не будут отличаться.
You should probably take the cheaper option, because otherwise they most likely won't differ much.
Summary based on the full transcript. Read the original for the full argument and data.Watch the original on youtube.com