--- id: "lib-2026-10-07-needle-an-open-benchmark-of-web-search-engines" type: "library" title: "NEEDLE: an open benchmark of web-search engines for agents" title_original: "NEEDLE: открытый бенчмарк движков для веб-поиска" lang: "RU" org: "Yandex for ML (Practical ML Conf)" source_type: "Talk" date: "2026-10-07" week: "2026-W41" url: "https://nosov.ai/library/lib-2026-10-07-needle-an-open-benchmark-of-web-search-engines/" original_url: "https://www.youtube.com/watch?v=nWGtr2w7-7g" tier: "B" tier_name: "Useful, limited" tier_reason: "Conference talk by Ilya Gusev, a known Russian NLP open-source author, with concrete prices, benchmark construction details and an engine-overlap analysis; the speaker's own start-up tops his own leaderboard." summary_basis: "transcript" original_minutes: "30" tags: ["ai technology", "benchmarks", "agent web search"] frames: ["verification-capacity"] --- # NEEDLE: an open benchmark of web-search engines for agents Original title: NEEDLE: открытый бенчмарк движков для веб-поиска Tier B (Useful, limited): Conference talk by Ilya Gusev, a known Russian NLP open-source author, with concrete prices, benchmark construction details and an engine-overlap analysis; the speaker's own start-up tops his own leaderboard. Summary based on the full transcript. ## In short A 30-minute Russian-language talk at Yandex's Practical ML Conf, published 7 October 2026, by Ilya Gusev, lead researcher at a web-search-for-agents start-up founded in December 2025 and author of the Saiga open models. He argues that the search API an agent calls has gone largely unmeasured, and that static benchmarks are easy to game because answers leak online, end up in training data or get cached by the engines. His team built NEEDLE, an open benchmark that regenerates queries from fresh feeds. Its overlap analysis shows that many agent-search vendors resell Brave or Google results. ## Key numbers - 300 combinations, ~$30,000: cost of one full agent benchmark run across models, harnesses and search engines at ~$100 per combination - >80%: share of GPT-5.6 Luna's search queries that use quotation marks in Artificial Analysis runs - $7 vs $0.30: price per 1,000 queries: Exa vs a Google-wrapper API - 600 of 6,000: AA-Omniscience questions that are public; the other two index components are fully public ## What it reports - Agents use two tools: search (query to documents) and fetch (URL to page text). Human-oriented search assumes rank order matters and returns 10 snippets; agents run dozens of queries in parallel and lean on `site:` operators and quotes. - Artificial Analysis search index scores: Perplexity 80, Parallel 75, the speaker's engine 67. Components: BrowseComp (1,300 multi-hop questions), DeepSearchQA and AA-Omniscience. - Failure modes seen: agents downloading published answer sets, recalling BrowseComp answers before any search, and engines able to run any seen query against competitors and cache good results. - The AA harness is custom, uses GPT-5.6 Luna, and fetches pages with a basic Python request that cannot read PDFs or get past captchas. - NEEDLE builds queries from news (RSS, Google Trends), finance (Wikidata and similar), US court cases, arXiv and other science feeds, plus filtered open agent logs from Hugging Face. Only that last slice, one in five (20%), is agentic. - Scoring uses an exact gold document where known, otherwise a 0-4 relevance grade by an LLM judge (GPT-5.6). - Results: the speaker's engine first, Exa best of the rest; then a Brave cluster (Parallel, You.com, Brave) and a Google-wrapper cluster (Serper, Tavily, TinyFish, Firecrawl). Shared wrong results show You.com reselling Brave, and Firecrawl and TinyFish drawing on Google. - Prices per 1,000 queries: Exa $7, the speaker's engine $1 for companies and $4 for individuals, a Google wrapper $0.30. ## How they got it A practitioner talk drawing on his team's benchmark work. The overlap measure counts the share of queries where two engines return the same irrelevant document, or more than three shared documents. Source: YouTube auto-captions; some names are garbled and only clearly stated figures are used here. ## Limits - The speaker sells a competing search API and his engine ranks first on his own benchmark, which he acknowledges. - He concedes NEEDLE is mostly non-agentic and its query mix has drifted from real user queries. ## Why it matters for companies When a company builds an agent that searches the web, the search provider shapes answer quality as much as the model, yet procurement usually looks only at the model. Several "different" vendors return the same underlying index, so paying more may buy only a wrapper. Public leaderboards for search are easy to game. Test candidate engines on your own recent queries before signing. ## Key passages in translation > Google оптимизировался под людей. Он оптимизировался под то, например, что порядок результатов выдачи важен, что первый результат выдачи важнее, чем, например, пятый. Для агентов это, вообще говоря, неверно. Google was optimized for people. It was optimized, for example, on the assumption that the order of results matters, that the first result is more important than, say, the fifth. For agents this is, generally speaking, not true. > агенту зачастую проще скачать правильный ответы из интернета, вместо того, чтобы искать по каким-то странным сайтам, проходить через много шагов It is often easier for the agent to download the correct answers from the internet than to search obscure sites and go through many steps. > если поисковый движок хоть раз видел запрос, можно в оффлайне запустить этот же запрос по всем конкурентам If a search engine has seen a query even once, it can run that same query offline against all its competitors. > Он должен быть динамичным. То есть запросы должны регулярно пересобираться на основе какой-то новой информации, чтобы их нельзя было запомнить в модели It has to be dynamic. That is, the queries must be rebuilt regularly from new information so that a model cannot memorize them. > вам нужно, наверное, лучше брать тот вариант ... который дешевле, потому что, скорее всего, в остальном они особо не будут отличаться. You should probably take the cheaper option, because otherwise they most likely won't differ much. Original: https://www.youtube.com/watch?v=nWGtr2w7-7g