Part 3: we deleted it and rewrote it
Мы удалили AI-менеджера для маркетплейсов и написали заново
B · Useful, limited A candid retrospective by the vendor Haimi explaining why its capped agent gave shallow answers despite a good judge score; evidence is the team's own reading of conversations plus one benchmark figure.
Summary based on the full text·Original: about 4 minutes to read
In short
Part 3 (30 October 2026, dated 31 October on the page) of Haimi's marketplace-seller assistant series. Four days after launching the capped agent from part 2, the team deleted it. A judge model scored it well, but reading the conversations showed confident answers built on a tenth of the shop's data, because tools returned up to 15 results and tables were truncated. Tool instructions made the model stop after two calls where twenty were needed. The team moved the loop to Pydantic AI, gave the agent the whole Wildberries API as one tool plus its documentation, removed truncation and usage advice, added web search, and kept one rule: every write waits for the owner's "yes", now as a single batch.
Key numbers
- 4.28judge score (1 to 5) for the marketplace manager agent on 8 typical tasks
- 7 of 9assistant roles where the agent beat the old phrase parser on the benchmark
- 582 / 121lines deleted and added when the in-house agent loop was replaced by Pydantic AI
- 307Wildberries API methods now reachable through one generic tool, with the official docs as a second tool
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- Benchmark: 8 typical tasks per role, judge scores 1 to 5; the agent beat the old parser in 7 of 9 roles; the marketplace manager scored 4.28.
- The judge missed the main failure: the agent's conclusion matched the data it saw, but it saw about ten products out of a hundred.
- Narrow wrappers blocked real requests, such as reviewing all reviews for six months; the "no tool" list grew faster than wrappers could be written.
- Raising limits to 50 steps and 15 minutes changed little, since each step still returned a fragment of data.
- The in-house loop (tool calls, retries, compression of old steps, tool groups) was ordinary ReAct, as a reader pointed out, and was replaced: 582 lines deleted, 121 added.
- New principle: give access, the tool and the docs; let the agent decide how to use them. One
wb_apitool covers 307 methods,wb_helpserves the official reference; MPStats is connected the same way. - Writes collect into one batch shown as "before to after"; the owner approves once; execution runs in order and stops at the first error. Reads run in parallel, writes one at a time.
- Successful API calls are saved as per-project "recipes"; a dropped model or search call resumes mid-turn.
How they got it
Team review of a benchmark run and of real conversations, with code excerpts. Results of the new version are deferred to part 4.
Limits
- The benchmark is the team's own (8 tasks per role, LLM judge); no per-task results.
- "Shallow" answers are described qualitatively; no counts of affected conversations.
- Vendor post; the new design's cost and accuracy are not reported.
Why it matters for companies
The most transferable finding is that an LLM judge that sees only the agent's inputs will approve a confident answer drawn from a sliver of the data. Evaluations need to check coverage as well as consistency. The second point is about where to spend engineering: the loop is a commodity, while the tools, write rules and approval flow are the part worth owning.
Key passages in translation
Russian original
Вывод звучит гладко, только сделан он по десятой части магазина. Судья такой ответ не ловит.
The conclusion sounds smooth, only it is drawn from a tenth of the shop. The judge does not catch such an answer.
Russian original
Даже там, где задача требовала двадцати вызовов, она делала два и отвечала тем, что есть.
Even where the task needed twenty calls, it made two and answered with what it had.
Russian original
Раньше мы давали агенту инструменты и объясняли, как ими пользоваться. Теперь даём доступ, инструмент и справку.
Before, we gave the agent tools and explained how to use them. Now we give it access, a tool and the documentation.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com