Classifying ~700 past inquiries to see what blocks Devin's first-pass investigation
Devinに問い合わせ調査を任せて見えてきた「原因未特定」の正体。過去の約700件を分類してみた
B · Useful, limited Canly engineering team classified about 700 resolved support escalations with a clear, published scheme; a useful operational analysis, but labels were assigned by an LLM and the figures are approximate by the author's own account.
Summary based on the full text·Original: about 10 minutes to read
In short
A Zenn post (21 October 2026) by Yamashita of Canly's store-marketing team. Canly uses Devin for first-pass investigation of customer-support escalations, with access to inquiry history, code and app logs but not the database. To see why many cases end as "cause unidentified", the team classified about 700 resolved escalations by type and by the evidence that settled the answer. The conclusion: what limits the agent is what it is allowed to see, more than how well it reasons.
Key numbers
- ~700resolved support inquiries escalated to engineering, classified by cause and deciding evidence
- 56%share of escalated inquiries that were not bugs
- 44%non-bug cases Devin could resolve with its current access (specs, code, app logs)
- 73%non-bug cases resolvable if Devin could read the production database (upper bound)
- 17%non-bug cases decided by the state of external services such as Google
As reported by the source.
Japanese·Web page·opens zenn.dev
What it reports
- Cause types: bugs 40%, spec or usage 22%, customer data state 20%, external services 14%, undetermined 4%. Non-bug cases total about 56%.
- Deciding evidence for non-bug cases: spec knowledge or past cases 33%, app logs 9%, code 2%, database data 29%, external service state 17%, uploaded files 6%, async jobs 2%, API access logs 1%.
- With current access Devin can settle about 44% of non-bug cases; with database read access the ceiling rises to about 73%, assuming it reads the right data correctly. About 41% of non-bug cases involved a database check at some point.
- The team expected infrastructure data (access logs, job queues) to be next most valuable; external service state (Google rejecting text it reads as a URL, review holds, listing limits) mattered far more.
- About 10% of all cases could have been answered by support staff themselves, often from the admin screen; a better inquiry form could reduce these without AI.
- A survey of six other companies' published setups found none giving the agent database access; all kept human review.
- Proposed controls for database access: read-only user on a replica with timeouts and row limits, column-restricted views with masking, query logging, prompt-injection defences for user-written text, write access limited to the inquiry thread, and confidence labels on answers.
How they got it
All resolved escalations with records (about 700) were labelled by an LLM on two axes, cause type and deciding evidence, using definitions published at the end of the post. The comparison of other companies draws on public articles as of September 2026.
Limits
- Labels were assigned by an LLM without a reported human validation sample; the author asks readers to treat figures as rough guides.
- The 73% figure is an upper bound that assumes perfect database use.
- One company, one product domain (store listings distributed to Google, Apple and Yahoo).
Why it matters for companies
Before buying a smarter model for support triage, count which evidence your resolved cases actually needed. This team found that access to data, and to partner systems, set the ceiling. The control list for giving an agent production read access is a reasonable starting checklist for any IT or security team facing the same request.
Key passages in translation
Japanese original
AIの一次調査がどこまで行けるかは、AIの賢さよりも、何を見られるようにしてあげるかで決まっている
How far AI first-pass investigation can go is decided less by how capable the AI is than by what we let it see.
Japanese original
全件を手作業で分類するのは難しいため、分類はLLMにやってもらいました。なので、数字はおおよその目安として見てもらえればと思います。
Classifying every case by hand was hard, so an LLM did the classification. Please treat the numbers as rough guides.
Japanese original
DBの次に効くのは、自社のインフラ側の情報ではなくて外部サービスの状態だった
After the database, the next most useful source was the state of external services, rather than our own infrastructure data.
Summary based on the full text. Read the original for the full argument and data.Read the original on zenn.dev