Automating sentiment on all X mentions with Jev
JevでXのネガポジ判定を全件自動化。CSだけでなくチーム全体でお客様の声を拾う
B · Useful, limited Engineering post from Nihon Toreka Center describing a working sentiment-monitoring tool with clear design choices and costs; practical, but two weeks of operation and no accuracy figures.
Summary based on the full text·Original: about 8 minutes to read
In short
A Zenn post (22 October 2026) by Sato of Nihon Toreka Center, which runs a trading-card lottery app. Customer support used to search X by hand for complaints. The team built x-pulse, which pulls mentions every five minutes, classifies each with TypeSafe's Jev model (which returns probabilities for typed questions rather than generated text) and posts negative ones to Slack. In about two weeks of use, threads on flagged posts pulled in product, logistics, legal and the official X account team.
Key numbers
- ~$5/monthJev classification cost; X API reads at $0.005 per post are the main expense
- 35%share of mentions that were campaign-entry replies to the official account in a 100-post test
- 22%share of posts left undeterminable before a campaign-reply category was added
- 6.3xslowdown from sending one post per request instead of 25 per request (also 1.3x costlier)
- ~0.7 sclassification time per post
As reported by the source.
Japanese·Web page·opens zenn.dev
What it reports
- Three questions per post: relevance (is the service the subject, not just named), one of eight categories, and five-level sentiment with an instruction to read sarcasm by intent.
- A 100-post test found campaign-entry replies made up about 35% of mentions; without a category for them, 22% of posts were undeterminable. An eighth category fixed this.
- 25 posts x 3 questions go in one request; single-post requests were 6.3x slower and 1.3x more expensive.
- Raw probabilities are stored; thresholds (negative at internal value 1.2 or below with confidence 0.6 or above; category confidence below 0.7 counts as undeterminable) apply at read time, so changing them recomputes history at no cost.
- Model pinned to jev-1.13.0 because thresholds are tuned to its distributions. Price listed as $0.042 per million input tokens; output is free.
- Jev costs about $5 a month; X API reads at $0.005 per post dominate cost. Classification takes about 0.7 s per post.
- Outcomes are anecdotal: faster guidance on app storage use, a changed LINE support path, early warning of a carrier tracking issue, and a keyword-editing screen shipped the same day a missed abbreviation was reported.
How they got it
Build-and-operate report from the engineering team, live since 3 October 2026. Question wording was tuned on 100 real posts and locked with golden tests; any wording change triggers a new accuracy check. Feedback comes from three internal managers.
Limits
- No precision, recall or agreement figures are published, although an accuracy check is described.
- The vendor states its model is trained mainly on English and that Japanese accuracy is not equivalent.
- About two weeks of operation; benefits are internal testimonials.
Why it matters for companies
The design choices transfer to any classification pipeline: store probabilities and apply thresholds later, pin model versions, batch requests, and let business staff edit search keywords without a deploy. The cost profile is a reminder that data access (here the X API) can cost far more than the model. The organisational gain was routing customer complaints to the owning department in one shared thread.
Key passages in translation
Japanese original
投稿本文は信頼できないデータとして扱っていて、Slack の mrkdwn の制御文字をエスケープし、リンクのプレビュー展開も無効にしています。
We treat post text as untrusted data: we escape Slack mrkdwn control characters and disable link previews.
Japanese original
こうしておくと、閾値を変えたときに Jev で再判定しなくても、過去の全期間の集計が変わります。
This way, when we change a threshold, the totals for the whole history update without re-classifying anything with Jev.
Japanese original
運用にかかるお金の大部分は X API の料金です。
Most of the running cost is the X API fees.
Summary based on the full text. Read the original for the full argument and data.Read the original on zenn.dev
- Tags
- Cost