Why our LLM cost tripled when we only swapped the model
모델만 바꿨는데 LLM 비용이 3배가 된 이유
A · Strong evidence Daangn backend engineer documents a production cost incident with before-and-after hit rates, the mechanism, a break-even formula and fixes tied to provider documentation; a well-documented case with numbers readers can check against their own logs.
Summary based on the full text·Original: about 15 minutes to read
In short
A Daangn (Karrot) tech blog post on Medium (21 October 2026) by Jaemin Seo ("Jay"), backend engineer on the Community Home team. The team uses an LLM to classify community posts by topic for its recommendation feed. In August 2026 it moved the job from Gemini to OpenAI GPT-5.6 for a 20% lower unit price. Responses were fine and nothing errored, but five days later cost per request was 3.2 times higher. GPT-5.6 places its automatic cache breakpoint at the end of the latest message, so the shared instructions were never cached on their own. The post explains prompt caching across OpenAI, Anthropic and Gemini and how to check it.
Key numbers
- 20%lower unit price that motivated the switch from Gemini to GPT-5.6 in August 2026
- 3.2xrise in cost per request five days after the switch
- 85.8% to 2.3%fall in prompt-cache hit rate (token basis) caused by the switch
- ~22%break-even hit rate below which caching costs more than no cache, with a 1.25x write premium
- ~95% / ~1/5hit rate after the fix and resulting cost per request versus the broken state
As reported by the source.
Korean·Web page·opens medium.com
What it reports
- Requests were a long fixed block (instructions and category list) followed by a short post body, sent as one message. Gemini's implicit cache read the shared prefix; GPT-5.5 placed breakpoints every 2,048 tokens and would have too.
- GPT-5.6 and later place one breakpoint at the end of the most recent message, so each cached chunk was "instructions + post A" and never matched "instructions + post B". Almost every request paid the write premium without ever reading.
- Hit rate fell from 85.8% to 2.3%. The only visible difference in responses was one cache field in the usage data.
- Pricing: reads are mostly 0.1x of the input price (0.05x for some models); OpenAI GPT-5.6+ charges 1.25x to write; Anthropic 1.25x for 5-minute and 2x for 1-hour caches; Gemini implicit has no write premium.
- Break-even: average multiplier 0.1h + 1.25(1 - h) exceeds 1 when hit rate h is below about 22%, independent of prompt length.
- Fix: the internal LLM platform now inserts an explicit breakpoint before a template's first variable. Hit rate rose to about 95% and cost per request fell to about one fifth.
- Things that silently break prefixes: timestamps or request IDs near the top, unstable JSON key order, changes to tool lists or their order, output schemas, and settings such as reasoning effort.
- Minimum cacheable lengths and cache lifetimes differ by provider and model; a middleware layer once dropped
cache_controlon some requests. - Model swaps should be treated as changes to operating conditions: compare cost per request, latency and token-basis hit rate on production traffic for at least a day.
How they got it
First-hand incident analysis from production usage data, with mechanism explained from model architecture and provider documentation as of October 2026. Hit rates are token-based. Code samples show the explicit-breakpoint fix for each provider.
Limits
- One workload (short varying input after a long fixed prefix); gains differ for chat or agent patterns.
- Provider pricing and cache rules change by model version; the author says to recheck documentation.
- No absolute spend figures; ratios only.
Why it matters for companies
Most firms track LLM unit prices, not hit rates, so a cheaper model can raise the bill with no error to flag it. Finance and platform teams should add token-basis cache hit rate and cost per request to model-change checklists and dashboards. Putting breakpoint placement in a shared platform, rather than leaving it to each team's prompts, is the control that held here.
Key passages in translation
Korean original
닷새 뒤 보니 단가는 내렸는데 요청당 비용은 3.2배가 돼 있었어요.
Five days later, the unit price had fallen but the cost per request had become 3.2 times higher.
Korean original
캐시가 깨져도 에러는 나지 않아요. 비용만 늘어요.
When the cache breaks, there is no error. Only the cost goes up.
Korean original
모델 교체는 코드 한 줄이 아니라 운영 조건을 바꾸는 일로 다뤄야 해요.
A model swap has to be treated as a change to operating conditions, not a one-line code change.
Summary based on the full text. Read the original for the full argument and data.Read the original on medium.com
- Tags
- Operations