---
id: "lib-2026-10-27-cutting-llm-cost-with-a-snap-snap-selective"
type: "library"
title: "Cutting LLM cost with a snap (SNAP: Selective Native AI Processing)"
title_original: "손가락 튕기기로 LLM 비용 줄이기"
lang: "KO"
org: "Toss (toss.tech)"
source_type: "Practitioner post"
date: "2026-10-27"
week: "2026-W44"
url: "https://nosov.ai/library/lib-2026-10-27-cutting-llm-cost-with-a-snap-snap-selective/"
original_url: "https://toss.tech/article/snap-llm-cost-reduction"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Toss ML engineers document a confidence-gated small-model fallback with offline thresholds, production coverage, token savings and load-test latency; method and numbers are specific, though quality is measured against LLM labels and no money figures are given."
summary_basis: "full-text"
original_minutes: "7"
tags: ["ai technology", "llm cost", "small model routing", "distillation"]
---

# Cutting LLM cost with a snap (SNAP: Selective Native AI Processing)

Original title: 손가락 튕기기로 LLM 비용 줄이기

Tier A (Strong evidence): Toss ML engineers document a confidence-gated small-model fallback with offline thresholds, production coverage, token savings and load-test latency; method and numbers are specific, though quality is measured against LLM labels and no money figures are given.

Summary based on the full text.

## In short

A Toss tech blog post (27 October 2026) by three engineers on SNAP (Selective Native AI Processing). Many LLM calls do simple jobs, such as deciding whether two product listings are the same item. SNAP trains small task-specific models from LLM logs and lets them answer only when their confidence clears a threshold set offline; everything else falls back to the LLM. The first deployment, a product-matching classifier, took about a third of daily requests in production, close to the offline estimate.

## Key numbers

- 34.74%: offline coverage at a 99.9% agreement target (34,963 of 100,648 validation requests)
- 51.59% / 72.35%: coverage if the agreement target is lowered to 99% / 95%
- about 33-37%: daily share of product-matching requests the small model handled in about 6 days of production
- about 7.6 billion: LLM input tokens avoided in that period (plus about 45.2 million output tokens)
- 7.24 ms: p50 latency at 598 requests per second on one L4 GPU, above the ~500/s production peak

## What it reports

- Tasks are identified by hashing the start of the system prompt, where role and task definition stay stable even when few-shot examples change.
- A daily Airflow batch groups logs by this fingerprint, totals calls, models, tokens and estimated cost, ranks tasks above a cost floor, and selects a fixed number (the example shows the top 30) for checks.
- Selected tasks pass only if responses are structured enough to extract labels and there is enough data per label for separate training and evaluation. Passing tasks get a trained model within a day, registered and served.
- First task: compare names and options of two products (brand, model, size, colour, contents) and decide whether they are the same sale unit.
- Thresholds are chosen on selective accuracy (agreement with the existing LLM on accepted requests) and coverage. At a 99.9% target: 34,963 of 100,648 accepted, 34 disagreements, 99.9028% agreement.
- The client calls the small model through the GenAI gateway first and calls the LLM only below threshold.
- About 6 days in production: 33-37% of daily requests handled; about 7.6 billion input and 45.2 million output tokens avoided.
- Load test from 10 to 1,200 requests a second: p50 9.02 ms and p99 15.23 ms at 1,203/s actual throughput, no failed or dropped requests.
- A similar full-replacement project needed 1-2 months of repeated test-set iterations; SNAP starts saving before a model handles everything.

## How they got it

Offline validation on 100,648 logged requests with LLM outputs as labels, a short production run with daily handling shares, and a client-side load test on one L4 GPU. The model and evaluation-data table in the post was not available in the saved copy.

## Limits

- Quality is agreement with the incumbent LLM, not with human-verified ground truth; LLM errors carry over as labels.
- Production results cover about 6 days and one task.
- Token savings are given; won or dollar savings and GPU cost are not.
- The cost-ranking table in the post uses illustrative values.

## Why it matters for companies

Most companies with LLM workloads have a long tail of repetitive classification and extraction calls that cost frontier-model prices. The pattern here is cheap to copy: fingerprint calls by system prompt, rank by spend, train small models on the LLM's own outputs, and route by a calibrated confidence threshold with LLM fallback. It turns cost reduction into a routine pipeline rather than a series of one-off migration projects, and the agreement target becomes an explicit business choice between quality and savings.

## Key passages in translation

> SNAP의 핵심은 해결해야 하는 문제의 모든 요청을 한 번에 Native Model로 처리하는 것이 아니라, 작은 모델도 높은 신뢰도로 처리할 수 있는 트래픽부터 빠르게 처리하는 것입니다.

The core of SNAP is to quickly take the traffic a small model can handle with high confidence first, instead of moving every request for a task to a native model at once.

> Confidence가 0.95라고 해서 정답률이 95%라는 뜻은 아니죠.

A confidence of 0.95 does not mean the answer is right 95% of the time.

> 모든 요청을 해결하는 모델을 완성하기 전에, 운영에서 LLM 호출을 줄이기 시작한 것입니다.

We started cutting LLM calls in production before finishing a model that solves every request.

Original: https://toss.tech/article/snap-llm-cost-reduction