Proving is harder than migrating: claims search index moved to CDC (옮기는 것보다 증명이 어렵다)
A · Strong evidence Kakao engineering post with an explicit three-axis verification design, full-population comparisons, measured latency and load figures, a defect found in the old system, and a stated rule for what was delegated to an agent; method shown and numbers checkable.
Summary based on the full text·Original: about 18 minutes to read
In short
A Kakao tech blog post (3 November 2026, author eric.signal) on moving the search index for commerce claims (cancellations, exchanges, returns) from notification-based change capture to change data capture (CDC) from the database log. The old setup missed changes that raised no notification and reset the "days in status" clock on unrelated edits. The harder part was proving the new index correct. The team checked missing documents, values and lag separately, delegated repeatable comparisons to an agent and kept judgment with people. They did not use speed as a success metric.
Key numbers
- about 42,000, 0 mismatchesnew changes compared field by field over 7 consecutive days between old and new index
- 14.3 s to 0.8 smedian absolute indexing lag before vs after the move to CDC (about 18x)
- 166 of 830,000documents differing in a full comparison, all traced to a non-deterministic flag in the old index (0.02%)
- about 82,000 to 74queries per full-index batch against the source database after fixing batch size and duplicate lookups
- 19 to 32tests in the suite during verification, one added for each problem found
As reported by the source.
Korean·Web page·opens tech.kakao.com
What it reports
- The claims screen is used for tens of thousands of lookups a day; each document has 40+ fields built from 17+ tables.
- Old design: service notifications, a 5-second polling job and a 12-hour catch-up batch. Notification and data change were not in one transaction, so the index could miss or misorder changes (dual write).
- A return on day 9 whose fault party was edited showed as 1 day in status, because the clock used the generic modified_at column. The new design records status-change times separately.
- New design: four paths (create, update, catch-up batch, full rebuild) sharing document-building code on Spark. Out-of-order events converge to the same result instead of having their order restored.
- Verification: document counts by year in both directions (0 difference outside the retention-boundary year, 0 documents only in the new index); field-level comparison of all ~42,000 new changes over 7 days (0 mismatches); median lag 14.3 s to 0.8 s, with the new index first on every measured document.
- A full comparison found 166 of 830,000 documents different, all on one flag. Both codebases matched; the old index picked a different row from a multi-row table on each run.
- After cut-over the full-index batch overloaded the source database: a hardcoded batch of 200 and an average of 45 duplicate lookups (16.43 million lookups for 366,000 real targets). Fixes cut queries from about 82,000 to 74, CPU 45%, and run time from 12.9 to 11.6 minutes.
- Delegation rule: hand to the agent only work whose result can be checked (tests, a design document kept true to fact, comparison scripts). The agent ran the daily comparisons and summaries; people decided which side was wrong and what to give up.
How they got it
The team's own measurements on production data: aggregate counts for the whole population, full field-level comparison for new changes, per-path samples for historical data, and absolute and relative lag. External sources cited include Kleppmann on dual writes, Netflix's DBLog, a test-generation benchmark (40.21% fault detection) and METR's developer study.
Limits
- Agreement between the two indexes does not prove both are right; the authors say a shared cause could produce the same wrong value.
- Historical data was sampled, not fully compared; the 7-day window only covers state transitions that happened in that week.
- Specific to one Kakao system; no cost of the agent work is given.
Why it matters for companies
Any migration of a system people make decisions from needs a proof plan, not only a cut-over plan. Separating missing records, wrong values and lag catches failures a single metric hides, and treating the old system as the answer key risks copying its defects. The delegation rule is worth adopting as policy: let agents run what can be checked, and keep the call on which side is wrong with a person.
Key passages in translation
Korean original
위임 기준은 작업의 난이도가 아니라 검증 가능성이었습니다.
The criterion for delegation was whether the result could be verified, not how hard the task was.
Korean original
기존 색인을 정답으로 전제하면 기존 결함을 그대로 복제하게 됩니다.
If you assume the old index is the right answer, you copy its defects as they are.
Korean original
작업이 얼마나 빨라졌는지는 성과 지표로 사용하지 않았습니다.
We did not use how much faster the work became as a performance metric.
Summary based on the full text. Read the original for the full argument and data.Read the original on tech.kakao.com