Building an AI analysis environment a team can trust: context design in the recommendation team
팀이 신뢰할 수 있는 AI 분석 환경 만들기: 추천팀의 컨텍스트 설계
B · Useful, limited Kakao recommendation engineer's design for shared context in agent-run metric analysis, with file layout and catalog examples; clear and reusable, but no adoption, accuracy or time figures, and the worked case is reconstructed.
Summary based on the full text·Original: about 8 minutes to read
In short
A Kakao tech blog post (28 October 2026) by a developer of KakaoTalk short-form video recommendation models. The team uses AI agents to write SQL on Hadoop and produce metric reports. Once several people used it, two developers got different "play count" numbers for the same metric: one agent used the video player's logs, the other the app's. Neither was wrong, but the numbers were not comparable. The fix moved knowledge out of personal instruction files into three shared contexts: a per-table data catalog, a common analyst procedure, and task background read from Jira.
Key numbers
- 3context types the agent reads before analysis: data catalog (what), analyst persona (how), task context (why)
- 6steps in the analyst persona: classify, design, trap check, analyse, interpret, report
- 5trap-check items passed in the worked A/B example before any conclusion
- 1catalog document per log table, holding usage rules not found in the schema
As reported by the source.
Korean·Web page·opens tech.kakao.com
What it reports
- Three gaps in the old setup: implicit data knowledge (which log, which rules) lived in people's heads; analysis methods and even report style varied by person and by model; the purpose and hypothesis of each task were not written down.
- The contexts are split because they change at different rates: metric definitions are reused across analyses, methods are reapplied, task context changes as the task moves.
- Data catalog, inspired by Google Cloud's Open Knowledge Format idea: one Markdown file per table with its purpose, metadata, tricky columns, aggregation rules and facts checked in data. Schema, partition and distribution facts are filled by LLM analysis; policy definitions, such as what counts as a valid view or a "cold" user, are written by people. The author compares it to a semantic layer.
- The example catalog states that the play log is the default source for weekly metrics, A/B tests and before-after comparisons, which resolves the play-count conflict, and records that one flag column is false in every row and must not be used.
- Analyst persona, built from past analyses: a fixed 6-step procedure, a trap check for logging or service changes before reading anything as user behaviour, and reports that lead with the conclusion and separate fact from estimate. It deliberately does not reuse past conclusions, because each task has different success metrics.
- Task context: agents read the Jira issue and comments, where people log observation, hypothesis, results, rejected hypotheses and next question.
- Worked case (reconstructed with figures removed): an A/B test of video embeddings as a new feature. The interim result was "no harm observed, gain not confirmed"; the agent proposed extending observation and judging by a pre-set rule. A later run met the rule; whether to apply it was a separate human decision on cost and benefit.
How they got it
The author's description of the team's setup, with the repository layout, a sample catalog file and a sample Jira comment. The A/B example is presented as a reconstructed case without real figures.
Limits
- No numbers on accuracy, agreement between analysts, time saved or how many people use it.
- One team, one domain (recommendation metrics).
- The worked example is reconstructed, not a live record.
Why it matters for companies
Self-service analysis with agents fails quietly when two people get two numbers for one metric and both look valid. This post gives a small, copyable structure: write down table-level usage rules that the schema cannot express, fix a shared analysis procedure with a trap check, and feed task purpose from the ticket system. It also draws a sensible line between facts an LLM may fill in and definitions only people may set.
Key passages in translation
Korean original
어느 한쪽이 틀린 데이터는 아니었지만, 쿼리가 실행되고 결과가 나왔다는 사실만으로는 두 숫자를 같은 의미로 비교할 수 없었습니다.
Neither data source was wrong, but the fact that a query ran and returned a result did not make the two numbers comparable.
Korean original
스키마·파티션·값의 분포 같은 사실은 LLM 분석결과로 채우고, 유효재생 기준처럼 정책에 해당하는 정의는 사람이 작성함
Facts such as schema, partitions and value distributions are filled from LLM analysis; policy definitions such as the valid-view rule are written by people.
Korean original
실제로 적용할지는 기대 효과와 전환에 따르는 운영·학습 비용을 견주어 사람이 따로 판단했습니다.
Whether to actually apply it was decided separately by people, weighing the expected effect against the operating and training costs of switching.
Summary based on the full text. Read the original for the full argument and data.Read the original on tech.kakao.com
- Tags
- AI enablement