Why a former store clerk ranked near the top of Krafton's AI ability test
크래프톤 AI 역량 평가에서 개발 경험 없는 매장 직원 출신이 상위권에 오른 이유
B · Useful, limited Builder Josh podcast with the two Krafton AI Frontier builders of the Probe hiring assessment; a detailed first-hand design account, but validity evidence is one internal hire and hackathon anecdotes.
Summary based on the full transcript·Original: about 48 minutes to listen
In short
An episode of the Korean HOW I AI podcast on the Builder Josh channel (13 October 2026, 48 minutes) with Koo Han-mo (product owner, ex-Bain) and Cho Jeong-seok (AI product engineer, ex-Naver Z) from Krafton's AI Frontier division. They built Probe, the first product of Krafton's "AI-native talent" platform Cofa. It replaces coding tests with a work simulation in which candidates use Claude Code or Codex, question stakeholder agents about a deliberately vague brief, and get scored by a panel of agents on how they work with AI.
Key numbers
- 20+evaluator agents running in parallel to score one candidate's task
- about 6 hourslength of the hackathon version of the Probe work simulation
- ~95 vs low 70scalibration scores when models take the test: Opus about 95, Haiku in the low 70s, Sonnet between
- about 2 yearsstore experience of the non-developer candidate who reached the top tier at a hackathon
As reported by the source.
Korean·Video·opens youtube.com
What it reports
- Distinction between AI-literate people, who speed up their existing workflow, and AI-native people, who redesign the workflow from the goal, including building a tool instead of buying one.
- Environment: IDE-like workspace with a terminal running Claude Code or Codex, browser preview, option to upload one's own skills; the brief is modelled on a real Krafton task.
- Stakeholder agents answer questions but will not hand over the solution; candidates who build context through small questions score well, while "tell me the four things I need to do" is deflected.
- More than 20 agents score each candidate on three axes, taken from a paper on technical, intent and cognitive debt: output quality and architecture; how the candidate steers AI; and whether they understand and can defend their own output.
- Token and cost use is tracked; the better candidates spend less.
- Calibration: when models take the test, scores follow benchmark order (Opus about 95, Haiku low 70s).
- At a hackathon with CJ Olive Young, a non-developer with about two years in an Olive Young store reached the top tier. Probe has been used for one internal hire and is planned for Krafton's open recruitment and an SK hynix hackathon.
How they got it
Interview with the product's builders. Transcript from Korean auto-captions. Outcomes are the speakers' own accounts.
Limits
- No predictive validity data: no link yet between Probe scores and later job performance.
- No information on fairness checks, candidate experience or adverse impact.
- The builders are promoting their own product.
Why it matters for companies
For HR and hiring teams this is a concrete design to borrow from: assess people doing real, ambiguous work with AI tools, score how they question, decompose and verify, and test whether they understand what the AI produced. Cost per candidate as a signal is a simple, cheap addition. The speakers' advice to HR is the right first step: agree internally what "AI ability" means, ideally by taking the test yourselves. Before using such tests in selection decisions, firms need their own validity and fairness evidence, which this source does not provide.
Key passages in translation
Korean original
저희 개발자 리드분께서 지금은 코드를 사랑하는 사람은 뽑고 싶지 않다라고 하시더라고요.
Our lead developer said that right now he doesn't want to hire people who love code.
Korean original
사실 되게 재밌는 거는 잘하시는 분들일수록 비용 적게 쓰시거든요.
What's actually really interesting is that the better people are, the less they spend.
Korean original
그 한 명의 과제를 평가하기 위해서 20개가 넘는 에이전트가 병렬적으로 돌아가거든요.
To evaluate one person's task, more than 20 agents run in parallel.
Summary based on the full transcript. Read the original for the full argument and data.Watch the original on youtube.com