Why our FDEs are fighting Voice AI
요즘 뜬다는 FDE, 왜 Voice AI랑 싸우고 있을까?
B · Useful, limited Channel Talk forward-deployed engineer describes an AI-plays-customer QA runner for a voice agent with its scoring design; specific and practical, but outcome figures are two colleagues' estimates.
Summary based on the full text·Original: about 6 minutes to read
In short
A Channel Talk tech blog post (29 October 2026) by a forward-deployed engineer who configures Voice ALF, the company's AI phone agent, for customers. Each conversation flow (a "Task") had to be tested by staff calling in as the customer, one call at a time, and flows that passed once failed on retest. The team built a runner where an AI plays the customer in text, many scenarios run in parallel, and an evaluator AI checks the transcript against the execution log on eight criteria. Deployment staff now make a fraction of the manual calls, and one runs an agent overnight to revise a Task until 10 tests pass.
Key numbers
- 10 x 20 = 200test calls requested in one case (10 scenarios, 20 runs each), the load that triggered the tool
- about 50 to 5manual test calls per voice Task for one deployment-team member after the tool
- about 20%another member's manual test calls compared with before
- 8pass criteria an evaluator AI checks; all must pass for a PASS
- 20 / 15 / 15points weighted to conversation flow, procedure completion and step-instruction compliance
As reported by the source.
Korean·Web page·opens tech.channel.io
What it reports
- Failure types seen in real calls: repeating a question, acting before the customer answered, and getting stuck at one step. Unlike chat, one person cannot hold several voice calls at once.
- The runner skips speech: it passes text between the customer AI and the agent's dialogue engine, below speech recognition and synthesis.
- First bug: announcement steps waited for a "playback finished" event that never came in text-only tests. The runner now checks the turn number on each text-done event and sends the matching speech-done signal.
- Task completion alone was misleading: one run stored a refund decision before the customer answered. The evaluator compares conversation and execution records, including interim stored values, and separates agent errors from runner errors.
- Stuck nodes were the most damaging issue in real QA, so scoring favours finishing correctly. Completion rate matters, but skipped answers or wrong data count as failure. The evaluator gives a reason per criterion and a program adds up the score.
- Preparation, execution, evaluation and reporting are bundled as one skill an agent can repeat.
- The new-deployment team uses it for new features; the team that sets up AI for each customer has adopted it too. It is now used while drafting Tasks, not only at the end.
- After passing, testers change questions or conditions to check the Task is not fitted to one scenario.
- Next: real phone-network tests by connecting the outbound calling product to the inbound agent, to cover recognition and interruptions.
How they got it
The engineer's description of the tool, with reported effects from two named colleagues on the deployment team. No aggregate data.
Limits
- Text-level tests skip speech recognition, synthesis and barge-in, which are still tested by hand.
- Savings are two people's estimates and vary by Task, as the post says.
- An AI evaluator grading an AI agent may share blind spots; no human audit rate is given.
Why it matters for companies
Voice agents are hard to test because calls do not parallelise and runs are not repeatable. This design, a simulated customer, an evaluator that reads execution logs rather than only the transcript, and a strict all-criteria pass rule, is reusable for any conversational agent that takes actions. The overnight "revise until tests pass" pattern raises the same risk as in coding agents, overfitting to the test set, which the team handles by changing scenarios after a pass.
Key passages in translation
Korean original
한 번 잘 끝난 상담도 다시 테스트하면 고객의 답을 기다리지 않거나 같은 질문을 반복하는 경우가 있었어요.
Even a conversation that went well once would, on retest, sometimes not wait for the customer's answer or repeat the same question.
Korean original
Task가 끝났다는 사실만으로는 상담이 올바르게 진행됐는지 알 수 없었어요.
The fact that a Task finished did not tell us whether the conversation had gone correctly.
Korean original
여덟 항목을 모두 충족해야 통과(PASS)하고, 하나라도 충족하지 못하면 실패(FAIL)로 판정해요.
All eight criteria must be met to pass; if even one is not met, the run is judged a failure.
Summary based on the full text. Read the original for the full argument and data.Read the original on tech.channel.io
- Tags
- Fde