InterviewPlayground: A Simulation Environment for Evaluating AI Interviewers
A · Strong evidence Johns Hopkins preprint that validates a simulation benchmark against 15 real studies with 450 human participants and expert-annotated measures.
Summary based on the full text·Original: about 35 minutes to read
In short
InterviewPlayground is a simulation environment for testing AI interviewers used in market research, polling and social science, built by Jonathan Ivey, Ziang Xiao, Anjalie Field and colleagues at Johns Hopkins with co-authors from Listen Labs. Simulated participants have personas, six behavioural traits and memory banks grounded in interview research. Performance in simulation predicts performance with real people with an average correlation of 0.86, though participant experience is predicted less well.
Key numbers
- 0.86average Pearson correlation between simulated and human-study results across 12 measures
- 450human participants in 15 real qualitative studies (5 AI interviewers x 3 topics)
- 0.60 / 0.67correlations for comfort level and overall experience, the weakest measures
- 0.83average correlation when simulated participants run on a model with 4B parameters
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Six simulated traits: knowledge, understanding, reflexivity, memory, verbosity, disclosure; derived from 20 characteristics found in a literature review.
- Each simulated participant gets 200 generated memories; five are retrieved per question, with noise added to simulate forgetting.
- Experts rated simulated participants 3.51 for realism versus 3.34 for humans on a 4-point scale (p = 0.28, not significant).
- 10 of 12 measures correlate strongly (r = 0.74-1); support and rapport statements r = 1.00, leading questions r = 0.98, interview guide coverage r = 0.95.
- Comfort (r = 0.60) and overall experience (r = 0.67) are moderate, with simulation underpredicting both.
- LLM judges agreed with median human ratings at 0.60-0.80 Krippendorff's alpha, against 0.62-0.70 among humans; based on 1,800 expert judgments.
- Most common failure in both settings: repeated questions, especially from InterviewGPT; LLMRoleplay achieved the best coverage.
How they got it
Five published AI interviewers, all on GPT-5.4 mini, replicated three published studies (obesity and weight management, Asian American politics, generative AI in knowledge work) with 30 participants each for up to 30 minutes. Humans were recruited through Listen Labs and paid $15 an hour; simulations used Gemini 3.1 Pro, plus a robustness check across seven simulator models. The design was fixed before comparison with human data, apart from setting trait distributions.
Limits
- Simulated participants voice discomfort directly, while humans tend to report it afterwards in surveys; experience measures remain weak.
- Only three topics and one interviewer base model; hostile or impaired participants were not simulated.
- Listen Labs, the recruitment platform, sells AI-led interviews and has co-authors on the paper.
Why it matters for companies
AI interviewers are moving into employee listening, exit interviews, customer research and candidate screening. This work shows a cheap way to compare vendors before exposing real people: run them against simulated respondents and look at coverage, leading questions and repetition. It also shows where simulation stops working: how people feel about being interviewed still needs real participants. Repetitive questioning is the first failure to check in any pilot.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Hiring