---
id: "lib-2026-10-10-probable-but-not-valid-a-psychometric-audit-of"
type: "library"
title: "Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents"
lang: "EN"
org: "arXiv (Kaunas UT)"
source_type: "Paper"
date: "2026-10-10"
week: "2026-W42"
url: "https://nosov.ai/library/lib-2026-10-10-probable-but-not-valid-a-psychometric-audit-of/"
original_url: "https://arxiv.org/abs/2610.14912"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Mantas Lukauskas (Kaunas University of Technology, Hostinger) and Viktorija Šarkauskaitė test 61 models against real respondents and statistical baselines on an open survey, with resampling intervals, controls and a documented scoring correction."
summary_basis: "full-text"
original_minutes: "60"
tags: ["work and ai", "synthetic respondents", "employee surveys", "psychometrics"]
---

# Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

Tier A (Strong evidence): Mantas Lukauskas (Kaunas University of Technology, Hostinger) and Viktorija Šarkauskaitė test 61 models against real respondents and statistical baselines on an open survey, with resampling intervals, controls and a documented scoring correction.

Summary based on the full text.

## In short

An arXiv paper (10 October 2026) by Mantas Lukauskas and Viktorija Šarkauskaitė. Can language models replace human respondents in an organisational-psychology survey? They test 14 new "decision models" that return a probability for each rating-scale point and 47 generative LLMs on a public Lithuanian survey, conditioning each on 100 real respondent profiles. The best model only ties a simple statistical baseline built from other respondents, newer releases are often worse, and every decision model forecasts individual answers worse than just using other people's answer frequencies.

## Key numbers

- PSS 0.813: best Psychometric Similarity Score (MiniMax M3) for a synthetic sample
- 0.824: score of a Gaussian copula fit only on other respondents, i.e. a plain statistical baseline
- 5 of 10: product lines where the newest model release scored below an earlier release
- 13 of 14: decision models scoring below the generative reference gpt-5.4-mini with rounded answers
- 1.0%: improvement of a profile regression over other respondents' answer frequencies in forecasting individuals

## What it reports

- Survey: n = 263, 68 items. Best generative model MiniMax M3 reaches PSS 0.813, level with a Gaussian copula fit on other respondents (0.824) and below an in-sample copula (0.927).
- A second sample of 100 real people scores below the best models, so at this size the metric cannot rank top models against real respondents.
- Newer is not better: in 5 of 10 product lines the latest release scores below an earlier one beyond resampling noise; none scores above.
- With rounded answers, 13 of 14 decision models score below gpt-5.4-mini; 33 of 47 generative models reproduce item correlations better than any decision model.
- Averaged probability vectors from decision models match human item distributions much better than rounded answers, and rank respondents about as well as a regression on 163 real people.
- As forecasts of each person's answers, every decision model is worse than other respondents' frequencies; the profile regression adds only 1.0%, so profile fields say little about individuals.
- The full decision-model run cost $1.27.

## How they got it

Each model received the same 100 respondent profiles and items. Synthetic samples were scored on item distributions, correlations, reliability, factor congruence, mediation and demographic effects, combined into a Psychometric Similarity Score with revised reverse-keying. Baselines: copula fits, a second human sample, a profile regression, controls without profiles and with attribute swaps. Individual forecasts were scored with a proper scoring rule.

## Limits

- One instrument, one language, one country, self-report data.
- With 47 generative cells, the top model is partly chance; ranking inside the top group is unstable.
- Mediation and demographic components are noisy at n = 100; generative models ran at minimal reasoning.
- Older models were not re-queried, so vendor-side changes since the original audit are not captured.

## Why it matters for companies

Vendors now sell "synthetic respondents" for employee surveys, market research and pilot testing. This audit shows the best models reach only the level of a cheap statistical model fit on real data, and that a model upgrade can make results worse without notice. Using synthetic answers to forecast how a specific employee or customer would respond is not supported. At most, synthetic samples help with sample-level pilots; any use should be validated against real responses and re-checked on every model version change.

Original: https://arxiv.org/abs/2610.14912