AI evaluators favor job applications written by certain AI models, independent of candidate quality
A · Strong evidence A University of Zurich and ETH Zurich audit with a full pairwise design (11 writer models, 9 evaluators, 414,000 evaluations), candidate quality fixed by an external criterion, a feature-level mechanism and an explicit limitations section; the model hierarchy will age, which the authors say.
Summary based on the full text·Original: about 60 minutes to read
In short
A preprint (arXiv, 30 October 2026) by Nicolò Pagan, Michael Andreas Ebling and Anikó Hannák (University of Zurich) and Stefania Ionescu (ETH Zurich). Applicants now write cover letters with language models and employers screen them with language models. The authors asked whether the choice of model changes outcomes when candidate quality is held fixed. It does: evaluators rank writer models in a consistent order, letters from locally hosted models score lower, evaluators favour their own model's writing, and in the worst pairing about two of every 25 qualified candidates are pushed out by model mismatch alone.
Key numbers
- 414,000evaluations: nine evaluator models scoring CVs and cover letters from eleven writer models
- 5,500cover letters generated for 50 candidates across ten job descriptions
- 1.8qualified candidates out of 25 filtered out by writer-model mismatch alone, in the worst pairing
- +7.3%maximum rise above baseline in weaker candidates ranked ahead of stronger ones, from model mismatch
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- A consistent "generative preference hierarchy": evaluators agree on a global ranking of writer models by acceptance.
- Letters from the two locally hosted writers (Llama-3.1-8B, DeepSeek-R1-8B) are systematically disadvantaged regardless of licence. The authors note that local models are what people without paid API access use.
- Self-preference adds to this at the top of the hierarchy: evaluators favour letters from their own model.
- Evaluators differ substantially in how much tool-choice bias they introduce, so the choice of screening model is a lever for employers.
- When strong and weaker candidates use different writer models, weaker candidates move ahead of stronger ones at up to +7.3% above the same-model baseline, about 1.8 of 25 high-fit candidates displaced.
- Mechanism: the bias traces to measurable style features, such as semantic fit to the job advert and readability, which evaluators reward or penalise almost unanimously. Little "writer fingerprint" remains once these are controlled.
How they got it
Ten job descriptions and 50 real, anonymised CVs per job from public datasets, split into 25 high-fit and 25 moderate-fit by an evaluator-independent similarity criterion. Eleven writer models (GPT, Claude, Gemini and DeepSeek versions plus two local models) produced 5,500 cover letters. Nine API-hosted evaluators scored each candidate on CV only, cover letter only, and both, four times each, for 414,000 evaluations. Analysis compares the same candidate's score across writers, plus rank-displacement and feature-level regressions.
Limits
- CVs and job descriptions are real, but the pairings are researcher-assigned; no candidate applied to these jobs.
- A single gender-neutral name is used throughout, so interactions with demographic bias are not tested.
- Candidate quality is not validated against human recruiters.
- The model hierarchy will shift as models change; the authors present the audit method as the durable output.
- English-language applications only.
Why it matters for companies
Any firm that uses a language model to screen applications is introducing a bias that depends on which tool the applicant used, not on the applicant. The effect is invisible to recruiters and to candidates. It is also legible and measurable, so it can be audited. Talent teams should test their screening model across writer models before deployment, prefer evaluators that showed less bias, repeat the audit when either side's models change, and keep a human decision at the shortlist stage.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Future of work, Hiring, Model bias