Same Person, Different Advice: Auditing LLM Financial Advice with Repeated Risk-Tolerance Responses
A · Strong evidence Four independent researchers audit seven models on real retest data from 800 households, with bootstrap intervals, robustness checks, a protocol fixed before results and all 77,014 model replies released; a clear, reusable method.
Summary based on the full text·Original: about 40 minutes to read
In short
An arXiv paper (16 October 2026) by Soovadeep Bakshi, Apurba Bose, Aishwarya Bose and Sandip Nandi, independent researchers. AI investment advice sets a client's stock allocation largely from one stated-risk question, but people do not answer that question the same way twice. The authors gave seven models from five providers 800 real household profiles where only the risk answer changed between the person's first and second response. Advice moved far more than the noise of repeating the prompt. One clarifying question removed almost all of the swing when the client corrected a stray answer.
Key numbers
- 24 to 35percentage-point move in recommended stock share when only the household's repeated risk answer changed
- 0.6 to 4.2points of variation when the same prompt was simply repeated
- 41%US panel households that gave a different risk-tolerance answer two years later
- 33.5 to 1.9average swing in points before and after one clarifying question, when the client then corrects the answer
- 22% vs 4%stocks advised to clients unwilling to take any risk once models were told the answer is noisy, versus without the note
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- In the US Survey of Consumer Finances panel, 41% of households chose a different risk answer in 2009 than in 2007; in a laboratory retest, a third of people moved two or more points on a 0-10 scale after six months.
- Models tested: Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6.1 Sol, GPT-6 Luna, Gemini 3.8 Flash, Grok 4.7 and a Mistral model.
- Recommended stock share moved 10 to 16 points on average, and 24 to 35 points when the answer changed, against 0.6 to 4.2 points for a repeated prompt. In the laboratory retest, the rating alone moved advice 11 to 18 points.
- Discounting the answer for its measured reliability would cut swings by a factor of 2.4 to 3.3. Swings are largest for clients under 40.
- Warning the model that the answer is noisy steadied some models but shifted weight to age and pulled cautious clients to the middle: clients unwilling to take any risk were advised 22% stocks instead of 4%.
- Three extra money questions reduced swings for some models without a statistically detectable change relative to differences across people. One scripted clarifying question cut the swing from 33.5 to 1.9 points while keeping the response to confirmed changes.
How they got it
Public data from the SCF 2007-09 panel (800 households, facts held at 2007 values) and the Basel-Berlin Risk Study retest. Models were queried through OpenRouter with one-turn allocation prompts; 77,014 replies are released with code. A classical test theory benchmark estimates how much advice should move given the answer's reliability. The protocol and the clarification test's analysis were fixed before results were seen.
Limits
- There is no ground truth for correct advice; the reliability benchmark assumes each model's response when told the answer is very reliable reflects its true mapping. At higher assumed reliability, Claude Sonnet 5.5 is statistically compatible with the benchmark.
- The SCF panel straddles the 2008 crisis, so some answer changes are real; the laboratory sample lacks age and wealth.
- Single prompts, English only, one point in time; outcomes for clients are not measured.
Why it matters for companies
Any firm deploying an AI adviser or a questionnaire-driven tool, in finance, benefits or pensions, inherits this problem: one noisy input drives a consequential output. The cheap fix is a confirmation step when an answer looks out of line with the rest of the profile. The audit method is the reusable part: ask the same tool about the same person with only one answer changed, and measure how far the recommendation moves. Regulated firms should add this same-person retest to suitability testing.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Audit