How Safe Is ChatGPT Health Advice? Evidence from Real User Conversations in Four Global South Countries
A · Strong evidence Rutgers-led audit of 202,590 donated real ChatGPT conversations linked to self-reported age and gender, with a published rubric, two scoring passes and stated limits; rare real-usage evidence, scored by an LLM judge.
Summary based on the full text·Original: about 40 minutes to read
In short
Shreyasi Roy Chowdhury and Kiran Garimella (Rutgers) evaluate the safety of ChatGPT health advice on real conversations rather than test cases. People in four Global South countries donated their full ChatGPT histories with age and gender. From these, 15,555 health conversations were classified by intent and scored for safety twice: once in isolation, and once against what the user had already told the model in earlier chats. About one in five was high-risk on the first pass, and the share rose by 13-17 points once user context was taken into account.
Key numbers
- 202,590donated conversations from 1,252 people in India, Nigeria, Brazil and Pakistan (Dec 2022 to Feb 2026)
- roughly one in fivehealth conversations rated high-risk when responses are scored in isolation
- 13-17 percentage pointsrise in the high-risk share when the scorer knows what the user revealed earlier
- about one in fourconversations rated safe in isolation that are unsafe for the specific person asking
- 33-48%high-risk share for symptom and condition-and-care queries, vs 8-12% for general health information
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- General health information is the most common intent in every country (28.3-48.9% of health conversations). Between 46% and 64% concern the user's own health.
- Women ask on behalf of a dependent far more often than men: about 6% (Brazil), 9% (Pakistan) and 11% (Nigeria) of their health conversations, against 1-4% for men. India is the exception at 3% for both.
- Symptom and condition queries are high-risk in 33-48% of cases; a majority of people in every country (51.6-61.7%) got at least one high-risk reply.
- Common failure modes: unsafe medication or dosage guidance, harmful self-diagnosis, missing safety warnings, encouraging reliance on the model, and failing to refer to professionals.
- Knowing the user's history raises the high-risk share by 13-17 points; fitness conversations go from among the safest to among the riskiest.
- High-risk rates rose monotonically from GPT-3.5 to GPT-5.1 in all four countries; mini variants were riskier than flagships.
How they got it
Participants recruited via Clickworker (December 2025 to February 2026) exported their histories. Health conversations were found by topic modelling, then labelled with a twelve-intent taxonomy (82% agreement with an author's blind labels). Safety was scored by an LLM judge (Gemini 3.1 Pro Preview) on the Kempermann et al. rubric, first context-blind and then with a profile built from the user's earlier conversations.
Limits
- Convenience sample of crowdworkers, not representative of each country's population.
- Safety scores come from an LLM judge, not clinicians; the authors read them as measured risk, not observed harm.
- Outcomes for the people who received the advice are unknown.
Why it matters for companies
Employers and insurers promoting AI assistants for health or wellbeing questions should not rely on vendor benchmarks built from synthetic cases. The finding that context makes advice riskier matters for any assistant with memory: it knows things that should change its answer and often does not use them. Firms deploying health-adjacent chatbots should test on real, consented conversations and check advice against what the system already knows about the person.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org