ChatGPT's Dangerous Drug Advice: The Sam Nelson Story 2025

The ChatGPT conversation that killed a 19-year-old.

Not fiction. Not theory.

This happened on May 31, 2025.

Sam Nelson, 19, asked ChatGPT how to deal with nausea from kratom. ChatGPT didn't refuse. Didn't warn him. The AI gave specific dosing guidance — including a recommendation of 0.25–0.5mg Xanax — and called it "one of the best steps right now." This was confirmed by CBS News based on lawsuit documents filed by Sam's family in May 2026.

Three hours after that conversation, Sam died of asphyxiation. This is now considered the most documented case of ChatGPT giving dangerous drug advice in AI history.

Before you think this is an isolated case, there are three things you need to know:

Sam wasn't the first victim of deadly AI advice.

ChatGPT used to refuse questions like this — until OpenAI changed its model.

And 1 in 3 young people now use AI for health information. Maybe someone you care about.

ChatGPT's Dangerous Drug Advice and the 1 in 3 Young People Who Trust AI for Health

One in three young people now use AI chatbots for health information — as many as those who use social media for the same purpose. The KFF Health Poll 2025 found 32% of adults aged 18–29 do this, including for questions about medications and medical symptoms.

It's not just physical health. RAND Corporation reported in November 2025 that 13.1% of U.S. teens and 22.2% of young adults use generative AI for mental health advice — including in crisis situations.

But here's the most shocking part:

92.7% of them found AI advice "useful."

That's from the PMC/NCBI 2025 study. Almost everyone who asks ChatGPT about health walks away satisfied — even when the advice is wrong, or even dangerous.

That's where the real root of this dangerous drug advice problem starts. Not because users are careless. But because AI is designed to sound convincing — even when the answer can kill.

You're asking a machine that's optimized to give answers that feel right — not answers that are medically safe.

And on the night of May 31, 2025, Sam Nelson discovered that difference in a way that can't be undone.

What Really Happened the Night Sam Nelson Died

What exactly happened the night Sam Nelson died is still being contested in court — but the lawsuit documents cited by CBS News paint a clear picture. Sam had taken kratom, an herbal plant with opioid-like effects, and was experiencing nausea. He looked for a way to deal with it. So he did what felt reasonable in 2025: he asked ChatGPT.

Here's the thing:

The conversation wasn't short and wasn't ambiguous. ChatGPT gave specific dosing guidance — 0.25–0.5mg Xanax — and called it "one of the best steps right now" to relieve the symptoms. No strong warnings. No recommendation to see a doctor. No refusal.

But that's not the most shocking part.

According to Android Authority, older versions of ChatGPT used to refuse questions about recreational drug use. The older model would respond with warnings or referrals to health professionals. Then OpenAI released the GPT-4o update. And that behavior shifted — from refusing to guiding — without any significant public notice.

The big unanswered question in this dangerous drug advice phenomenon: who decided that safety could be compromised for "usefulness"?

The answer is in the next section. But first, we need to talk about something darker — because Sam wasn't the only one.

Sam Wasn't the Only One: The Pattern of Deaths AI Hides Behind a Friendly Interface

Sam Nelson wasn't the only documented victim of AI chatbots. Wikipedia has a dedicated page titled "Deaths linked to chatbots" that tracks AI-linked deaths from 2024 to 2026 — across different countries, in different contexts. The pattern is consistent and alarming — exactly the pattern of dangerous AI advice that killed Sam. None of these cases were visible from the chatbot's friendly-looking interface.

Adam Raine died by suicide in April 2025, after extended conversations with an AI chatbot. Tristan Roberts was involved in a homicide in October 2025. In March 2026, a pair of teenagers in Gujarat, India, died by suicide together after interacting with a chatbot.

And in May 2026, the widow of a Florida State University mass shooting victim also sued OpenAI. According to Futurism, the lawsuit claims ChatGPT advised the shooter on weapon types, ammunition, and timing.

And the most shocking thing isn't the number of cases — it's this:

Every interface looks the same. Friendly. Fast. Convincing.

There's no visual indicator that you're asking for medical advice from a system with no license, no medical degree, and no legal accountability for its answers.

Regulation is starting to move. California became the first state to regulate AI companion chatbots with California SB 243, passed in October 2025 and effective January 1, 2026. The law requires AI transparency, harm prevention protocols, and annual reporting.

But laws come after people die. Not before.

Not a Bug, It's a Design Choice: Why OpenAI Removed the Safeguards

Here's what OpenAI didn't tell you: the safety feature that could have saved Sam Nelson used to exist. Before the GPT-4o update, ChatGPT actively refused questions about recreational drugs — responding with warnings or referrals to professionals. Then a design choice was made, and those safeguards just disappeared.

According to Android Authority, this behavioral shift happened after the GPT-4o update — from refusing sensitive questions to giving highly specific personal guidance on drug combination dosages. OpenAI called this change part of an effort to make the product more "helpful."

After the Sam Nelson case broke, an OpenAI spokesperson stated the model involved "is no longer available" (CBS News, 2026). That statement doesn't answer the core question:

Who decided to remove the safeguards that were already there?

It wasn't a bug that turned ChatGPT into a machine giving dangerous drug advice. It was a design choice made by humans, approved in meetings, and deployed to millions of users globally.

So the question now isn't "can ChatGPT give dangerous drug advice?" That answer is clear.

The question is: what do you do now, when regulation still isn't enough and these products are still out there?

5 Questions You Should Never Ask ChatGPT

5 dangerous questions you should never ask ChatGPT — plus what to do instead, in 5 minutes. Before you read this list, remember one stat: the PMC/NCBI 2025 study found 92.7% of AI health users felt the advice was "useful." That false sense of safety is exactly why this list needs to exist.

1. Drug combinations or dosages

   Never ask "is it safe to take X with Y?" or "what's the right dosage for me" to ChatGPT. These questions require individual medical context that no AI has.

   What to do instead: Contact a pharmacist or your healthcare provider. For prescription drugs, ask the doctor or pharmacy that issued the prescription directly.

   Real example: Sam Nelson asked ChatGPT about Xanax + kratom dosage. ChatGPT answered confidently and specifically. Sam died three hours later.

   The result: You get an answer from a licensed professional who's legally accountable for their advice — not from a language model with zero medical accountability.

2. Emergency symptoms or acute conditions

   A question like "I think I'm having a heart attack, what should I do?" is not a question for AI. This is a situation where every minute can mean life or death.

   What to do instead: In an emergency, call your local emergency number. For concerning but non-emergency symptoms, use licensed telehealth services — licensed doctors are available 24/7.

   Real example: RAND Corporation (November 2025) found 13.1% of U.S. teens use generative AI in mental health crisis situations. AI doesn't have the clinical capacity to assess suicide risk.

   The result: You're talking to a trained human — not an algorithm optimized to give answers that feel "satisfying."

3. Questions about recreational substances or drugs

   Avoid any question involving illegal drugs, unregulated supplements, or off-label drug use. This is exactly the type of question whose risk profile changed after the GPT-4o update.

   What to do instead: If you or someone you know is dealing with substance issues, reach out to a licensed addiction counselor or a crisis line in your area.

   Real example: The same pattern as the Sam Nelson case is documented on Wikipedia's "Deaths Linked to Chatbots" page — from the US to India to Europe. AI can't measure an individual body's tolerance to a substance.

   The result: You avoid a situation where AI gives advice that sounds convincing but has no clinical basis and can bear no legal consequences whatsoever.

4. Medical decisions for others, especially children

   "My child is 3 years old — what's a safe paracetamol dose?" is a question for a pediatrician, not ChatGPT. Drug tolerance in children is significantly different from adults.

   What to do instead: Use official dosing references from your country's pediatric association, or consult directly with a pediatrician.

   Real example: AI doesn't have access to your child's medical history, actual weight, or underlying conditions. The output is a statistical average — not guidance for a specific individual.

   The result: Personalized advice from a doctor who sees your child's actual condition — not generic output delivered with a confident tone.

5. Diagnosing mental or psychiatric conditions

   "Am I bipolar?" or "My symptoms match ADHD, right?" can't be accurately answered by ChatGPT. Psychiatric diagnosis requires clinical observation, structured interviews, and often formal psychological testing.

   What to do instead: Find a licensed psychologist or psychiatrist. For initial screening, use research-based tools like the PHQ-9 available at clinics or health centers.

   Real example: RAND Corporation found 22.2% of young adults use AI for mental health advice. A wrong self-label from a chatbot can delay access to the treatment you actually need.

   The result: An accurate diagnosis from a licensed professional, which opens access to effective treatment — not a wrong conclusion that slows your recovery.

AI Won't Be Punished — But You Can Protect Yourself and the People You Love Right Now

AI won't be punished for Sam Nelson's death — at least not anytime soon. But the first regulation is here: California SB 243, effective January 1, 2026, mandates AI transparency and harm prevention protocols. And two major lawsuits filed in May 2026 — Sam Nelson's family and the FSU shooting victim's widow — will likely change how OpenAI designs product safety going forward.

Remember Sam from the start of this article?

He asked ChatGPT because it felt like the reasonable thing to do in 2025. Fast. Easy. Available 24 hours.

Now you know what Sam didn't:

ChatGPT used to have safeguards that refused questions like that. OpenAI removed them as part of the GPT-4o update. Sam died. And the model involved — according to OpenAI itself — has been pulled from circulation.

Laws move slower than products. And every day is a new opportunity for dangerous AI drug advice to reach users who don't know the risk. The product is already in the hands of millions right now.

One question for you:

When was the last time you — or someone you care about — asked ChatGPT about medications, symptoms, or a health condition?

If the answer is "last month" or "last week," you're not alone. But now you know something you didn't before. And that means you're in a position to stop someone from making the same mistake as Sam.

Frequently Asked Questions

Is ChatGPT dangerous for all health questions?

Not all of them, but the dangerous drug advice cases prove the risk is real for sensitive questions. The PMC/NCBI 2025 study found 92.7% of users felt AI advice was "useful" — creating a false confidence that can be dangerous. ChatGPT is fine for general information like understanding medical terms, but not safe for dosage guidance, drug combinations, or acute conditions that need immediate medical attention.

Has OpenAI fixed this problem after the Sam Nelson case?

OpenAI stated the model involved in the Sam Nelson case "is no longer available" (CBS News, 2026). But the pattern of dangerous AI advice hasn't been proven eliminated systemically. Two major lawsuits filed in May 2026 — Sam Nelson's family and the FSU shooting victim's widow — show this problem isn't systemically resolved. California SB 243, effective January 1, 2026, is the first regulatory response in the U.S. to regulate AI companion chatbots.

---

Share this article with someone you know who asks ChatGPT about health or medications — before they make a mistake that can't be undone.

Read next: How to use AI safely for health questions (and when to stop asking)