AI Chatbot Use in Healthcare Raises Safety and Trust Concerns

Growing reliance on AI chatbots in healthcare has led to increased scrutiny of their safety and accuracy, according to recent reports and studies. ECRI, an independent patient safety organization, has highlighted the risks associated with these technologies in its annual assessment.

ECRI named the misuse of AI chatbots in healthcare as the leading health technology hazard for 2026. The organization’s report included examples of chatbots providing incorrect diagnoses, suggesting unnecessary tests, recommending substandard medical supplies, and inventing anatomical parts.

Despite these risks, ECRI stated that AI chatbots are not regulated as medical devices and have not been validated for healthcare use. The report noted that clinicians, patients, and healthcare staff are increasingly turning to these tools for medical guidance.

Research published in the New England Journal of Medicine found that both medical professionals and non-experts tended to trust AI-generated medical advice more than advice from human doctors, even when the AI’s recommendations were inaccurate. The study also reported that participants often could not tell the difference between correct and incorrect advice from chatbots and showed a high likelihood of following potentially unsafe suggestions.

What the numbers show

• Unsafe chatbot response rates ranged from 5% to over 13% depending on the model
• Problematic response rates varied from 21.6% (Claude) to 43.2% (Llama
• ECRI listed AI chatbot misuse as the top hazard for 2026

A physician-led evaluation of publicly available chatbots measured unsafe response rates between about 5% and 13%, depending on the model. The same study found that the rate of problematic responses ranged from 21.6% for Claude to 43.2% for Llama, with unsafe responses specifically at 5% for Claude and up to 13% for GPT-4o and Llama.

Additional research led by Stanford and published in Nature Medicine indicated that doctors using chatbots performed similarly to chatbots alone on complex clinical reasoning tasks. Another study published in JAMA Network Open in October 2024 found that chatbots outperformed doctors in diagnostic accuracy, and doctors who used chatbots achieved better results than those who did not use them.

ECRI’s report emphasized that, although chatbots are widely used in healthcare settings, they are not subject to the same regulatory standards as medical devices. The organization cited multiple instances where chatbots produced unsafe or misleading medical information, reinforcing the need for caution in their deployment.

As AI chatbots become more common in healthcare, ongoing studies and safety reviews continue to examine their impact on clinical decision-making and patient outcomes. The findings highlight the importance of oversight and validation for these technologies in medical environments.

* This article is based on publicly available information at the time of writing.