Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions

Luo, M., et al. (2024, May 26).

arXiv.org.

https://arxiv.org/abs/2405.16402

Abstract

The integration of Large Language Models (LLMs) into the healthcare domain has the potential to significantly enhance patient care and support through the development of empathetic, patient-facing chatbots. This study investigates an intriguing question Can ChatGPT respond with a greater degree of empathy than those typically offered by physicians? To answer this question, we collect a de-identified dataset of patient messages and physician responses from Mayo Clinic and generate alternative replies using ChatGPT. Our analyses incorporate novel empathy ranking evaluation (EMRank) involving both automated metrics and human assessments to gauge the empathy level of responses. Our findings indicate that LLM-powered chatbots have the potential to surpass human physicians in delivering empathetic communication, suggesting a promising avenue for enhancing patient care and reducing professional burnout. The study not only highlights the importance of empathy in patient interactions but also proposes a set of effective automatic empathy ranking metrics, paving the way for the broader adoption of LLMs in healthcare.

A preprint is here.

Here are some thoughts:

The research explores an innovative approach to assessing empathy in healthcare communication by comparing responses from physicians and ChatGPT, a large language model (LLM). The study focuses on prostate cancer patient interactions, utilizing a real-world dataset from Mayo Clinic to investigate whether AI-powered chatbots can potentially deliver more empathetic responses than human physicians.

The researchers developed a novel methodology called EMRank, which employs multiple evaluation techniques to measure empathy. This approach includes both automated metrics using LLaMA (another language model) and human assessments. By using zero-shot, one-shot, and few-shot learning strategies, they created a flexible framework for ranking empathetic communication that could be generalized across different healthcare domains.

Key findings suggest that LLM-powered chatbots like ChatGPT have significant potential to surpass human physicians in delivering empathetic communication. The study's unique contributions include using real patient data, developing innovative automatic empathy ranking metrics, and incorporating patient evaluations to validate the assessment methods. By demonstrating the capability of AI to generate compassionate responses, the research opens new avenues for enhancing patient care and potentially reducing professional burnout among healthcare providers.

The methodology carefully addressed privacy concerns by de-identifying patient and physician information, and controlled for response length to ensure a fair comparison. Ultimately, the study represents a promising step towards integrating artificial intelligence into healthcare communication, highlighting the potential of LLMs to provide supportive, empathetic interactions in medical contexts.