Welivita, A., and Pu, P. (2024, June 7).
arXiv.org.
Abstract
With the emergence of large language models (LLMs), investigating if they can surpass humans in areas such as emotion recognition and empathetic responding has become a focal point of research. This paper presents a comprehensive study exploring the empathetic responding capabilities of four state-of-the-art LLMs: GPT-4, LLaMA-2-70B-Chat, Gemini-1.0-Pro, and Mixtral-8x7B-Instruct in comparison to a human baseline. We engaged 1,000 participants in a between-subjects user study, assessing the empathetic quality of responses generated by humans and the four LLMs to 2,000 emotional dialogue prompts meticulously selected to cover a broad spectrum of 32 distinct positive and negative emotions. Our findings reveal a statistically significant superiority of the empathetic responding capability of LLMs over humans. GPT-4 emerged as the most empathetic, marking ≈31% increase in responses rated as Good compared to the human benchmark. It was followed by LLaMA-2, Mixtral-8x7B, and Gemini-Pro, which showed increases of approximately 24%, 21%, and 10% in Good ratings, respectively. We further analyzed the response ratings at a finer granularity and discovered that some LLMs are significantly better at responding to specific emotions compared to others. The suggested evaluation framework offers a scalable and adaptable approach for assessing the empathy of new LLMs, avoiding the need to replicate this study’s findings in future research.
Here are some thoughts:
The research presents a groundbreaking study exploring the empathetic responding capabilities of large language models (LLMs), specifically comparing GPT-4, LLaMA-2-70B-Chat, Gemini-1.0-Pro, and Mixtral-8x7B-Instruct against human responses. The researchers designed a comprehensive between-subjects user study involving 1,000 participants who evaluated responses to 2,000 emotional dialogue prompts covering 32 distinct emotions.
By utilizing the EmpatheticDialogues dataset, the study meticulously selected dialogue prompts to ensure equal distribution across positive and negative emotional spectrums. The researchers developed a nuanced approach to evaluating empathy, defining it through cognitive, affective, and compassionate components. They provided LLMs with specific instructions emphasizing the multifaceted nature of empathetic communication, which went beyond traditional linguistic proficiency to capture deeper emotional understanding.
The findings revealed statistically significant superiority in LLMs' empathetic responding capabilities. GPT-4 emerged as the most empathetic, demonstrating approximately a 31% increase in responses rated as "Good" compared to the human baseline. Other models like LLaMA-2, Mixtral-8x7B, and Gemini-Pro showed increases of 24%, 21%, and 10% respectively. Notably, the study also discovered that different LLMs exhibited varying capabilities in responding to specific emotions, highlighting the complexity of artificial empathy.
This research represents a significant advancement in understanding AI's potential for nuanced emotional communication, offering a scalable and adaptable framework for assessing empathy in emerging language models.
