Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Health Care Professionals

Choudhury, A., & Chaudhry, Z. (2024).

Journal of medical Internet research, 26, e56764.

https://doi.org/10.2196/56764

Abstract

As the health care industry increasingly embraces large language models (LLMs), understanding the consequence of this integration becomes crucial for maximizing benefits while mitigating potential pitfalls. This paper explores the evolving relationship among clinician trust in LLMs, the transition of data sources from predominantly human-generated to artificial intelligence (AI)–generated content, and the subsequent impact on the performance of LLMs and clinician competence. One of the primary concerns identified in this paper is the LLMs’ self-referential learning loops, where AI-generated content feeds into the learning algorithms, threatening the diversity of the data pool, potentially entrenching biases, and reducing the efficacy of LLMs. While theoretical at this stage, this feedback loop poses a significant challenge as the integration of LLMs in health care deepens, emphasizing the need for proactive dialogue and strategic measures to ensure the safe and effective use of LLM technology. Another key takeaway from our investigation is the role of user expertise and the necessity for a discerning approach to trusting and validating LLM outputs. The paper highlights how expert users, particularly clinicians, can leverage LLMs to enhance productivity by off-loading routine tasks while maintaining a critical oversight to identify and correct potential inaccuracies in AI-generated content. This balance of trust and skepticism is vital for ensuring that LLMs augment rather than undermine the quality of patient care. We also discuss the risks associated with the deskilling of health care professionals. Frequent reliance on LLMs for critical tasks could result in a decline in health care providers’ diagnostic and thinking skills, particularly affecting the training and development of future professionals. The legal and ethical considerations surrounding the deployment of LLMs in health care are also examined. We discuss the medicolegal challenges, including liability in cases of erroneous diagnoses or treatment advice generated by LLMs. The paper references recent legislative efforts, such as The Algorithmic Accountability Act of 2023, as crucial steps toward establishing a framework for the ethical and responsible use of AI-based technologies in health care. In conclusion, this paper advocates for a strategic approach to integrating LLMs into health care. By emphasizing the importance of maintaining clinician expertise, fostering critical engagement with LLM outputs, and navigating the legal and ethical landscape, we can ensure that LLMs serve as valuable tools in enhancing patient care and supporting health care professionals. This approach addresses the immediate challenges posed by integrating LLMs and sets a foundation for their maintainable and responsible use in the future.

The abstract provides a sufficient summary.

Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation

Stade, E. C.,  et al. (2024).

Npj Mental Health Research, 3(1).

https://doi.org/10.1038/s44184-024-00056-z

Abstract

Large language models (LLMs) such as Open AI’s GPT-4 (which power ChatGPT) and Google’s Gemini, built on artificial intelligence, hold immense potential to support, augment, or even eventually automate psychotherapy. Enthusiasm about such applications is mounting in the field as well as industry. These developments promise to address insufficient mental healthcare system capacity and scale individual access to personalized treatments. However, clinical psychology is an uncommonly high stakes application domain for AI systems, as responsible and evidence-based therapy requires nuanced expertise. This paper provides a roadmap for the ambitious yet responsible application of clinical LLMs in psychotherapy. First, a technical overview of clinical LLMs is presented. Second, the stages of integration of LLMs into psychotherapy are discussed while highlighting parallels to the development of autonomous vehicle technology. Third, potential applications of LLMs in clinical care, training, and research are discussed, highlighting areas of risk given the complex nature of psychotherapy. Fourth, recommendations for the responsible development and evaluation of clinical LLMs are provided, which include centering clinical science, involving robust interdisciplinary collaboration, and attending to issues like assessment, risk detection, transparency, and bias. Lastly, a vision is outlined for how LLMs might enable a new generation of studies of evidence-based interventions at scale, and how these studies may challenge assumptions about psychotherapy.

The article is linked above.

Here are some thoughts.

This article examines the potential of large language models (LLMs), such as GPT-4 and Google’s Gemini, to support and transform behavioral healthcare, particularly psychotherapy. LLMs could enhance access to care by automating administrative tasks like documentation and session summaries, assisting with treatment planning, and supporting clinician training. The authors propose a phased integration of LLMs, starting with low-risk assistive roles, moving toward collaborative functions with human oversight, and potentially, though more controversially, fully autonomous psychotherapy.

While LLMs offer promising opportunities to improve efficiency and scale mental health services, the authors emphasize the need for cautious, evidence-based development due to significant ethical, safety, and accountability concerns. They call for ongoing collaboration between clinicians, researchers, and technologists to ensure LLM use in mental healthcare prioritizes patient safety, transparency, and effectiveness through rigorous testing and gradual implementation.

Why Most Resist AI Companions

De Freitas, J., et al. (2025).

(Working Paper No. 25–030).

Here is a link.

Abstract

Chatbots are now able to form emotional relationships with people and alleviate loneliness—a growing public health concern. Behavioral research provides little insight into whether everyday people are likely to use these applications and why. We address this question by focusing on the context of “AI companion” applications, designed to provide people with synthetic interaction partners. Study 1 shows that people believe AI companions are more capable than human companions in advertised respects relevant to relationships (being more available and nonjudgmental). Even so, they view them as incapable of realizing the underlying values of relationships, like mutual caring, judging them as not ‘true’ relationships. Study 2 provides further insight into this belief: people believe relationships with AI companions are one-sided

(rather than mutual), because they see AI as incapable of understanding and feeling emotion. Study 3 finds that actually interacting with an AI companion increases acceptance by changing beliefs about the AI’s advertised capabilities, but not about its ability to achieve the true values of relationships, demonstrating the resilience of this belief against intervention. In short, despite the potential loneliness-reducing benefits of AI companions, we uncover fundamental psychological barriers to adoption, suggesting these benefits will not be easily realized.

Here are some thoughts:

The research explores why people remain reluctant to adopt AI companions, despite the growing public health crisis of loneliness and the promise that AI might offer support. Through a series of studies, the authors identify deep-seated psychological barriers to embracing AI as a substitute or supplement for human connection. Specifically, people tend to view AI companions as fundamentally incapable of embodying the core features of meaningful relationships—such as mutual care, genuine emotional understanding, and shared experiences. While participants often acknowledged some of the practical benefits of AI companionship, such as constant availability and non-judgmental interaction, they consistently doubted that AI could offer authentic or reciprocal relationships. Even when people interacted directly with AI systems, their impressions of the AI’s functional abilities improved, but their skepticism around the emotional and relational authenticity of AI companions remained firmly in place. These findings suggest that the resistance is not merely technological or unfamiliarity-based, but rooted in beliefs about what makes relationships "real."

For psychologists, this research is particularly important because it sheds light on how people conceptualize emotional connection, authenticity, and support—core concerns in both clinical and social psychology. As mental health professionals increasingly confront issues of social isolation, understanding the limitations of AI in replicating genuine human connection is critical. Psychologists might be tempted to view AI companions as possible interventions for loneliness, especially for individuals who are socially isolated or homebound. However, this paper underscores that unless these deep psychological barriers are acknowledged and addressed, such tools may be met with resistance or prove insufficient in fulfilling emotional needs. Furthermore, the study contributes to a broader understanding of human-technology relationships, offering insights into how people emotionally and cognitively differentiate between human and artificial agents. This knowledge is crucial for designing future interventions, therapeutic tools, and technologies that are sensitive to the human need for authenticity, reciprocity, and emotional depth in relationships.

AI can help people feel heard, but an AI label diminishes this impact

Yin, Y., Jia, N., & Wakslak, C. J. (2024).

PNAS, 121(14).

https://doi.org/10.1073/pnas.2319112121

Abstract

People want to “feel heard” to perceive that they are understood, validated, and valued. Can AI serve the deeply human function of making others feel heard? Our research addresses two fundamental issues: Can AI generate responses that make human recipients feel heard, and how do human recipients react when they believe the response comes from AI? We conducted an experiment and a follow-up study to disentangle the effects of actual source of a message and the presumed source. We found that AI-generated messages made recipients feel more heard than human-generated messages and that AI was better at detecting emotions. However, recipients felt less heard when they realized that a message came from AI (vs. human). Finally, in a follow-up study where the responses were rated by third-party raters, we found that compared with humans, AI demonstrated superior discipline in offering emotional support, a crucial element in making individuals feel heard, while avoiding excessive practical suggestions, which may be less effective in achieving this goal. Our research underscores the potential and limitations of AI in meeting human psychological needs. These findings suggest that while AI demonstrates enhanced capabilities to provide emotional support, the devaluation of AI responses poses a key challenge for effectively leveraging AI’s capabilities.

Significance

As AI becomes more embedded in daily life, understanding its potential and limitations in meeting human psychological needs becomes more pertinent. Our research explores the fundamental human desire to “feel heard.” It reveals that while AI can generate responses that make people feel heard, individuals feel more heard when they believe a response comes from a fellow human. These findings highlight the potential of AI to augment human capacity for understanding and communication while also raising important conceptual questions about the meaning of being heard, as well as practical questions about how to best to leverage AI’s capabilities to support greater human flourishing.

Here are some thoughts.

This study explores whether people can feel heard by AI, examining recipients' perspectives and related perceptions and emotions after receiving a response from AI or a human. It was hypothesized that AI would be better than humans at detecting and understanding emotions. However, the researchers also expected that people would feel less heard by AI due to the perception that AI lacks a mind and cannot think or feel, and because of negative attitudes towards AI.

The study found that AI was statistically significantly better at eliciting feelings of being heard, with recipients reporting feeling more heard, perceiving the response to be more accurate, feeling more understood by the responder, and feeling more connected to the responder when the response was generated by AI. However, recipients reported more positive reactions when they believed the response came from a human, demonstrating a devaluation of AI-generated responses.

The researchers conclude that AI can make people feel heard, but this is influenced both by the quality of the response and the perception of the responder. They suggest that as people encounter and use AI more, they may feel more heard by AI, but feelings of connection may still depend on perceiving AI as having a mind.

Being Human in the Age of Generative AI: Young People’s Ethical Concerns about Writing and Living with Machines.

Higgs, J. M., & Stornaiuolo, A. (2024b).

Reading Research Quarterly.

https://doi.org/10.1002/rrq.552

Abstract

The recent unveiling of chatbots such as ChatGPT has catalyzed vigorous debates about generative AI's impact on how learners read, write, and communicate. Largely missing from these debates is careful consideration of how young people are experiencing AI in their everyday lives and how they are making sense of the questions that these rapidly evolving cultural tools raise about ethics, power, and social participation. Engaging cultural-historical perspectives on technology, the present study drew on student survey and focus group data from English language arts classes in two culturally and linguistically diverse high schools to answer the following questions: (1) How are young people using AI in their everyday lives, if at all?; (2) What do young people identify as key considerations related to AI-mediated writing?; and (3) What ethical and critical considerations, if any, inform young people's sensemaking of and practices with AI? Young people reported using generative AI for diverse purposes in and out of school, including to accomplish routine organizational and information tasks, to entertain themselves through experimenting with AI technologies, and to catalyze their thinking and writing processes. Survey and focus group participants' responses suggested their regular navigation of ethical and critical dimensions of AI use and their contemplation of what it means to be human through and with advancing technologies. Young people also reported a lack of opportunity to examine AI practices and perspectives in school, suggesting the important role schools can play in supporting youths' development of AI ethics.

Here are some thoughts.

This study explored high school students' use and understanding of artificial intelligence (AI) writing tools. Researchers collected survey and focus group data from English language arts classes in two high schools. They found that students use AI tools for various purposes in and out of school, including completing tasks, exploring AI technologies, and enhancing their thinking and writing. Students also reported regularly considering the ethical and critical implications of AI use, such as the impact on authenticity, creativity, and what it means to be human. The study suggests that schools have an important role in helping students develop AI ethics.

Why should humanities education persist in an AI age?

Johannes Steizinger

The Conversation

Originally published 3 Feb 25

Since the launch of ChatGPT in November 2022, the use of artificial intelligence (AI) chatbots has become rampant among students in higher education.

While some might be ambivalent about the impact of generative AI on higher education, many instructors in the humanities scramble to adapt their classes to the new reality and have declared a crisis of their teaching model.

Professors and students alike argue that unrestricted use of generative AI threatens the purpose of an education in disciplines like philosophy, history or literature. They say that, as a society, we should care about this loss of intellectual competencies.

But why is it important that traditional learning not become obsolete — as some predict?

Today, when corrupt leaders promote AI development, AI reflects repressive political biases. There are serious concerns about AI disinformation, so it’s critical to consider the original purpose of modern universities.

I consider this question as a historian of philosophy who has examined how modern ideas have intersected with democratic and fascist societies.

The info is here.

Here are some thoughts:

The article argues that while AI excels at technical and data-driven tasks, it cannot replicate the deeply human skills fostered by the humanities, such as critical thinking, ethical reasoning, creativity, and self-reflection. These skills are not only essential for navigating the complexities of an AI-driven society but also align closely with the core concerns of psychology, making the article particularly important for psychologists.

One of the key takeaways for psychologists is the article’s emphasis on human-centric skills, such as empathy, emotional intelligence, and ethical reasoning. These are areas where AI falls short, and they are also foundational to psychological practice. Whether in therapy, research, or education, psychologists rely on these skills to understand and support human behavior and well-being. The article’s focus on the humanities as a means of developing these abilities reinforces their importance in both personal and professional contexts. Additionally, as AI becomes more integrated into society, psychologists are increasingly called upon to address ethical dilemmas related to its use, such as algorithmic bias, privacy concerns, and the psychological impact of AI on individuals. The humanities provide a valuable framework for exploring these ethical questions, which aligns with the ethical responsibilities of psychologists.

The article also highlights the role of humanities education in fostering emotional intelligence and well-being, which are central to mental health. In a world where AI may dehumanize certain interactions, the ability to connect with others on an emotional level becomes even more critical. Psychologists can draw on this perspective to advocate for educational approaches that prioritize emotional and social learning, ensuring that individuals are equipped to thrive in an AI-driven world. Furthermore, the article bridges the gap between technology and the humanities, suggesting that interdisciplinary collaboration is essential for addressing modern challenges. This resonates with the work of psychologists, who often operate at the intersection of multiple fields, integrating insights from the humanities, social sciences, and technology to better understand and address human behavior.

Another important aspect of the article is its discussion of the future of work. As AI automates many technical tasks, the demand for uniquely human skills—such as creativity, critical thinking, and interpersonal communication—will grow. Psychologists can play a key role in helping individuals and organizations adapt to these changes, and the article provides a strong rationale for why humanities education is a vital part of this preparation. Finally, the article emphasizes the role of the humanities in fostering self-development and a deeper understanding of one’s identity and purpose. This aligns with psychological theories of self-actualization and personal growth, making it particularly relevant for psychologists who work in areas like counseling, coaching, and personal development.

Simulating 500 million years of evolution with a language model

Hayes, T., Rao, R., et al. (2025).

Science.

https://doi.org/10.1126/science.ads0018

Abstract

More than three billion years of evolution have produced an image of biology encoded into the space of natural proteins. Here we show that language models trained at scale on evolutionary data can generate functional proteins that are far away from known proteins. We present ESM3, a frontier multimodal generative language model that reasons over the sequence, structure, and function of proteins. ESM3 can follow complex prompts combining its modalities and is highly responsive to alignment to improve its fidelity. We have prompted ESM3 to generate fluorescent proteins. Among the generations that we synthesized, we found a bright fluorescent protein at a far distance (58% sequence identity) from known fluorescent proteins, which we estimate is equivalent to simulating five hundred million years of evolution.

A pdf of the research is here.

Here are some thoughts:

A groundbreaking advancement in evolutionary biology and artificial intelligence has emerged with the development of ESM3, a cutting-edge multimodal generative language model capable of simulating the evolution of proteins over hundreds of millions of years. ESM3 leverages principles of language modeling to reason across the sequence, structure, and function of proteins, enabling the creation of novel proteins with unprecedented diversity and functionality. This innovation is built on scalable architecture, utilizing 98 billion parameters trained on billions of protein sequences and structures. Through this extensive training, ESM3 generates proteins that align with complex biological prompts, uncovering regions of protein design previously unexplored by natural evolution.

Among its remarkable achievements, ESM3 successfully created a novel fluorescent protein named esmGFP, which is evolutionarily distinct from known proteins, effectively simulating over 500 million years of natural evolutionary progress. Using token-based training, ESM3 predicts and generates protein sequences and structures with extraordinary fidelity to natural patterns. The model’s iterative fine-tuning process enhances its biological alignment, improving its ability to solve intricate design challenges such as ligand binding and tertiary coordination tasks. Moreover, ESM3 enables programmable control, offering scientists the ability to design proteins with specified traits, such as fluorescence, while maintaining their functional integrity.

This innovative approach holds transformative potential for biotechnology, facilitating the rapid design of proteins for applications ranging from medicine to materials science. ESM3’s ability to simulate and surpass the constraints of natural evolution marks a new frontier in computational biology, driven by the synergy of artificial intelligence and evolutionary science. By unlocking new possibilities in protein design, ESM3 is poised to redefine the boundaries of what is achievable in both theoretical and applied biosciences.

How Moral Case Deliberation Supports Good Clinical Decision Making

Inguaggiato, G., et al. (2019).

The AMA Journal of Ethic, 21(10),

E913-919.

https://doi.org/10.1001/amajethics.2019.913

Abstract

In clinical decision making, facts are presented and discussed, preferably in the context of both evidence-based medicine and patients’ values. Because clinicians’ values also have a role in determining the best courses of action, we argue that reflecting on both patients’ and professionals’ values fosters good clinical decision making, particularly in situations of moral uncertainty. Moral case deliberation, a form of clinical ethics support, can help elucidate stakeholders’ values and how they influence interpretation of facts. This article demonstrates how this approach can help clarify values and contribute to good clinical decision making through a case example.

Here are some thoughts:

This article discusses how moral case deliberation (MCD) supports good clinical decision-making. It argues that while evidence-based medicine and patient values are crucial, clinicians' values also play a significant role, especially in morally uncertain situations. MCD, a form of clinical ethics support, helps clarify the values of all stakeholders and how these values influence the interpretation of facts. The article explains how MCD differs from shared decision-making, emphasizing its focus on ethical dilemmas and understanding moral uncertainty among caregivers rather than reaching a shared decision with the patient. Through dialogue and a structured approach, MCD facilitates a deeper understanding of the situation, leading to better-informed and morally sensitive clinical decisions. The article uses a case study from a neonatal intensive care unit to illustrate how MCD can help resolve disagreements and uncertainties by exploring the different values held by nurses and physicians.

Surviving and thriving in spite of hate: Burnout and resiliency in clinicians working with patients attracted by violent extremism

Rousseau, C.,  et al. (2025).

The American journal of orthopsychiatry,

10.1037/ort0000832.

Advance online publication.

https://doi.org/10.1037/ort0000832

Abstract

Violent extremism (VE) is often manifested through hate discourses, which are hurtful for their targets, shatter social cohesion, and provoke feelings of impending threat. In a clinical setting, these discourses may affect clinicians in different ways, eroding their capacity to provide care. This clinical article describes the subjective experiences and the coping strategies of clinicians engaged with individuals attracted by VE. A focus group was held with eight clinicians and complemented with individual interviews and field notes. Clinicians reported four categories of personal consequences. First, results show that the effect of massive exposure to hate discourses is associated with somatic manifestations and with the subjective impression of being dirty. Second, clinicians endorse a wide range of work-related affects, ranging from intense fear, anger, and irritation to sadness and numbing. Third, they perceive that their work has relational consequences on their families and friends. Last, clinicians also describe that their work transforms their vision of the world. In terms of coping strategies, team relations and a community of practice were identified as supportive. With time, the pervasive uncertainty, the relative lack of institutional support, and the work-related emotional burden are associated with disengagement and burnout, in particular in practitioners working full-time with this clientele. Working with clients attracted to or engaged in VE is very demanding for clinicians. To mitigate the emotional burden of being frequently confronted with hate and threats, team relations, decreasing clinical exposure, and avoiding heroic positions help prevent burnout.

A pdf is here.

Here are some thoughts:

The article explores the psychological impact on clinicians treating individuals drawn to violent extremism (VE). It documents how prolonged exposure to hate discourse can lead to somatic symptoms (e.g., nausea, headaches), emotional exhaustion, hypervigilance, and a sense of being "contaminated" by hate. Clinicians reported struggling with moral dilemmas, fearing responsibility if a patient acts violently, and experiencing disruptions in their personal relationships.

Despite these challenges, team support, supervision, humor, and structured work boundaries were identified as critical resilience factors. The study highlights the need for institutional backing and clinician training to manage moral distress, avoid burnout, and sustain ethical engagement with patients who espouse extremist views.