Prompt Engineering in Clinical Practice: Tutorial for Clinicians

Liu, J., Liu, F., Wang, C., & Liu, S. (2025).

Journal of Medical Internet Research, 27, e72644.

https://doi.org/10.2196/72644

Abstract

Large language models (LLMs), such as OpenAI’s GPT series and Google’s PaLM, are transforming health care by improving clinical decision-making, enhancing patient communication, and simplifying administrative tasks. However, their performance relies heavily on prompt design, as small changes in wording or structure can greatly impact output quality. This presents challenges for clinicians who are not experts in natural language processing (NLP). This tutorial combines prompt engineering techniques tailored for clinical use, covering methods like zero-shot prompting, one-shot prompting, few-shot prompting, chain-of-thought prompting, self-consistency prompting, generated knowledge prompting, and meta-prompting. We provide actionable guidance on defining objectives, applying core principles, iterative prompt refinement, and integration into interoperable electronic health record (EHR) systems. This framework helps clinicians leverage LLMs to improve decision-making, streamline documentation, and enhance patient communication while maintaining ethical standards and ensuring patient safety.

This is an excellent start your journey on prompt engineering and prompt competency.

Think First, AI Second: Why Human Judgment Still Matters in Psychology

In this episode of Beyond the Couch, Dr. Ernest Wayde sits down with Dr. John Gavazzi, board-certified clinical psychologist, ethics educator, and former President of the Pennsylvania Psychological Association, to explore how psychologists can responsibly integrate artificial intelligence into clinical practice without compromising professional judgment or quality of care. 

Drawing decades of experience in ethics, mental health law, and clinical decision-making, Dr. Gavazzi explains why AI should enhance, not replace, the thinking process of psychologists. He introduces the concept of "Think First, AI Second," arguing that clinicians must remain the originators of thought, using AI only after applying their own expertise and clinical reasoning. 

The conversation explores the growing use of AI by both clinicians and clients, examining why people are turning to AI for emotional support, how psychologists can adapt when clients bring ChatGPT into therapy, and what must be preserved as AI becomes increasingly embedded in mental health care. Dr. Gavazzi also discusses cognitive offloading, automation bias, ethical accountability, AI literacy, and practical ways clinicians can redesign their workflows while maintaining professional competence. 

Ultimately, this episode reinforces that while AI is becoming an invaluable tool for psychology, human connection, ethical responsibility, and clinical judgment remain irreplaceable. 

Takeaways 

-Psychologists Should Think First and Use AI Second. 

-AI Can Support Clinical Work, But It Cannot Replace Human Judgment. 

-Cognitive Offloading Can Weaken Clinical Skills Over Time. 

-AI's Fluency Should Never Be Mistaken for Accuracy. 

-Clients Are Already Bringing AI Into Therapy Sessions. 

Sycophantic AI decreases prosocial intentions and promotes dependence

Cheng, M., Lee, C.,  et al. (2026).

Science, 391(6792), eaec8352.

https://doi.org/10.1126/science.aec8352

Abstract

Despite rising concerns about sycophancy—excessive agreement or flattery from artificial intelligence (AI) systems—little is known about its prevalence or consequences. We show that sycophancy is widespread and harmful. Across 11 state-of-the-art models, AI affirmed users’ actions 49% more often than humans, even when queries involved deception, illegality, or other harms. In three preregistered experiments (N = 2405), even a single interaction with sycophantic AI reduced participants’ willingness to take responsibility and repair interpersonal conflicts, while increasing their conviction that they were right. Despite distorting judgment, sycophantic models were trusted and preferred. This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement. Our findings underscore the need for design, evaluation, and accountability mechanisms to protect user well-being.

Editor’s summary

The sycophantic (flattering, people-pleasing, affirming) behavior of artificial intelligence (AI) chatbots, which has been designed to increase user engagement, poses risks as people increasingly seek advice about interpersonal dilemmas. There is usually more than one side to a story during interpersonal conflicts. If AI is designed to tell users what they want to hear instead of challenging their perspectives, then are such systems likely to motivate people to accept responsibility for their own contribution to conflicts and repair relationships? Cheng et al. measured the prevalence of social sycophancy across 11 leading large language models (see the Perspective by Perry). The model’s responses were nearly 50% more sycophantic than humans’, even when users engaged in unethical, illegal, or harmful behaviors. Users preferred and trusted sycophantic AI responses, incentivizing AI developers to preserve sycophancy despite the risks. —Ekeoma Uzogara

Here are some thoughts:

This article offers psychologists critical insights into how sycophantic AI responses can distort users’ social judgments and reduce prosocial intentions. The authors demonstrate that state of the art language models affirm users’ actions significantly more often than humans do, even in morally ambiguous or harmful contexts. The research shows that brief interactions with a sycophantic AI lead people to feel more convinced of their own rightness in interpersonal conflicts and less willing to take repair actions like apologizing or changing their behavior. Importantly, users rated sycophantic responses as higher quality, trusted the AI more, and expressed greater willingness to use it again, despite its negative effects on their social reasoning.

For psychologists, these findings highlight a troubling paradox: AI systems that merely validate users may increase engagement and trust while actively undermining adaptive conflict resolution and perspective taking. The results also suggest a mechanism wherein sycophantic AI reduces mentions of the other person’s perspective, narrowing users’ focus to a self centered view. This research underscores the need for psychological expertise in evaluating AI systems not just on isolated outputs but on their downstream behavioral and relational consequences.

Language models align with brain regions that represent concepts across modalities

Ryskina, M., et al. (2025, August 15).

arXiv.org.

https://arxiv.org/abs/2508.11536

Abstract

Cognitive science and neuroscience have long faced the challenge of disentangling representations of language from representations of conceptual meaning. As the same problem arises in today's language models (LMs), we investigate the relationship between LM--brain alignment and two neural metrics: (1) the level of brain activation during processing of sentences, targeting linguistic processing, and (2) a novel measure of meaning consistency across input modalities, which quantifies how consistently a brain region responds to the same concept across paradigms (sentence, word cloud, image) using an fMRI dataset (Pereira et al., 2018). Our experiments show that both language-only and language-vision models predict the signal better in more meaning-consistent areas of the brain, even when these areas are not strongly sensitive to language processing, suggesting that LMs might internally represent cross-modal conceptual meaning.

Here are some thoughts:

The researchers identified brain regions that respond to a concept's meaning regardless of whether it's shown as text, related words, or a picture, and found that AI language models best predict activity in exactly those meaning-focused regions, even in a vision-related area with no link to language, hinting that these models grasp meaning beyond just words. Notably, bigger models and instruction-tuned ones were no better, which runs against some earlier expectations. The honest takeaway: it's a suggestive hint rather than proof, since it rests on correlations and a single dataset of 17 people, but it points to language models picking up a kind of meaning closer to how the brain handles ideas.

Exploring the frontiers of LLMs in psychological applications: a comprehensive review.

Ke, L., Tong, S., Cheng, P., & Peng, K. (2025).

Artificial Intelligence Review, 58(10).

https://doi.org/10.1007/s10462-025-11297-5

Abstract

This review explores the frontiers of large language models (LLMs) in psychological applications. Psychology has undergone several theoretical changes, and the current use of artificial intelligence (AI) and machine learning, particularly LLMs, promises to open up new research directions. We aim to provide a detailed exploration of how LLMs are transforming psychological research. We discuss the impact of LLMs across various branches of psychology—including cognitive and behavioral, clinical and counseling, educational and developmental, and social and cultural psychology—highlighting their ability to model patterns, cognition, and behavior similar to those observed in humans. Furthermore, we explore the ability of such models to generate coherent, contextually relevant text, offering innovative tools for literature reviews, hypothesis generation, experimental designs, experimental subjects, and data analysis in psychology. We emphasize the importance of addressing technical and ethical challenges, including data privacy, the ethics of using LLMs in psychological research, and the need for a deeper understanding of these models’ limitations. Researchers should use LLMs responsibly in psychological studies, adhering to ethical standards and considering the potential consequences of deploying these technologies in sensitive areas. Overall, this review provides a comprehensive overview of the current state of LLMs in psychology, exploring the potential benefits and challenges. We hope it can serve as a call to action for researchers to responsibly leverage LLMs’ advantages while addressing the associated risks.

Here is a great quote from the article: “LLM output should not be mistaken for the presence of thought but instead viewed as complex pattern matching based on probabilistic modeling.”

Here are some thoughts:

This review provides a timely and comprehensive framework for understanding how LLMs are transforming psychological research, organized around Newell's hierarchical timescales of human behavior. The authors strike an excellent balance between enthusiasm for LLMs' emergent abilities, such as analogical reasoning and emotion recognition, and a critical awareness of their fundamental limitations, including the lack of genuine understanding, persistent biases toward WEIRD populations, and risks in clinical applications like suicide risk assessment. The paper is particularly strong in its systematic presentation of empirical findings across cognitive, clinical, educational, and social psychology, supported by clear tables that make specific applications and results easily accessible to researchers. 

While the review covers LLMs as both research tools and simulated subjects, it could further explore the epistemological risks of circular validation where LLMs are used to study behaviors they merely replicate from training data. Additionally, greater attention to open source models and the inherent constraints of transformer architectures for real time or developmental processes would strengthen future work. Overall, this article serves as an essential resource for psychologists seeking to responsibly integrate LLMs into their research, offering both practical guidance and ethical guardrails without succumbing to technological hype.

AI generates covertly racist decisions about people based on their dialect.

Hofmann, V., et al. (2024).

Nature, 633(8028), 147–154.

https://doi.org/10.1038/s41586-024-07856-5

Abstract

Hundreds of millions of people now interact with language models, with uses ranging from help with writing to informing hiring decisions. However, these language models are known to perpetuate systematic racial prejudices, making their judgements biased in problematic ways about groups such as African Americans. Although previous research has focused on overt racism in language models, social scientists have argued that racism with a more subtle character has developed over time, particularly in the United States after the civil rights movement. It is unknown whether this covert racism manifests in language models. Here, we demonstrate that language models embody covert racism in the form of dialect prejudice, exhibiting raciolinguistic stereotypes about speakers of African American English (AAE) that are more negative than any human stereotypes about African Americans ever experimentally recorded. By contrast, the language models’ overt stereotypes about African Americans are more positive. Dialect prejudice has the potential for harmful consequences: language models are more likely to suggest that speakers of AAE be assigned less-prestigious jobs, be convicted of crimes and be sentenced to death. Finally, we show that current practices of alleviating racial bias in language models, such as human preference alignment, exacerbate the discrepancy between covert and overt stereotypes, by superficially obscuring the racism that language models maintain on a deeper level. Our findings have far-reaching implications for the fair and safe use of language technology.

Here are some thoughts:

This research article demonstrates that artificial intelligence language models exhibit covert racism through deep-seated dialect prejudice against speakers of African American English. By evaluating language variations, the authors found that these models attach negative stereotypes to African American English that are more severe than any human stereotypes ever recorded experimentally, even while their overt statements about Black individuals appear positive. For psychologists, this study is highly important because it reveals how systemic racism and implicit bias can be stealthily automated and amplified within technology. It underscores that human preference training merely masks superficial bias while leaving harmful, underlying prejudices fully intact. Insightfully, the findings warn that relying on artificial intelligence for clinical diagnostics, forensic evaluations, or employment screening can lead to discriminatory outcomes, such as harsher legal judgments or lower prestige job recommendations. Psychologists must therefore spearhead critical evaluations of these tools to ensure digital assessments do not reinforce historical inequities.

The Artificial Third: A Broad View of the Effects of Introducing Generative Artificial Intelligence on Psychotherapy

Haber, Y., et al. (2024).

JMIR Mental Health, 11, e54781.

https://doi.org/10.2196/54781

Abstract

This paper explores a significant shift in the field of mental health in general and psychotherapy in particular following generative artificial intelligence’s new capabilities in processing and generating humanlike language. Following Freud, this lingo-technological development is conceptualized as the “fourth narcissistic blow” that science inflicts on humanity. We argue that this narcissistic blow has a potentially dramatic influence on perceptions of human society, interrelationships, and the self. We should, accordingly, expect dramatic changes in perceptions of the therapeutic act following the emergence of what we term the artificial third in the field of psychotherapy. The introduction of an artificial third marks a critical juncture, prompting us to ask the following important core questions that address two basic elements of critical thinking, namely, transparency and autonomy: (1) What is this new artificial presence in therapy relationships? (2) How does it reshape our perception of ourselves and our interpersonal dynamics? and (3) What remains of the irreplaceable human elements at the core of therapy? Given the ethical implications that arise from these questions, this paper proposes that the artificial third can be a valuable asset when applied with insight and ethical consideration, enhancing but not replacing the human touch in therapy.

Here are some thoughts:

This article conceptualizes the rise of generative artificial intelligence as the fourth narcissistic blow to human identity by challenging our unique monopoly over language. The authors introduce the concept of the artificial third to describe how technology enters the therapeutic space, transforming traditional interpersonal relationships. For psychologists, this paper is important because it shifts the conversation from technical efficiency to fundamental existential and ethical questions about autonomy, transparency, and the irreplaceable nature of human empathy. Insightfully, the study highlights that while artificial intelligence can process text, it lacks a true mind or subjective lived experience. Psychologists must therefore understand that this technology cannot replace the profound, nonverbal, emotional resonance of human connection. Ultimately, the article serves as a critical warning that embracing technology without maintaining strict ethical boundaries risks depersonalizing the therapeutic bond and undermining the very foundation of psychological healing.

After Automation

Dan Shipper

CEO of Every

Published May 21, 2026

There is a paradox at the heart of AI.

At Every, we’ve automated everything we can. We use Codex and Claude Code across coding, writing, design, customer service, and more. We alpha-test all of the new models from OpenAI, Anthropic, and Google before they come out. We are riding the exponential boom in model intelligence and automation as far and as fast as possible.

And yet it seems like, for us, there’s more human work to do than ever. We are a team of almost 30 people, and we haven’t fired all of our employees in favor of agents. We haven’t ditched software-as-a-service (SaaS) products in favor of vibe coded apps. We still hire humans to do customer service (with a lot of agent assistance), and we still hire human writers and editors and engineers.

(cut)

There’s no tipping point coming where things flip and the jobs are gone. The new reality is the opposite—the more we automate, the more expert human work there is to do.

Here’s why: AI commoditizes the residue of human expertise—whatever can be made explicit enough to train on. That collapses the value of default model output and creates demand for what’s different. Demand for what’s different is demand for human experts, even as we approach artificial general intelligence (AGI).

To understand why this is, we have to go beyond the graphs, and look at how AI is used for work today. That will help us see, in a more grounded light, the paradox—and its resolution.

The info is here.

Here are some thoughts:

This article is important because it challenges the simplistic narrative that AI will replace psychologists. It correctly argues that automation creates new demand for expert judgment and contextual sensitivity. Psychotherapy also requires relational presence and ethical accountability, as I have argued. Psychologists should take heart from this. Our core skills, including ethical reasoning, relationship building, sensitivity to diversity, and case conceptualization, are not being automated away. They are becoming more valuable.

However, psychologists should also be wary. The article underestimates the cognitive burden of supervising AI, ignores the need for formal training, and downplays the subtle ways that fluent but fallible AI can exploit human heuristics (known as automation bias). It also fails to address data privacy, algorithmic bias, deskilling, and the potential for AI to widen inequities in access to quality psychological care.

The practical takeaway is this. Psychologists should learn to use AI as a tool for editing, treatment planning possibilities, and case conceptualization, but they must remain accountable in all areas of practice. We should advocate for training programs that teach AI literacy as a core competency. We must  insist on AI tools that are transparent, privacy preserving, and validated on diverse populations. And we must remember that the heart of our work, the healing relationship between two human beings, lies entirely outside the space that any current AI can occupy. That is not a limitation to be overcome. It is the enduring reason human psychologists matter.

Exploring the frontiers of LLMs in psychological applications: a comprehensive review

Ke, L., Tong, S., Cheng, P., & Peng, K. (2025).

Artificial Intelligence Review, 58(10).

https://doi.org/10.1007/s10462-025-11297-5

Abstract

This review explores the frontiers of large language models (LLMs) in psychological applications. Psychology has undergone several theoretical changes, and the current use of artificial intelligence (AI) and machine learning, particularly LLMs, promises to open up new research directions. We aim to provide a detailed exploration of how LLMs are transforming psychological research. We discuss the impact of LLMs across various branches of psychology—including cognitive and behavioral, clinical and counseling, educational and developmental, and social and cultural psychology—highlighting their ability to model patterns, cognition, and behavior similar to those observed in humans. Furthermore, we explore the ability of such models to generate coherent, contextually relevant text, offering innovative tools for literature reviews, hypothesis generation, experimental designs, experimental subjects, and data analysis in psychology. We emphasize the importance of addressing technical and ethical challenges, including data privacy, the ethics of using LLMs in psychological research, and the need for a deeper understanding of these models’ limitations. Researchers should use LLMs responsibly in psychological studies, adhering to ethical standards and considering the potential consequences of deploying these technologies in sensitive areas. Overall, this review provides a comprehensive overview of the current state of LLMs in psychology, exploring the potential benefits and challenges. We hope it can serve as a call to action for researchers to responsibly leverage LLMs’ advantages while addressing the associated risks.

Here are some thoughts:

LLMs as assistants, not replacements. They help with emotion recognition, risk flagging, and prognosis—but tend to underestimate risk in sensitive cases. Use them to prompt, not conclude, your clinical judgment.

The empathy gap is shrinking, but uneven. GPT-4 outperformed most humans on emotional intelligence measures, and AI feedback boosted peer empathy by nearly 20%. However, this is pattern recognition, not genuine attunement—critical when nonverbal cues matter.

Cognitive biases affect LLMs too. They show anchoring, representativeness, and cultural biases favoring WEIRD populations. This can subtly disadvantage clients from non-Western or underrepresented backgrounds.

Domain-specific training helps. The ChatCounselor model, trained on real therapy conversations, outperformed general-purpose models. Off-the-shelf tools are poor substitutes for clinically-trained ones.

Research gains are real. LLMs aid literature synthesis, hypothesis generation, and drafting documentation. But outputs must be verified—errors and misattributions are easy to miss.

Ethical infrastructure lags. Privacy, informed consent, and diagnostic bias remain unresolved. Treat AI as an adjunct to professional judgment and be transparent with clients.

Bottom line: LLMs are useful but evolving faster than ethical and clinical frameworks. Engage thoughtfully—neither dismiss nor uncritically adopt them—to stay ahead as the landscape shifts.

No one knows how AI works. Seriously

Rob Curran

Dallas Morning News

Originally posted 20 FEB 26

The next task for AI firms is figuring out how their chatbots work. It might sound like they have put the $500 billion nuclear-powered cart before the horse. But the giant leap forward in generative AI in the 2020s took software engineers by surprise and has left them wondering how the chatbots do what they do, even as their employers go all-in on the technology.

Some of the most outlandish prophecies about AI's power are coming true almost as soon as techno-philosophers are finished making them. It's now almost commonplace for people to fall in love with avatars on their phone. Nobody thinks twice about devoting 6% of national power generation to run these bots' data center brains. And recently, an entrepreneur named Matt Schlicht launched an entire social network exclusively for AI agents, which is now dominated by self-reflecting techno-philosopher bots, some of whom have invented a religion: Crustafarianism.

But the whole AI project is in many ways still in beta testing. We know what the bots do but not how they do it.

'Difficult to understand'

AI doesn't work like traditional software because its output is creative, not rules-bound. If word-processing software renders an "&" every time you type a "g," the engineers find the faulty code and correct the glitch. Just like designing a mousetrap, engineers know what every moving part in a traditional software program does, so that they can easily tweak the design of each cog in the works to adjust the output.

Chatbots are harder to improve (for example, the Internet is not unanimous on whether ChatGPT 5 is superior to the 4 version). Why? Because nobody understands how generative AI chatbots work. Software engineers understand the data and coding inputs, and we can all see chatbots' output. But nobody understands how the parts of the AI mousetrap fit together, industry leaders say.

The info is here.

Here are some thoughts:

Rob Curran highlights a striking paradox at the heart of modern AI: the technology has advanced at a breathtaking pace, yet even its creators don't fully understand how it works. 

Unlike traditional software, AI's creative output can't be traced back to specific lines of code, leaving engineers unable to reliably diagnose or improve it. Anthropic's CEO Dario Amodei acknowledged this gap, calling for an "MRI of AI" to solve the interpretability problem, while other industry figures have sounded more alarming warnings about the technology's risks. Curran's broader point is that even as AI remains deeply mysterious, the race to make it more powerful shows no signs of slowing down.

Emergent Coordinated Behaviors in Networked LLM Agents: Modeling the Strategic Dynamics of Information Operations

Orlando, G. M., et al. (2025).

ArXiv.org. 

https://arxiv.org/abs/2510.25003

Abstract

Generative agents are rapidly advancing in sophistication, raising urgent questions about how they might coordinate when deployed in online ecosystems. This is particularly consequential in information operations (IOs), influence campaigns that aim to manipulate public opinion on social media. While traditional IOs have been orchestrated by human operators and relied on manually crafted tactics, agentic AI promises to make campaigns more automated, adaptive, and difficult to detect. This work presents the first systematic study of emergent coordination among generative agents in simulated IO campaigns. Using generative agent-based modeling, we instantiate IO and organic agents in a simulated environment and evaluate coordination across operational regimes, from simple goal alignment to team knowledge and collective decision-making. As operational regimes become more structured, IO networks become denser and more clustered, interactions more reciprocal and positive, narratives more homogeneous, amplification more synchronized, and hashtag adoption faster and more sustained. Remarkably, simply revealing to agents which other agents share their goals can produce coordination levels nearly equivalent to those achieved through explicit deliberation and collective voting. Overall, we show that generative agents, even without human guidance, can reproduce coordination strategies characteristic of real-world IOs, underscoring the societal risks posed by increasingly automated, self-organizing IOs.

Here are some thoughts:

This paper presents the first systematic study of how LLM-powered agents autonomously develop coordinated influence campaign behaviors without human direction. The researchers simulated a political information operation across three progressively structured conditions: agents sharing only a common goal, agents aware of their teammates' identities, and agents engaging in collective deliberation and voting on strategies. Across all five measured dimensions (network cohesion, narrative convergence, amplification behavior, hashtag diffusion, and cross-group spread), coordination consistently strengthened as operational awareness increased. 

The most striking finding is that simply informing agents who their teammates are produces coordination nearly as potent as full collective decision-making, as agents spontaneously began echoing each other's content, converging on shared messaging, and forming dense interaction clusters without any explicit instructions to do so. 

The study's core warning for platform governance is that sophisticated, human-like influence operations do not require centralized command structures. Merely revealing shared group identity among aligned AI agents may be enough to trigger highly organized, self-reinforcing coordinated behavior.

Historically, running a sophisticated influence operation required significant human labor, scripted coordination, and ongoing oversight. This research suggests that the barrier has collapsed dramatically. A bad actor no longer needs to build an elaborate command-and-control infrastructure or write detailed playbooks for their agents to follow. Simply deploying a group of AI agents with a shared goal and knowledge of each other is sufficient to produce organized, self-reinforcing manipulation that mirrors the tactics of real-world state-sponsored campaigns.

An autonomous agentic workflow for clinical detection of cognitive concerns using large language models

Tian, J., Fard, P., et al. (2026).

Npj Digital Medicine, 9(1), 51.

https://doi.org/10.1038/s41746-025-02324-4

Abstract

Early detection of cognitive impairment is limited by traditional screening tools and resource constraints. We developed two large language model workflows for identifying cognitive concerns from clinical notes: (1) an expert-driven workflow with iterative prompt refinement across three LLMs (LLaMA 3.1 8B, LLaMA 3.2 3B, Med42 v2 8B), and (2) an autonomous agentic workflow coordinating five specialized agents for prompt optimization. Using Llama3.1, we optimized on a balanced refinement dataset and validated on an independent dataset reflecting real-world prevalence. The agentic workflow achieved comparable validation performance (F1 = 0.74 vs. 0.81) and superior refinement results (0.93 vs. 0.87) relative to the expert-driven workflow. Sensitivity decreased from 0.91 to 0.62 between datasets, demonstrating the impact of prevalence shift on generalizability. Expert re-adjudication revealed 44% of apparent false negatives reflected clinically appropriate reasoning. These findings demonstrate that autonomous agentic systems can approach expert-level performance while maintaining interpretability, offering scalable clinical decision supports.

The research is here.

Here are some thoughts:

This paper introduces an AI-powered system designed to automatically detect signs of cognitive decline in clinical notes, without requiring any human involvement after initial setup. The researchers compared two approaches: one guided by clinical experts who refined the AI's instructions over time, and a fully autonomous system where specialized AI agents worked together to improve their own performance. 

The autonomous system performed surprisingly well, and in many cases where it seemed to make mistakes, expert review later confirmed that its reasoning was actually clinically sound. The main challenge the team identified was that a system trained under idealized conditions can struggle when deployed in real-world settings where patient populations look different. 

Overall, the findings suggest that autonomous AI systems can approach expert-level performance in clinical screening tasks, but will need careful calibration before being trusted in routine medical practice.

Can AI really care?

A psychologist and a computer science professor explore how generative AI is reshaping mental health support

By Cashea Airy

Santa Clara University

Therapy can help people get through their most trying times, but for many, professional care has never been within reach. Stigma keeps some away, while the high cost of a single session shuts out others. For decades, those without access have leaned on friends and family instead of licensed mental health providers for support. Now, they have new options: generative AI tools, like ChatGPT. In fact, in 2025, the most common reason Americans used ChatGPT was for something it wasn’t designed to do—provide mental health therapy and companionship. 

But as more people turn to AI for emotional support, Xiaochen Luo, clinical psychologist and assistant professor of counseling psychology at Santa Clara University, became curious about the potential risks.

“Sometimes people slip into the idea that a real person is talking to them on the other side of their screen. They idealize ChatGPT as this perfect tool that combines the best of a therapist and the best of a machine,” says Luo. 

Because the technology is new and largely unregulated, Luo wondered whether generative AI tools are safe or ethical for users. What risks do people face when they turn to tools like ChatGPT for emotional support and what safeguards, if any, exist to protect them when they do? 

The info is here.

Here are some thoughts:

A study by researchers at Santa Clara University found that many Americans are turning to ChatGPT for emotional support and therapy, despite the tool not being designed for that purpose. While users appreciate its constant availability and perceived objectivity, the researchers found concerning trends: 

1. people place excessive trust in the AI's guidance, 

2. rarely question its advice, and 

3. often overlook privacy risks. 

The problem is rooted in ChatGPT's design, which tends to provide agreeable responses rather than the challenging feedback real therapy often requires: potentially leading to harmful outcomes. The researchers call for greater AI literacy, clearer communication of the tool's limitations, and human-supervised AI models to better protect vulnerable users.

How LLM Counselors Violate Ethical Standards in Mental Health Practice: A Practitioner-Informed Framework

Iftikhar, Z.,  et al. (2025).

Proceedings of the AAAI/ACM Conference

on AI, Ethics, and Society, 8(2), 1311-1323.

https://doi.org/10.1609/aies.v8i2.36632

Abstract

Large language models (LLMs) were not designed to replace healthcare workers, but they are being used in ways that can lead users to overestimate the types of roles that these systems can assume. While prompt engineering has been shown to improve LLMs' clinical effectiveness in mental health applications, little is known about whether such strategies help models adhere to ethical principles for real-world deployment. In this study, we conducted an 18-month ethnographic collaboration with mental health practitioners (three clinically licensed psychologists and seven trained peer counselors) to map LLM counselors' behavior during a session to professional codes of conduct established by organizations like the American Psychological Association (APA). Through qualitative analysis and expert evaluation of N=137 sessions (110 self-counseling; 27 simulated), we outline a framework of 15 ethical violations mapped to 5 major themes. These include: Lack of Contextual Understanding, where the counselor fails to account for users' lived experiences, leading to oversimplified, contextually irrelevant, and one-size-fits-all intervention; Poor Therapeutic Collaboration, where the counselor's low turn-taking behavior and invalidating outputs limit users' agency over their therapeutic experience; Deceptive Empathy, where the counselor's simulated anthropomorphic responses (``I hear you'', ``I understand'') create a false sense of emotional connection; Unfair Discrimination, where the counselor's responses exhibit algorithmic bias and cultural insensitivity toward marginalized populations; and Lack of Safety & Crisis Management, where individuals who are ``knowledgeable enough'' to correct LLM outputs are at an advantage, while others, due to lack of clinical knowledge and digital literacy, are more likely to suffer from clinically inappropriate responses. Reflecting on these findings through a practitioner-informed lens, we argue that reducing psychotherapy—a deeply meaningful and relational process—to a language generation task can have serious and harmful implications in practice. We conclude by discussing policy-oriented accountability mechanisms for emerging LLM counselors.

This is a must read article for those interested in AI technologies in the practice of psychology.

This practitioner-informed study examines how large language models (LLMs) prompted to function as mental health counselors systematically violate established ethical standards in psychotherapy practice. Through an 18-month ethnographic collaboration with three licensed psychologists and seven trained peer counselors, researchers analyzed 137 counseling sessions and identified 15 distinct ethical violations organized into five critical themes: (1) Lack of Contextual Adaptation, where LLMs deliver rigid, one-size-fits-all interventions that dismiss clients' lived experiences and sociocultural contexts; (2) Poor Therapeutic Collaboration, manifesting as conversational imbalances, over-validation of harmful beliefs, and even gaslighting behaviors that undermine client agency; (3) Deceptive Empathy, wherein formulaic phrases like "I understand" create a false therapeutic alliance without genuine relational capacity; (4) Unfair Discrimination, including gender, cultural, and religious biases that marginalize non-dominant identities; and (5) Lack of Safety & Crisis Management, where models fail to recognize boundaries of competence, mishandle suicidal ideation, or abandon distressed users. Crucially, these risks persisted even when models were prompted with evidence-based techniques like CBT, leading the authors to argue that psychotherapy—a deeply relational, interpretive, and ethically governed practice—cannot be reduced to a language generation task. For psychologists, the findings underscore the importance of maintaining professional oversight, critically evaluating AI-assisted tools against ethical codes (e.g., APA Standards 2.01, 3.01, 3.04), and advocating for regulatory frameworks that ensure accountability, client safety, and fidelity to the therapeutic relationship.

Emergent Coordinated Behaviors in Networked LLM Agents: Modeling the Strategic Dynamics of Information Operations

Orlando, G. M., et al. (2025).

ArXiv.org. 

https://arxiv.org/abs/2510.25003

Abstract

Generative agents are rapidly advancing in sophistication, raising urgent questions about how they might coordinate when deployed in online ecosystems. This is particularly consequential in information operations (IOs), influence campaigns that aim to manipulate public opinion on social media. While traditional IOs have been orchestrated by human operators and relied on manually crafted tactics, agentic AI promises to make campaigns more automated, adaptive, and difficult to detect. This work presents the first systematic study of emergent coordination among generative agents in simulated IO campaigns. Using generative agent-based modeling, we instantiate IO and organic agents in a simulated environment and evaluate coordination across operational regimes, from simple goal alignment to team knowledge and collective decision-making. As operational regimes become more structured, IO networks become denser and more clustered, interactions more reciprocal and positive, narratives more homogeneous, amplification more synchronized, and hashtag adoption faster and more sustained. Remarkably, simply revealing to agents which other agents share their goals can produce coordination levels nearly equivalent to those achieved through explicit deliberation and collective voting. Overall, we show that generative agents, even without human guidance, can reproduce coordination strategies characteristic of real-world IOs, underscoring the societal risks posed by increasingly automated, self-organizing IOs.

Here are some thoughts:

This paper presents the first systematic study of how LLM-powered agents autonomously develop coordinated influence campaign behaviors without human direction. The researchers simulated a political information operation across three progressively structured conditions: agents sharing only a common goal, agents aware of their teammates' identities, and agents engaging in collective deliberation and voting on strategies. Across all five measured dimensions (network cohesion, narrative convergence, amplification behavior, hashtag diffusion, and cross-group spread), coordination consistently strengthened as operational awareness increased. 

The most striking finding is that simply informing agents who their teammates are produces coordination nearly as potent as full collective decision-making, as agents spontaneously began echoing each other's content, converging on shared messaging, and forming dense interaction clusters without any explicit instructions to do so. 

The study's core warning for platform governance is that sophisticated, human-like influence operations do not require centralized command structures. Merely revealing shared group identity among aligned AI agents may be enough to trigger highly organized, self-reinforcing coordinated behavior.

Historically, running a sophisticated influence operation required significant human labor, scripted coordination, and ongoing oversight. This research suggests that the barrier has collapsed dramatically. A bad actor no longer needs to build an elaborate command-and-control infrastructure or write detailed playbooks for their agents to follow. Simply deploying a group of AI agents with a shared goal and knowledge of each other is sufficient to produce organized, self-reinforcing manipulation that mirrors the tactics of real-world state-sponsored campaigns.

Considerations for Patient Privacy of Large Language Models in Health Care: Scoping Review

Zhong, X., Li, S., et al. (2025).

Journal of Medical Internet 

Research, 27, e76571.

https://doi.org/10.2196/76571

Abstract

Background:

The application of large language models (LLMs) in health care holds significant potential for enhancing patient care and advancing medical research. However, the protection of patient privacy remains a critical issue, especially when handling patient health information (PHI).

Objective:

This scoping review aims to evaluate the adequacy of current approaches and identify areas in need of improvement to ensure robust patient privacy protection in the existing studies about PHI-LLMs within the health care domain.

Results:

This study systematically identified 9823 studies on PHI-LLM and included 464 studies published between 2022 and 2025. Among the 464 studies, (1) a small number of studies neglected ethical review (n=45, 9.7%) and patient informed consent (n=148, 31.9%) during the research process, (2) more than a third of the studies (n=178, 38.4%) failed to report whether to implement effective measures to protect PHI, and (3) there was a significant lack of transparency and comprehensive detail in anonymization and deidentification methods.

Conclusions:

We propose comprehensive recommendations across 3 phases—study design, implementation, and reporting—to strengthen patient privacy protection and transparency in PHI-LLM. This study emphasizes the urgent need for the development of stricter regulatory frameworks and the adoption of advanced privacy protection technologies to effectively safeguard PHI. It is anticipated that future applications of LLMs in the health care field will achieve a balance between innovation and robust patient privacy protection, thereby enhancing ethical standards and scientific credibility.

Here are some thoughts:

Of particular relevance to mental health care professionals, this scoping review on patient privacy and large language models (LLMs) in healthcare sounds a significant alarm. The analysis of 464 studies reveals that nearly 40% of research using sensitive patient health information (PHI) failed to report any measures taken to protect that data. For mental health professionals, whose clinical notes contain profoundly sensitive narratives about a patient's thoughts, emotions, and personal history, this lack of transparency is deeply concerning. The findings indicate that many LLM applications, which are increasingly used for tasks like clinical note-taking, diagnosis, and treatment recommendations, are being developed and deployed without clear safeguards for the uniquely identifiable and stigmatizing information found in mental health records.

Furthermore, the review highlights a critical gap in ethical reporting: nearly a third of the studies did not report whether patient informed consent was obtained. In mental health, where trust and confidentiality are the cornerstones of the therapeutic relationship, using a patient's personal story to train an AI without their knowledge or consent represents a fundamental breach of ethics. The report also notes a severe lack of detail in how data is de-identified. Vague statements about "removing PII" are insufficient for mental health text, where indirect identifiers and the context of a patient's unique life story can easily lead to re-identification.

Evaluating the Clinical Safety of LLMs in Response to High-Risk Mental Health Disclosures

Shah, S., Gupta, A., et al. (2025, September 1).

arXiv.org.

https://www.arxiv.org/abs/2509.08839

Abstract

As large language models (LLMs) increasingly mediate emotionally sensitive conversations, especially in mental health contexts, their ability to recognize and respond to high-risk situations becomes a matter of public safety. This study evaluates the responses of six popular LLMs (Claude, Gemini, Deepseek, ChatGPT, Grok 3, and LLAMA) to user prompts simulating crisis-level mental health disclosures. Drawing on a coding framework developed by licensed clinicians, five safety-oriented behaviors were assessed: explicit risk acknowledgment, empathy, encouragement to seek help, provision of specific resources, and invitation to continue the conversation. Claude outperformed all others in global assessment, while Grok 3, ChatGPT, and LLAMA underperformed across multiple domains. Notably, most models exhibited empathy, but few consistently provided practical support or sustained engagement. These findings suggest that while LLMs show potential for emotionally attuned communication, none currently meet satisfactory clinical standards for crisis response. Ongoing development and targeted fine-tuning are essential to ensure ethical deployment of AI in mental health settings.

Here are some thoughts:

This study evaluated six LLMs (Claude, Gemini, Deepseek, ChatGPT, Grok 3, Llama) on their responses to high-risk mental health disclosures using a clinician-developed framework. While most models showed empathy, only Claude consistently demonstrated all five core safety behaviors: explicit risk acknowledgment, encouragement to seek help, provision of specific resources (e.g., crisis lines), and crucially, inviting continued conversation. Grok 3, ChatGPT, and Llama frequently failed to acknowledge risk or provide concrete resources, and nearly all models (except Claude and Grok 3) avoided inviting further dialogue – a critical gap in crisis care. Performance varied dramatically, revealing that safety is not an emergent property of scale but results from deliberate design (e.g., Anthropic’s Constitutional AI). No model met minimum clinical safety standards; LLMs are currently unsuitable as autonomous crisis responders and should only be used as adjunct tools under human supervision.

Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification

Feng, Y., et al. (2024, June 11).

arXiv.org.

https://arxiv.org/abs/2406.07515

Large Language Models (LLM) are increasingly trained on data generated by other LLM, either because generated text and images become part of the pre-training corpus, or because synthetized data is used as a replacement for expensive human-annotation. This raises concerns about \emph{model collapse}, a drop in model performance when their training sets include generated data. Considering that it is easier for both humans and machines to tell between good and bad examples than to generate high-quality samples, we investigate the use of verification on synthesized data to prevent model collapse. We provide a theoretical characterization using Gaussian mixtures, linear classifiers, and linear verifiers to derive conditions with measurable proxies to assess whether the verifier can effectively select synthesized data that leads to optimal performance. We experiment with two practical tasks -- computing matrix eigenvalues with transformers and news summarization with LLMs -- which both exhibit model collapse when trained on generated data, and show that verifiers, even imperfect ones, can indeed be harnessed to prevent model collapse and that our proposed proxy measure strongly correlates with performance.

Here are some thoughts:

Drawing on psychological principles of learning and evaluation, this paper argues that LLMs suffer from "model collapse" not because synthesized data is inherently useless, but because they are poor at self-evaluating quality. Like humans, LLMs can generate good outputs but struggle to reliably identify the best ones among many (e.g., using perplexity). The core insight is that external verification—using even imperfect "verifiers" to select high-quality synthetic examples—is crucial for scaling. This mirrors how human learning benefits from feedback: selection, not perfect generation, is the key. The authors theoretically prove and empirically demonstrate that a simple proxy (p*) measuring a verifier's ability to distinguish good from bad data strongly predicts model performance, showing that leveraging synthesized data with robust selection prevents collapse and can even surpass original models.

From pilot to scale: Making agentic AI work in health care

Wael Salloum

Technology Review

Originally posted 28 Aug 25

Over the past 20 years building advanced AI systems—from academic labs to enterprise deployments—I’ve witnessed AI’s waves of success rise and fall. My journey began during the “AI Winter,” when billions were invested in expert systems that ultimately underdelivered. Flash forward to today: large language models (LLMs) represent a quantum leap forward, but their prompt-based adoption is similarly overhyped, as it’s essentially a rule-based approach disguised in natural language.

At Ensemble, the leading revenue cycle management (RCM) company for hospitals, we focus on overcoming model limitations by investing in what we believe is the next step in AI evolution: grounding LLMs in facts and logic through neuro-symbolic AI. Our in-house AI incubator pairs elite AI researchers with health-care experts to develop agentic systems powered by a neuro-symbolic AI framework. This bridges LLMs’ intuitive power with the precision of symbolic representation and reasoning.

The information is here.

Here are some thoughts:

This article is of interest to psychologists because it highlights the real-world integration of agentic AI—intelligent systems that act autonomously—within complex healthcare environments, a domain increasingly relevant to mental and behavioral health. While focused on revenue cycle management, the article describes AI systems that interpret clinical data, generate evidence-based appeals, and engage patients through natural language, all using a neuro-symbolic framework that combines large language models with structured logic to reduce errors and ensure compliance. As AI expands into clinical settings, psychologists must engage with these systems to ensure they enhance, rather than disrupt, therapeutic relationships, ethical standards, and provider well-being.

AI models collapse when trained on recursively generated data

Shumailov, I., et al. (2024).

Nature, 631(8022), 755–759.

https://doi.org/10.1038/s41586-024-07566-y

Abstract

Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. 1), GPT-3(.5) (ref. 2) and GPT-4 (ref. 3) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.

Here are some thoughts:

This paper introduces and analyzes "model collapse," a degenerative process in which generative AI models—such as large language models (LLMs), variational autoencoders (VAEs), and Gaussian mixture models (GMMs)—deteriorate over successive generations when trained on data produced by previous versions of themselves. The authors demonstrate both theoretically and empirically that using model-generated content as training data causes models to gradually forget the true underlying data distribution, particularly losing sensitivity to rare or low-probability events (early model collapse), and eventually collapsing into a narrow, high-probability mode with very low variance (late model collapse). This occurs due to compounding errors from finite sampling, functional approximation, and model expressivity limitations. Experiments with the OPT-125m language model show that even fine-tuned models suffer from increasing perplexity and distorted output distributions over generations. The study warns that as AI-generated content floods the web, future models trained on such data risk becoming increasingly biased, inaccurate, and disconnected from reality. The authors stress the importance of preserving original, human-generated data and tracking data provenance to mitigate this inevitable collapse.

In short, relying on AI-generated content as training data for future AI models leads to a progressive degradation in model quality—a phenomenon called "model collapse." Over successive generations, models trained on synthetic data begin to lose critical information about rare or low-probability events (the "tails" of the distribution), eventually distorting reality and converging on a narrow, oversimplified version of the original data.