How AI can distort clinical decision-making to prioritize profits over patients

Katie Palmer

STATnews.com

Originally posted 3 March 25

More than a decade ago, Ken Mandl was on a call with a pharmaceutical company and the leader of a social network for people with diabetes. The drug maker was hoping to use the platform to encourage its members to get a certain lab test.

The test could determine a patient's need for a helpful drug. But in that moment, said Mandl, director of the computational health informatics program at Boston Children's Hospital, "I could see this focus on a biomarker as a way to increase sales of the product." To describe the phenomenon, he coined the term "biomarkup": the way commercial interests can influence the creation, adoption, and interpretation of seemingly objective measures of medical status.

These days, Mandl has been thinking about how the next generation of quantified outputs in health could be gamed: artificial intelligence tools.

"It is easy to imagine a new generation of Al-based revenue cycle management model tools that achieve higher reimbursements by nudging clinicians toward more lucrative care pathways," Mandl wrote in a recent perspective in NEJM AI. "Al-based decision support interventions are vulnerable across their entire development life cycle and could be manipulated to favor specific products or services."

The info is paywalled here.

Here are some thoughts:

Dr. Ken Mandl raises a critical concern about the potential for "biomarkup" in the age of artificial intelligence within healthcare. This concept, initially describing how commercial interests can manipulate seemingly objective medical measures, now extends to AI tools. Mandl warns that AI-driven systems, designed for tasks like revenue cycle management or clinical decision support, could be subtly manipulated to prioritize financial gain over patient well-being. This manipulation might involve nudging clinicians towards more lucrative care pathways or tuning algorithms to generate more referrals, particularly in fee-for-service models. The issue is exacerbated in direct-to-consumer healthcare, where profit motives may be even stronger and regulatory oversight potentially weaker. The ease with which financial outcomes can be measured, compared to patient outcomes, further compounds the problem, creating a risk of AI implementation being driven primarily by return on investment. Mandl emphasizes the urgent need for transparency in AI decision frameworks, ethical development practices, and careful regulatory oversight to safeguard patient interests and ensure that AI serves its intended purpose of improving healthcare, not just increasing profits.

Risk of Attempted and Completed Suicide in Persons Diagnosed With Headache

Elser, H., Farkas, D. K., et al. (2025).

JAMA Neurology.

https://doi.org/10.1001/jamaneurol.2024.4974

Abstract

Importance  Although past research suggests an association between migraine and attempted suicide, there is limited research regarding risk of attempted and completed suicide across headache disorders.

Objective  To examine the risk of attempted and completed suicide associated with diagnosis of migraine, tension-type headache, posttraumatic headache, and trigeminal autonomic cephalalgia (TAC).

Design, Setting, and Participants  This was a population-based cohort study of Danish citizens from 1995 to 2020. The setting was in Denmark, with a population of 5.6 million people. Persons 15 years and older who were diagnosed with headache were matched by sex and birth year to persons without headache diagnosis with a ratio of 5:1. Data analysis was conducted from May 2023 to May 2024.

Conclusions and Relevance  Results of this cohort study revealing the robust and persistent association of headache diagnoses with attempted and completed suicide suggest that behavioral health evaluation and treatment may be important for these patients.

Here are some thoughts:

This study identified a significant association between headache diagnoses and elevated risks of both attempted and completed suicide. The analysis revealed a robust and persistent link, with individuals diagnosed with headaches facing a disproportionately higher likelihood of suicidal behavior compared to the general population. While the study did not specify headache subtypes, the findings underscore the need for heightened mental health screening and intervention in patients with headache disorders. Researchers emphasized integrating suicide risk assessments into routine clinical care for this vulnerable population.

Implications for Practice

The results align with broader calls to address mental health comorbidities in chronic pain conditions. Primary care providers, in particular, are urged to adopt proactive strategies, such as safety planning and risk screening, to mitigate suicide risk in patients with headaches. Psychologists also need to identify headaches as a risk for suicide.

WundtGPT: Shaping Large Language Models To Be An Empathetic, Proactive Psychologist

Ren, C., Zhang, Y., He, D., & Qin, J. 

(2024, June 16).

arXiv.org. https://arxiv.org/abs/2406.15474

Abstract

Large language models (LLMs) are raging over the medical domain, and their momentum has carried over into the mental health domain, leading to the emergence of few mental health LLMs. Although such mental health LLMs could provide reasonable suggestions for psychological counseling, how to develop an authentic and effective doctor-patient relationship (DPR) through LLMs is still an important problem. To fill this gap, we dissect DPR into two key attributes, i.e., the psychologist's empathy and proactive guidance. We thus present WundtGPT, an empathetic and proactive mental health large language model that is acquired by fine-tuning it with instruction and real conversation between psychologists and patients. It is designed to assist psychologists in diagnosis and help patients who are reluctant to communicate face-to-face understand their psychological conditions. Its uniqueness lies in that it could not only pose purposeful questions to guide patients in detailing their symptoms but also offer warm emotional reassurance. In particular, WundtGPT incorporates Collection of Questions, Chain of Psychodiagnosis, and Empathy Constraints into a comprehensive prompt for eliciting LLMs' questions and diagnoses. Additionally, WundtGPT proposes a reward model to promote alignment with empathetic mental health professionals, which encompasses two key factors: cognitive empathy and emotional empathy. We offer a comprehensive evaluation of our proposed model. Based on these outcomes, we further conduct the manual evaluation based on proactivity, effectiveness, professionalism and coherence. We notice that WundtGPT can offer professional and effective consultation. The model is available at huggingface. 

The research is here.

Here are some thoughts:

WundtGPT is an innovative large language model (LLM) specifically designed for mental health tasks. The model addresses three critical limitations in existing mental health LLMs: lack of goal-oriented diagnosis, insufficient proactive questioning, and ambiguous conceptualization of empathy.

The researchers developed WundtGPT by fine-tuning it using instruction and real-world conversation datasets between psychologists and patients. Its unique capabilities include posing purposeful questions to guide patients in detailing their symptoms and offering warm emotional reassurance. The model incorporates a comprehensive prompt strategy that includes a Collection of Questions, Chain of Psychodiagnosis, and Empathy Constraints.

A key innovation is the model's reward system, which promotes alignment with empathetic mental health professionals by encompassing two critical factors: cognitive empathy and emotional empathy. For cognitive empathy, the model uses an emotional detection task, while emotional empathy is aligned through reinforcement learning from human feedback.

The researchers evaluated WundtGPT from two perspectives: its ability to provide proactive diagnosis and deliver warm psychological consultation. The evaluation involved emotional benchmarking and expert assessments of the model's proactivity, effectiveness, professionalism, and coherence. Experimental results demonstrated that WundtGPT exhibits superior performance compared to baseline LLMs in simulated medical consultation scenarios.

Notably, WundtGPT is claimed to be the first proactive LLM specifically designed for mental health tasks, capable of assisting psychologists in diagnosis and helping patients who are reluctant to communicate face-to-face understand their psychological conditions.

Large Language Models Pass the Turing Test

Jones, C. R., & Bergen, B. K. (2025, March 31).

arXiv.org.

https://arxiv.org/abs/2503.23674

Abstract

We evaluated 4 systems (ELIZA, GPT-4o, LLaMa-3.1-405B, and GPT-4.5) in two randomised, controlled, and pre-registered Turing tests on independent populations. Participants had 5 minute conversations simultaneously with another human participant and one of these systems before judging which conversational partner they thought was human. When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant. LLaMa-3.1, with the same prompt, was judged to be the human 56% of the time -- not significantly more or less often than the humans they were being compared to -- while baseline models (ELIZA and GPT-4o) achieved win rates significantly below chance (23% and 21% respectively). The results constitute the first empirical evidence that any artificial system passes a standard three-party Turing test. The results have implications for debates about what kind of intelligence is exhibited by Large Language Models (LLMs), and the social and economic impacts these systems are likely to have.

Here are some thoughts:

The study highlights significant advancements in AI technology, particularly in the capabilities of large language models (LLMs), as demonstrated by their ability to pass the Turing test. GPT-4.5 and LLaMa-3.1-405B, when given specific persona prompts, achieved win rates of 73% and 56%, respectively, meaning they were judged to be human more often than actual human participants in some cases. This marks the first robust empirical evidence that an AI system can pass the standard three-party Turing test, a major milestone in AI development. The success of these models underscores their ability to convincingly mimic human conversation, blurring the line between human and machine interaction.

A key factor in their performance was the use of tailored prompts. Models instructed to adopt a humanlike persona—such as a young, introverted individual familiar with internet culture—significantly outperformed those without such guidance. This adaptability demonstrates the flexibility of modern LLMs and their capacity to refine behavior based on contextual instructions. In contrast, older systems like ELIZA and GPT-4o performed poorly, with win rates of just 23% and 21%, highlighting the rapid progress in AI conversational abilities. The study also challenges the "ELIZA effect," showing that contemporary LLMs succeed not through superficial imitation but by replicating nuanced human conversational patterns.

Human interrogators often relied on social and emotional cues—such as humor, personality, and linguistic style—rather than traditional measures of intelligence to distinguish humans from AI. Despite some effective strategies, like "jailbreak" prompts or probing for inconsistencies, most participants struggled to reliably identify AI, further emphasizing the sophistication of these models. The findings suggest that LLMs can now effectively substitute for humans in short conversations, raising both opportunities and concerns. On one hand, this capability could enhance customer service, education, and entertainment. On the other, it poses ethical risks, including the potential for AI to be used in deception, social engineering, or the spread of misinformation.

Looking ahead, the study calls for further research into longer interactions, expert interrogators, and cultural common ground to better understand the limits of AI’s humanlike abilities. It also reignites philosophical debates about whether passing the Turing test truly reflects intelligence or merely advanced imitation. As AI continues to evolve, these advancements underscore the need for careful consideration of their societal impact, ethical implications, and the future of human-AI interaction.

Artificial intelligence for Mental health care: clinical applications, barriers, facilitators, and artificial wisdom.

Lee, E. E., Torous, J. et al. (2021).

Biological Psychiatry Cognitive

Neuroscience and Neuroimaging, 6(9), 856–864.

https://doi.org/10.1016/j.bpsc.2021.02.001

Abstract

Artificial intelligence (AI) is increasingly employed in health care fields such as oncology, radiology, and dermatology. However, the use of AI in mental health care and neurobiological research has been modest. Given the high morbidity and mortality in people with psychiatric disorders, coupled with a worsening shortage of mental health care providers, there is an urgent need for AI to help identify high-risk individuals and provide interventions to prevent and treat mental illnesses. While published research on AI in neuropsychiatry is rather limited, there is a growing number of successful examples of AI's use with electronic health records, brain imaging, sensor-based monitoring systems, and social media platforms to predict, classify, or subgroup mental illnesses as well as problems such as suicidality. This article is the product of a study group held at the American College of Neuropsychopharmacology conference in 2019. It provides an overview of AI approaches in mental health care, seeking to help with clinical diagnosis, prognosis, and treatment, as well as clinical and technological challenges, focusing on multiple illustrative publications. Although AI could help redefine mental illnesses more objectively, identify them at a prodromal stage, personalize treatments, and empower patients in their own care, it must address issues of bias, privacy, transparency, and other ethical concerns. These aspirations reflect human wisdom, which is more strongly associated than intelligence with individual and societal well-being. Thus, the future AI or artificial wisdom could provide technology that enables more compassionate and ethically sound care to diverse groups of people.

Here are some thoughts:

The paper explores AI’s potential in mental health, where its adoption has lagged behind other medical fields due to challenges like data sensitivity, diagnostic complexity, and ethical concerns. AI applications, including machine learning (ML) and natural language processing (NLP), have demonstrated promise in diagnosing mental illnesses, predicting suicide risk, and personalizing treatments using data from electronic health records (EHRs), brain imaging, wearable sensors, and social media. AI has also been explored for optimizing psychiatric treatments by predicting patient responses to antidepressants, cognitive behavioral therapy (CBT), and electroconvulsive therapy. However, there are major obstacles, including the lack of FDA-approved AI applications in psychiatry, concerns about biased training data, and challenges in integrating AI into clinical practice. Programs like REACH VET, which identifies veterans at high risk for suicide, show AI’s potential, but widespread adoption requires overcoming clinician skepticism and ensuring AI models are transparent and equitable.

The concept of Artificial Wisdom (AW) is introduced as an evolution of AI that goes beyond intelligence to incorporate ethical decision-making, empathy, and fairness. While AI can process vast amounts of data, it lacks human wisdom, which is essential for compassionate and just mental healthcare. The paper argues that future AI should not only improve efficiency but also align with human values to ensure patient well-being. This will require collaboration among computer scientists, psychiatrists, and ethicists to develop unbiased, transparent AI models governed by strong regulatory frameworks. Ultimately, AI should complement clinicians by automating administrative tasks and enhancing diagnostic accuracy, allowing professionals to focus on patient relationships. If ethical and technical challenges are addressed, AI and AW have the potential to transform mental healthcare, making it more personalized, efficient, and equitable.

Can AI replace psychotherapists? Exploring the future of mental health care.

Zhang, Z., & Wang, J. (2024).

Frontiers in psychiatry, 15, 1444382.

https://doi.org/10.3389/fpsyt.2024.1444382

In the current technological era, Artificial Intelligence (AI) has transformed operations across numerous sectors, enhancing everything from manufacturing automation to intelligent decision support systems in financial services. In the health sector, particularly, AI has not only refined the accuracy of disease diagnoses but has also ushered in groundbreaking advancements in personalized medicine. The mental health field, amid a global crisis characterized by increasing demand and insufficient resources, is witnessing a significant paradigm shift facilitated by AI, presenting novel approaches that promise to reshape traditional mental health care models (see Figure 1 ).

Mental health, once a stigmatized aspect of health care, is now recognized as a critical component of overall well-being, with disorders such as depression becoming leading causes of global disability (WHO). Traditional mental health care, reliant on in-person consultations, is increasingly perceived as inadequate against the growing prevalence of mental health issues. AI’s role in mental health care is multifaceted, encompassing predictive analytics, therapeutic interventions, clinician support tools, and patient monitoring systems. For instance, AI algorithms are increasingly used to predict treatment outcomes by analyzing patient data. Meanwhile, AI-powered interventions, such as virtual reality exposure therapy and chatbot-delivered cognitive behavioral therapy, are being explored, though they are at varying stages of validation. Each of these applications is evolving at its own pace, influenced by technological advancements and the need for rigorous clinical validation.

The article is linked above.

Here are some thoughts: 

This article explores the evolving role of artificial intelligence (AI) in mental health care, particularly its potential to support or even replace some functions of human psychotherapists. With global demand for mental health services rising and traditional care systems under strain, AI is emerging as a tool to enhance diagnosis, personalize treatments, and provide therapeutic interventions through technologies like chatbots and virtual reality therapy. While early research shows promise, particularly in managing conditions such as anxiety and depression, existing studies are limited and call for larger, long-term trials to determine effectiveness and safety. The authors emphasize that while AI may supplement mental health care and address gaps in service delivery, it must be integrated responsibly, with careful attention to algorithmic bias, ethical considerations, and the irreplaceable human elements of psychotherapy, such as empathy and nuanced judgment.

Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Health Care Professionals

Choudhury, A., & Chaudhry, Z. (2024).

Journal of medical Internet research, 26, e56764.

https://doi.org/10.2196/56764

Abstract

As the health care industry increasingly embraces large language models (LLMs), understanding the consequence of this integration becomes crucial for maximizing benefits while mitigating potential pitfalls. This paper explores the evolving relationship among clinician trust in LLMs, the transition of data sources from predominantly human-generated to artificial intelligence (AI)–generated content, and the subsequent impact on the performance of LLMs and clinician competence. One of the primary concerns identified in this paper is the LLMs’ self-referential learning loops, where AI-generated content feeds into the learning algorithms, threatening the diversity of the data pool, potentially entrenching biases, and reducing the efficacy of LLMs. While theoretical at this stage, this feedback loop poses a significant challenge as the integration of LLMs in health care deepens, emphasizing the need for proactive dialogue and strategic measures to ensure the safe and effective use of LLM technology. Another key takeaway from our investigation is the role of user expertise and the necessity for a discerning approach to trusting and validating LLM outputs. The paper highlights how expert users, particularly clinicians, can leverage LLMs to enhance productivity by off-loading routine tasks while maintaining a critical oversight to identify and correct potential inaccuracies in AI-generated content. This balance of trust and skepticism is vital for ensuring that LLMs augment rather than undermine the quality of patient care. We also discuss the risks associated with the deskilling of health care professionals. Frequent reliance on LLMs for critical tasks could result in a decline in health care providers’ diagnostic and thinking skills, particularly affecting the training and development of future professionals. The legal and ethical considerations surrounding the deployment of LLMs in health care are also examined. We discuss the medicolegal challenges, including liability in cases of erroneous diagnoses or treatment advice generated by LLMs. The paper references recent legislative efforts, such as The Algorithmic Accountability Act of 2023, as crucial steps toward establishing a framework for the ethical and responsible use of AI-based technologies in health care. In conclusion, this paper advocates for a strategic approach to integrating LLMs into health care. By emphasizing the importance of maintaining clinician expertise, fostering critical engagement with LLM outputs, and navigating the legal and ethical landscape, we can ensure that LLMs serve as valuable tools in enhancing patient care and supporting health care professionals. This approach addresses the immediate challenges posed by integrating LLMs and sets a foundation for their maintainable and responsible use in the future.

The abstract provides a sufficient summary.

Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation

Stade, E. C.,  et al. (2024).

Npj Mental Health Research, 3(1).

https://doi.org/10.1038/s44184-024-00056-z

Abstract

Large language models (LLMs) such as Open AI’s GPT-4 (which power ChatGPT) and Google’s Gemini, built on artificial intelligence, hold immense potential to support, augment, or even eventually automate psychotherapy. Enthusiasm about such applications is mounting in the field as well as industry. These developments promise to address insufficient mental healthcare system capacity and scale individual access to personalized treatments. However, clinical psychology is an uncommonly high stakes application domain for AI systems, as responsible and evidence-based therapy requires nuanced expertise. This paper provides a roadmap for the ambitious yet responsible application of clinical LLMs in psychotherapy. First, a technical overview of clinical LLMs is presented. Second, the stages of integration of LLMs into psychotherapy are discussed while highlighting parallels to the development of autonomous vehicle technology. Third, potential applications of LLMs in clinical care, training, and research are discussed, highlighting areas of risk given the complex nature of psychotherapy. Fourth, recommendations for the responsible development and evaluation of clinical LLMs are provided, which include centering clinical science, involving robust interdisciplinary collaboration, and attending to issues like assessment, risk detection, transparency, and bias. Lastly, a vision is outlined for how LLMs might enable a new generation of studies of evidence-based interventions at scale, and how these studies may challenge assumptions about psychotherapy.

The article is linked above.

Here are some thoughts.

This article examines the potential of large language models (LLMs), such as GPT-4 and Google’s Gemini, to support and transform behavioral healthcare, particularly psychotherapy. LLMs could enhance access to care by automating administrative tasks like documentation and session summaries, assisting with treatment planning, and supporting clinician training. The authors propose a phased integration of LLMs, starting with low-risk assistive roles, moving toward collaborative functions with human oversight, and potentially, though more controversially, fully autonomous psychotherapy.

While LLMs offer promising opportunities to improve efficiency and scale mental health services, the authors emphasize the need for cautious, evidence-based development due to significant ethical, safety, and accountability concerns. They call for ongoing collaboration between clinicians, researchers, and technologists to ensure LLM use in mental healthcare prioritizes patient safety, transparency, and effectiveness through rigorous testing and gradual implementation.

Why Most Resist AI Companions

De Freitas, J., et al. (2025).

(Working Paper No. 25–030).

Here is a link.

Abstract

Chatbots are now able to form emotional relationships with people and alleviate loneliness—a growing public health concern. Behavioral research provides little insight into whether everyday people are likely to use these applications and why. We address this question by focusing on the context of “AI companion” applications, designed to provide people with synthetic interaction partners. Study 1 shows that people believe AI companions are more capable than human companions in advertised respects relevant to relationships (being more available and nonjudgmental). Even so, they view them as incapable of realizing the underlying values of relationships, like mutual caring, judging them as not ‘true’ relationships. Study 2 provides further insight into this belief: people believe relationships with AI companions are one-sided

(rather than mutual), because they see AI as incapable of understanding and feeling emotion. Study 3 finds that actually interacting with an AI companion increases acceptance by changing beliefs about the AI’s advertised capabilities, but not about its ability to achieve the true values of relationships, demonstrating the resilience of this belief against intervention. In short, despite the potential loneliness-reducing benefits of AI companions, we uncover fundamental psychological barriers to adoption, suggesting these benefits will not be easily realized.

Here are some thoughts:

The research explores why people remain reluctant to adopt AI companions, despite the growing public health crisis of loneliness and the promise that AI might offer support. Through a series of studies, the authors identify deep-seated psychological barriers to embracing AI as a substitute or supplement for human connection. Specifically, people tend to view AI companions as fundamentally incapable of embodying the core features of meaningful relationships—such as mutual care, genuine emotional understanding, and shared experiences. While participants often acknowledged some of the practical benefits of AI companionship, such as constant availability and non-judgmental interaction, they consistently doubted that AI could offer authentic or reciprocal relationships. Even when people interacted directly with AI systems, their impressions of the AI’s functional abilities improved, but their skepticism around the emotional and relational authenticity of AI companions remained firmly in place. These findings suggest that the resistance is not merely technological or unfamiliarity-based, but rooted in beliefs about what makes relationships "real."

For psychologists, this research is particularly important because it sheds light on how people conceptualize emotional connection, authenticity, and support—core concerns in both clinical and social psychology. As mental health professionals increasingly confront issues of social isolation, understanding the limitations of AI in replicating genuine human connection is critical. Psychologists might be tempted to view AI companions as possible interventions for loneliness, especially for individuals who are socially isolated or homebound. However, this paper underscores that unless these deep psychological barriers are acknowledged and addressed, such tools may be met with resistance or prove insufficient in fulfilling emotional needs. Furthermore, the study contributes to a broader understanding of human-technology relationships, offering insights into how people emotionally and cognitively differentiate between human and artificial agents. This knowledge is crucial for designing future interventions, therapeutic tools, and technologies that are sensitive to the human need for authenticity, reciprocity, and emotional depth in relationships.