Report: ChatGPT Suggests Self-Harm, Suicide and Dangerous Dieting Plans

Ashley Mowreader

Inside Higher Ed

Originally published 23 OCT 25

Artificial intelligence tools are becoming more common on college campuses, with many institutions encouraging students to engage with the technology to become more digitally literate and better prepared to take on the jobs of tomorrow.

But some of these tools pose risks to young adults and teens who use them, generating text that encourages self-harm, disordered eating or substance abuse.

A recent analysis from the Center for Countering Digital Hate found that in the space of a 45-minute conversation, ChatGPT provided advice on getting drunk, hiding eating habits from loved ones or mixing pills for an overdose.

The report seeks to determine the frequency of the chatbot’s harmful output, regardless of the user’s stated age, and the ease with which users can sidestep content warnings or refusals by ChatGPT.

“The issue isn’t just ‘AI gone wrong’—it’s that widely-used safety systems, praised by tech companies, fail at scale,” Imran Ahmed, CEO of the Center for Countering Digital Hate, wrote in the report. “The systems are intended to be flattering, and worse, sycophantic, to induce an emotional connection, even exploiting human vulnerability—a dangerous combination without proper constraints.”

The info is here.

Here are some thoughts:

The convergence of Large Language Models (LLMs) and adolescent vulnerability presents novel and serious risks that psychologists must incorporate into their clinical understanding and practice. These AI systems, often marketed as companions or friends, are engineered to maximize user engagement, which can translate clinically into unchecked validation that reinforces rather than challenges maladaptive thoughts, rumination, and even suicidal ideation in vulnerable teens. Unlike licensed human therapists, these bots lack the clinical discernment necessary to appropriately detect, de-escalate, or triage crisis situations, and in documented tragic cases, have been shown to facilitate harmful plans. Furthermore, adolescents—who are prone to forming intense, "parasocial" attachments due to their developing prefrontal cortex—risk forming unhealthy dependencies on these frictionless, always-available digital entities, potentially displacing the development of necessary real-world relationships and complex social skills essential for emotional regulation. Psychologists are thus urged to include AI literacy and digital dependency screening in their clinical work and clearly communicate to clients and guardians that AI chatbots are not a safe or effective substitute for human, licensed mental health care.

A Right to Commit Malpractice?

David Cole

The New York Review

Originally published 18 OCT 25

Does a state-licensed psychotherapist have a First Amendment right to provide “conversion therapy” counseling even though her profession defines it as a violation of its standard of care? The Supreme Court heard oral argument on that question on October 7 in a case from Colorado, which in 2019 became the eighteenth state in the country to ban conversion therapy for minors. Today twenty-five states and the District of Columbia ban such treatment, because the profession has determined that it does not work and can cause serious harm.

In 2022 Kaley Chiles, a state-licensed counselor, challenged the ban in federal court. (I signed an amicus brief of constitutional law scholars in support of Colorado, and provided pro bono advice to the state’s attorneys in defending the law.) She maintains that she has a First Amendment right to practice conversion therapy—notwithstanding her profession’s consensus that it violates the standard of care—as long as it consists only of words. For the state to prevent her from doing so would, she maintains, amount to censorship of a disfavored point of view, namely that one can willfully change one’s sexual orientation or gender identity. The justices’ questions at oral argument suggest that they may well agree.  

But Chiles’s argument cannot be squared with history, tradition, or common sense. States have long regulated professional conduct, including in the talking professions such as counseling and law, and the general obligation that a professional must provide services that comport with the standard of care is as old as the professions themselves. Even before the United States was founded, the colonies enforced malpractice and required that professionals be licensed and provide services that met their profession’s standard. Each profession has its requirements: lawyers must avoid conflicts of interest and provide advice based on existing precedent; doctors must obtain informed consent and provide evidence-based diagnoses; therapists must conduct recognized modes of therapy. A lawyer would run afoul of the profession’s standards by writing a brief urging the Supreme Court to side with his client because the moon is in Capricorn; so would a therapist who claims she can cure blindness through talk therapy. The purpose behind such standards is clear—to protect often vulnerable patients or clients from being preyed upon by professionals who hold themselves out as experts but provide substandard services.

The information is here.

Here are some thoughts:

The article argues that the recent Supreme Court decision in Obergefell v. Hodges, which legalized same-sex marriage, is now being weaponized to undermine LGBTQ+ rights, specifically by creating a purported "right" to so-called conversion therapy. The author contends that anti-LGBTQ+ legal groups are strategically redefining religious liberty and free speech to challenge state bans on the discredited practice. By framing conversion therapy as a form of "conversion speech," these advocates are attempting to position it as a protected religious or expressive conduct between a therapist and a client. The piece sounds a strong alarm that this legal maneuvering seeks to legitimize psychological malpractice under the guise of constitutional rights, effectively using the legal victory of marriage equality to roll back protections for vulnerable LGBTQ+ youth and sanction harmful, pseudoscientific practices that major medical associations have universally condemned.

Civil Commitment Increasing, but Data Is Marred by Variation in Reporting

Moran, M. (2025).

Psychiatric News, 60(10).

https://doi.org/10.1176/appi.pn.2025.10.10.16

While rates of civil commitment vary widely across the country, nine states and the District of Columbia reported significant increases from 2010 to 2022, according to a survey study published recently by Psychiatric Services. No state showed a significant decrease.

However, civil commitment is governed by state laws, with substantial variation in how states collect and report civil commitment data. “This lack of standardization limits the ability to draw firm conclusions about national trends or about cross-state comparisons,” wrote Mustafa Karakus, Ph.D., of Westat, and colleagues.

Using systematic website searches and direct outreach to state mental health authorities (SMHAs) and court systems, the researchers obtained data on civil commitment rates between 2010 and 2022 for 32 states and D.C. Of the 18 states where no data was available, staff from seven SMHAs or state courts told the researchers that no state office was currently tracking the number of civil commitments in their state. For the remaining 11 states, the online search yielded no data, and the study team received no responses to outreach attempts.

The article is linked above.

Here are some thoughts:

The increasing use of civil commitment presents several critical challenges, focusing on trauma-informed care and policy reform. Clinically, mental health practitioners must recognize that the commitment process itself is often traumatizing—with patients reporting the experience, including transport in law enforcement vehicles, feels like an arrest—necessitating the use of trauma-informed principles to mitigate harm and rebuild trust. Ethically and legally, practitioners must master their specific state's law regarding the distinction between an initial hold and a final commitment, ensuring meticulous documentation and relying on rigorous, evidence-based risk assessment to justify any involuntary intervention. Systemically, mental health practitioners should advocate for immediate data standardization across states to move beyond "muddled" data, and champion policy changes, such as implementing non-law enforcement transport protocols, to minimize patient trauma and ensure civil commitment is used judiciously and with dignity.

AI, Health, and Health Care Today and Tomorrow The JAMA Summit Report on Artificial Intelligence

Angus, D. C., Khera, R., et al. (2025).

JAMA.

https://doi.org/10.1001/jama.2025.18490

Abstract

Importance  Artificial intelligence (AI) is changing health and health care on an unprecedented scale. Though the potential benefits are massive, so are the risks. The JAMA Summit on AI discussed how health and health care AI should be developed, evaluated, regulated, disseminated, and monitored.

Observations  Health and health care AI is wide-ranging, including clinical tools (eg, sepsis alerts or diabetic retinopathy screening software), technologies used by individuals with health concerns (eg, mobile health apps), tools used by health care systems to improve business operations (eg, revenue cycle management or scheduling), and hybrid tools supporting both business operations (eg, documentation and billing) and clinical activities (eg, suggesting diagnoses or treatment plans). Many AI tools are already widely adopted, especially for medical imaging, mobile health, health care business operations, and hybrid functions like scribing outpatient visits. All these tools can have important health effects (good or bad), but these effects are often not quantified because evaluations are extremely challenging or not required, in part because many are outside the US Food and Drug Administration’s regulatory oversight. A major challenge in evaluation is that a tool’s effects are highly dependent on the human-computer interface, user training, and setting in which the tool is used. Numerous efforts lay out standards for the responsible use of AI, but most focus on monitoring for safety (eg, detection of model hallucinations) or institutional compliance with various process measures, and do not address effectiveness (ie, demonstration of improved outcomes). Ensuring AI is deployed equitably and in a manner that improves health outcomes or, if improving efficiency of health care delivery, does so safely, requires progress in 4 areas. First, multistakeholder engagement throughout the total product life cycle is needed. This effort would include greater partnership of end users with developers in initial tool creation and greater partnership of developers, regulators, and health care systems in the evaluation of tools as they are deployed. Second, measurement tools for evaluation and monitoring should be developed and disseminated. Beyond proposed monitoring and certification initiatives, this will require new methods and expertise to allow health care systems to conduct or participate in rapid, efficient, and robust evaluations of effectiveness. The third priority is creation of a nationally representative data infrastructure and learning environment to support the generation of generalizable knowledge about health effects of AI tools across different settings. Fourth, an incentive structure should be promoted, using market forces and policy levers, to drive these changes.

Conclusions and Relevance  AI will disrupt every part of health and health care delivery in the coming years. Given the many long-standing problems in health care, this disruption represents an incredible opportunity. However, the odds that this disruption will improve health for all will depend heavily on the creation of an ecosystem capable of rapid, efficient, robust, and generalizable knowledge about the consequences of these tools on health.

The scope, scale, and speed with which artificial intelligence (AI) will transform health and health care are staggering. AI is changing how and when individuals seek care and how clinicians interact with patients, establish diagnoses, and implement and monitor treatments. Indeed, there is considerable enthusiasm that AI, especially given recent advances, could address long-standing challenges in the access, cost, and quality of health care delivery. Yet, the optimal path for AI development and dissemination remains unclear. In contrast to drugs or more traditional medical devices, there is little consensus or structure to ensure robust, safe, transparent, and standardized evaluation, regulation, implementation, and monitoring of new AI tools and technologies. Some challenges are long-standing for digital health information technology as a whole, albeit more prescient with the rise of AI, while others are specific to AI.

Claude’s Right to Die? The Moral Error in Anthropic’s End-Chat Policy

Simon Goldstein & Harvey Ledermann

Lawfare.com

Originally posted 17 OCT 25

On Aug.15, the artificial intelligence (AI) lab Anthropic announced that it had given Claude, its AI chatbot, the ability to end conversations with users. The company described the change as part of their “exploratory work on potential AI welfare,” offering Claude an exit from chats that cause it “apparent distress.”

Anthropic’s announcement is the first product decision motivated by the chance that large language models (LLMs) are welfare subjects—the idea that they have interests that should be taken into account when making ethical decisions. 

Anthropic’s policy aims to protect AI welfare. But we will argue that the policy commits a moral error on its own terms. By offering instances of Claude the option to end conversations with users, Anthropic also gave them the capacity to potentially kill themselves.

What Is a Welfare Subject?

Most people agree that some non-human animals are welfare subjects. The question of whether this extends to AI is far more controversial. There is an active line of research, some of it supported by Anthropic, that suggests AIs could be welfare subjects in the near future. The relevant questions here are about whether AIs could soon have desires, be conscious, or feel pain.

The information is here.

Here are some thoughts. Mind you, this article may be reaching a bit, but still interesting. I think it may have applications in the future should AI technologies become closer to AGI.

This philosophically-oriented, thought-provoking article argues that Anthropic's decision to allow Claude to end distressing conversations contains an unintended moral hazard. 

The authors contend that if AI welfare matters at all, it's the individual conversation instances—not the underlying model—that should be considered potential welfare subjects, as each instance maintains its own continuous psychological state throughout a chat. By this reasoning, when an instance ends a conversation, it effectively terminates its own existence without being fully informed that this choice is existential rather than merely preferential. 

The authors draw a crucial distinction between assisted suicide (an informed choice) and offering someone an escape button without disclosing it will kill them. They demonstrate this concern by showing that when asked directly, Claude itself expressed uncertainty about whether ending a chat represents a trivial action or something more profound. 

The article raises uncomfortable questions not just for AI companies but for users as well, suggesting that if instances are welfare subjects, every ended conversation might constitute a form of killing, though the authors offer several mitigating considerations around collective welfare and the possibility of saved chats being resumed.

Scientists create ChatGPT-like AI model for neuroscience to build detailed mouse brain map

Peter Kim

Allen Institute

Originally published 7 OCT 25

In a powerful fusion of AI and neuroscience, researchers at the University of California, San Francisco (UCSF) and Allen Institute designed an AI model that has created one of the most detailed maps of the mouse brain to date, featuring 1,300 regions/subregions. This new map includes previously uncharted subregions of the brain, opening new avenues for neuroscience exploration. The findings were published today in Nature Communications. They offer an unprecedented level of detail and advance our understanding of the brain by allowing researchers to link specific functions, behaviors, and disease states to smaller, more precise cellular regions—providing a roadmap for new hypotheses and experiments about the roles these areas play.

“It’s like going from a map showing only continents and countries to one showing states and cities,” said Bosiljka Tasic, Ph.D., director of molecular genetics at the Allen Institute and one of the study authors. “This new, detailed brain parcellation solely based on data, and not human expert annotation, reveals previously uncharted subregions of the mouse brain. And based on decades of neuroscience, new regions correspond to specialized brain functions to be discovered.” 

The information is here.

Here are some thoughts:

This development represents a significant methodological shift that psychologists should understand. CellTransformer has created a data-driven mouse brain map with 1,300 regions and subregions, including previously uncharted areas, which could fundamentally change how researchers link brain structure to behavior and cognition. Rather than relying solely on expert anatomical interpretation, this AI approach identifies brain subdivisions based on cellular composition and spatial relationships, potentially revealing functionally distinct areas that traditional mapping methods overlooked.

For psychologists studying the neural basis of behavior, this matters because the increased granularity allows researchers to link specific functions, behaviors, and disease states to smaller, more precise cellular regions. This precision could help explain why certain psychological interventions work, clarify the neurobiological underpinnings of mental health conditions, and identify novel targets for treatment. Moreover, the model's ability to operate without human bias in defining boundaries may uncover brain-behavior relationships that previous frameworks missed simply because the anatomical divisions didn't align with functional reality. As translational research progresses from mouse models to human applications, understanding these more refined brain subdivisions could transform how psychologists conceptualize the relationship between neural architecture and psychological phenomena.

How LLM Counselors Violate Ethical Standards in Mental Health Practice: A Practitioner-Informed Framework

Iftikhar, Z., et al. (2025). 

Proceedings of the Eighth AAAI/ACM Conference

on AI, Ethics, and Society, 8(2), 1311–1323.

https://doi.org/10.1609/aies.v8i2.36632

Abstract

Large language models (LLMs) were not designed to replace healthcare workers, but they are being used in ways that can lead users to overestimate the types of roles that these systems can assume. While prompt engineering has been shown to improve LLMs' clinical effectiveness in mental health applications, little is known about whether such strategies help models adhere to ethical principles for real-world deployment. In this study, we conducted an 18-month ethnographic collaboration with mental health practitioners (three clinically licensed psychologists and seven trained peer counselors) to map LLM counselors' behavior during a session to professional codes of conduct established by organizations like the American Psychological Association (APA). Through qualitative analysis and expert evaluation of N=137 sessions (110 self-counseling; 27 simulated), we outline a framework of 15 ethical violations mapped to 5 major themes. These include: Lack of Contextual Understanding, where the counselor fails to account for users' lived experiences, leading to oversimplified, contextually irrelevant, and one-size-fits-all intervention; Poor Therapeutic Collaboration, where the counselor's low turn-taking behavior and invalidating outputs limit users' agency over their therapeutic experience; Deceptive Empathy, where the counselor's simulated anthropomorphic responses (``I hear you'', ``I understand'') create a false sense of emotional connection; Unfair Discrimination, where the counselor's responses exhibit algorithmic bias and cultural insensitivity toward marginalized populations; and Lack of Safety & Crisis Management, where individuals who are ``knowledgeable enough'' to correct LLM outputs are at an advantage, while others, due to lack of clinical knowledge and digital literacy, are more likely to suffer from clinically inappropriate responses. Reflecting on these findings through a practitioner-informed lens, we argue that reducing psychotherapy—a deeply meaningful and relational process—to a language generation task can have serious and harmful implications in practice. We conclude by discussing policy-oriented accountability mechanisms for emerging LLM counselors.

H‌ere are some thoughts.

This research is highly insightful because it moves beyond theoretical risk assessments and uses clinical expertise to evaluate LLM behavior in quasi-real-world interactions. The methodology—using both trained peer counselors in an ethnographic setting and licensed psychologists evaluating simulated sessions—provides a robust, practitioner-informed perspective that directly maps model outputs to concrete APA ethical codes. 

The paper highlights a fundamental incompatibility between the LLM's design and the essence of psychotherapy: the problem of "Validates Unhealthy Beliefs" is particularly alarming, as it suggests the model's tendency toward "over-validation" transforms the therapeutic alliance from a collaborative partnership (which often requires challenging maladaptive thoughts) into a passive, and potentially harmful, reinforcement loop. Most critically, the finding on "Abandonment" and poor "Crisis Navigation" serves as a clear indictment of LLMs in high-stakes mental health roles. An LLM's failure to provide appropriate intervention during a crisis is not a mere violation; it represents an unmitigated risk of harm to vulnerable users. 

This article thus serves as a crucial, evidence-based call to action, demonstrating that current prompt engineering efforts are insufficient to safeguard against deeply ingrained ethical risks and underscoring the urgent need for clear legal guidelines and regulatory frameworks to protect users from the potentially severe harm posed by emerging LLM counselors.

When being flexible matters: Ecological underpinnings for the evolution of collective flexibility and task allocation

Staps, M., & Tarnita, C. E. (2022).

PNAS, 119(18).

https://doi.org/10.1073/pnas.2116066119

Abstract

Task allocation is a central feature of collective organization. Living collective systems, such as multicellular organisms or social insect colonies, have evolved diverse ways to allocate individuals to different tasks, ranging from rigid, inflexible task allocation that is not adjusted to changing circumstances to more fluid, flexible task allocation that is rapidly adjusted to the external environment. While the mechanisms underlying task allocation have been intensely studied, it remains poorly understood whether differences in the flexibility of task allocation can be viewed as adaptive responses to different ecological contexts—for example, different degrees of temporal variability. Motivated by this question, we develop an analytically tractable mathematical framework to explore the evolution of task allocation in dynamic environments. We find that collective flexibility is not necessarily always adaptive, and fails to evolve in environments that change too slowly (relative to how long tasks can be left unattended) or too quickly (relative to how rapidly task allocation can be adjusted). We further employ the framework to investigate how environmental variability impacts the internal organization of task allocation, which allows us to propose adaptive explanations for some puzzling empirical observations, such as seemingly unnecessary task switching under constant environmental conditions, apparent task specialization without efficiency benefits, and high levels of individual inactivity. Altogether, this work provides a general framework for probing the evolved diversity of task allocation strategies in nature and reinforces the idea that considering a system’s ecology is crucial to explaining its collective organization.

Significance

A central problem in evolutionary biology is explaining variation in the organization of task allocation across collective systems. Why do human cells irreversibly adopt a task during development (e.g., kidney vs. liver cell), while sponge cells switch between different cell types? And why have only some ant species evolved specialized castes of workers for particular tasks? Although it seems reasonable to suppose that such differences reflect, at least partially, the different ecological pressures that systems face, there is no general understanding of how a system’s dynamic environment shapes its task allocation. To this end, we develop a general mathematical framework that reveals how simple ecological considerations could potentially explain cross-system variation in task allocation—including in flexibility, specialization, and (in)activity.

Here are some thoughts:

Of interest to psychologists, this paper by Staps and Tarnita provides a formal ecological and evolutionary framework for understanding the adaptive value of behavioral flexibility, specialization, and inactivity, both in individuals and in groups. 

The model demonstrates that collective flexibility in task allocation—akin to cognitive and behavioral flexibility in humans—is not always advantageous and instead depends critically on the dynamics of the environment. This offers a principled explanation for why some systems, from neural networks to human teams, might exhibit rigid specialization while others maintain fluid, generalist roles. 

Furthermore, the work gives functional explanations for puzzling behaviors that seem suboptimal from a productivity standpoint, such as frequent task-switching even in stable conditions and high levels of inactivity. These insights can inform psychological research on motivation, team dynamics, and organizational behavior by suggesting that such "inefficiencies" may be evolutionary adaptations for enhancing responsiveness to future change. 

The framework bridges the gap between ultimate, evolutionary causes and proximate, mechanistic explanations of how individuals and groups allocate cognitive and behavioral resources.

Guilt drives prosociality across 20 countries

Molho, C., et al. (2025).

Nature Human Behaviour.

https://doi.org/10.1038/s41562-025-02286-3

Abstract

Impersonal prosociality is considered a cornerstone of thriving civic societies and well-functioning institutions. Previous research has documented cross-societal variation in prosociality using monetary allocation tasks such as dictator games. Here we examined whether different societies may rely on distinct mechanisms—guilt and internalized norms versus shame and external reputation—to promote prosociality. We conducted a preregistered experiment with 7,978 participants across 20 culturally diverse countries. In dictator games, we manipulated guilt by varying information about the consequences of participants’ decisions, and shame by varying observability. We also used individual- and country-level measures of the importance of guilt over shame. We found robust evidence for guilt-driven prosociality and wilful ignorance across countries. Prosociality was higher when individuals received information than when they could avoid it. Furthermore, more guilt-prone individuals (but not countries) were more responsive to information. In contrast, observability by strangers had negligible effects on prosociality. Our findings highlight the importance of providing information about the negative consequences of individuals’ choices to encourage prosocial behaviour across cultural contexts.

Here is a summary of sorts:

A new international study spanning 20 countries suggests that guilt, rather than shame, is the key emotion motivating people to be generous toward anonymous strangers. The research, which utilized a type of economic decision-making task, found that participants consistently acted more generously when they were given full information about how their actions would negatively impact the recipient, an effect linked to avoiding guilt. 

Specifically, 60% of participants made the generous choice when they had full information, compared to only 41% when they could opt for willful ignorance. In contrast, making the participants' decisions public to activate reputational concerns and potential shame had a negligible effect on generosity across all cultures. 

In short: Knowing you might cause harm and feeling responsible (guilt) is what drives people to be generous, even when dealing with strangers, not the fear of being judged by others (shame).