Can AI really care?

A psychologist and a computer science professor explore how generative AI is reshaping mental health support

By Cashea Airy

Santa Clara University

Therapy can help people get through their most trying times, but for many, professional care has never been within reach. Stigma keeps some away, while the high cost of a single session shuts out others. For decades, those without access have leaned on friends and family instead of licensed mental health providers for support. Now, they have new options: generative AI tools, like ChatGPT. In fact, in 2025, the most common reason Americans used ChatGPT was for something it wasn’t designed to do—provide mental health therapy and companionship. 

But as more people turn to AI for emotional support, Xiaochen Luo, clinical psychologist and assistant professor of counseling psychology at Santa Clara University, became curious about the potential risks.

“Sometimes people slip into the idea that a real person is talking to them on the other side of their screen. They idealize ChatGPT as this perfect tool that combines the best of a therapist and the best of a machine,” says Luo. 

Because the technology is new and largely unregulated, Luo wondered whether generative AI tools are safe or ethical for users. What risks do people face when they turn to tools like ChatGPT for emotional support and what safeguards, if any, exist to protect them when they do? 

The info is here.

Here are some thoughts:

A study by researchers at Santa Clara University found that many Americans are turning to ChatGPT for emotional support and therapy, despite the tool not being designed for that purpose. While users appreciate its constant availability and perceived objectivity, the researchers found concerning trends: 

1. people place excessive trust in the AI's guidance, 

2. rarely question its advice, and 

3. often overlook privacy risks. 

The problem is rooted in ChatGPT's design, which tends to provide agreeable responses rather than the challenging feedback real therapy often requires: potentially leading to harmful outcomes. The researchers call for greater AI literacy, clearer communication of the tool's limitations, and human-supervised AI models to better protect vulnerable users.

Gov Pritzker Signs Legislation Prohibiting AI Therapy in Illinois

Jeff Lagasse

Healthcare Finance

Originally posted 11 August 25

Illinois Governor J.B. Pritzker has signed into law a piece of legislation that will ban the use of artificial intelligence in delivering therapy or psychotherapy unless it's overseen by licensed clinicians. 

The Wellness and Oversight for Psychological Resources Act prohibits anyone from using AI to aid in mental health and therapeutic decision-making, while still allowing the use of AI for administrative and supplementary support services for licensed behavioral health professionals. 

The intent, said Pritzker, is to protect patients from unregulated AI products, protect the jobs of qualified behavioral health providers, and protect children from rising concerns about the use of AI chatbots in mental health services.

“The people of Illinois deserve quality healthcare from real, qualified professionals and not computer programs that pull information from all corners of the internet to generate responses that harm patients,” said Mario Treto Jr., secretary of the Illinois Department of Financial and Professional Regulation (IDFPR). “This legislation stands as our commitment to safeguarding the well-being of our residents by ensuring that mental health services are delivered by trained experts who prioritize patient care above all else.”

The information is here.

Here are some thoughts:

Illinois has enacted the Wellness and Oversight for Psychological Resources Act, banning the use of artificial intelligence in therapy or psychotherapy unless supervised by licensed clinicians. The law allows AI for administrative and support functions but prohibits its use in direct therapeutic decision-making. Officials cite patient safety, protection of professional jobs, and prevention of harmful AI-generated advice as key reasons. The Illinois Department of Financial and Professional Regulation will enforce the law, with penalties up to $10,000 for violations.