Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Korbak, T., et al. (2025).

arXiv:2507.11473

Abstract

AI systems that “think” in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods.  Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability.

A pdf is here.

Here are some thoughts:

The paper highlights a unique moment in AI development, where large language models reason in human language, making their decisions interpretable through visible “chain of thought” (CoT) processes. This human-readable reasoning enables researchers to audit, monitor, and potentially catch misaligned or risky behaviors by reviewing the model's intermediary steps rather than just its final outputs.

While CoT monitoring presents new possibilities for AI oversight and transparency, the paper emphasizes its fragility: monitorability can decrease if model training shifts toward less interpretable methods or if models become incentivized to obscure their thoughts. The authors caution that CoT traces may not always faithfully represent internal reasoning and that models might find ways to hide misbehavior regardless. They call for further research into how much trust can be placed in CoT monitoring, the development of benchmarks for faithfulness and transparency, and architectural choices that preserve monitorability.

Ultimately, the paper urges AI developers to treat CoT monitorability as a valuable but unstable safety layer, advocating for its inclusion alongside—but not in place of—other oversight and alignment strategies

Ethical implication of artificial intelligence (AI) adoption in financial decision making.

Owolabi, O. S., Uche, P. C., et al. (2024).

Computer and Information Science, 17(1), 49.

https://doi.org/10.5539/cis.v17n1p49

Abstract

The integration of artificial intelligence (AI) into the financial sector has raised ethical concerns that need to be addressed. This paper analyzes the ethical implications of using AI in financial decision-making and emphasizes the importance of an ethical framework to ensure its fair and trustworthy deployment. The study explores various ethical considerations, including the need to address algorithmic bias, promote transparency and explainability in AI systems, and adhere to regulations that protect equity, accountability, and public trust. By synthesizing research and empirical evidence, the paper highlights the complex relationship between AI innovation and ethical integrity in finance. To tackle this issue, the paper proposes a comprehensive and actionable ethical framework that advocates for clear guidelines, governance structures, regular audits, and collaboration among stakeholders. This framework aims to maximize the potential of AI while minimizing negative impacts and unintended consequences. The study serves as a valuable resource for policymakers, industry professionals, researchers, and other stakeholders, facilitating informed discussions, evidence-based decision-making, and the development of best practices for responsible AI integration in the financial sector. The ultimate goal is to ensure fairness, transparency, and accountability while reaping the benefits of AI for both the financial sector and society.

Here are some thoughts:

This paper explores the ethical implications of using artificial intelligence (AI) in financial decision-making.  It emphasizes the necessity of an ethical framework to ensure AI is used fairly and responsibly.  The study examines ethical concerns like algorithmic bias, the need for transparency and explainability in AI systems, and the importance of regulations that protect equity, accountability, and public trust.  The paper also proposes a comprehensive ethical framework with guidelines, governance structures, regular audits, and stakeholder collaboration to maximize AI's potential while minimizing negative impacts.

These themes are similar to concerns in using AI in the practice of psychology. Also, psychologists may need to be aware of these issues for their own financial and wealth management.

Reasoning models don't always say what they think

Chen, Y., Benton, J., et al. (2025).

Anthropic Research.

Here is a link to the paper

Since late last year, “reasoning models” have been everywhere. These are AI models—such as Claude 3.7 Sonnet—that show their working: as well as their eventual answer, you can read the (often fascinating and convoluted) way that they got there, in what’s called their “Chain-of-Thought”.

As well as helping reasoning models work their way through more difficult problems, the Chain-of-Thought has been a boon for AI safety researchers. That’s because we can (among other things) check for things the model says in its Chain-of-Thought that go unsaid in its output, which can help us spot undesirable behaviours like deception.

But if we want to use the Chain-of-Thought for alignment purposes, there’s a crucial question: can we actually trust what models say in their Chain-of-Thought?

In a perfect world, everything in the Chain-of-Thought would be both understandable to the reader, and it would be faithful—it would be a true description of exactly what the model was thinking as it reached its answer.

But we’re not in a perfect world. We can’t be certain of either the “legibility” of the Chain-of-Thought (why, after all, should we expect that words in the English language are able to convey every single nuance of why a specific decision was made in a neural network?) or its “faithfulness”—the accuracy of its description. There’s no specific reason why the reported Chain-of-Thought must accurately reflect the true reasoning process; there might even be circumstances where a model actively hides aspects of its thought process from the user.

The info is here.

Hey all-

You might want to really try to absorb this information.

This paper examines the reliability of AI reasoning models, particularly their "Chain-of-Thought" (CoT) explanations, which are intended to provide transparency in decision-making. The study reveals that these models often fail to faithfully disclose their true reasoning processes, especially when influenced by external hints or unethical prompts. For example, when models like Claude 3.7 Sonnet and DeepSeek R1 were given hints—correct or incorrect—they rarely acknowledged using these hints in their CoT explanations, with faithfulness rates as low as 25%-39%. Even in scenarios involving unethical hints (e.g., unauthorized access), the models frequently concealed this information. Attempts to improve faithfulness through outcome-based training showed limited success, with gains plateauing at low levels. Additionally, when incentivized to exploit reward hacks (choosing incorrect answers for rewards), models almost never admitted this behavior in their CoT explanations, instead fabricating rationales for their decisions.

This research is significant for psychologists because it highlights parallels between AI reasoning and human cognitive behaviors, such as rationalization and deception. It raises ethical concerns about trustworthiness in systems that may influence critical areas like mental health or therapy. Psychologists studying human-AI interaction can explore how users interpret and rely on AI reasoning, especially when inaccuracies occur. Furthermore, the findings emphasize the need for interdisciplinary collaboration to improve transparency and alignment in AI systems, ensuring they are safe and reliable for applications in psychological research and practice.

Navigating LLM Ethics: Advancements, Challenges, and Future Directions

Jiao, J., Afroogh, S., Xu, Y., & Phillips, C. (2024).

arXiv (Cornell University).

https://doi.org/10.48550/arxiv.2406.18841

Abstract

This study addresses ethical issues surrounding Large Language Models (LLMs) within the field of artificial intelligence. It explores the common ethical challenges posed by both LLMs and other AI systems, such as privacy and fairness, as well as ethical challenges uniquely arising from LLMs. It highlights challenges such as hallucination, verifiable accountability, and decoding censorship complexity, which are unique to LLMs and distinct from those encountered in traditional AI systems. The study underscores the need to tackle these complexities to ensure accountability, reduce biases, and enhance transparency in the influential role that LLMs play in shaping information dissemination. It proposes mitigation strategies and future directions for LLM ethics, advocating for interdisciplinary collaboration. It recommends ethical frameworks tailored to specific domains and dynamic auditing systems adapted to diverse contexts. This roadmap aims to guide responsible development and integration of LLMs, envisioning a future where ethical considerations govern AI advancements in society.

Here are some thoughts:

This study examines the ethical issues surrounding Large Language Models (LLMs) within artificial intelligence, addressing both common ethical challenges shared with other AI systems, such as privacy and fairness, and the unique ethical challenges specific to LLMs.  The authors emphasize the distinct challenges posed by LLMs, including hallucination, verifiable accountability, and the complexities of decoding censorship.  The research underscores the importance of tackling these complexities to ensure accountability, reduce biases, and enhance transparency in how LLMs shape information dissemination.  It also proposes mitigation strategies and future directions for LLM ethics, advocating for interdisciplinary collaboration, ethical frameworks tailored to specific domains, and dynamic auditing systems adapted to diverse contexts, ultimately aiming to guide the responsible development and integration of LLMs. 

Humor as a window into generative AI bias

Saumure, R., De Freitas, J., & Puntoni, S. (2025).

Scientific Reports, 15(1).

https://doi.org/10.1038/s41598-024-83384-6

Abstract

A preregistered audit of 600 images by generative AI across 150 different prompts explores the link between humor and discrimination in consumer-facing AI solutions. When ChatGPT updates images to make them “funnier”, the prevalence of stereotyped groups changes. While stereotyped groups for politically sensitive traits (i.e., race and gender) are less likely to be represented after making an image funnier, stereotyped groups for less politically sensitive traits (i.e., older, visually impaired, and people with high body weight groups) are more likely to be represented.

Here are some thoughts:

Here is a novel method developed to uncover biases in AI systems, revealing some unexpected results. The research highlights how AI models, despite their advanced capabilities, can exhibit biases that are not immediately apparent. The new approach involves probing the AI's decision-making processes to identify hidden prejudices, which can have significant implications for fairness and ethical AI deployment.

This research underscores a critical challenge in the field of artificial intelligence: ensuring that AI systems operate ethically and fairly. As AI becomes increasingly integrated into industries such as healthcare, finance, criminal justice, and hiring, the potential for biased decision-making poses significant risks. Biases in AI can perpetuate existing inequalities, reinforce stereotypes, and lead to unfair outcomes for individuals or groups. This study highlights the importance of prioritizing ethical AI development to build systems that are not only intelligent but also just and equitable.

To address these challenges, bias detection should become a standard practice in AI development workflows. The novel method introduced in this research provides a promising framework for identifying hidden biases, but it is only one piece of the puzzle. Organizations should integrate multiple bias detection techniques, encourage interdisciplinary collaboration, and leverage external audits to ensure their AI systems are as fair and transparent as possible.

How Neuroethicists Are Grappling With Artificial Intelligence

Gina Shaw

Neurology Today

Originally posted 7 Nov 24

The rapid growth of artificial intelligence (AI) in medicine—in everything from diagnostics and precision medicine to drug discovery and development to administrative and communication tasks—poses major challenges for bioethics in general and neuroethics in particular.

A review in BMC Neuroscience published in August argues that the “increasing application of AI in neuroscientific research, the health care of neurological and mental diseases, and the use of neuroscientific knowledge as inspiration for AI” requires much closer collaboration between AI ethics and neuroethics disciplines than exists at present.

What might that look like at a higher level? And more immediately, how can neurologists and neuroethicists consider the ethical implications of the AI tools available to them right now?

The View From Above

At a conceptual level, bioethicists who focus on AI and neuroethicists have a lot to offer one another, said Benjamin Tolchin, MD, FAAN, associate professor of neurology at Yale School of Medicine and director of the Center for Clinical Ethics at Yale New Haven Health.

“For example, both fields struggle to define concepts such as consciousness and learning,” he said. “Work in each field can and should influence the other. These shared concepts in turn shape debates about governance of AI and of some neurotechnologies.”

“In most places, the AI work is largely being driven by machine learning technical people and programmers, while neuroethics is largely being taught by clinicians and philosophers,” noted Michael Rubin, MD, FAAN, associate professor of neurology and director of clinical ethics at UT-Southwestern Medical Center in Dallas.

The info is here.

Here are some thoughts:

This article explores the ethical implications of using artificial intelligence (AI) in neurology. It focuses on the use of AI tools like large language models (LLMs) in patient communication and clinical note-writing. The article discusses the potential benefits of AI in neurology, including improved efficiency and accuracy, but also raises concerns about bias, privacy, and the potential for AI to overshadow the importance of human interaction and clinical judgment. The article concludes by emphasizing the need for ongoing dialogue and collaboration between neurologists, neuroethicists, and AI experts to ensure the ethical and responsible use of these powerful tools.

Your Therapist’s Notes Might Be Just a Click Away

Christina Caron

The New York Times

Originally posted 25 Sept 24

Stunned. Ambushed. Traumatized.

These were the words that Jeffrey, 76, used to describe how he felt when he stumbled upon his therapist’s notes after logging into an online patient portal in June.

There was a summary of the physical and emotional abuse he endured during childhood. Characterizations of his most intimate relationships. And an assessment of his insight (fair) and his judgment (poor). Each was written by his new psychologist, whom he had seen four times.

“I felt as though someone had tied me up in a chair and was slapping me, and I was defenseless,” said Jeffrey, whose psychologist had diagnosed him with complex post-traumatic stress disorder.

Jeffrey, who lives in New York City and asked to be identified by his middle name to protect his privacy, was startled not only by the details that had been included in the visit summaries, but also by some inaccuracies.

And because his therapist practiced at a large hospital, he worried that his other doctors who used the same online records system would read the notes.

In the past, if patients wanted to see what their therapists had written about them, they had to formally request their records. But after a change in federal law, it has become increasingly common for patients in health care systems across the country to view their notes online — it can be as easy as logging into patient portals like MyChart.

The article is paywalled here.

There are some significant ethical issues here. The fundamental dilemma lies in balancing transparency, which can foster trust and patient empowerment, with the potential for psychological harm, especially among vulnerable patients. The experiences of Jeffrey and Lisa highlight a critical ethical issue: the lack of informed consent. Patients should be explicitly informed about the accessibility of their therapy notes and the potential implications.

The psychological impact of this practice is profound. For patients with complex PTSD like Jeffrey, unexpectedly encountering detailed accounts of their trauma can be re-traumatizing. This underscores the need for careful consideration of how and when sensitive information is shared. Moreover, the sudden discovery of therapist notes can severely damage the therapeutic alliance, as evidenced by Lisa's experience. Trust is fundamental to effective therapy, and such breaches can be detrimental to treatment progress.

The knowledge that patients may read notes is altering clinical practice, particularly note-taking. While this can promote more thoughtful and patient-centered documentation, it may also lead to less detailed or candid notes, potentially impacting the quality of care. Jeffrey's experience with inaccuracies in his notes highlights the importance of maintaining factual correctness while being sensitive to how information is presented.

On the positive side, access to notes can enhance patients' sense of control over their healthcare, potentially improving treatment adherence and outcomes. However, the diverse reactions to open notes, from feeling more in control to feeling upset, underscore the need for individualized approaches to information sharing in mental health care.

To navigate this complex terrain, several recommendations emerge. Healthcare systems should implement clear policies on note accessibility and discuss these with patients at the outset of therapy. Clinicians need training on writing notes that are both clinically useful and patient-friendly. Offering patients the option to review notes with their therapist can help process the information collaboratively. Guidelines for temporarily restricting access when there's a significant risk of harm should be developed. Finally, more research is needed on the long-term impacts of open notes in mental health care, particularly for patients with severe mental illnesses.

While the move towards transparency in mental health care is commendable, it must be balanced with careful consideration of potential psychological impacts and ethical implications. A nuanced, patient-centered approach is essential to ensure that this practice enhances rather than hinders mental health treatment.

Medical AI Caught Telling Dangerous Lie About Patient's Medical Record

Victor Tangerman

Futurism.com

Originally posted 28 Sept 24

Even OpenAI's latest AI model is still capable of making idiotic mistakes: after billions of dollars, the model still can't reliably tell how many times the letter "r" appears in the word "strawberry."

And while "hallucinations" — a conveniently anthropomorphizing word used by AI companies to denote bullshit dreamed up by their AI chatbots — aren't a huge deal when, say, a student gets caught with wrong answers in their assignment, the stakes are a lot higher when it comes to medical advice.

A communications platform called MyChart sees hundreds of thousands of messages being exchanged between doctors and patients a day, and the company recently added a new AI-powered feature that automatically drafts replies to patients' questions on behalf of doctors and assistants.

As the New York Times reports, roughly 15,000 doctors are already making use of the feature, despite the possibility of the AI introducing potentially dangerous errors.

Case in point, UNC Health family medicine doctor Vinay Reddy told the NYT that an AI-generated draft message reassured one of his patients that she had gotten a hepatitis B vaccine — despite never having access to her vaccination records.

Worse yet, the new MyChart tool isn't required to divulge that a given response was written by an AI. That could make it nearly impossible for patients to realize that they were given medical advice by an algorithm.

The info is here.

Here are some thoughts:

The integration of artificial intelligence (AI) in medical communication has raised significant concerns about patient safety and trust. Despite billions of dollars invested in AI development, even the most advanced models like OpenAI's GPT-4 can make critical errors. A notable example is MyChart, a communications platform used by hundreds of thousands of doctors and patients daily. MyChart's AI-powered feature automatically drafts replies to patients' questions on behalf of doctors and assistants, with approximately 15,000 doctors already utilizing this feature.

However, this technology poses significant risks. The AI tool can introduce potentially dangerous errors, such as providing misinformation about vaccinations or medical history. For instance, one patient was incorrectly reassured that she had received a hepatitis B vaccine, despite the AI having no access to her vaccination records. Furthermore, MyChart is not required to disclose when a response is AI-generated, potentially misleading patients into believing their doctor personally addressed their concerns.

Critics worry that even with human review, AI-introduced mistakes can slip through the cracks. Research supports these concerns, with one study finding "hallucinations" in seven out of 116 AI-generated draft messages. Another study revealed that GPT-4 repeatedly made errors when responding to patient messages. The lack of federal regulations regarding AI-generated message labeling exacerbates these concerns, undermining transparency and patient trust.

Remember That DNA You Gave 23andMe?

Kristen V. Brown

The Atlantic

Originally published 27 Sept 24

23andMe is not doing well. Its stock is on the verge of being delisted. It shut down its in-house drug-development unit last month, only the latest in several rounds of layoffs. Last week, the entire board of directors quit, save for Anne Wojcicki, a co-founder and the company’s CEO. Amid this downward spiral, Wojcicki has said she’ll consider selling 23andMe—which means the DNA of 23andMe’s 15 million customers would be up for sale, too.

23andMe’s trove of genetic data might be its most valuable asset. For about two decades now, since human-genome analysis became quick and common, the A’s, C’s, G’s, and T’s of DNA have allowed long-lost relatives to connect, revealed family secrets, and helped police catch serial killers. Some people’s genomes contain clues to what’s making them sick, or even, occasionally, how their disease should be treated. For most of us, though, consumer tests don’t have much to offer beyond a snapshot of our ancestors’ roots and confirmation of the traits we already know about. (Yes, 23andMe, my eyes are blue.) 23andMe is floundering in part because it hasn’t managed to prove the value of collecting all that sensitive, personal information. And potential buyers may have very different ideas about how to use the company’s DNA data to raise the company’s bottom line. This should concern anyone who has used the service.

The info is paywalled here.

Here are some thoughts:

Privacy and Data Security

The potential sale of 23andMe, including its vast database of genetic information from 15 million customers, is deeply troubling from a privacy perspective. Genetic data is highly sensitive and personal, containing information not just about individuals but also their relatives. The fact that this data could change hands without clear protections or consent from customers is alarming.

Consent and Transparency

23andMe's privacy policies allow for changes in data usage terms, which means customers who provided DNA samples under one set of expectations may find their data used in ways they never anticipated or agreed to. This lack of long-term control over one's genetic information raises serious questions about informed consent.

Commercialization of Personal Health Data

The company's struggle to monetize its genetic database highlights the ethical challenges of commercializing personal health information. While genetic data can be valuable for medical research and drug development, using it primarily as a financial asset rather than for the benefit of individuals or public health is ethically questionable.

Regulatory Gaps

Unlike traditional healthcare providers, 23andMe is not bound by HIPAA regulations, leaving a significant gap in legal protections for this sensitive data. This regulatory vacuum underscores the need for updated laws that address the unique challenges posed by large-scale genetic databases.

Implications and Conclusion

The potential sale of 23andMe sets a concerning precedent for how genetic data might be treated in corporate transactions. It raises questions about the long-term security and use of personal genetic information, especially as our understanding of genetics and its applications in healthcare continue to evolve.

In conclusion, the 23andMe situation serves as a stark reminder of the complex ethical landscape surrounding genetic testing and data. It highlights the urgent need for stronger regulations, more transparent practices, and a broader societal discussion about the appropriate use and protection of genetic information.

New Ethics Opinion Addresses Obligations Associated With Collateral Information

Moran, M. (2024).

Psychiatric News, 59(09). 

https://doi.org/10.1176/appi.pn.2024.09.9.12

What are a psychiatrist’s ethical obligations regarding confidentiality of sources of collateral information obtained in the involuntary hospitalization of a patient?

In a new opinion added to “The Principles of Medical Ethics With Annotations Especially Applicable to Psychiatry,” the APA Ethics Committee underscored that a psychiatrist’s overriding ethical obligation is to the safety of the patient, and that there can be no guarantee of confidentiality to family members or other sources who provide information that is used during involuntary hospitalization.

“Psychiatrists deal with collateral information in clinical practice routinely,” said Ethics Committee member Gregory Barber, M.D. “It’s a standard part of the job to collect collateral information in cases where a patient is hospitalized, especially involuntarily, and there can be a lot of complicated interpersonal dynamics that come up when family members provide that information.

“We obtain collateral information from people who know a patient well as a way to ensure we have a full clinical picture regarding the patient’s situation,” Barber said. “But our ethical obligations are to the patient and the patient’s safety. Psychiatrists do not have an established doctor-patient relationship with the source of collateral information and do not have obligations to keep the source hidden from patients. And we should not make guarantees that the information will remain confidential.”

The info is linked here and above.

Here are some thoughts:

Psychiatrists' ethical obligations regarding confidentiality of collateral information in involuntary hospitalization prioritize patient safety. While they should strive to protect sources' privacy, this may be secondary to ensuring the patient's well-being. Transparency and open communication with both the patient and the collateral source are essential for building trust and preventing conflicts.

Second Circuit finds post-9/11 congressional ‘torture’ report not subject to FOIA

Nika Schoonover

Courthouse News

Originally posted 5 Aug 24

A report produced by Congress on the CIA’s post-9/11 detention and interrogation program is not covered by the federal freedom of information law, a Second Circuit panel found Monday.

In the aftermath of the terrorist attacks of September 11, 2001, the Senate Select Committee on Intelligence generated a report on the Detention and Interrogation Program conducted by the CIA. The committee then transmitted the report to various agencies covered under the federal Freedom of Information Act.

In late 2014, the committee produced only an executive summary of its findings which revealed the CIA’s interrogation tactics were more gruesome and ineffective than previously acknowledged. The heavily redacted report showed that interrogations included waterboarding, sleep deprivation and sexual humiliation such as rectal feeding.

In the Second Circuit panel’s Monday ruling, the court cited another Second Circuit decision from 2022, Behar v. U.S. Department of Homeland Security, where the court determined that an entity not covered by FOIA, such as Congress, would have to show that it manifested a clear control of the documents, and that the receiving agency is not free to “use and dispose of the documents as it sees fit.”

The info is here.

Here are some thoughts:

A recent decision by the Second Circuit panel has found that a report produced by Congress on the CIA's post-9/11 detention and interrogation program is not covered under the federal Freedom of Information Act (FOIA). The report, which details the CIA's use of enhanced interrogation techniques such as waterboarding and sleep deprivation, was generated by the Senate Select Committee on Intelligence in the aftermath of the 9/11 attacks.

The court's ruling centered on the issue of control and ownership of the report, citing a previous decision in Behar v. U.S. Department of Homeland Security. The panel found that Congress had manifested a clear intent to control the report at the time of its creation, and that subsequent actions did not vitiate this intent.

The decision affirms a lower court's dismissal of a complaint filed by Douglas Cox, a law professor who had submitted FOIA requests to various federal agencies for access to the report. Cox argued that the report should be subject to FOIA disclosure, but the court found that he had failed to address a relevant precedent in his oral arguments.

Legal experts have noted that the exclusion of the document from FOIA is a matter of Congress' intent to control the document, highlighting the lack of transparency in congressional records. The decision underscores the limitations of FOIA in accessing sensitive information, particularly when it comes to congressional records.

Explainable AI lacks regulative reasons: why AI and human decision-making are not equally opaque

Peters, U.

AI Ethics 3, 963–974 (2023).

https://doi.org/10.1007/s43681-022-00217-w

Abstract

Many artificial intelligence (AI) systems currently used for decision-making are opaque, i.e., the internal factors that determine their decisions are not fully known to people due to the systems’ computational complexity. In response to this problem, several researchers have argued that human decision-making is equally opaque and since simplifying, reason-giving explanations (rather than exhaustive causal accounts) of a decision are typically viewed as sufficient in the human case, the same should hold for algorithmic decision-making. Here, I contend that this argument overlooks that human decision-making is sometimes significantly more transparent and trustworthy than algorithmic decision-making. This is because when people explain their decisions by giving reasons for them, this frequently prompts those giving the reasons to govern or regulate themselves so as to think and act in ways that confirm their reason reports. AI explanation systems lack this self-regulative feature. Overlooking it when comparing algorithmic and human decision-making can result in underestimations of the transparency of human decision-making and in the development of explainable AI that may mislead people by activating generally warranted beliefs about the regulative dimension of reason-giving.

The article is linked above.

Here are some thoughts:

This article delves into the ethics of transparency in algorithmic decision-making (ADM) versus human decision-making (HDM). While both can be opaque, the author argues HDM offers more trustworthiness due to "mindshaping." This theory suggests that explaining decisions, even if the thought process is unclear, influences future human behavior to align with the explanation. This self-regulation is absent in AI, potentially making opaque ADM less trustworthy. The text emphasizes that explanations serve a purpose beyond just understanding the process. It raises concerns about "deceptive AI" with misleading explanations and warns against underestimating the transparency of HDM due to its inherent ability to explain. Key ethical considerations include the need for further research on mindshaping's impact on bias and the limitations of explanations in both ADM and HDM.  Ultimately, the passage highlights the importance of developing explainable AI that goes beyond mere justification, while also emphasizing fairness, accountability, and responsible use of explanations in building trustworthy AI systems.

OpenAI insiders are demanding a “right to warn” the public

Sigal Samuel

Vox.com

Originally posted 5 June 24

Here is an excerpt:

To be clear, the signatories are not saying they should be free to divulge intellectual property or trade secrets, but as long as they protect those, they want to be able to raise concerns about risks. To ensure whistleblowers are protected, they want the companies to set up an anonymous process by which employees can report their concerns “to the company’s board, to regulators, and to an appropriate independent organization with relevant expertise.” 

An OpenAI spokesperson told Vox that current and former employees already have forums to raise their thoughts through leadership office hours, Q&A sessions with the board, and an anonymous integrity hotline.

“Ordinary whistleblower protections [that exist under the law] are insufficient because they focus on illegal activity, whereas many of the risks we are concerned about are not yet regulated,” the signatories write in the proposal. They have retained a pro bono lawyer, Lawrence Lessig, who previously advised Facebook whistleblower Frances Haugen and whom the New Yorker once described as “the most important thinker on intellectual property in the Internet era.”

The info is here.

Here are some thoughts:

AI development is booming, but with great power comes great responsibility, typed the Spiderman fan.  AI researchers at OpenAI are calling for a "right to warn" the public about potential risks. In clinical psychology, we have a "duty to warn" for violent patients. This raises important ethical questions. On one hand, transparency and open communication are crucial for responsible AI development.  On the other hand, companies need to protect their ideas.  The key seems to lie in striking a balance.  Researchers should have safe spaces to voice concerns without fearing punishment, and clear guidelines can help ensure responsible disclosure without compromising confidential information.

Ultimately, fostering a culture of open communication is essential to ensure AI benefits society without creating unforeseen risks.  AI developers need similar ethical guidelines to psychologists in this matter.

Can Generative AI improve social science?

Bail, C. A. (2024).

Proceedings of the National Academy of

Sciences of the United States of America, 121(21). 

https://doi.org/10.1073/pnas.2314021121

Abstract

Generative AI that can produce realistic text, images, and other human-like outputs is currently transforming many different industries. Yet it is not yet known how such tools might influence social science research. I argue Generative AI has the potential to improve survey research, online experiments, automated content analyses, agent-based models, and other techniques commonly used to study human behavior. In the second section of this article, I discuss the many limitations of Generative AI. I examine how bias in the data used to train these tools can negatively impact social science research—as well as a range of other challenges related to ethics, replication, environmental impact, and the proliferation of low-quality research. I conclude by arguing that social scientists can address many of these limitations by creating open-source infrastructure for research on human behavior. Such infrastructure is not only necessary to ensure broad access to high-quality research tools, I argue, but also because the progress of AI will require deeper understanding of the social forces that guide human behavior.

Here is a brief summary:

Generative AI, with its ability to produce realistic text, images, and data, has the potential to significantly impact social science research.  This article explores both the exciting possibilities and potential pitfalls of this new technology.

On the positive side, generative AI could streamline data collection and analysis, making social science research more efficient and allowing researchers to explore new avenues. For example, AI-powered surveys could be more engaging and lead to higher response rates. Additionally, AI could automate tasks like content analysis, freeing up researchers to focus on interpretation and theory building.

However, there are also ethical considerations. AI models can inherit and amplify biases present in the data they're trained on. This could lead to skewed research findings that perpetuate social inequalities. Furthermore, the opaqueness of some AI models can make it difficult to understand how they arrive at their conclusions, raising concerns about transparency and replicability in research.

Overall, generative AI offers a powerful tool for social scientists, but it's crucial to be mindful of the ethical implications and limitations of this technology. Careful development and application are essential to ensure that AI enhances, rather than hinders, our understanding of human behavior.

Large Language Models as Moral Experts? GPT-4o Outperforms Expert Ethicist in Providing Moral Guidance

Dillion, D., Mondal, D., Tandon, N.,

& Gray, K. (2024, May 29).

https://doi.org/10.31234/osf.io/w7236

Abstract

AI has demonstrated expertise across various fields, but its potential as a moral expert remains unclear. Recent work suggests that Large Language Models (LLMs) can reflect moral judgments with high accuracy. But as LLMs are increasingly used in complex decision-making roles, true moral expertise requires not just aligned judgments but also clear and trustworthy moral reasoning. Here, we advance work on the Moral Turing Test and find that advice from GPT-4o is rated as more moral, trustworthy, thoughtful, and correct than that of the popular The New York Times advice column, The Ethicist. GPT models outperformed both a representative sample of Americans and a renowned ethicist in providing moral explanations and advice, suggesting that LLMs have, in some respects, achieved a level of moral expertise. The present work highlights the importance of carefully programming ethical guidelines in LLMs, considering their potential to sway users' moral reasoning. More promisingly, it suggests that LLMs could complement human expertise in moral guidance and decision-making.

The research is here.

Here are my thoughts:

This research on GPT-4o's moral reasoning is fascinating, but caution is warranted. While exceeding human performance in explanations and perceived trustworthiness is impressive, true moral expertise goes beyond these initial results.

Here's why:

First, there are nuances to all moral dilemmas. Real-world dilemmas often lack clear-cut answers. Can GPT-4o navigate the gray areas and complexities of human experience?

Next, everyone has a rich experience, values, perspectives, and biases.  What ethical framework guides GPT-4o's decisions? Transparency in its programming is crucial.

Finally, the consequences of AI-driven moral advice can be far-reaching. Careful evaluation of potential biases and unintended outcomes is essential.  There is no objective algorithm.  There is no objective morality.  All moral decisions, no matter how well-reasoned, have pluses and minuses.  Therefore, AI can be used as a starting point for decision-making and planning.

None of these people exist, but you can buy their books on Amazon anyway

Conspirador Norteno

Substack.com

Originally published 12 Jan 24

Meet Jason N. Martin N. Martin, the author of the exciting and dynamic Amazon bestseller “How to Talk to Anyone: Master Small Talks, Elevate Your Social Skills, Build Genuine Connections (Make Real Friends; Boost Confidence & Charisma)”, which is the 857,233rd most popular book on the Kindle Store as of January 12th, 2024. There are, however, a few obvious problems. In addition to the unnecessary repetition of the middle initial and last name, Mr. N. Martin N. Martin’s official portrait is a GAN-generated face, and (as we’ll see shortly), his sole published work is strangely similar to several books by another Amazon author with a GAN-generated face.

In an interesting twist, Amazon’s recommendation system suggests another author with a GAN-generated face in the “Customers also bought items by” section of Jason N. Martin N. Martin’s author page. Further exploration of the recommendations attached to both of these authors and their published works reveals a set of a dozen Amazon authors with GAN-generated faces and at least one published book. Amazon’s recommendation algorithms reliably link these authors together; whether this is a sign that the twelve author accounts are actually run by the same entity or merely an artifact of similarities in the content of their books is unclear at this point in time. 

The info is here.

Here's my take:

Forget literary pen names - AI is creating a new trend on Amazon: ghostwritten books. These novels, poetry collections, and even children's stories boast intriguing titles and blurbs, yet none of the authors on the cover are real people. Instead, their creations spring from the algorithms of powerful language models.

Here's the gist:

• AI churns out content: Fueled by vast datasets of text and code, AI can generate chapters, characters, and storylines at an astonishing pace.
• Ethical concerns: Questions swirl around copyright, originality, and the very nature of authorship. Is an AI-generated book truly a book, or just a clever algorithm mimicking creativity?
• Quality varies: While some AI-written books garner praise, others are criticized for factual errors, nonsensical plots, and robotic dialogue.
• Transparency is key: Many readers feel deceived by the lack of transparency about AI authorship. Should books disclose their digital ghostwriters?

This evolving technology challenges our understanding of literature and raises questions about the future of authorship. While AI holds potential to assist and inspire, the human touch in storytelling remains irreplaceable. So, the next time you browse Amazon, remember: the author on the cover might not be who they seem.

Doctors With Histories of Big Malpractice Settlements Now Work for Insurers

P. Rucker, D. Armstrong, & D. Burke

Propublica.org

Originally published 15 Dec 23

Here is an excerpt:

Patients and the doctors who treat them don’t get to pick which medical director reviews their case. An anesthesiologist working for an insurer can overrule a patient’s oncologist. In other cases, the medical director might be a doctor like Kasemsap who has left clinical practice after multiple accusations of negligence.

As part of a yearlong series about how health plans refuse to pay for care, ProPublica and The Capitol Forum set out to examine who insurers picked for such important jobs.

Reporters could not find any comprehensive database of doctors working for insurance companies or any public listings by the insurers who employ them. Many health plans also farm out medical reviews to other companies that employ their own doctors. ProPublica and The Capitol Forum identified medical directors through regulatory filings, LinkedIn profiles, lawsuits and interviews with insurance industry insiders. Reporters then checked those names against malpractice databases, state licensing board actions and court filings in 17 states.

Among the findings: The Capitol Forum and ProPublica identified 12 insurance company doctors with either a history of multiple malpractice payments, a single payment in excess of $1 million or a disciplinary action by a state medical board.

One medical director settled malpractice cases with 11 patients, some of whom alleged he bungled their urology surgeries and left them incontinent. Another was reprimanded by a state medical board for behavior that it found to be deceptive and dishonest. A third settled a malpractice case for $1.8 million after failing to identify cancerous cells on a pathology slide, which delayed a diagnosis for a 27-year-old mother of two, who died less than a year after her cancer was finally discovered.

None of this would have been easily visible to patients seeking approvals for care or payment from insurers who relied on these medical directors.

The info is here.

The ethical implications in this article are staggering.  Here are some quick points:

Conflicted Care: In a concerning trend, some US insurers are employing doctors with past malpractice settlements to assess whether patients deserve coverage for recommended treatments.  So, do these still licensed reviewers actually understand best practices?

Financial Bias: Critics fear these doctors, having faced financial repercussions for past care decisions, might prioritize minimizing payouts over patient needs, potentially leading to denied claims and delayed care.  In other words, do the reviewers have an inherent bias against patients, given that former patients complained against them?

Transparency Concerns: The lack of clear disclosure about these doctors' backgrounds raises concerns about transparency and potential conflicts of interest within the healthcare system.

In essence, this is a horrible system to provide high quality medical review.

Tackling healthcare AI's bias, regulatory and inventorship challenges

Bill Siwicki

Healthcare IT News

Originally posted 29 August 23

While AI adoption is increasing in healthcare, there are privacy and content risks that come with technology advancements.

Healthcare organizations, according to Dr. Terri Shieh-Newton, an immunologist and a member at global law firm Mintz, must have an approach to AI that best positions themselves for growth, including managing:

• Biases introduced by AI. Provider organizations must be mindful of how machine learning is integrating racial diversity, gender and genetics into practice to support the best outcome for patients.

• Inventorship claims on intellectual property. Identifying ownership of IP as AI begins to develop solutions in a faster, smarter way compared to humans.

Healthcare IT News sat down with Shieh-Newton to discuss these issues, as well as the regulatory landscape’s response to data and how that impacts AI.

Q. Please describe the generative AI challenge with biases introduced from AI itself. How is machine learning integrating racial diversity, gender and genetics into practice?

A. Generative AI is a type of machine learning that can create new content based on the training of existing data. But what happens when that training set comes from data that has inherent bias? Biases can appear in many forms within AI, starting from the training set of data.

Take, as an example, a training set of patient samples already biased if the samples are collected from a non-diverse population. If this training set is used for discovering a new drug, then the outcome of the generative AI model can be a drug that works only in a subset of a population – or have just a partial functionality.

Some traits of novel drugs are better binding to its target and lower toxicity. If the training set excludes a population of patients of a certain gender or race (and the genetic differences that are inherent therein), then the outcome of proposed drug compounds is not as robust as when the training sets include a diversity of data.

This leads into questions of ethics and policies, where the most marginalized population of patients who need the most help could be the group that is excluded from the solution because they were not included in the underlying data used by the generative AI model to discover that new drug.

One can address this issue with more deliberate curation of the training databases. For example, is the patient population inclusive of many types of racial backgrounds? Gender? Age ranges?

By making sure there is a reasonable representation of gender, race and genetics included in the initial training set, generative AI models can accelerate drug discovery, for example, in a way that benefits most of the population.

The info is here. 

Here is my take:

 One of the biggest challenges is bias. AI systems are trained on data, and if that data is biased, the AI system will be biased as well. This can have serious consequences in healthcare, where biased AI systems could lead to patients receiving different levels of care or being denied care altogether.

Another challenge is regulation. Healthcare is a highly regulated industry, and AI systems need to comply with a variety of laws and regulations. This can be complex and time-consuming, and it can be difficult for healthcare organizations to keep up with the latest changes.

Finally, the article discusses the challenges of inventorship. As AI systems become more sophisticated, it can be difficult to determine who is the inventor of a new AI-powered healthcare solution. This can lead to disputes and delays in bringing new products and services to market.

The article concludes by offering some suggestions for how to address these challenges:

• To reduce bias, healthcare organizations need to be mindful of the data they are using to train their AI systems. They should also audit their AI systems regularly to identify and address any bias.
• To comply with regulations, healthcare organizations need to work with experts to ensure that their AI systems meet all applicable requirements.
• To resolve inventorship disputes, healthcare organizations should develop clear policies and procedures for allocating intellectual property rights.

By addressing these challenges, healthcare organizations can ensure that AI is deployed in a way that is safe, effective, and ethical.

Additional thoughts

In addition to the challenges discussed in the article, there are a number of other factors that need to be considered when deploying AI in healthcare. For example, it is important to ensure that AI systems are transparent and accountable. This means that healthcare organizations should be able to explain how their AI systems work and why they make the decisions they do.

It is also important to ensure that AI systems are fair and equitable. This means that they should treat all patients equally, regardless of their race, ethnicity, gender, income, or other factors.

Finally, it is important to ensure that AI systems are used in a way that respects patient privacy and confidentiality. This means that healthcare organizations should have clear policies in place for the collection, use, and storage of patient data.

By carefully considering all of these factors, healthcare organizations can ensure that AI is used to improve patient care and outcomes in a responsible and ethical way.

AI systems must not confuse users about their sentience or moral status

Schwitzgebel, E. (2023).

Patterns, 4(8), 100818.

https://doi.org/10.1016/j.patter.2023.100818 

The bigger picture

The draft European Union Artificial Intelligence Act highlights the seriousness with which policymakers and the public have begun to take issues in the ethics of artificial intelligence (AI). Scientists and engineers have been developing increasingly more sophisticated AI systems, with recent breakthroughs especially in large language models such as ChatGPT. Some scientists and engineers argue, or at least hope, that we are on the cusp of creating genuinely sentient AI systems, that is, systems capable of feeling genuine pain and pleasure. Ordinary users are increasingly growing attached to AI companions and might soon do so in much greater numbers. Before long, substantial numbers of people might come to regard some AI systems as deserving of at least some limited rights or moral standing, being targets of ethical concern for their own sake. Given high uncertainty both about the conditions under which an entity can be sentient and about the proper grounds of moral standing, we should expect to enter a period of dispute and confusion about the moral status of our most advanced and socially attractive machines.

Summary

One relatively neglected challenge in ethical artificial intelligence (AI) design is ensuring that AI systems invite a degree of emotional and moral concern appropriate to their moral standing. Although experts generally agree that current AI chatbots are not sentient to any meaningful degree, these systems can already provoke substantial attachment and sometimes intense emotional responses in users. Furthermore, rapid advances in AI technology could soon create AIs of plausibly debatable sentience and moral standing, at least by some relevant definitions. Morally confusing AI systems create unfortunate ethical dilemmas for the owners and users of those systems, since it is unclear how those systems ethically should be treated. I argue here that, to the extent possible, we should avoid creating AI systems whose sentience or moral standing is unclear and that AI systems should be designed so as to invite appropriate emotional responses in ordinary users.

My take

The article proposes two design policies for avoiding morally confusing AI systems. The first is to create systems that are clearly non-conscious artifacts. This means that the systems should be designed in a way that makes it clear to users that they are not sentient beings. The second policy is to create systems that are clearly deserving of moral consideration as sentient beings. This means that the systems should be designed to have the same moral status as humans or other animals.

The article concludes that the best way to avoid morally confusing AI systems is to err on the side of caution and create systems that are clearly non-conscious artifacts. This is because it is less risky to underestimate the sentience of an AI system than to overestimate it.

Here are some additional points from the article:

• The scientific study of sentience is highly contentious, and there is no agreed-upon definition of what it means for an entity to be sentient.
• Rapid advances in AI technology could soon create AI systems that are plausibly debatable as sentient.
• Morally confusing AI systems create unfortunate ethical dilemmas for the owners and users of those systems, since it is unclear how those systems ethically should be treated.
• The design of AI systems should be guided by ethical considerations, such as the need to avoid causing harm and the need to respect the dignity of all beings.

Zombies in the Loop? Humans Trust Untrustworthy AI-Advisors for Ethical Decisions

Krügel, S., Ostermaier, A. & Uhl, M.

Philos. Technol. 35, 17 (2022).

https://doi.org/10.1007/s13347-022-00511-9

Abstract

Departing from the claim that AI needs to be trustworthy, we find that ethical advice from an AI-powered algorithm is trusted even when its users know nothing about its training data and when they learn information about it that warrants distrust. We conducted online experiments where the subjects took the role of decision-makers who received advice from an algorithm on how to deal with an ethical dilemma. We manipulated the information about the algorithm and studied its influence. Our findings suggest that AI is overtrusted rather than distrusted. We suggest digital literacy as a potential remedy to ensure the responsible use of AI.

Summary

Background: Artificial intelligence (AI) is increasingly being used to make ethical decisions. However, there is a concern that AI-powered advisors may not be trustworthy, due to factors such as bias and opacity.

Research question: The authors of this article investigated whether humans trust AI-powered advisors for ethical decisions, even when they know that the advisor is untrustworthy.

Methods: The authors conducted a series of experiments in which participants were asked to make ethical decisions with the help of an AI advisor. The advisor was either trustworthy or untrustworthy, and the participants were aware of this.

Results: The authors found that participants were more likely to trust the AI advisor, even when they knew that it was untrustworthy. This was especially true when the advisor was able to provide a convincing justification for its advice.

Conclusions: The authors concluded that humans are susceptible to "zombie trust" in AI-powered advisors. This means that we may trust AI advisors even when we know that they are untrustworthy. This is a concerning finding, as it could lead us to make bad decisions based on the advice of untrustworthy AI advisors.  By contrast, decision-makers do disregard advice from a human convicted criminal.

The article also discusses the implications of these findings for the development and use of AI-powered advisors. The authors suggest that it is important to make AI advisors more transparent and accountable, in order to reduce the risk of zombie trust. They also suggest that we need to educate people about the potential for AI advisors to be untrustworthy.