Automation bias and assistive AI.

Khera, R., Simon, M. A., & Ross, J. S. (2023).

JAMA, 330(23), 2255. 

https://doi.org/10.1001/jama.2023.22557

At the point of care, artificial intelligence (AI) algorithms have been developed to augment diagnostic decisions and suggest appropriate care pathways, by leveraging complex information in a patient’s electronic health record, such as imaging, documentation, and diagnostic testing. With an increasing number of technologies integrated into the diagnosis, management, and even treatment of patients, the promise of AI to enhance accuracy, reduce errors, reduce clinician burnout, and improve clinical workflows may appear imminent.

MostAI algorithms aredesigned tobe assistive technologies—augmenting, not replacing, clinicians’

decision-making. AI models are imperfect and lack the broader clinical context that may be relevant for patient care. The expectation is that the diagnostic performance of clinicians supported by AI will exceed those of clinicians without such support.

A pdf is here.

Here are some thoughts:

This article highlights a critical problem with artificial intelligence in medicine: automation bias. This is when clinicians trust an AI’s recommendation too much, even when it is clearly wrong or contradicts their own judgment. The authors show that biased AI models can significantly lower the quality of patient care, and simply explaining how the AI works does not fix the issue. Clinicians, often working under time pressure, may defer to the tool instead of using their own expertise, which can lead to direct patient harm.

The key takeaway is that keeping a human “in the loop” is not enough to ensure safety. Current regulatory approaches focus too much on the AI’s technical accuracy and not enough on how real clinicians actually use these tools in practice. The authors argue that better training, higher safety standards, and truly interpretable AI are needed. Without these changes, the excitement around medical AI risks overshadowing its primary goal: improving patient care, not undermining it.

A review of neuro-symbolic AI integrating reasoning and learning for advanced cognitive systems

Nawaz, U., Anees-Ur-Rahaman, M., & Saeed, Z. (2025).

Intelligent Systems With Applications, 26, 200541.

https://doi.org/10.1016/j.iswa.2025.200541

Abstract

Neuro-symbolic AI represents the convergence of two principal paradigms in artificial intelligence: neural networks, which are efficient in data-driven learning, and symbolic reasoning, which offers explainability and logical inference. This hybrid methodology combines the adaptability of neural networks with symbolic AI's interpretability and formal reasoning abilities, which provide a practical framework for advanced cognitive systems. This paper analyzes the present condition of neuro-symbolic AI, emphasizing essential techniques that combine reasoning and learning. We explore models such as Logic Tensor Networks, Differentiable Logic Programs, and Neural Theorem Provers. The study analyzes their impact on the advancement of cognitive systems in natural language processing, robotics, and decision-making. The paper examines the challenges faced by neuro-symbolic AI, such as scalability, integration with multimodal data, and maintaining interpretability without compromising efficiency. By evaluating the strengths and weaknesses of many methodologies, we comprehensively understand the field's development and its potential to revolutionize intelligent systems. In addition, we identify emerging research areas, including the incorporation of ethical frameworks and the development of adaptive dynamic neuro-symbolic systems that respond in real-time. This review aims to guide future research by providing insights into the potential of neuro-symbolic AI to influence the development of the next generation of intelligent, explainable, and adaptive systems.

Here are some thoughts:

This research is important because it provides a comprehensive, state-of-the-art analysis of the most promising path forward for creating truly intelligent, reliable, and understandable AI systems. It acknowledges the power of deep learning while rigorously addressing its most critical shortcomings—lack of reasoning, explainability, and data efficiency. For anyone working on or relying on AI in critical areas like medicine, finance, or autonomous systems, understanding neuro-symbolic AI is becoming essential.

Neuro-Symbolic AI is a hybrid approach to artificial intelligence that combines neural networks (which learn patterns from data) with symbolic reasoning (which uses logic and rules to think and explain decisions). In decision-science terms, this process is merging Type 1 and Type 2 thinking in order to reason more coherently.

In equation format: Neuro-Symbolic AI = Neural Learning (pattern recognition) + Symbolic Reasoning (logic & explainability).

The Hermeneutic Turn of AI: Are Machines Capable of Interpreting?

Demichelis, R. (2024, November 19).

arXiv.org.

https://arxiv.org/abs/2411.12517

This article aims to demonstrate how the approach to computing is being disrupted by deep learning (artificial neural networks), not only in terms of techniques but also in our interactions with machines. It also addresses the philosophical tradition of hermeneutics (Don Ihde, Wilhelm Dilthey) to highlight a parallel with this movement and to demystify the idea of human-like AI.

The paper is here.

Here are some thoughts:

This paper examines how modern AI systems, like ChatGPT, have evolved from simply executing commands to interpreting ambiguous human language. The paper draws on the tradition of hermeneutics to argue that while AI can mimic interpretation through data processing, it lacks the genuine understanding and imaginative insight characteristic of human cognition. This mechanical approximation of interpretation raises important concerns regarding transparency, bias, and ethical oversight, prompting a reevaluation of how we define knowledge and meaning in the age of AI.

How The Software Industry Must Marry Ethics With Artificial Intelligence

Christian Pedersen
Forbes.com
Originally posted July 15, 2019

Here is an excerpt:

Companies developing software used to automate business decisions and processes, military operations or other serious work need to address explainability and human control over AI as they weave it into their products. Some have started to do this.

As AI is introduced into existing software environments, those application environments can help. Many will have established preventive and detective controls and role-based security. They can track who made what changes to processes or to the data that feeds through those processes. Some of these same pathways can be used to document changes made to goals, priorities or data given to AI.

But software vendors have a greater opportunity. They can develop products that prevent bad use of AI, but they can also use AI to actively protect and aid people, business and society. AI can be configured to solve for anything from overall equipment effectiveness or inventory reorder point to yield on capital. Why not have it solve for nonfinancial, corporate social responsibility metrics like your environmental footprint or your environmental or economic impact? Even a common management practice like using a balanced scorecard could help AI strive toward broader business goals that consider the well-being of customers, employees, customers, suppliers and other stakeholders.

The info is here.

Ethics in the Age of Artificial Intelligence

Shohini Kundu
Scientific American
Originally published July 3, 2019

Here is an except:

Unfortunately, in that decision-making process, AI also took away the transparency, explainability, predictability, teachability and auditability of the human move, replacing it with opacity. The logic for the move is not only unknown to the players, but also unknown to the creators of the program. As AI makes decisions for us, transparency and predictability of decision-making may become a thing of the past.

Imagine a situation in which your child comes home to you and asks for an allowance to go see a movie with her friends. You oblige. A week later, your other child comes to you with the same request, but this time, you decline. This will immediately raise the issue of unfairness and favoritism. To avoid any accusation of favoritism, you explain to your child that she must finish her homework before qualifying for any pocket money.

Without any explanation, there is bound to be tension in the family. Now imagine replacing your role with an AI system that has gathered data from thousands of families in similar situations. By studying the consequence of allowance decisions on other families, it comes to the conclusion that one sibling should get the pocket money while the other sibling should not.

But the AI system cannot really explain the reasoning—other than to say that it weighed your child’s hair color, height, weight and all other attributes that it has access to in arriving at a decision that seems to work best for other families. How is that going to work?

The info is here.

How To Prevent AI Ethics Councils From Failing

Manoj Saxena
www.forbes.com
Originally posted April 30, 2019

There's nothing “artificial” about Artificial Intelligence-infused systems. These systems are real and are already impacting everyone today though automated predictions and decisions. However, these digital brains may demonstrate unpredictable behaviors that can be disruptive, confusing, offensive, and even dangerous. Therefore, before getting started with an AI strategy, it’s critical for companies to consider AI through the lens of building systems you can trust.

Educate on the criticality of a ‘people and ethics first’ approach

AI systems often function in oblique, invisible ways that may harm the most vulnerable. Examples of such harm include loss of privacy, discrimination, loss of skills, economic impact, the security of critical infrastructure, and long-term effects on social well-being.

The ‘’technology and monetization first approach’’ to AI needs to evolve to a “people and ethics first” approach. Ethically aligned design is a set of societal and policy guidelines for the development of intelligent and autonomous systems to ensure such systems serve humanity’s values and ethical principles.

Multiple noteworthy organizations and countries have proposed guidelines for developing trusted AI systems going forward. These include the IEEE, World Economic Forum, the Future of Life Institute, Alan Turing Institute, AI Global, and the Government of Canada. Once you have your guidelines in place, you can then start educating everyone internally and externally about the promise and perils of AI and the need for an AI ethics council.

The info is here.

Artificial Intelligence and Black‐Box Medical Decisions: Accuracy versus Explainability

Alex John London
The Hastings Center Report
Volume49, Issue1, January/February 2019, Pages 15-21

Abstract

Although decision‐making algorithms are not new to medicine, the availability of vast stores of medical data, gains in computing power, and breakthroughs in machine learning are accelerating the pace of their development, expanding the range of questions they can address, and increasing their predictive power. In many cases, however, the most powerful machine learning techniques purchase diagnostic or predictive accuracy at the expense of our ability to access “the knowledge within the machine.” Without an explanation in terms of reasons or a rationale for particular decisions in individual cases, some commentators regard ceding medical decision‐making to black box systems as contravening the profound moral responsibilities of clinicians. I argue, however, that opaque decisions are more common in medicine than critics realize. Moreover, as Aristotle noted over two millennia ago, when our knowledge of causal systems is incomplete and precarious—as it often is in medicine—the ability to explain how results are produced can be less important than the ability to produce such results and empirically verify their accuracy.

The info is here.

Pop Culture, AI And Ethics

Phaedra Boinodiris
Forbes.com
Originally published February 24, 2019

Here is an excerpt:


5 Areas of Ethical Focus

The guide goes on to outline five areas of ethical focus or consideration:

Accountability – there is a group responsible for ensuring that REAL guests in the hotel are interviewed to determine their needs. When feedback is negative this group implements a feedback loop to better understand preferences. They ensure that at any point in time, a guest can turn the AI off.

Fairness – If there is bias in the system, the accountable team must take the time to train with a larger, more diverse set of data.Ensure that the data collected about a user's race, gender, etc. in combination with their usage of the AI, will not be used to market to or exclude certain demographics.

Explainability and Enforced Transparency – if a guest doesn’t like the AI’s answer, she can ask how it made that recommendation using which dataset. A user must explicitly opt in to use the assistant and provide the guest options to consent on what information to gather.

User Data Rights – The hotel does not own a guest’s data and a guest has the right to have the system purges at any time. Upon request, a guest can receive a summary of what information was gathered by the Ai assistant.

Value Alignment – Align the experience to the values of the hotel. The hotel values privacy and ensuring that guests feel respected and valued. Make it clear that the AI assistant is not designed to keep data or monitor guests. Relay how often guest data is auto deleted. Ensure that the AI can speak in the guest’s respective language.

The info is here.

Computer Says No: Part 2 - Explainability

Jasmine Leonard
theRSA.org
Originally posted March 23, 2018

Here is an expert:

The trouble is, since many decisions should be explainable, it’s tempting to assume that automated decision systems should also be explainable.  But as discussed earlier, automated decisions systems don’t actually make decisions; they make predictions.  And when a prediction is used to guide a decision, the prediction is itself part of the explanation for that decision.  It therefore doesn’t need to be explained itself, it merely needs to be justifiable.

This is a subtle but important distinction.  To illustrate it, imagine you were to ask your doctor to explain her decision to prescribe you a particular drug.  She could do so by saying that the drug had cured many other people with similar conditions in the past and that she therefore predicted it would cure you too.  In this case, her prediction that the drug will cure you is the explanation for her decision to prescribe it.  And it’s a good explanation because her prediction is justified – not on the basis of an explanation of how the drug works, but on the basis that it’s proven to be effective in previous cases.  Indeed, explanations of how drugs work are often not available because the biological mechanisms by which they operate are poorly understood, even by those who produce them.  Moreover, even if your doctor could explain how the drug works, unless you have considerable knowledge of pharmacology, it’s unlikely that the explanation would actually increase your understanding of her decision to prescribe the drug.

If explanations of predictions are unnecessary to justify their use in decision-making then, what else can justify the use of a prediction made by an automated decision system?  The best answer, I believe, is that the system is shown to be sufficiently accurate.  What “sufficiently accurate” means is obviously up for debate, but at a minimum I would suggest that it means the system’s predictions are at least as accurate as those produced by a trained human.  It also means that there are no other readily available systems that produce more accurate predictions.

The article is here.