Automation bias: a systematic review of frequency, effect mediators, and mitigators

Goddard, K., Roudsari, A., & Wyatt, J. C. (2011).

Journal of the American Medical

Informatics Association, 19(1), 121–127.

https://doi.org/10.1136/amiajnl-2011-000089

Abstract

Automation bias (AB)—the tendency to over-rely on automation—has been studied in various academic fields. Clinical decision support systems (CDSS) aim to benefit the clinical decision-making process. Although most research shows overall improved performance with use, there is often a failure to recognize the new errors that CDSS can introduce. With a focus on healthcare, a systematic review of the literature from a variety of research fields has been carried out, assessing the frequency and severity of AB, the effect mediators, and interventions potentially mitigating this effect. This is discussed alongside automation-induced complacency, or insufficient monitoring of automation output. A mix of subject specific and freetext terms around the themes of automation, human–automation interaction, and task performance and error were used to search article databases. Of 13 821 retrieved papers, 74 met the inclusion criteria. User factors such as cognitive style, decision support systems (DSS), and task specific experience mediated AB, as did attitudinal driving factors such as trust and confidence. Environmental mediators included workload, task complexity, and time constraint, which pressurized cognitive resources. Mitigators of AB included implementation factors such as training and emphasizing user accountability, and DSS design factors such as the position of advice on the screen, updated confidence levels attached to DSS output, and the provision of information versus recommendation. By uncovering the mechanisms by which AB operates, this review aims to help optimize the clinical decision-making process for CDSS developers and healthcare practitioners.

Here are some thoughts:

This systematic review examines the frequency, mediators, and mitigators of automation bias, which is the tendency for users to over rely on automated decision support systems as a heuristic replacement for vigilant information seeking and processing. The authors reviewed 74 studies across healthcare, aviation, and human computer interaction fields and found that automation bias is a robust effect, with a meta analysis showing that erroneous clinical decision support system advice increased the risk of incorrect decisions by 26%. Key mediators include user factors such as cognitive style, trust, confidence, and task specific experience, as well as environmental factors like workload, task complexity, and time pressure that strain cognitive resources. Mitigators include increasing user accountability, providing training, updating confidence levels alongside advice, positioning advice less prominently on screen, and presenting information rather than direct recommendations. The review concludes that automation bias and the related concept of automation induced complacency represent distinct but overlapping attentional phenomena that can introduce new errors even when decision support systems improve overall performance, highlighting the need for careful design and implementation strategies.

Transforming clinical reasoning—the role of AI in supporting human cognitive limitations

Greengrass C. J. (2026).

Frontiers in digital health, 7, 1715440.

https://doi.org/10.3389/fdgth.2025.1715440

Abstract

Clinical reasoning is foundational to medical practice, requiring clinicians to synthesise complex information, recognise patterns, and apply causal reasoning to reach accurate diagnoses and guide patient management. However, human cognition is inherently limited by factors such as limitations in working memory capacity, constraints in cognitive load, a general reliance on heuristics; with an inherent vulnerability to biases including anchoring, availability bias, and premature closure. Cognitive fatigue and cognitive overload, particularly apparent in high-pressure environments, further compromise diagnostic accuracy and efficiency. Artificial intelligence (AI) presents a transformative opportunity to overcome these limitations by supplementing and supporting decision-making. With AI's advanced computational capabilities, these systems can analyse large datasets, detect subtle or atypical patterns, and provide accurate evidence-based diagnoses. Furthermore, by leveraging machine learning and probabilistic modelling, AI reduces dependence on incomplete heuristics and potentially mitigates cognitive biases. It also ensures consistent performance, unaffected by fatigue or information overload. These attributes likely make AI an invaluable tool for enhancing the accuracy and efficiency of diagnostic reasoning. Through a narrative review, this article examines the cognitive limitations inherent in diagnostic reasoning and considers how AI can be positioned as a collaborative partner in addressing them. Drawing on the concept of Mutual Theory of Mind, the author identifies a set of indicators that should inform the design of future frameworks for human–AI interaction in clinical decision-making. These highlight how AI could dynamically adapt to human reasoning states, reduce bias, and promote more transparent and adaptive diagnostic support in high-stakes clinical environments.

Here are some thoughts:

This article examines how artificial intelligence can support clinical diagnostic reasoning by compensating for inherent human cognitive limitations such as limited working memory capacity, cognitive load, reliance on heuristics, and susceptibility to biases like anchoring and premature closure. The author integrates cognitive psychology concepts including dual process theory (System 1 intuitive pattern recognition versus System 2 analytical reasoning), cognitive load theory, and Bayesian reasoning to analyze how AI systems can reduce cognitive burden, provide external schema repositories, offer transparent explainable outputs, and support metacognitive monitoring. While AI offers advantages in processing vast data streams, maintaining multiple hypotheses, and performing consistently without fatigue, the review acknowledges current limitations of large language models including poor probabilistic reasoning and potential for algorithmic or transferred bias. The article concludes that AI should function as a collaborative partner within a Mutual Theory of Mind framework, enhancing rather than replacing human judgment, provided that ethical standards and clinician training keep pace with technological development.

Using AI-Based Virtual simulated patients for training in psychopathological interviewing: Cross-Sectional Observational study.

García-Torres, D., et al. (2025).

JMIR Medical Education, 11, e78857.

https://doi.org/10.2196/78857

Abstract

Background:

Virtual simulated patients (VSPs) powered by generative artificial intelligence (GAI) offer a promising tool for training clinical interviewing skills; yet, little is known about how different system- and user-level variables shape students’ perceptions of these interactions.

Objective:

We aim to study psychology students’ perceptions of GAI-driven VSPs and examine how demographic factors, system parameters, and interaction characteristics influence such perceptions.

Methods:

We conducted a total of 1832 recorded interactions involving 156 psychology students with 13 GAI-generated VSPs configured with varying temperature settings (0.1, 0.5, 0.9). For each student, we collected age and sex; for each interview, we recorded interview length (total number of question–answer turns), number of connectivity failures, the specific VSP consulted, and the model temperature. After every interview, students provided a 1-10 global rating and open-ended comments regarding strengths and areas for improvement. At the end of the training sequence, they also reported perceived improvement in diagnostic ability. Statistical analyses assessed the influence of different variables on global ratings: demographics, interaction-level data, and GAI temperature setting. Sentiment analysis was conducted to evaluate the VSPs’ clinical realism.

Results:

Statistical analysis showed that female students rated the tool significantly higher (mean rating 9.25/10) than male students (mean rating 8.94/10; Kruskal-Wallis test, H=8.7; P=.003). On the other side, no significant correlation was found between global rating and age (r=0.02, 95% CI –0.03 to 0.06; P=.42), interview length (r=0.04, 95% CI –0.2 to 0.10; P=.18), or frequency of participation (Kruskal-Wallis test, H=4.62; P=.20). A moderate negative correlation emerged between connectivity failures and ratings (r=–0.26, 95% CI –0.41 to –0.10; P=.002). Temperature settings significantly influenced ratings (Kruskal-Wallis test, H=6.93; P=.03; η²=0.02), with higher scores at temperature 0.9 compared with 0.1 (Dunn’s test, P=.04). Concerning learning outcomes, self-perceived improvement in diagnostic ability was reported by 94% (94/100) of students; however, final practical examination scores (mean 6.67, SD 1.42) did not differ significantly from those of the previous cohort without VSP training (mean 6.42, SD 1.56). Sentiment analysis indicated predominantly negative sentiment in GAI responses (median negativity 0.8903, IQR 0.306-0.961), consistent with clinical realism.

Conclusions:

GAI-driven VSPs were well-received by psychology students, with student gender and system-level variables (particularly temperature settings and connection stability) shaping user evaluations. Although participants perceived the training as beneficial for their diagnostic skills, objective examination performance did not significantly differ from the previous cohort. However, lack of randomization limits the generalization of the results obtained, and further experiments are required.

Here are some thoughts:

This study is important because it demonstrates a promising application of AI in clinical training, using generative AI-powered virtual simulated patients to help psychology students practice psychopathological interviewing in a safe, low-stakes environment. The platform was highly rated by students and 94% reported meaningful improvement in their ability to identify clinically relevant symptoms. Higher AI temperature settings, which produce more natural and varied responses, were associated with greater student satisfaction, while connectivity failures reduced ratings, underscoring the importance of technical reliability. Although students found VSP-based sessions more challenging than traditional paper cases, final exam scores were comparable between cohorts, suggesting the AI simulation provides a more realistic learning experience rather than a less effective one. For practicing psychologists and educators, this study offers early empirical support for integrating AI-driven patient simulation into clinical training, while highlighting the need for randomized studies and careful calibration of AI parameters before broad adoption.

The moon, the ghetto and artificial intelligence: Reducing systemic racism in computational algorithms.

Fountain, J. (2022).

Government Information Quarterly, 39(2), 101645.

https://doi.org/10.1016/j.giq.2021.101645 

Abstract

Computational algorithms and automated decision making systems that include them offer potential to improve public policy and organizations. But computational algorithms based on biased data encode those biases into algorithms, models and their outputs. Systemic racism is institutionalized bias with respect to race, ethnicity and related attributes. Such bias is located in data that encode the results and outputs of decisions that have been discriminatory, in procedures and processes that may intentionally or unintentionally disadvantage people based on race, and in policies that may discriminate by race. Computational algorithms may exacerbate systemic racism if they are not designed, developed, and used–that is, enacted–with attention to identifying and remedying bias specific to race. Advancing social equity in digital governance requires systematic, ongoing efforts to assure that automated decision making systems, and their enactment in complex public organizational arrangements, are free from bias.

Highlights

• Computational algorithms are powerful tools but may replicate biases.

• Biases, including systemic racism, in underlying data bias algorithms

• Automated decision making systems that discriminate harm people.

• Careful scrutiny of data, processes, variables and algorithms may reduce bias.

AI in psychoeducational assessment: a study of report generation.

Farmer, R. L., et al. (2025).

Online preprint. 

https://doi.org/10.13140/RG.2.2.31345.95844

Abstract

Artificial intelligence (AI) is poised to reshape school psychology, with report writing as a primary area of impact. A national sample of school psychologists using AI at work (n = 100) reported on its role in documentation. Of those using AI for report writing (n = 45), most applied it to discrete tasks such as rewriting sections for clarity (69%) or generating recommendations (67%). Far fewer used AI for interpretive or diagnostic purposes, and none relied on it to generate entire reports. Nearly all users (94%) edited AI-generated content before use. On average, practitioners who used AI for report writing saved 6.3 hours per week (95% CI: 4.6–8.1), more than double the savings of those using AI for other tasks (≈3 hours). AI shows promise for reducing documentation burden and reclaiming time for direct services, though its use raises ethical concerns requiring further guidance and oversight.

Here are some thoughts:

This preprint is important to practicing psychologists because AI is already being widely adopted in school psychology (roughly half of practitioners report using it at work) and those using it specifically for report writing are saving an estimated six hours per week, meaningful relief in a profession burdened by high caseloads and heavy documentation demands. However, the study also reveals real risks: a subset of practitioners are entering student-identifying information into AI platforms in likely violation of FERPA, and some are using AI to assist with diagnostic and eligibility decisions that currently lack any empirical validation for AI use. The research draws a clear distinction between acceptable applications (drafting, editing, improving clarity) and problematic ones (interpreting test scores, rendering classifications), and while it is encouraging that 94% of users edit AI-generated content before use, the authors argue this oversight needs to be formalized rather than left to individual habit, making this study a timely and practical reference for any psychologist navigating AI in their work.

“Feasible but Fragile”: An inflection point for artificial intelligence in mental health care.

Clegg, K. (2025).

Journal of Medical Internet Research, 27, e89202.

https://doi.org/10.2196/89202

On November 18, 2025, a congressional hearing was held in Washington, DC, by the US House Energy and Commerce’s Subcommittee on Oversight and Investigations, examining the risks and benefits of artificial intelligence (AI) chatbots.

Marlynn Wei, MD, JD; Jennifer King, PhD; and John Torous, MBI, MD (director of digital psychiatry at Beth Israel Deaconess Medical School and associate professor of psychiatry at Harvard Medical School) provided expert testimony at the congressional hearing. I sat down with Torous to discuss his reflections on the future of AI in mental health.

An Inflection Point

Following on the heels of several lawsuits and mounting concerns about the safety of commercially available AI chatbots and their widespread “off-label” use as psychological support, November’s congressional hearing was somewhat anomalous—in a good way.

“I actually am optimistic,” says Torous, “because we never saw a congressional oversight committee form in the early days when social media came out or when apps came out or when VR [virtual reality] came out. It’s exciting to see a body like Congress taking the time and attention to try to understand what the issue is.”

He remarks that it’s a different trajectory than we’ve seen over the past 25 years of digital health innovation, one that simultaneously signals that “we’re seeing the end of AI exceptionalism in mental health.” It suggests that regulators are taking the risks seriously and that AI—whether purpose-built or used de facto—will not be exempt from the same scrutiny applied to other clinical tools.

It’s also a potential inflection point from the otherwise rapid and underregulated growth and proliferation of AI tools for mental health, including many chatbots whose safety and efficacy remain to be definitively established. The shape of the trajectory now—whether these tools succeed or fail to materialize their potential for improving mental health care—depends on what we do from here.

The information is here.

Key Takeaways

• The future of artificial intelligence (AI) tools in mental health care is at an inflection point; regulators are taking both the potential benefits and the risks of these tools seriously.

• Whether these tools succeed or fail to meet their potential for improving mental health care depends on the extent to which stakeholders are able to successfully seize the moment and collaborate on transparent, high-quality research; establish and incentivize safety and efficacy; adopt patient-centric benchmarks; and think beyond traditional therapeutic models.

After Automation

Dan Shipper

CEO of Every

Published May 21, 2026

There is a paradox at the heart of AI.

At Every, we’ve automated everything we can. We use Codex and Claude Code across coding, writing, design, customer service, and more. We alpha-test all of the new models from OpenAI, Anthropic, and Google before they come out. We are riding the exponential boom in model intelligence and automation as far and as fast as possible.

And yet it seems like, for us, there’s more human work to do than ever. We are a team of almost 30 people, and we haven’t fired all of our employees in favor of agents. We haven’t ditched software-as-a-service (SaaS) products in favor of vibe coded apps. We still hire humans to do customer service (with a lot of agent assistance), and we still hire human writers and editors and engineers.

(cut)

There’s no tipping point coming where things flip and the jobs are gone. The new reality is the opposite—the more we automate, the more expert human work there is to do.

Here’s why: AI commoditizes the residue of human expertise—whatever can be made explicit enough to train on. That collapses the value of default model output and creates demand for what’s different. Demand for what’s different is demand for human experts, even as we approach artificial general intelligence (AGI).

To understand why this is, we have to go beyond the graphs, and look at how AI is used for work today. That will help us see, in a more grounded light, the paradox—and its resolution.

The info is here.

Here are some thoughts:

This article is important because it challenges the simplistic narrative that AI will replace psychologists. It correctly argues that automation creates new demand for expert judgment and contextual sensitivity. Psychotherapy also requires relational presence and ethical accountability, as I have argued. Psychologists should take heart from this. Our core skills, including ethical reasoning, relationship building, sensitivity to diversity, and case conceptualization, are not being automated away. They are becoming more valuable.

However, psychologists should also be wary. The article underestimates the cognitive burden of supervising AI, ignores the need for formal training, and downplays the subtle ways that fluent but fallible AI can exploit human heuristics (known as automation bias). It also fails to address data privacy, algorithmic bias, deskilling, and the potential for AI to widen inequities in access to quality psychological care.

The practical takeaway is this. Psychologists should learn to use AI as a tool for editing, treatment planning possibilities, and case conceptualization, but they must remain accountable in all areas of practice. We should advocate for training programs that teach AI literacy as a core competency. We must  insist on AI tools that are transparent, privacy preserving, and validated on diverse populations. And we must remember that the heart of our work, the healing relationship between two human beings, lies entirely outside the space that any current AI can occupy. That is not a limitation to be overcome. It is the enduring reason human psychologists matter.

Can AI train the next generation of counsellors? Ethical challenges and opportunities

Athanassopoulos, L. (2026).

Journal of Psychology and AI, 2(1).

https://doi.org/10.1080/29974100.2025.2606673

Abstract

This study explores the feasibility of integrating artificial intelligence (AI) into counsellor training to enhance feedback quality and scalability. Using Natural Language Processing (NLP), simulated counselling transcripts were analysed across three therapeutic modalities: Person-Centred Therapy (PCT), Pluralistic Therapy, and Cognitive Behavioural Therapy (CBT). NLP, a branch of AI that combines computational linguistics and machine learning, enables systems to interpret and generate human language. The researcher, a qualified psychotherapist and educator, constructed simulated transcripts that were anonymously reviewed by colleagues practising in the respective modalities. The fine-tuned NLP system evaluated key therapeutic markers, including empathy, relational depth, cognitive restructuring, and responsiveness to client preferences. It also demonstrated safeguarding potential by detecting linguistic indicators of suicidal ideation. Findings suggest that AI has the potential to identify modality-specific therapeutic elements and provide consistent, actionable feedback aligned with training benchmarks. However, challenges remain in capturing nonverbal cues and ensuring adaptability across diverse contexts and practitioner styles. Ethical integration within reflective, practitioner-led training frameworks is essential. Overall, AI has the potential to augment human supervision by offering timely, structured, and scalable insights, provided its use is ethically governed and firmly embedded within reflective, practitioner-led training frameworks.

Here are some thoughts:

The study found that the AI showed genuine promise in identifying modality-specific therapeutic markers such as empathy and congruence in PCT, cognitive distortions in CBT, and collaborative decision-making in Pluralistic Therapy, and could provide structured, consistent feedback aligned with professional training standards. Notably, the system also demonstrated potential as a safeguarding tool by detecting linguistic indicators of suicidal ideation and emotional distress, areas where trainee counsellors may lack experience.

However, the study also highlights significant limitations. The AI struggled with nuanced emotional attunement, nonverbal cues, and the subtler relational dimensions of therapy that are central to effective practice. Ethical concerns around algorithmic bias, data privacy, cultural adaptability, and the risk of over-reliance on automated feedback are also raised. The author concludes that AI holds real transformative potential for counsellor education, but must function as a supplement to rather than a replacement for human supervision, and must be embedded within ethically governed, practitioner-led training frameworks.

ChatGPT Clinical Use in Mental Health Care: Scoping Review of Empirical evidence.

Balan, R., & Gumpel, T. P. (2025).

JMIR Mental Health, 12, e81204.

https://doi.org/10.2196/81204

Abstract

Background:

As mental health challenges continue to rise globally, there is an increasing interest in the use of GPT models, such as ChatGPT, in mental health care. A few months after its release, tens of thousands of users interacted with GPT-based therapy bots, with mental health support identified as the primary use case. ChatGPT offers scalable and immediate support through natural language processing capabilities, but their clinical applicability, safety, and effectiveness remain underexplored.

Objective:

This scoping review aims to provide a comprehensive overview of the main clinical applications of ChatGPT in mental health care, along with the existing empirical evidence for its performance.

Methods:

A systematic search was conducted in 8 electronic databases in April 2025 to identify primary studies. Eligible studies included primary research, reporting on the evaluation of a ChatGPT clinical application implemented for a mental health care–specific purpose.

Results:

In total, 60 studies were included in this scoping review. The results highlighted that most applications used generic ChatGPT and focused on the detection of mental health problems and counseling and treatment. At the same time, only a minority of studies investigated ChatGPT use in clinical decision facilitation and prognosis tasks. Most of the studies were prompt experiments, in which standardized text inputs—designed to mimic clinical scenarios, patient descriptions, or practitioner queries—are submitted to ChatGPT to evaluate its performance in mental health-related tasks. In terms of performance, ChatGPT shows good accuracy in binary diagnostic classification and differential diagnosis, simulating therapeutic conversation, providing psychoeducation, and conducting specific therapeutic strategies. However, ChatGPT has significant limitations, particularly with more complex clinical presentations and its overly pessimistic prognostic outputs. Nevertheless, overall, when compared to mental health experts or other artificial intelligence models, ChatGPT approximates or surpasses their performance in conducting various clinical tasks. Finally, custom ChatGPT use was associated with better performance, especially in counseling and treatment tasks.

Conclusions:

While ChatGPT offers promising capabilities for mental health screening, psychoeducation, and structured therapeutic interactions, its current limitations highlight the need for caution in clinical adoption. These limitations also underscore the need for rigorous evaluation frameworks, model refinement, and safety protocols before broader clinical integration. Moreover, the variability in performance across versions, tasks, and diagnostic categories also invites a more nuanced reflection on the conditions under which ChatGPT can be safely and effectively integrated into mental health settings.