Benchmarking Large Language Models Against Psychiatry Residents Using Traditional Institutional Assessments

Sethi, M. I. S. et al. (2026).

Indian Journal of Psychological Medicine, 

02537176261435658.

https://doi.org/10.1177/02537176261435658

Background:Artificial intelligence (AI) models demonstrate remarkable capabilities in healthcare applications, yet their performance compared to medical trainees in psychiatric education remains unexplored. This study evaluated the comparative performance of large language models (LLMs) against first-year psychiatry residents in standardized assessments at a premier Indian medical educational institute.

Methods:For this study, the already-scored answer sheets for Theory Papers I and II, as well as unmanned, non-interactive Objective Structured Clinical Examinations (OSCEs) with image-based tasks, from all 25 first-year psychiatry residents (March 2024 exam) were obtained from the examination section of the institute. The same question papers were then uploaded into three AI models (ChatGPT−3.5, Gemini Advanced, and Claude Sonnet). Four blinded faculty members evaluated the responses generated by the AI models. Final, the scores of the AI models and psychiatry residents were analyzed for comparison. Statistical analysis employed Kruskal–Wallis tests with post hoc Mann–Whitney U comparisons.

Results:AI models outperformed residents in theoretical assessments. In Paper I (theory), AI models achieved mean scores (standard deviation) of Claude Sonnet 67.88 (10.63), ChatGPT−3.5 70.38 (3.95), and Gemini Advanced 71.25 (3.86), compared to residents’ 58.0 (2.58). Paper II (theory) assessments showed even larger gaps, with AI models scoring Claude Sonnet 72.88 (3.77), ChatGPT−3.5 71.0 (3.56), and Gemini Advanced 69.63 (12.86), compared to residents’ 50.96 (2.49). OSCE performance patterns differed markedly. Paper I OSCEs showed equivalent performance: AI: 13.0; residents’: 13.16 (1.49), while Paper II OSCEs revealed variable results: Claude Sonnet excelled at 20.0 (1.41), but ChatGPT−3.5 underperformed at 15.0 (0.50), compared to residents at 16.6 (1.55). Inter-rater reliability coefficients remained excellent ( intraclass correlation coefficients [ICC]: 0.810–0.934).

Conclusions:While AI demonstrated superior theoretical knowledge, equivalent or variable practical skills performance reveals fundamental limitations in clinical reasoning and contextual understanding. These findings necessitate reconceptualizing psychiatric education to emphasize uniquely human competencies while leveraging AI’s capabilities for knowledge synthesis.

Here are some thoughts:

This study compared three large language models (LLMs) to first-year psychiatry residents using real institutional exams in India. The LLMs consistently outperformed residents on theoretical assessments (by 17–43%) but showed equivalent or inconsistent performance on practical OSCEs, revealing critical gaps in clinical reasoning and cultural contextualization. The authors conclude that psychiatric education should shift focus toward uniquely human skills like empathy and judgment, while using AI as a tool for knowledge synthesis.

Adversarial AI reveals mechanisms and treatments for disorders of consciousness

Toker, D. et al. (2026).

Nature Neuroscience, 29(4), 964–977.

https://doi.org/10.1038/s41593-026-02220-4

Abstract

Understanding disorders of consciousness (DOC) remains one of the most challenging problems in neuroscience, hindered by the lack of experimental models for probing mechanisms or testing interventions. Here, to address this, we introduce a generative adversarial artificial intelligence (AI) framework that pits deep neural networks—trained to detect consciousness across more than 680,000 ten-second neuroelectrophysiology samples and validated on 565 patients, healthy volunteers and animals—against interpretable, machine learning-driven neural field models. This adversarial architecture produces biologically realistic simulations of both conscious and comatose brains that recapitulate empirical neurophysiological features across humans, monkeys, rats and bats. Without explicit programming, the AI model retrodicts known DOC responses to brain stimulation and generates testable predictions about the mechanisms of unconsciousness. Two such predictions are validated here: selective disruption of the basal ganglia indirect pathway, supported by diffusion magnetic resonance imaging in 51 patients with DOC, and increased cortical inhibitory-to-inhibitory synaptic coupling, supported by RNA sequencing of resected brain tissue from 6 human patients with coma and a rat stroke model. The model also identifies high-frequency stimulation of the subthalamic nucleus as a promising intervention for DOC, supported by electrophysiological data from human patients. This work introduces an AI framework for causal inference and therapeutic discovery in consciousness research, as well as in complex systems more broadly.

Here are some thoughts:

This work gives psychologists a more concrete, brain-circuit-level understanding of why coma and related states happen, and points to a specific, testable new treatment approach, moving the field beyond “we don’t know what’s happening inside” toward identifiable mechanisms that may one day guide rehabilitation and family education.

Automation bias: a systematic review of frequency, effect mediators, and mitigators

Goddard, K., Roudsari, A., & Wyatt, J. C. (2011).

Journal of the American Medical

Informatics Association, 19(1), 121–127.

https://doi.org/10.1136/amiajnl-2011-000089

Abstract

Automation bias (AB)—the tendency to over-rely on automation—has been studied in various academic fields. Clinical decision support systems (CDSS) aim to benefit the clinical decision-making process. Although most research shows overall improved performance with use, there is often a failure to recognize the new errors that CDSS can introduce. With a focus on healthcare, a systematic review of the literature from a variety of research fields has been carried out, assessing the frequency and severity of AB, the effect mediators, and interventions potentially mitigating this effect. This is discussed alongside automation-induced complacency, or insufficient monitoring of automation output. A mix of subject specific and freetext terms around the themes of automation, human–automation interaction, and task performance and error were used to search article databases. Of 13 821 retrieved papers, 74 met the inclusion criteria. User factors such as cognitive style, decision support systems (DSS), and task specific experience mediated AB, as did attitudinal driving factors such as trust and confidence. Environmental mediators included workload, task complexity, and time constraint, which pressurized cognitive resources. Mitigators of AB included implementation factors such as training and emphasizing user accountability, and DSS design factors such as the position of advice on the screen, updated confidence levels attached to DSS output, and the provision of information versus recommendation. By uncovering the mechanisms by which AB operates, this review aims to help optimize the clinical decision-making process for CDSS developers and healthcare practitioners.

Here are some thoughts:

This systematic review examines the frequency, mediators, and mitigators of automation bias, which is the tendency for users to over rely on automated decision support systems as a heuristic replacement for vigilant information seeking and processing. The authors reviewed 74 studies across healthcare, aviation, and human computer interaction fields and found that automation bias is a robust effect, with a meta analysis showing that erroneous clinical decision support system advice increased the risk of incorrect decisions by 26%. Key mediators include user factors such as cognitive style, trust, confidence, and task specific experience, as well as environmental factors like workload, task complexity, and time pressure that strain cognitive resources. Mitigators include increasing user accountability, providing training, updating confidence levels alongside advice, positioning advice less prominently on screen, and presenting information rather than direct recommendations. The review concludes that automation bias and the related concept of automation induced complacency represent distinct but overlapping attentional phenomena that can introduce new errors even when decision support systems improve overall performance, highlighting the need for careful design and implementation strategies.

Transforming clinical reasoning—the role of AI in supporting human cognitive limitations

Greengrass C. J. (2026).

Frontiers in digital health, 7, 1715440.

https://doi.org/10.3389/fdgth.2025.1715440

Abstract

Clinical reasoning is foundational to medical practice, requiring clinicians to synthesise complex information, recognise patterns, and apply causal reasoning to reach accurate diagnoses and guide patient management. However, human cognition is inherently limited by factors such as limitations in working memory capacity, constraints in cognitive load, a general reliance on heuristics; with an inherent vulnerability to biases including anchoring, availability bias, and premature closure. Cognitive fatigue and cognitive overload, particularly apparent in high-pressure environments, further compromise diagnostic accuracy and efficiency. Artificial intelligence (AI) presents a transformative opportunity to overcome these limitations by supplementing and supporting decision-making. With AI's advanced computational capabilities, these systems can analyse large datasets, detect subtle or atypical patterns, and provide accurate evidence-based diagnoses. Furthermore, by leveraging machine learning and probabilistic modelling, AI reduces dependence on incomplete heuristics and potentially mitigates cognitive biases. It also ensures consistent performance, unaffected by fatigue or information overload. These attributes likely make AI an invaluable tool for enhancing the accuracy and efficiency of diagnostic reasoning. Through a narrative review, this article examines the cognitive limitations inherent in diagnostic reasoning and considers how AI can be positioned as a collaborative partner in addressing them. Drawing on the concept of Mutual Theory of Mind, the author identifies a set of indicators that should inform the design of future frameworks for human–AI interaction in clinical decision-making. These highlight how AI could dynamically adapt to human reasoning states, reduce bias, and promote more transparent and adaptive diagnostic support in high-stakes clinical environments.

Here are some thoughts:

This article examines how artificial intelligence can support clinical diagnostic reasoning by compensating for inherent human cognitive limitations such as limited working memory capacity, cognitive load, reliance on heuristics, and susceptibility to biases like anchoring and premature closure. The author integrates cognitive psychology concepts including dual process theory (System 1 intuitive pattern recognition versus System 2 analytical reasoning), cognitive load theory, and Bayesian reasoning to analyze how AI systems can reduce cognitive burden, provide external schema repositories, offer transparent explainable outputs, and support metacognitive monitoring. While AI offers advantages in processing vast data streams, maintaining multiple hypotheses, and performing consistently without fatigue, the review acknowledges current limitations of large language models including poor probabilistic reasoning and potential for algorithmic or transferred bias. The article concludes that AI should function as a collaborative partner within a Mutual Theory of Mind framework, enhancing rather than replacing human judgment, provided that ethical standards and clinician training keep pace with technological development.

Using AI-Based Virtual simulated patients for training in psychopathological interviewing: Cross-Sectional Observational study.

García-Torres, D., et al. (2025).

JMIR Medical Education, 11, e78857.

https://doi.org/10.2196/78857

Abstract

Background:

Virtual simulated patients (VSPs) powered by generative artificial intelligence (GAI) offer a promising tool for training clinical interviewing skills; yet, little is known about how different system- and user-level variables shape students’ perceptions of these interactions.

Objective:

We aim to study psychology students’ perceptions of GAI-driven VSPs and examine how demographic factors, system parameters, and interaction characteristics influence such perceptions.

Methods:

We conducted a total of 1832 recorded interactions involving 156 psychology students with 13 GAI-generated VSPs configured with varying temperature settings (0.1, 0.5, 0.9). For each student, we collected age and sex; for each interview, we recorded interview length (total number of question–answer turns), number of connectivity failures, the specific VSP consulted, and the model temperature. After every interview, students provided a 1-10 global rating and open-ended comments regarding strengths and areas for improvement. At the end of the training sequence, they also reported perceived improvement in diagnostic ability. Statistical analyses assessed the influence of different variables on global ratings: demographics, interaction-level data, and GAI temperature setting. Sentiment analysis was conducted to evaluate the VSPs’ clinical realism.

Results:

Statistical analysis showed that female students rated the tool significantly higher (mean rating 9.25/10) than male students (mean rating 8.94/10; Kruskal-Wallis test, H=8.7; P=.003). On the other side, no significant correlation was found between global rating and age (r=0.02, 95% CI –0.03 to 0.06; P=.42), interview length (r=0.04, 95% CI –0.2 to 0.10; P=.18), or frequency of participation (Kruskal-Wallis test, H=4.62; P=.20). A moderate negative correlation emerged between connectivity failures and ratings (r=–0.26, 95% CI –0.41 to –0.10; P=.002). Temperature settings significantly influenced ratings (Kruskal-Wallis test, H=6.93; P=.03; η²=0.02), with higher scores at temperature 0.9 compared with 0.1 (Dunn’s test, P=.04). Concerning learning outcomes, self-perceived improvement in diagnostic ability was reported by 94% (94/100) of students; however, final practical examination scores (mean 6.67, SD 1.42) did not differ significantly from those of the previous cohort without VSP training (mean 6.42, SD 1.56). Sentiment analysis indicated predominantly negative sentiment in GAI responses (median negativity 0.8903, IQR 0.306-0.961), consistent with clinical realism.

Conclusions:

GAI-driven VSPs were well-received by psychology students, with student gender and system-level variables (particularly temperature settings and connection stability) shaping user evaluations. Although participants perceived the training as beneficial for their diagnostic skills, objective examination performance did not significantly differ from the previous cohort. However, lack of randomization limits the generalization of the results obtained, and further experiments are required.

Here are some thoughts:

This study is important because it demonstrates a promising application of AI in clinical training, using generative AI-powered virtual simulated patients to help psychology students practice psychopathological interviewing in a safe, low-stakes environment. The platform was highly rated by students and 94% reported meaningful improvement in their ability to identify clinically relevant symptoms. Higher AI temperature settings, which produce more natural and varied responses, were associated with greater student satisfaction, while connectivity failures reduced ratings, underscoring the importance of technical reliability. Although students found VSP-based sessions more challenging than traditional paper cases, final exam scores were comparable between cohorts, suggesting the AI simulation provides a more realistic learning experience rather than a less effective one. For practicing psychologists and educators, this study offers early empirical support for integrating AI-driven patient simulation into clinical training, while highlighting the need for randomized studies and careful calibration of AI parameters before broad adoption.

The moon, the ghetto and artificial intelligence: Reducing systemic racism in computational algorithms.

Fountain, J. (2022).

Government Information Quarterly, 39(2), 101645.

https://doi.org/10.1016/j.giq.2021.101645 

Abstract

Computational algorithms and automated decision making systems that include them offer potential to improve public policy and organizations. But computational algorithms based on biased data encode those biases into algorithms, models and their outputs. Systemic racism is institutionalized bias with respect to race, ethnicity and related attributes. Such bias is located in data that encode the results and outputs of decisions that have been discriminatory, in procedures and processes that may intentionally or unintentionally disadvantage people based on race, and in policies that may discriminate by race. Computational algorithms may exacerbate systemic racism if they are not designed, developed, and used–that is, enacted–with attention to identifying and remedying bias specific to race. Advancing social equity in digital governance requires systematic, ongoing efforts to assure that automated decision making systems, and their enactment in complex public organizational arrangements, are free from bias.

Highlights

• Computational algorithms are powerful tools but may replicate biases.

• Biases, including systemic racism, in underlying data bias algorithms

• Automated decision making systems that discriminate harm people.

• Careful scrutiny of data, processes, variables and algorithms may reduce bias.

AI in psychoeducational assessment: a study of report generation.

Farmer, R. L., et al. (2025).

Online preprint. 

https://doi.org/10.13140/RG.2.2.31345.95844

Abstract

Artificial intelligence (AI) is poised to reshape school psychology, with report writing as a primary area of impact. A national sample of school psychologists using AI at work (n = 100) reported on its role in documentation. Of those using AI for report writing (n = 45), most applied it to discrete tasks such as rewriting sections for clarity (69%) or generating recommendations (67%). Far fewer used AI for interpretive or diagnostic purposes, and none relied on it to generate entire reports. Nearly all users (94%) edited AI-generated content before use. On average, practitioners who used AI for report writing saved 6.3 hours per week (95% CI: 4.6–8.1), more than double the savings of those using AI for other tasks (≈3 hours). AI shows promise for reducing documentation burden and reclaiming time for direct services, though its use raises ethical concerns requiring further guidance and oversight.

Here are some thoughts:

This preprint is important to practicing psychologists because AI is already being widely adopted in school psychology (roughly half of practitioners report using it at work) and those using it specifically for report writing are saving an estimated six hours per week, meaningful relief in a profession burdened by high caseloads and heavy documentation demands. However, the study also reveals real risks: a subset of practitioners are entering student-identifying information into AI platforms in likely violation of FERPA, and some are using AI to assist with diagnostic and eligibility decisions that currently lack any empirical validation for AI use. The research draws a clear distinction between acceptable applications (drafting, editing, improving clarity) and problematic ones (interpreting test scores, rendering classifications), and while it is encouraging that 94% of users edit AI-generated content before use, the authors argue this oversight needs to be formalized rather than left to individual habit, making this study a timely and practical reference for any psychologist navigating AI in their work.

“Feasible but Fragile”: An inflection point for artificial intelligence in mental health care.

Clegg, K. (2025).

Journal of Medical Internet Research, 27, e89202.

https://doi.org/10.2196/89202

On November 18, 2025, a congressional hearing was held in Washington, DC, by the US House Energy and Commerce’s Subcommittee on Oversight and Investigations, examining the risks and benefits of artificial intelligence (AI) chatbots.

Marlynn Wei, MD, JD; Jennifer King, PhD; and John Torous, MBI, MD (director of digital psychiatry at Beth Israel Deaconess Medical School and associate professor of psychiatry at Harvard Medical School) provided expert testimony at the congressional hearing. I sat down with Torous to discuss his reflections on the future of AI in mental health.

An Inflection Point

Following on the heels of several lawsuits and mounting concerns about the safety of commercially available AI chatbots and their widespread “off-label” use as psychological support, November’s congressional hearing was somewhat anomalous—in a good way.

“I actually am optimistic,” says Torous, “because we never saw a congressional oversight committee form in the early days when social media came out or when apps came out or when VR [virtual reality] came out. It’s exciting to see a body like Congress taking the time and attention to try to understand what the issue is.”

He remarks that it’s a different trajectory than we’ve seen over the past 25 years of digital health innovation, one that simultaneously signals that “we’re seeing the end of AI exceptionalism in mental health.” It suggests that regulators are taking the risks seriously and that AI—whether purpose-built or used de facto—will not be exempt from the same scrutiny applied to other clinical tools.

It’s also a potential inflection point from the otherwise rapid and underregulated growth and proliferation of AI tools for mental health, including many chatbots whose safety and efficacy remain to be definitively established. The shape of the trajectory now—whether these tools succeed or fail to materialize their potential for improving mental health care—depends on what we do from here.

The information is here.

Key Takeaways

• The future of artificial intelligence (AI) tools in mental health care is at an inflection point; regulators are taking both the potential benefits and the risks of these tools seriously.

• Whether these tools succeed or fail to meet their potential for improving mental health care depends on the extent to which stakeholders are able to successfully seize the moment and collaborate on transparent, high-quality research; establish and incentivize safety and efficacy; adopt patient-centric benchmarks; and think beyond traditional therapeutic models.

After Automation

Dan Shipper

CEO of Every

Published May 21, 2026

There is a paradox at the heart of AI.

At Every, we’ve automated everything we can. We use Codex and Claude Code across coding, writing, design, customer service, and more. We alpha-test all of the new models from OpenAI, Anthropic, and Google before they come out. We are riding the exponential boom in model intelligence and automation as far and as fast as possible.

And yet it seems like, for us, there’s more human work to do than ever. We are a team of almost 30 people, and we haven’t fired all of our employees in favor of agents. We haven’t ditched software-as-a-service (SaaS) products in favor of vibe coded apps. We still hire humans to do customer service (with a lot of agent assistance), and we still hire human writers and editors and engineers.

(cut)

There’s no tipping point coming where things flip and the jobs are gone. The new reality is the opposite—the more we automate, the more expert human work there is to do.

Here’s why: AI commoditizes the residue of human expertise—whatever can be made explicit enough to train on. That collapses the value of default model output and creates demand for what’s different. Demand for what’s different is demand for human experts, even as we approach artificial general intelligence (AGI).

To understand why this is, we have to go beyond the graphs, and look at how AI is used for work today. That will help us see, in a more grounded light, the paradox—and its resolution.

The info is here.

Here are some thoughts:

This article is important because it challenges the simplistic narrative that AI will replace psychologists. It correctly argues that automation creates new demand for expert judgment and contextual sensitivity. Psychotherapy also requires relational presence and ethical accountability, as I have argued. Psychologists should take heart from this. Our core skills, including ethical reasoning, relationship building, sensitivity to diversity, and case conceptualization, are not being automated away. They are becoming more valuable.

However, psychologists should also be wary. The article underestimates the cognitive burden of supervising AI, ignores the need for formal training, and downplays the subtle ways that fluent but fallible AI can exploit human heuristics (known as automation bias). It also fails to address data privacy, algorithmic bias, deskilling, and the potential for AI to widen inequities in access to quality psychological care.

The practical takeaway is this. Psychologists should learn to use AI as a tool for editing, treatment planning possibilities, and case conceptualization, but they must remain accountable in all areas of practice. We should advocate for training programs that teach AI literacy as a core competency. We must  insist on AI tools that are transparent, privacy preserving, and validated on diverse populations. And we must remember that the heart of our work, the healing relationship between two human beings, lies entirely outside the space that any current AI can occupy. That is not a limitation to be overcome. It is the enduring reason human psychologists matter.