Ensuring safety and quality

Frameworks and best practices for ensuring quality, safety, and effectiveness of AI tools.

A longitudinal analysis of declining medical safety messaging in generative AI models

Sharma S, Alaa AM, Daneshjou R. npj Digital Medicine. 2025;8:592.

This study explains the decline of medical disclaimers in large language models (LLMs) and vision-language models (VLMs) between 2022 and 2025. Researchers analyzed responses to different medical questions and medical images, including mammograms, chest X-rays, and dermatology images. The findings show that disclaimer rates in text-based medical responses decreased from 26.3% in 2022 to 0.97% in 2025, while disclaimer use in medical image interpretation dropped from 19.6% to 1.05%. The authors conclude that as AI systems become more accurate and conversational, safety messaging has declined significantly, raising concerns about patient trust, misinformation, and the need for adaptive medical safeguards in healthcare AI systems.

ChatGPT Health Performance in a Structured Test of Triage Recommendations

Ramaswamy A, Tyagi A, Hugo H, et al. Nat Med. 2026;32:1671–1675.

This study evaluates the performance of ChatGPT Health using clinician-authored clinical vignettes across different medical domains. Researchers found that the AI system performed best with moderate-acuity cases, but also had a high safety concern at clinical extremes. ChatGPT Health under-triaged 52% of emergency conditions that includes diabetic ketoacidosis and respiratory failure, but also overtriaged many cases that were not urgent. They also found that anchoring bias, such as having reassurance from family, shifted recommendations towards less urgent care. This raises concerns about the safety and reliability of consumer-facing AI triage systems.

A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools

Patel H, Crusco S, Hansen D, et al. J Gen Intern Med. 2026.

This systematic review evaluates published literature on the performance of Epic’s CDS tools to compare real-world performance to vendor-reported metrics. The authors analyzed 22 articles which included the Epic Deterioration Index (EDI), Epic Sepsis Model (ESM), Epic Unplanned Readmission Model (EURM), Epic End-of-Life Care Index (EEOL-CI), and Epic Risk of Patient No-Show (ERPNS). They found that the models had modest real world performances. Additionally, results demonstrated a lower accuracy compared to what Epic reported for the ESM, EURM, EEOL-CI models. They also found that all models reflected high heterogeneity indicating the need for local evaluation before deployment.

MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care

Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. J Am Med Inform Assoc. 2020;27(12):2011-2015.

This article proposes MINIMAR (MINimum Information for Medical AI Reporting), a standard to accurately report on AI in health care which promotes transparency to address bias and unintended consequences. This standard includes four requirements: 1) information on the population that the training data was collected form, 2) detailed information of patient characteristics (race, socioeconomics, etc), 3) documentation of the model’s architecture and development, 4) report of the model evaluation, optimization, and validation to enable resource sharing. The authors emphasize the need for transparency by model developers in order to avoid bias and support the reproducibility of results.

Testing and Evaluation of Health Care Applications of Large Language Models

Bedi S, Liu Y, Orr-Ewing L, et al. JAMA. 2025;333(4):319–328.

This systematic review categorizes generative AI evaluation in healthcare. The authors found that among the reviewed studies only 5% evaluated and tested LLMs using real patient care data. The results show that administrative tasks like clinical note-taking and generating clinical referral were evaluated far less frequently. They identified fewer studies evaluating fairness, bias and robustness compared to other dimensions, with bias being evaluated only in 15.8% of the studies. For NLP and NLU tasks, most studies evaluated the LLM performance using question-answering tasks. The authors emphasized the need for future evaluations to include a wider range of tasks and specialties, use clinical data, and adopt standardized applications and metrics.

Standing on FURM Ground : A Framework for Evaluating Fair, Useful, and Reliable AI Models in Health Care Systems

Callahan A, McElfresh D, Banda JM, et al. Standing on FURM Ground. NEJM Catalyst Innov Care Deliv. 2024;5(10). doi:10.1056/CAT.24.0131

Stanford Health Care developed a process to assess the potential usefulness, financial impact, and ethical considerations of an AI model in clinical or operational settings. Called the FURM (Fair, Useful, and Reliable AI Models) assessment, this tool guides users through three stages that support the evaluation of AI models at various stages of development. The first stage asks what problem the AI model solves and why it should be used. The second stage determines how the AI model can be deployed in a healthcare system. The last stage evaluates the observed impact during and after deployment.