Which explanations do clinicians prefer? A comparative evaluation of XAI understandability and actionability in predicting the need for hospitalization.

Publisher: BioMed Central Country of Publication: England NLM ID: 101088682 Publication Model: Electronic Cited Medium: Internet ISSN: 1472-6947 (Electronic) Linking ISSN: 14726947 NLM ISO Abbreviation: BMC Med Inform Decis Mak Subsets: MEDLINE

Original Publication: London : BioMed Central, [2001-

COVID-19*/therapy ; COVID-19*/diagnosis ; Attitude of Health Personnel* ; Hospitalization*/statistics & numerical data ; Machine Learning*; Humans ; Male ; Female ; Italy ; Adult ; Surveys and Questionnaires ; Middle Aged ; SARS-CoV-2

Background: This study aims to address the gap in understanding clinicians' attitudes toward explainable AI (XAI) methods applied to machine learning models using tabular data, commonly found in clinical settings. It specifically explores clinicians' perceptions of different XAI methods from the ALFABETO project, which predicts COVID-19 patient hospitalization based on clinical, laboratory, and chest X-ray at time of presentation to the Emergency Department. The focus is on two cognitive dimensions: understandability and actionability of the explanations provided by explainable-by-design and post-hoc methods.
Methods: A questionnaire-based experiment was conducted with 10 clinicians from the IRCCS Policlinico San Matteo Foundation in Pavia, Italy. Each clinician evaluated 10 real-world cases, rating predictions and explanations from three XAI tools: Bayesian networks, SHapley Additive exPlanations (SHAP), and AraucanaXAI. Two cognitive statements for each method were rated on a Likert scale, as well as the agreement with the prediction. Two clinicians answered the survey during think-aloud interviews.
Results: Clinicians demonstrated generally positive attitudes toward AI, but high compliance rates (86% on average) indicate a risk of automation bias. Understandability and actionability are positively correlated, with SHAP being the preferred method due to its simplicity. However, the perception of methods varies according to specialty and expertise.
Conclusions: The findings suggest that SHAP and AraucanaXAI are promising candidates for improving the use of XAI in clinical decision support systems (DSSs), highlighting the importance of clinicians' expertise, specialty, and setting on the selection and development of supportive XAI advice. Finally, the study provides valuable insights into the design of future XAI DSSs.
(© 2025. The Author(s).)

Declarations. Ethics approval and consent to participate: The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of Fondazione IRCCS Policlinico San Matteo (Protocol “P-20200072983”, approved on 30 September 2020). All study participants gave their informed consent to be included in the study and are co-authors of the present manuscript. Consent for publication: Not applicable. Competing interests: The authors declare no competing interests.

Muehlematter UJ, Bluethgen C, Vokinger KN. FDA-cleared artificial intelligence and machine learning-based medical devices and their 510(k) predicate networks. Lancet Digit Health. 2023;5(9):e618–26. https://doi.org/10.1016/S2589-7500(23)00126-7 . (PMID: 10.1016/S2589-7500(23)00126-737625896)
Joshi G, Jain A, Araveeti SR, Adhikari S, Garg H, Bhandari M. FDA-Approved artificial intelligence and machine learning (AI/ML)-Enabled medical devices: an updated landscape. Electronics. 2024;13(3):498. https://doi.org/10.3390/electronics13030498 . (PMID: 10.3390/electronics13030498)
Combi C, Amico B, Bellazzi R, et al. A manifesto on explainability for artificial intelligence in medicine. Artif Intell Med. 2022;133:102423. https://doi.org/10.1016/j.artmed.2022.102423 . (PMID: 10.1016/j.artmed.2022.10242336328669)
Longo L, Brcic M, Cabitza F, et al. Explainable artificial intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions. Inform Fusion. 2024;106:102301. https://doi.org/10.1016/j.inffus.2024.102301 . (PMID: 10.1016/j.inffus.2024.102301)
Adadi A, Berrada M. Peeking inside the Black-Box: A survey on explainable artificial intelligence (XAI). IEEE Access. 2018;6:52138–60. https://doi.org/10.1109/ACCESS.2018.2870052 . (PMID: 10.1109/ACCESS.2018.2870052)
Kop M. EU Artificial Intelligence Act: The European Approach to AI. Social Science Research Network; 2021. Accessed December 19, 2021. https://papers.ssrn.com/abstract=3930959.
Cabitza F, Campagner A, Ronzio L, et al. Rams, hounds and white boxes: investigating human–AI collaboration protocols in medical diagnosis. Artif Intell Med. 2023;138:102506. https://doi.org/10.1016/j.artmed.2023.102506 . (PMID: 10.1016/j.artmed.2023.10250636990586)
Cabitza F, Fregosi C, Campagner A, Natali C. Explanations considered harmful: the impact of misleading explanations on accuracy in hybrid Human-AI decision making. Commun Comput Inform Sci. 2024;2156 CCIS:255–69. https://doi.org/10.1007/978-3-031-63803-9_14 . (PMID: 10.1007/978-3-031-63803-9_14)
Cabitza F, Campagner A, Malgieri G, et al. Quod Erat demonstrandum? - Towards a typology of the concept of explanation for the design of explainable AI. Expert Syst Appl. 2023;213:118888. https://doi.org/10.1016/j.eswa.2022.118888 . (PMID: 10.1016/j.eswa.2022.118888)
Lundberg SM, Lee SI. A Unified Approach to Interpreting Model Predictions. In: Advances in Neural Information Processing Systems. Curran Associates, Inc.; 2017. Accessed October 17, 2024;30. https://papers.nips.cc/paper_files/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html.
Parimbelli E, Buonocore TM, Nicora G, Michalowski W, Wilk S, Bellazzi R. Why did AI get this one wrong? — Tree-based explanations of machine learning model predictions. Artif Intell Med. 2023;135:102471. https://doi.org/10.1016/j.artmed.2022.102471 . (PMID: 10.1016/j.artmed.2022.10247136628785)
Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1(5):206–15. https://doi.org/10.1038/s42256-019-0048-x . (PMID: 10.1038/s42256-019-0048-x356030109122117)
Lambert SI, Madi M, Sopka S, et al. An integrative review on the acceptance of artificial intelligence among healthcare professionals in hospitals. Npj Digit Med. 2023;6(1):1–14. https://doi.org/10.1038/s41746-023-00852-5 . (PMID: 10.1038/s41746-023-00852-5)
Chen M, Zhang B, Cai Z, et al. Acceptance of clinical artificial intelligence among physicians and medical students: A systematic review with cross-sectional survey. Front Med (Lausanne). 2022;9:990604. https://doi.org/10.3389/fmed.2022.990604 . (PMID: 10.3389/fmed.2022.99060436117979)
Shamszare H, Choudhury A. Clinicians’ perceptions of artificial intelligence: focus on workload, risk, trust, clinical decision making, and clinical integration. Healthcare. 2023;11(16):2308. https://doi.org/10.3390/healthcare11162308 . (PMID: 10.3390/healthcare111623083762850610454426)
Sivaraman V, Bukowski LA, Levin J, Kahn JM, Perer A, Ignore. Trust, or negotiate: Understanding clinician acceptance of AI-Based treatment recommendations in health care. Published Online January. 2023;31. https://doi.org/10.48550/arXiv.2302.00096 .
Komorowski M, Celi LA, Badawi O, Gordon AC, Faisal AA. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nat Med. 2018;24(11):1716–20. https://doi.org/10.1038/s41591-018-0213-5 . (PMID: 10.1038/s41591-018-0213-530349085)
Du Y, Antoniadi AM, McNestry C, McAuliffe FM, Mooney C. The role of XAI in Advice-Taking from a clinical decision support system: A comparative user study of feature Contribution-Based and Example-Based explanations. Appl Sci. 2022;12(20):10323. https://doi.org/10.3390/app122010323 . (PMID: 10.3390/app122010323)
Cabitza F, Campagner A, Natali C, Parimbelli E, Ronzio L, Cameli M. Painting the black box white: experimental findings from applying XAI to an ECG reading setting. Mach Learn Knowl Extr. 2023;5(1):269–86. https://doi.org/10.3390/make5010017 . (PMID: 10.3390/make5010017)
Evans T, Retzlaff CO, Geißler C, et al. The explainability paradox: challenges for xAI in digital pathology. Future Generation Comput Syst. 2022;133:281–96. https://doi.org/10.1016/j.future.2022.03.009 . (PMID: 10.1016/j.future.2022.03.009)
Karagoz G, van Kollenburg G, Ozcelebi T, Meratnia N. Evaluating How Explainable AI Is Perceived in the Medical Domain: A Human-Centered Quantitative Study of XAI in Chest X-Ray Diagnostics. In: Chen H, Zhou Y, Xu D, Vardhanabhuti VV, eds. Trustworthy Artificial Intelligence for Healthcare. Springer Nature Switzerland; 2024:92–108. https://doi.org/10.1007/978-3-031-67751-9_8.
Hoffman RR, Mueller ST, Klein G, Litman J. Measures for explainable AI: explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance. Front Comput Sci. 2023;5. https://doi.org/10.3389/fcomp.2023.1096257 .
Catalano M, Bortolotto C, Nicora G, et al. Performance of an AI algorithm during the different phases of the COVID pandemics: what can we learn from the AI and vice versa. Eur J Radiol Open. 2023;11:100497. https://doi.org/10.1016/j.ejro.2023.100497 . (PMID: 10.1016/j.ejro.2023.1004973736077010278371)
Nicora G, Catalano M, Bortolotto C, et al. Bayesian networks in the management of hospital admissions: A comparison between explainable AI and black box AI during the pandemic. J Imaging. 2024;10(5):117. https://doi.org/10.3390/jimaging10050117 . (PMID: 10.3390/jimaging100501173878657111122655)
Arrieta AB, Díaz-Rodríguez N, Del Ser J, et al. Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Published Online Dec. 2019;26. https://doi.org/10.48550/arXiv.1910.10045 .
Famiglini L, Campagner A, Barandas M, La Maida GA, Gallazzi E, Cabitza F. Evidence-based XAI: an empirical approach to design more effective and explainable decision support systems. Comput Biol Med. 2024;170:108042. https://doi.org/10.1016/j.compbiomed.2024.108042 . (PMID: 10.1016/j.compbiomed.2024.10804238308866)
Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989;13(3):319–40. https://doi.org/10.2307/249008 . (PMID: 10.2307/249008)
Nourani M, King J, Ragan E. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. Proc AAAI Conf Hum Comput Crowdsourcing. 2020;8:112–21. https://doi.org/10.1609/hcomp.v8i1.7469 . (PMID: 10.1609/hcomp.v8i1.7469)
Kim A, Yang M, Zhang J. When algorithms Err: differential impact of early vs. Late errors on users’ reliance on algorithms. Published Online July. 2020;1. https://doi.org/10.2139/ssrn.3691575 .
Gamborg ML, Mehlsen M, Paltved C, Vetter SS, Musaeus P. Clinical decision-making and adaptive expertise in residency: a think-aloud study. BMC Med Educ. 2023;23(1):22. https://doi.org/10.1186/s12909-022-03990-8 . (PMID: 10.1186/s12909-022-03990-8366356699835279)
Thompson S, Moorley C, Barratt J. A comparative study on the clinical decision-making processes of nurse practitioners vs. medical Doctors using scenarios in a secondary care environment. J Adv Nurs. 2017;73(5):1097–110. https://doi.org/10.1111/jan.13206 . (PMID: 10.1111/jan.1320627859497)
Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Hum Factors. 2015;2(2):e14. https://doi.org/10.2196/humanfactors.4537 . (PMID: 10.2196/humanfactors.4537270255404797671)
Anjara SG, Janik A, Dunford-Stenger A, et al. Examining explainable clinical decision support systems with think aloud protocols. PLoS ONE. 2023;18(9):e0291443. https://doi.org/10.1371/journal.pone.0291443 . (PMID: 10.1371/journal.pone.02914433770813510501571)
Noushad B, Van Gerven PWM, de Bruin ABH. Twelve tips for applying the think-aloud method to capture cognitive processes. Med Teach. 2024;46(7):892–7. https://doi.org/10.1080/0142159X.2023.2289847 . (PMID: 10.1080/0142159X.2023.228984738071621)
Herm LV. Impact of explainable AI on cognitive load: insights from an empirical study. Published Online April. 2023;18. https://doi.org/10.48550/arXiv.2304.08861 .
Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–3. (PMID: 15883903)
Bertrand A, Belloum R, Eagan JR, Maxwell W. How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. ACM; 2022:78–91. https://doi.org/10.1145/3514094.3534164.
Kim J, Maathuis H, Sent D. Human-centered evaluation of explainable AI applications: a systematic review. Front Artif Intell. 2024;7:1456486. https://doi.org/10.3389/frai.2024.1456486 . (PMID: 10.3389/frai.2024.14564863948415411525002)
Sloman S, Zemla JC, Lagnado D, Bechlivanidis C, Hemmatian B. Are humans intuitive philosophers?? In: Grimm SR, editor. Varieties of Understanding. 1st ed. Oxford University PressNew York; 2019. pp. 231–50. https://doi.org/10.1093/oso/9780190860974.003.0012 .
Bussone A, Stumpf S, O’Sullivan D. The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. In: 2015 International Conference on Healthcare Informatics. 2015:160–169. https://doi.org/10.1109/ICHI.2015.26.
Lötsch J, Kringel D, Ultsch A. Explainable artificial intelligence (XAI) in biomedicine: making AI decisions trustworthy for physicians and patients. BioMedInformatics. 2022;2(1):1–17. https://doi.org/10.3390/biomedinformatics2010001 . (PMID: 10.3390/biomedinformatics2010001)
Vasilyeva N, Wilkenfeld D, Lombrozo T. Goals affect the perceived quality of explanations. Cognitive Science. Published online. 2015. Accessed April 9, 2025. https://www.semanticscholar.org/paper/Goals-Affect-the-Perceived-Quality-of-Explanations-Vasilyeva-Wilkenfeld/6493889436a14776682118ba668e0c0c2bf56d10.
Goddard K, Roudsari A, Wyatt JC. Automation bias: empirical results assessing influencing factors. Int J Med Informatics. 2014;83(5):368–75. https://doi.org/10.1016/j.ijmedinf.2014.01.001 . (PMID: 10.1016/j.ijmedinf.2014.01.001)
Vasconcelos H, Jörke M, Grunde-McLaughlin M, Gerstenberg T, Bernstein MS, Krishna R. Explanations can reduce overreliance on AI systems during Decision-Making. Proc ACM Hum-Comput Interact. 2023;7(CSCW1):1291–129. https://doi.org/10.1145/3579605 . (PMID: 10.1145/3579605)
Dopp AR, Parisi KE, Munson SA, Lyon AR. Aligning implementation and user-centered design strategies to enhance the impact of health services: results from a concept mapping study. Implement Sci Commun. 2020;1:17. https://doi.org/10.1186/s43058-020-00020-w . (PMID: 10.1186/s43058-020-00020-w328851797427975)

H53D23008090001 European Union - Next Generation EU; #PNC0000007 Italian Ministry of Research

Keywords: Clinical decision-making; Human-AI collaboration; Interpretability; Questionnaire; Think-aloud protocol; User study

Date Created: 20250716 Date Completed: 20250717 Latest Revision: 20250716

20250717

10.1186/s12911-025-03045-0

40671048