Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians.

Publisher: Public Library of Science Country of Publication: United States NLM ID: 101285081 Publication Model: eCollection Cited Medium: Internet ISSN: 1932-6203 (Electronic) Linking ISSN: 19326203 NLM ISO Abbreviation: PLoS One Subsets: MEDLINE

Original Publication: San Francisco, CA : Public Library of Science

Education, Medical*/methods; Humans ; Students, Medical

Background: ChatGPT is a large language model (LLM) trained on over 400 billion words from books, articles, and websites. Its extensive training draws from a large database of information, making it valuable as a diagnostic aid. Moreover, its capacity to comprehend and generate human language allows medical trainees to interact with it, enhancing its appeal as an educational resource. This study aims to investigate ChatGPT's diagnostic accuracy and utility in medical education.
Methods: 150 Medscape case challenges (September 2021 to January 2023) were inputted into ChatGPT. The primary outcome was the number (%) of cases for which the answer given was correct. Secondary outcomes included diagnostic accuracy, cognitive load, and quality of medical information. A qualitative content analysis was also conducted to assess its responses.
Results: ChatGPT answered 49% (74/150) cases correctly. It had an overall accuracy of 74%, a precision of 48.67%, sensitivity of 48.67%, specificity of 82.89%, and an AUC of 0.66. Most answers were considered low cognitive load 51% (77/150) and most answers were complete and relevant 52% (78/150).
Discussion: ChatGPT in its current form is not accurate as a diagnostic tool. ChatGPT does not necessarily give factual correctness, despite the vast amount of information it was trained on. Based on our qualitative analysis, ChatGPT struggles with the interpretation of laboratory values, imaging results, and may overlook key information relevant to the diagnosis. However, it still offers utility as an educational tool. ChatGPT was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Additionally, answers were easy to understand, showcasing a potential benefit in simplifying complex concepts for medical learners. Our results should guide future research into harnessing ChatGPT's potential educational benefits, such as simplifying medical concepts and offering guidance on differential diagnoses and next steps.
Competing Interests: The authors have declared that no competing interests exist.
(Copyright: © 2024 Hadi et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

N Engl J Med. 2019 Apr 4;380(14):1347-1358. (PMID: 30943338)
Resuscitation. 2023 Apr;185:109729. (PMID: 36773836)
Healthcare (Basel). 2023 Mar 19;11(6):. (PMID: 36981544)
J Am Med Inform Assoc. 2014 Mar-Apr;21(2):221-30. (PMID: 24201027)
Multimed Tools Appl. 2023;82(3):3713-3744. (PMID: 35855771)
J Med Syst. 2023 Mar 04;47(1):33. (PMID: 36869927)
J Educ Eval Health Prof. 2023;20:1. (PMID: 36627845)
Am J Med. 2018 Feb;131(2):129-133. (PMID: 29126825)
Nat Med. 2019 Jan;25(1):24-29. (PMID: 30617335)
Ophthalmol Sci. 2023 May 05;3(4):100324. (PMID: 37334036)
Med Educ. 2005 Jan;39(1):98-106. (PMID: 15612906)
J Am Coll Radiol. 2023 Oct;20(10):990-997. (PMID: 37356806)
Acad Med. 1999 Aug;74(8):890-5. (PMID: 10495728)
Nature. 2023 Feb;614(7947):224-226. (PMID: 36737653)
Narra J. 2023 Apr;3(1):e103. (PMID: 38450035)
Yearb Med Inform. 2008;:128-44. (PMID: 18660887)
AMA J Ethics. 2019 Feb 1;21(2):E146-152. (PMID: 30794124)
JAMA. 2020 Feb 11;323(6):509-510. (PMID: 31845963)
N Engl J Med. 2018 Mar 15;378(11):981-983. (PMID: 29539284)
Belitung Nurs J. 2023 Feb 12;9(1):1-5. (PMID: 37469634)
AMIA Jt Summits Transl Sci Proc. 2020 May 30;2020:191-200. (PMID: 32477638)
N Engl J Med. 2020 Apr 30;382(18):1679-1681. (PMID: 32160451)
Cureus. 2023 Feb 19;15(2):e35179. (PMID: 36811129)
JAMA. 2018 Dec 4;320(21):2199-2200. (PMID: 30398550)
Eur J Nucl Med Mol Imaging. 2023 May;50(6):1549-1552. (PMID: 36892666)
Science. 2019 Oct 25;366(6464):447-453. (PMID: 31649194)
JMIR Med Educ. 2023 Feb 8;9:e45312. (PMID: 36753318)
Pak J Med Sci. 2023 Mar-Apr;39(2):605-607. (PMID: 36950398)
PLOS Digit Health. 2023 Feb 9;2(2):e0000198. (PMID: 36812645)

Date Created: 20240731 Date Completed: 20240731 Latest Revision: 20240802

20240802

PMC11290643

10.1371/journal.pone.0307383

39083523