A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports.

Publisher: Nature Publishing Group Country of Publication: England NLM ID: 101563288 Publication Model: Electronic Cited Medium: Internet ISSN: 2045-2322 (Electronic) Linking ISSN: 20452322 NLM ISO Abbreviation: Sci Rep Subsets: MEDLINE

Original Publication: London : Nature Publishing Group, copyright 2011-

Medicine* ; Musculoskeletal Diseases*; Male ; Humans ; Middle Aged ; Pilot Projects ; Retrospective Studies ; Language ; Magnetic Resonance Imaging

Large language models (LLMs) have shown potential in various applications, including clinical practice. However, their accuracy and utility in providing treatment recommendations for orthopedic conditions remain to be investigated. Thus, this pilot study aims to evaluate the validity of treatment recommendations generated by GPT-4 for common knee and shoulder orthopedic conditions using anonymized clinical MRI reports. A retrospective analysis was conducted using 20 anonymized clinical MRI reports, with varying severity and complexity. Treatment recommendations were elicited from GPT-4 and evaluated by two board-certified specialty-trained senior orthopedic surgeons. Their evaluation focused on semiquantitative gradings of accuracy and clinical utility and potential limitations of the LLM-generated recommendations. GPT-4 provided treatment recommendations for 20 patients (mean age, 50 years ± 19 [standard deviation]; 12 men) with acute and chronic knee and shoulder conditions. The LLM produced largely accurate and clinically useful recommendations. However, limited awareness of a patient's overall situation, a tendency to incorrectly appreciate treatment urgency, and largely schematic and unspecific treatment recommendations were observed and may reduce its clinical usefulness. In conclusion, LLM-based treatment recommendations are largely adequate and not prone to 'hallucinations', yet inadequate in particular situations. Critical guidance by healthcare professionals is obligatory, and independent use by patients is discouraged, given the dependency on precise data input.
(© 2023. The Author(s).)

Erratum in: Sci Rep. 2024 Mar 5;14(1):5431. (PMID: 38443449)

N Engl J Med. 2023 Mar 30;388(13):1233-1239. (PMID: 36988602)
Radiology. 2023 May;307(4):e230424. (PMID: 37014239)
Radiology. 2023 May;307(4):e230725. (PMID: 37014240)
JAMA. 2023 Mar 14;329(10):842-844. (PMID: 36735264)
Curr Probl Diagn Radiol. 2023 Aug 30;:. (PMID: 37758604)
Semin Arthritis Rheum. 2012 Feb;41(4):604-10. (PMID: 22035623)
Diagn Interv Radiol. 2023 Oct 03;30(2):80-90. (PMID: 37789676)
J Med Internet Res. 2023 Aug 22;25:e48659. (PMID: 37606976)
Rheumatology (Oxford). 2006 Aug;45(8):1039-41. (PMID: 16829534)
Radiology. 2023 Jun;307(5):e231259. (PMID: 37367439)
Radiology. 2023 Oct;309(1):e230806. (PMID: 37787671)
Arthritis. 2013;2013:370231. (PMID: 23365745)
Arthritis Rheum. 1997 May;40(5):884-92. (PMID: 9153550)
J Bone Joint Surg Am. 2015 Jan 21;97(2):156-68. (PMID: 25609443)
J Orthop Trauma. 2013 Oct;27(10):543-51. (PMID: 23760182)
JAMA Intern Med. 2023 Jun 1;183(6):589-596. (PMID: 37115527)
NPJ Digit Med. 2021 Sep 21;4(1):140. (PMID: 34548621)
Br J Sports Med. 2018 Aug;52(15):956. (PMID: 29514819)
J Orthop. 2018 Aug 16;15(3):837-841. (PMID: 30140130)
JAMA Netw Open. 2023 Aug 1;6(8):e2325000. (PMID: 37578798)
Curr Probl Diagn Radiol. 2024 Mar-Apr;53(2):215-225. (PMID: 37891083)
PLOS Digit Health. 2023 Feb 9;2(2):e0000198. (PMID: 36812645)

ODELIA, 101057091 European Union's Horizon Europe programme; COMFORT, 101079894 European Union's Horizon Europe programme; TR 1700/7-1 Deutsche Forschungsgemeinschaft; NE 2136/3-1 Deutsche Forschungsgemeinschaft; DEEP LIVER, ZMVI1-2520DAT111 Bundesministerium für Gesundheit; #70113864 Max-Eder-Programme of the German Cancer Aid; PEARL, 01KD2104C German Federal Ministry of Education and Research; CAMINO, 01EO2101 German Federal Ministry of Education and Research; SWAG, 01KD2215A German Federal Ministry of Education and Research; TRANSFORM LIVER, 031L0312A German Federal Ministry of Education and Research; TANGERINE, 01KT2302 through ERA-NET Transcan German Federal Ministry of Education and Research; SECAI, 57616814 Deutscher Akademischer Austauschdienst; Transplant.KI, 01VSF21048 German Federal Joint Committee; ODELIA, 101057091 European Union's Horizon Europe and innovation programme; GENIAL, 101096312 European Union's Horizon Europe and innovation programme; NIHR, NIHR213331 National Institute for Health and Care Research

Date Created: 20231118 Date Completed: 20231120 Latest Revision: 20240308

20240308

PMC10656559

10.1038/s41598-023-47500-2

37978240