Exploring the performance and explainability of fine-tuned BERT models for neuroradiology protocol assignment.

Publisher: BioMed Central Country of Publication: England NLM ID: 101088682 Publication Model: Electronic Cited Medium: Internet ISSN: 1472-6947 (Electronic) Linking ISSN: 14726947 NLM ISO Abbreviation: BMC Med Inform Decis Mak Subsets: MEDLINE

Original Publication: London : BioMed Central, [2001-

Problem Solving* ; Natural Language Processing*; Humans

Background: Deep learning has demonstrated significant advancements across various domains. However, its implementation in specialized areas, such as medical settings, remains approached with caution. In these high-stake environments, understanding the model's decision-making process is critical. This study assesses the performance of different pretrained Bidirectional Encoder Representations from Transformers (BERT) models and delves into understanding its decision-making within the context of medical image protocol assignment.
Methods: Four different pre-trained BERT models (BERT, BioBERT, ClinicalBERT, RoBERTa) were fine-tuned for the medical image protocol classification task. Word importance was measured by attributing the classification output to every word using a gradient-based method. Subsequently, a trained radiologist reviewed the resulting word importance scores to assess the model's decision-making process relative to human reasoning.
Results: The BERT model came close to human performance on our test set. The BERT model successfully identified relevant words indicative of the target protocol. Analysis of important words in misclassifications revealed potential systematic errors in the model.
Conclusions: The BERT model shows promise in medical image protocol assignment by reaching near human level performance and identifying key words effectively. The detection of systematic errors paves the way for further refinements to enhance its safety and utility in clinical settings.
(© 2024. The Author(s).)

J Am Coll Radiol. 2016 Oct;13(10):1210-1214. (PMID: 27313127)
Int J Med Inform. 2023 Feb;170:104956. (PMID: 36512987)
Annu Rev Biomed Eng. 2017 Jun 21;19:221-248. (PMID: 28301734)
N Engl J Med. 2018 Mar 15;378(11):981-983. (PMID: 29539284)
J Am Coll Radiol. 2011 Jan;8(1):19-25. (PMID: 21211760)
Bioinformatics. 2020 Feb 15;36(4):1234-1240. (PMID: 31501885)
NPJ Digit Med. 2018 Oct 18;1:59. (PMID: 31304338)
J Magn Reson Imaging. 2019 Jun;49(7):e14-e25. (PMID: 30145852)
J Biomed Inform. 2018 Nov;87:12-20. (PMID: 30217670)
Acad Radiol. 2017 Feb;24(2):160-166. (PMID: 27889399)
J Am Coll Radiol. 2020 Sep;17(9):1149-1158. (PMID: 32278847)
Brief Bioinform. 2018 Nov 27;19(6):1236-1246. (PMID: 28481991)
J Am Med Inform Assoc. 2018 May 1;25(5):568-571. (PMID: 29092082)
J Am Coll Radiol. 2014 May;11(5):440-1. (PMID: 24793037)

Keywords: BERT; Explanations; Healthcare; Interpretability; Machine learning

Date Created: 20240207 Date Completed: 20240209 Latest Revision: 20240210

20240210

PMC10848624

10.1186/s12911-024-02444-z

38326769