Recalibrating prognostic models to improve predictions of in-hospital child mortality in resource-limited settings.

Publisher: Blackwell Scientific Publications Country of Publication: England NLM ID: 8709766 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1365-3016 (Electronic) Linking ISSN: 02695022 NLM ISO Abbreviation: Paediatr Perinat Epidemiol Subsets: MEDLINE

Original Publication: Oxford ; Boston : Blackwell Scientific Publications, [c1987-

Resource-Limited Settings* ; Child Mortality*; Humans ; Child ; Prognosis ; Hospital Mortality ; Hospitals

Background: In an external validation study, model recalibration is suggested once there is evidence of poor model calibration but with acceptable discriminatory abilities. We identified four models, namely RISC-Malawi (Respiratory Index of Severity in Children) developed in Malawi, and three other predictive models developed in Uganda by Lowlaavar et al. (2016). These prognostic models exhibited poor calibration performance in the recent external validation study, hence the need for recalibration.
Objective: In this study, we aim to recalibrate these models using regression coefficients updating strategy and determine how much their performances improve.
Methods: We used data collected by the Clinical Information Network from paediatric wards of 20 public county referral hospitals. Missing data were multiply imputed using chained equations. Model updating entailed adjustment of the model's calibration performance while the discriminatory ability remained unaltered. We used two strategies to adjust the model: intercept-only and the logistic recalibration method.
Results: Eligibility criteria for the RISC-Malawi model were met in 50,669 patients, split into two sets: a model-recalibrating set (n = 30,343) and a test set (n = 20,326). For the Lowlaavar models, 10,782 patients met the eligibility criteria, of whom 6175 were used to recalibrate the models and 4607 were used to test the performance of the adjusted model. The intercept of the recalibrated RISC-Malawi model was 0.12 (95% CI 0.07, 0.17), while the slope of the same model was 1.08 (95% CI 1.03, 1.13). The performance of the recalibrated models on the test set suggested that no model met the threshold of a perfectly calibrated model, which includes a calibration slope of 1 and a calibration-in-the-large/intercept of 0.
Conclusions: Even after model adjustment, the calibration performances of the 4 models did not meet the recommended threshold for perfect calibration. This finding is suggestive of models over/underestimating the predicted risk of in-hospital mortality, potentially harmful clinically. Therefore, researchers may consider other alternatives, such as ensemble techniques to combine these models into a meta-model to improve out-of-sample predictive performance.
(© 2023 The Authors. Paediatric and Perinatal Epidemiology published by John Wiley & Sons Ltd.)

J Biomed Inform. 2009 Apr;42(2):377-81. (PMID: 18929686)
Am Health Drug Benefits. 2009 Sep;2(6):218-22. (PMID: 25126292)
Stat Med. 2011 Feb 20;30(4):377-99. (PMID: 21225900)
Stat Med. 2021 Aug 30;40(19):4230-4251. (PMID: 34031906)
Med Decis Making. 2006 Nov-Dec;26(6):565-74. (PMID: 17099194)
Stat Med. 1996 Feb 28;15(4):361-87. (PMID: 8668867)
J Clin Epidemiol. 2006 Oct;59(10):1092-101. (PMID: 16980150)
J Infect. 2016 Nov;73(5):393-401. (PMID: 27519619)
J Clin Epidemiol. 2008 Jan;61(1):76-86. (PMID: 18083464)
Stat Methods Med Res. 2018 Jan;27(1):185-197. (PMID: 27460537)
Stat Med. 2004 Aug 30;23(16):2567-86. (PMID: 15287085)
Wellcome Open Res. 2020 May 27;5:106. (PMID: 32724864)
J Glob Health. 2021 Oct 09;11:04062. (PMID: 34737862)
J Clin Epidemiol. 2016 Aug;76:175-82. (PMID: 26964707)
Paediatr Perinat Epidemiol. 2023 May;37(4):313-321. (PMID: 36745113)
Stat Methods Med Res. 2007 Jun;16(3):219-42. (PMID: 17621469)
Eur Heart J. 2014 Aug 1;35(29):1925-31. (PMID: 24898551)
Med Decis Making. 2015 Feb;35(2):162-9. (PMID: 25155798)
Stat Med. 2019 Mar 30;38(7):1276-1296. (PMID: 30357870)
Stat Med. 1991 Aug;10(8):1213-26. (PMID: 1925153)
J Clin Epidemiol. 2016 Jan;69:245-7. (PMID: 25981519)
PLoS One. 2016 Dec 28;11(12):e0168126. (PMID: 28030608)
PLoS One. 2016 Mar 10;11(3):e0150683. (PMID: 26963914)
BMC Med. 2019 Dec 16;17(1):230. (PMID: 31842878)
BMJ Open. 2020 Oct 19;10(10):e035045. (PMID: 33077558)
Br J Cancer. 2016 Mar 15;114(6):623-30. (PMID: 26954719)
Clin Infect Dis. 2005 Apr 15;40(8):1079-86. (PMID: 15791504)
Intensive Care Med. 2014 Mar;40(3):438-41. (PMID: 24526318)
Clin Infect Dis. 2020 Jul 11;71(2):372-380. (PMID: 31504308)
Arch Dis Child. 2020 Jul;105(7):648-654. (PMID: 32169853)
Age Ageing. 1996 Nov;25(6):479-89. (PMID: 9003886)
J Clin Epidemiol. 2016 Nov;79:76-85. (PMID: 27262237)
BMJ. 2016 Jan 25;352:i6. (PMID: 26810254)
N Engl J Med. 2021 Jul 15;385(3):283-286. (PMID: 34260843)

United Kingdom WT_ Wellcome Trust

Keywords: model recalibration; paediatric mortality; prediction

Date Created: 20230206 Date Completed: 20230510 Latest Revision: 20240320

20240320

PMC10946771

10.1111/ppe.12948

36745113