Mechanical ventilation is a routine therapy in the paediatric cardiac intensive care unit. Separation from mechanical ventilation represents a vulnerable transition for patients with critical cardiovascular disease. Extubation can exacerbate underlying cardiac dysfunction and worsen other circulatory pathophysiology, potentially necessitating reintubation (extubation failure) and leading to poor outcomes. Children experiencing extubation failure are more likely to have prolonged duration of mechanical ventilation and length of stay, increased likelihood for tracheostomy, higher mortality rates, and higher hospital costs.Reference Farias, Retta and Alía1–Reference Epstein, Ciubotaru and Wong4 Thus, determining new approaches to predict extubation readiness and avoid extubation failure may help improve clinical outcomes and resource utilisation.
Efforts to improve medical decision-making around the time of extubation have largely been unsuccessful to date. Extubation readiness trials are commonly used in paediatric intensive care, but previous work demonstrates that standard extubation readiness trials fail to enhance clinician judgement.Reference Randolph, Wypij and Venkataraman5 Evidence to support any given method in paediatric intensive care patients remains limited.Reference Randolph, Wypij and Venkataraman5–Reference Ferreira, Sugo and Aragon8 The typical variables assessed during an extubation readiness trial – oxygen saturation, respiratory rate, heart rate, and tidal volumeReference Randolph, Wypij and Venkataraman5,Reference Newth, Venkataraman and Willson7,Reference Abu-Sultaneh, Hole and Tori9 – can vary greatly both at baseline and in response to stress amongst patients with structural heart disease. An increasing number of paediatric cardiac intensive care units have access to high-frequency clinical data systems, which can lead to more intricate characterisation of these variables. However, subtle changes in physiologic variables may not be easily identifiable to clinicians in real time. Tree-based machine learning methods offer an intuitive framework for classification and are adept at making predictions using complex data.Reference Banerjee, Reynolds and Andersson10 Capitalising on these methods and the availability of high-frequency clinical data may help to determine physiologic patterns that predict extubation failure across patients with varying cardiac lesions.
In this context, we applied tree-based machine learning methods to develop and evaluate a prediction tool for extubation failure in the cardiac intensive care unit. To accomplish this analysis, we linked our local Pediatric Cardiac Critical Care Consortium (PC4) clinical registry data with continuously captured physiologic data from the T3 software platform (Etiometry, Inc., Boston, MA). We hypothesised that we could identify physiologic variables during extubation readiness trials associated with extubation failure in cardiac intensive care unit patients and develop a tool that could be implemented in the cardiac intensive care unit to improve assessment of extubation readiness.
Materials and methods
Data sources
The Pediatric Cardiac Critical Care Consortium clinical registry is a voluntary quality improvement collaborative that collects data on all patients with primary cardiac disease admitted to the cardiac intensive care unit attending service of participating hospitals.Reference Gaies, Cooper and Tabbutt11 Participating hospitals are audited on a regular schedule and audit results suggest complete, accurate and timely submission of data across hospitals, with the most recent published results demonstrating a major discrepancy rate of 0.6% across 29,476 fields.Reference Gaies, Donohue and Willis12 Patient demographics, comorbidities, surgical procedures/interventions, critical care therapies, and complications are collected in the database. We abstracted clinical information about patients from our institution only for our analyses.
T3 (Etiometry, Inc.; Boston, MA) is a commercial software application that allows for the continuous collection and visualisation of physiologic data from patients in an intensive care setting.13 Data at our institution are streamed at 1-minute intervals from physiologic monitors, ventilator data, and other inputs, providing a higher frequency dataset compared to the electronic medical record. The T3 data at the University of Michigan were cleaned through joint efforts of our analysis team and Etiometry, Inc. All physiologic data were filtered for the removal of implausible physiologic data. T3 ventilator data were compared to the clinical registry data to make sure the data were clinically aligned. Any runs that were tagged as incongruent between T3 and Pediatric Cardiac Critical Care Consortium by the programme we developed to compare them were examined further and brought to the attention of Etiometry, Inc. Multiple updates of the data warehouse took place in order to obtain the most accurate data possible.
The University of Michigan Medical School Institutional Review Board (IRBMED) provided oversight for this study, which was approved with waiver of informed consent (Approval Number: HUM00153234).
Inclusion and exclusion criteria
The study population included patients admitted to the paediatric cardiac intensive care unit at the University of Michigan between 12/2016 and 3/2018 for both medical and surgical indications. The episode of analysis was a course of mechanical ventilation. We retrospectively analysed patients who had at least one ventilation episode ≥12 hours and an extubation readiness trial (minimum: 30 minutes; maximum: 3 hours) within 14 hours of extubation. Timing of the extubation is at the discretion of the clinical team after utilising all clinical inputs (including vent setting, sedative medication load, amount of endotracheal tube leak). We used 12 hours as the minimum episode length as literature has shown that risk of extubation failure increases with length of ventilation,Reference Gaies, Tabbutt and Schwartz14 so those extubated shortly after intubation likely have lower extubation failure rates and represent a simpler clinical decision for physicians. Only the first qualifying mechanical ventilation episode that ended in a planned extubation during each hospitalisation of our qualifying patients was analysed. Patients were excluded from the analysis if they were ventilated via tracheostomy or extubated due to withdrawal of care.
Outcome and predictor variables
The primary outcome of this analysis was extubation failure. Extubation failure was defined as reintubation within 48 hours of a planned extubation.Reference Gaies, Cooper and Tabbutt11 We utilised local Pediatric Cardiac Critical Care Consortium registry data for patient and clinical outcome data. We included admission type (medical vs. surgical) and timing of ventilation relative to surgery (pre- vs. post-operative) in the analysis. History of a paralysed diaphragm, vocal cord dysfunction, extracorporeal membrane oxygenation, and number of cardiac ventricles was recorded. Mechanical ventilation-related variables included the length of ventilation prior to extubation, intercurrent surgical procedure during the episode, and the number of previous extubations the patient had during that cardiac intensive care unit encounter. Other clinical and demographic variables were included that were readily available in the database and thought to be clinically applicable.
Though T3 collects data over the entirety of each patient’s cardiac intensive care unit hospitalisation, we focused our analysis on physiologic data during the last extubation readiness trial prior to extubation of each patient’s qualifying run of mechanical ventilation. At our institution, extubation readiness trials are performed in appropriate patients by turning off the inspiratory pressure and mandatory vent rate for patients on minimal positive end-expiratory pressure (5–6 cmH2O) and pressure support (8–10 cmH2O) in order to simulate breathing on one’s own for a minimum of 30 minutes. Patients may not undergo extubation readiness trial testing despite being on minimal settings at the discretion of the physician if there are factors outside of their ventilator support that may increase the risk of extubation failure (i.e., other signs of clinical worsening) or if they require to stay intubated for other aspects of their medical care. Inability for a patient to maintain these settings while maintaining normal vital signs/oxygen saturation for their physiology indicates a failed trial and leads to early termination. SAS Version 9.4 (SAS Institute, Cary, NC) was used to create a programme that automatically scans the T3 files to identify extubation readiness trials of appropriate length based on ventilator settings described above. The programme compared each patient’s extubation readiness trial times to their extubation time as listed in Pediatric Cardiac Critical Care Consortium, and the extubation readiness trial closest to extubation was evaluated. This programme was validated through repeated comparison of extubation readiness trial timing with the electronic medical record.
Physiologic variables were selected from the T3 database a priori, based on clinical importance. Variables chosen were either ones typically assessed in clinical practice or predicted to be relevant to extubation readiness. The physiologic variables studied were heart rate, systolic/diastolic/mean blood pressure, respiratory rate, oxygen saturation, central venous pressure, dynamic compliance, volume delivered by the ventilator (i.e., the measured volume observed during the extubation readiness trial with pressure-supported spontaneous ventilation), fraction of inspired oxygen, oxygen saturation/fraction of inspired oxygen ratio, mean airway pressure, and positive end-expiratory pressure. To acknowledge that some of these physiologic variables vary across age groups, we analysed statistical summaries of these variables across the extubation readiness trial for each patient rather than solely the raw data itself: slope over the entire extubation readiness trial, average piece-wise slope, and the standard deviation of each variable. Specifically, in regard to slope, slope over the entire extubation readiness trial was calculated using ordinary least squares regression. Average piece-wise slope was calculated as the mean slope between each pair of consecutive measurements during the extubation readiness trial. For variables that did not depend on age, we also examined the mean, minimum, and maximum values in addition to the summary statistics used for their age-dependent counterparts.
Machine learning approach
We utilised three machine learning methods to predict extubation failure.Reference Banerjee, Reynolds and Andersson10 As an initial step, we performed classification and regression tree analyses on our data, which creates a single decision tree where each decision point/node is a split in the predictor variable that the algorithm determines will maximise within-node homogeneity of the resulting “daughter nodes.”Reference Breiman, Friedman and Olshen15 We expanded on our analysis by also performing tree-based ensemble analyses, namely Boosting and Random Forest.Reference Hastie, Tibshirani and Friedman16–Reference Freund and Schapire18 While classification and regression tree analyses tend to be the most clinically intuitive due to their output of a decision tree, these ensemble methods tend to create more robust classifiers as they use many trees (in this case, 1000 trees based on 1000 bootstrapped samples) to make predictions. Their output instead is a variable importance plot that gives weight to the different predictor variables based on many different trees. Our variable importance plots display the twenty highest-weighted variables.
We implemented tree-growing control by only considering splits that resulted in daughter nodes of at least five patients. In addition, for the classification and regression tree and Boosting algorithms, we only allowed nodes with 15 or more patients to be split and only considered splits that reduced node impurity by a factor of 0.01 (using the “cp” parameter in the rpart package). Classification and regression tree and Boosting used surrogate splits, and Random Rorest used a fast imputation approach to handle missing covariate information.
To enhance the training dataset for the classifiers both in terms of total sample size and number of events, we trained our algorithms using a bootstrapped sample (with replacement) from the original population. This final sample was 4× the size of the original study population.
We employed an iterative approach by first building classifiers based just on clinical data (Pediatric Cardiac Critical Care Consortium database). Next, we developed classifiers using solely the physiologic summary measures (T3 database). Lastly, we utilised machine learning to build classifiers based on the combined clinical and physiologic information (Pediatric Cardiac Critical Care Consortium + T3 database). In each of the above scenarios, classifiers were built using classification and regression tree, Random Forest, and Boosting.
Prediction performance was assessed through a 10-fold cross-validation stratified by extubation failure status. Metrics used to assess prediction performance included area under the receiver operating characteristic curve, prediction accuracy (prediction accuracy: percentage of the time the classifier correctly predicts a patient’s extubation outcome), positive predictive value, and negative predictive value.
All analyses were performed using R version 4.1.1 using the rpart, randomForestSRC, adabag, mlr, and ggplot2 packages.Reference Ishwaran and Kogalur19–Reference Alfaro, Gamez and Garcia22
Results
Population characteristics
There were 178 ventilator episodes from our institution meeting inclusion criteria for this study. The overall extubation failure rate for the cohort was 11.2% (N = 20 failures). Demographic and clinical characteristics of the patients are described in Table 1.
Table 1. Study population demographics and characteristics
* Before extubation in the same cardiac intensive care unit encounter as extubation
Machine learning outputs
Performance and prediction accuracy of all machine learning methods are summarised in Table 2. Classifier area under the receiver operating characteristic curve values ranged from 0.40 to 0.78, with Random Forest using both clinical and demographic data having the highest area under the receiver operating characteristic curve (area under the receiver operating characteristic curve: 0.78). Prediction accuracies were higher for ensemble methods (range: 0.88–0.89) than using classification and regression tree for all three data groups.
Table 2. Machine learning algorithm results using clinical/demographic data, physiologic data, and both data sources
The classification tree based on the Pediatric Cardiac Critical Care Consortium-derived clinical and demographic information is displayed in Figure 1. This classifier split patients based on age, weight, length of admission, length of episode of mechanical ventilation, previous extubations in that encounter, and presence of pulmonary hypertension or a paralysed diaphragm. Applying the Random Forest algorithm to Pediatric Cardiac Critical Care Consortium data identified duration of mechanical ventilation, patient length and weight, age at time of cardiac intensive care unit admission, and number of ventricles as the five variables of greatest predictive value. Boosting identified similar Pediatric Cardiac Critical Care Consortium variables as being of highest importance, with the inclusion of race instead of number of ventricles.
Figure 1. Classification and regression tree (CART) result using clinical/demographic data.
The classification tree based on only the T3 physiologic data split patients on a variety of physiologic variables, with prominent inclusion of variables related to respiratory rate, central venous pressure, and oxygen saturation. When applying Random Forest methodology to T3 data, the variables of greatest importance in predicting extubation readiness were oxygen saturation mean, minimum, and maximum, in addition to standard deviation and maximum of dynamic compliance. When applying Boosting to the physiologic T3 data, the oxygen saturation mean, slopes of heart and respiratory rate, central venous pressure minimum, and age were identified as most important.
Applying classification and regression tree to both datasets (Pediatric Cardiac Critical Care Consortium and T3) generated a classifier with similar physiologic variables compared to the T3 classifier (Fig 2). The only clinical/demographic variables represented were patient weight and the duration of ventilation. Prediction accuracy metrics based on the 10-fold cross-validation are displayed in Table 2. When using Random Forest on the combined dataset, which had the highest area under the receiver operating characteristic curve, the variable importance plot (Fig 3) illustrates physiologic variables rather than demographic variables being the majority of the twenty highest-weighted variables in determining extubation readiness. Physiologic variables had overall higher importance compared to clinical and demographic data as well when using Boosting. However, overall predictive accuracy was similar when comparing the physiologic or combined dataset to clinical/demographic data alone.
Figure 2. Classification and regression tree (CART) result using both physiologic and clinical/demographic data.
Figure 3. Random Forest variable importance plot using both physiologic and clinical/demographic data.
Discussion
We applied machine learning methods to clinical registry and streamed physiologic data to build a classifier for prediction of extubation failure at the time of extubation readiness testing with the aim to identify extubations that are unlikely to be successful. Our results indicate that in patients with critical cardiovascular disease, unique, high-frequency physiologic variables might aid in the prediction of extubation failure. We demonstrated the potential importance of assessing physiologic variables that are not routinely analysed during standard extubation readiness trials as well as how computational methodologies of extracting information on routine vital signs may lead to increased discriminative capabilities. However, our most accurate classifier had an accuracy of 89%, and our population’s extubation failure rate was 11%, indicating a similar degree of misidentification of ready subjects. While we believe that these methods show promise to improve clinical care, classifier performance must improve before being applied as a clinical tool.
Machine learning methods have been used to create prediction and decision-making tools in various clinical domains.Reference Norrie23–Reference Jalali, Bender and Rehman25 Examples include the prediction of periventricular leukomalacia after neonatal cardiac surgeryReference Jalali, Licht and Nataraj26,Reference Goodman, Lessler and Cosgrove27 and the prediction of clinical deterioration in those with haematologic malignancies.Reference Hu, Wong and Correa24 Additionally, analysis of granular physiologic data with machine learning methods can reveal physiologic changes that are not visible to clinicians. For instance, the HeRO score derived from high fidelity electrocardiogram data has helped clinicians identify neonatal sepsis and reduce mortality rates by 20%.Reference Fairchild28 Uncovering these silent patterns illustrates the promise of these approaches.
Despite the evidence of machine learning’s utility, there is currently a paucity of research to unlock the value of high-frequency physiologic data collected every day in paediatric cardiac intensive care units. The variable importance plots from our work demonstrate that physiologic variables might have more importance than clinical and demographic data, suggesting the value of analysing these variables with advanced statistical methodologies. This is particularly compelling as while we consider our T3 data as high frequency compared to electronic medical record documentation as it is streamed at 1-minute intervals, there are many platforms that collect even higher fidelity physiologic data. The physiologic complexity of paediatric cardiac patients and the wealth of available data from various monitoring techniques may make this patient population particularly suited to apply machine learning methods in an effort to develop predictive algorithms that improve patient care.Reference Olive and Owens29
Our work highlighted the importance of identifying unique features with biologic plausibility not robustly interrogated during traditional extubation readiness trials. While features such as central venous pressure, standard deviation of blood pressure, and extremes of oxygen saturation may be recorded or derivable from recorded measurements during an extubation readiness trial, most parameters to determine a “pass” or “fail” are related to respiratory mechanics and function. While these vital signs are often available during an extubation readiness trial, trend data and measures of variability may not be recognised. An advantage to the algorithms we created is that they not only include a larger number of variables and trends than clinicians can feasibly integrate in real-time for decision-making, but they also innately create cut points at different nodes that maximise clinical utility, which is important as we do not have literature on how many of these variable trends over an extubation readiness trial correlate to extubation readiness. Furthermore, T3 has a graphical user interface for clinicians that can display real-time data and algorithm results. For example, the University of Michigan cardiac intensive care unit T3 graphical user interface currently displays an inadequate oxygen delivery index that is derived from streamed physiologic data. This interface increases the feasibility of a real-time extubation readiness algorithm. Optimising the number of physiologic clues clinicians can utilise in the decision-making process could potentially lead to more accurate predictions and improve clinical care.
There are important limitations to this analysis. Focusing the study on our institution limited the sample size and number of events. We addressed this by enhancing our learning set through a bootstrapped sample of the original population as well as employing training and a 10-fold cross-validation. Despite this, our classifiers’ performance was below that necessary to support real-time clinical decision-making. Currently, it is logistically challenging to use the data from the T3 repository across institutions; however, we aim to use T3 data from multiple cardiac intensive care units in the future, allowing for larger, more heterogeneous populations that would improve the accuracy and applicability of these approaches.
Another limitation is that while physiologic variables were measured at a frequency of once per minute, they underwent conversion into summary statistics in order to be included in our analysis. While our techniques focused on optimising clinical interpretability, reducing the physiologic data to summary measures inherently creates limitations. It is possible that our methods resulted in loss of important clinical information and negatively affected our classifier performance. Also, we aimed for these summary statistics to allow us to compare physiologic variables across age groups that naturally have different baselines. However, we recognise this is an imperfect means of age adjustment as certain variables are known to also have different variability across age ranges.
When analysing potential predictors of extubation failure, we were limited to candidate variables that were either in the Pediatric Cardiac Critical Care Consortium or T3 clinical registry. Though comprehensive, there may be other significant predictors of extubation failure that were not included in our analyses. For instance, T3 as well as certain other high fidelity software systems in cardiac intensive care units currently are able to access NIRS data, which is a variable that has significant potential to predict outcomes but was not available in our dataset.
In summary, we applied machine learning methods in an attempt to generate information that could assist clinicians in the cardiac intensive care unit when assessing patients for extubation readiness. Several physiologic variables that are not routinely scrutinised during standard extubation readiness trials were identified as potential predictors of extubation failure. Our experience suggests that the application of machine learning to the plethora of data collected in the cardiac intensive care unit holds promise for predicting extubation failure, but it is subject to the same limitations of sample size as traditional statistical approaches. Larger patient cohorts might yield analyses to inform a clinically applicable tool that improves medical decision-making for mechanically ventilated patients.
Acknowledgements
We acknowledge the data collection team for Pediatric Cardiac Critical Care Consortium at the University of Michigan. Sydney Rooney at the time of the study was a research fellow supported by the Sarnoff Cardiovascular Research Foundation.
Financial support
This study was supported in part by funding from the University of Michigan Congenital Heart Center, CHAMPS for Mott, and the Michigan Institute for Clinical & Health Research (NIH/NCATS, grant number: UL1TR002240). Sara Pasquali receives funding from the National Heart, Lung, and Blood Institute (grant number: R01HL12226; PI Pasquali). Dr. Banerjee's research was partially funded by a grant BCBS 002983 from the Blue Cross Blue Shield Foundation of Michigan.
Conflicts of interest
None.
