The development of databases to track the outcomes of children with cardiovascular disease has been ongoing for much of the last two decades, paralleled by the rise of databases in the intensive care unit. While the breadth of data available in national, regional and local databases has grown exponentially, the ability to identify meaningful measurements of outcomes for patients with cardiovascular disease is still in its early stages.
Crude mortality rates are no longer sufficient for defining outcome for children in cardiac intensive care units (CICUs). Examining complex processes such as the care of patients with congenital cardiac disease in tertiary level cardiac intensive care units is exceedingly difficult, but absolutely necessary to future improvements in care. While most surgical procedures are successful with a very low mortality, there are numerous areas where improvement can occur. This improvement cannot occur without adequate, well-organised data. This organization of data requires databases that track critical care, specifically targeting markers of morbidity in critically ill children. Furthermore, the care of infants and children undergoing cardiac surgical repair involves many other factors such as the following factors:
• the intensive care unit team care
• cardiology input
• pre-operative condition
• cardiopulmonary bypass, and
• surgical techniques.
Each of these factors has potential impact on the post-operative intensive care unit and outcomes. In an era where financial reimbursement may become tied to measured outcome, it is essential we develop clarity and logic in such a complex and multi-factorial arena.
No one centre operates on sufficient volumes of any congenital lesion, over a sufficiently concise period of time, to see meaningful trends in a rapid and timely fashion; therefore, it is critical that these datasets are tracked nationally and internationally in order to detect clinically relevant changes as they occur. Collective databases allow more significant information on larger homogeneous groups of patients to be assembled, thereby giving a more accurate assessment of lesion-specific outcome. The success of the international Databases of The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery, and the national Central Cardiac Audit Database in the United Kingdom, are attestations to this.
Definition of the cardiac patient
For databases to be meaningful, a uniform definition of a cardiac patient must first be established. Even at centres with dedicated cardiac intensive care units, children with cardiac disease are a distinct and important constituency among the multitude of patients that are found in a paediatric intensive care unit (PICU); however their identification continues to be difficult and controversial. The definition hinges, to some extent, on the user of the information. Congenital cardiac surgeons may be more interested in patients who have had or are going to have surgery in the near term. Neonatologists may only be interested in neonates with congenital cardiac lesions prior to operative intervention. Cardiologists may be interested in surgical patients, but would also like to understand the outcomes of nonsurgical patients with isolated arrhythmias and cardiomyopathies. Cardiologists may also be more interested in the long-range outcomes, years following surgical correction. Cardiac intensivists may need to know all of the above. And, in reality, all subspecialists eventually need to know all of the above! For meaningful comparisons to be made, a database must be able to accommodate the audience it serves. This complexity increases when one considers cross institutional comparisons and becomes even greater when considering international comparisons.
To cross this divide and reduce complexity, cardiovascular databases need to be both flexible and comprehensive. It should be possible to analyse patients of interest to the relevant constituencies and, at a minimum, divide patients into cardiac surgical and cardiac medical categories. For a patient to be identified as a cardiac surgical patient, they should be able to have a procedural code from the database of The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery, as well as a diagnostic code during their admission. In a similar fashion, a cardiac medical patient should have a diagnostic code from the database of The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery assigned during their admission. While this is a relatively simple construct, there will, as expected, be some patients whose assignation will engender debate, for example, a patient admitted to the hospital three months after a Fontan operation who presents with an arrhythmia. Accordingly, there will need to be the ability for some “hybrid” patients. These definitions should apply regardless of the type of hospital, intensive care unit, or primary caregiver, and the database should allow flexibility for the user constituency to determine patients of interest.
What is a cardiac intensive care unit?
Databases of intensive care have, and continue to, struggle with the concept of ‘What is a cardiac intensive care unit?’ The practitioners of cardiac intensive care within these units have clear ideas of what they consider a cardiac intensive care unit; however, this clear idea tends to differ from clinician to clinician. Despite this lack of a consistent consensus definition, most will agree that they would like to compare their outcomes among similar units. This desire then engenders the question, ‘what is a cardiac intensive care unit’ and how does a database allow valid comparisons.
Numerous questions exist:
• What is a cardiac intensive care unit’
• Does a cardiac intensive care unit require four walls distinct from those of the paediatric intensive care unit?
• Does a cardiac intensive care unit need a dedicated staff of nurses, doctors, respiratory therapists, pharmacists, social workers, and other team members, who only work in the cardiac intensive care unit?
• Can a cardiac intensive care unit be a ‘virtual unit’, in that it has cardiac patients managed within the walls of a paediatric intensive care unit, though cared for by distinct staff dedicated to cardiac intensive care?
• Can a cardiac intensive care unit be a ‘virtual unit’ with care provided by the same staff that provides care in the paediatric intensive care unit?
Numerous other possible modifications to these schema exist, all of which could be called a cardiac intensive care unit by an institution. This variable definition of a cardiac intensive care unit clearly complicates the ability to compare outcomes meaningfully across cardiac intensive care units, although such comparisons may help us to define the minimum requirements of a cardiac intensive care unit and best practice in the future. Ultimately, the comparison group that a unit desires to compare itself to will be determined on a local level. A database that is tailored to service cardiac intensive care units should have the ability to compare across the universe of cardiac intensive care units, as well as to a more customizable group of units. Alternatively, databases that are designed for the universe of intensive care unit patients should be able to compare across ‘cardiac’ patients independently of unit assignation.
Additional complexity is encountered when factoring in pre-operative neonates, some of whom reside in traditional cardiac intensive care units, others in neonatal intensive care units (NICUs) on neonatology services, and still others in neonatal intensive care units on a cardiac intensive care service. As there is no clear answer in defining a cardiac intensive care unit, it will be necessary for those caring for cardiac patients to consider the comparison group to meaningfully compare their patient outcomes with other cardiac intensive care units and pediatric intensive care units. Thus, for a database to successfully compare the outcomes of the treatments provided to cardiovascular patients and the universe of cardiac intensive care units, it must capture sufficient details regarding the characteristics of the “unit” to allow valid comparison. In addition the question that is being answered by the comparison must be clear. A comprehensive understanding of the spectrum of outcomes will be necessary to assure that individual teams are providing the best possible care for their patients.
The difficulties
Many challenges and difficulties are encountered when creating databases, as exemplified by the following non-comprehensive list:
• time constraints on staff
• ease of data entry
• human error
• quality of the data, and
• dependence on technology
Intensive care unit databases must be validated to be reliable and meaningful. A common problem is the need for greater facilitation of collection of data. Often inadequate staffing is provided for the collection of data, particularly in environments that are “resource poor”. Institutions that have inadequate staffing and resources for the collection of data will suffer from incomplete or incorrect collection of information. Facilitating the entry of data with easier methods of collection, such as eliminating multiple entries of the same data by clinical staff, would be beneficial. Obviously, it is imperative that the data collected is clearly defined, meaningful, and accurate, if it is to be productive for analyzing care and ultimately the performance of a given unit. The old adage that “the quality of results that are produced is only as good as the data that is entered” is never more pertinent. It is best practice to review and discuss the results from an institutional database as a team at regular intervals in order to ensure the quality of the data, and to identify trends.
Databases: United States – Virtual Pediatric Intensive Care Unit Performance System (VPS)
The Virtual Pediatric Intensive Care Unit Performance System (VPS) is a clinically based database system for the paediatric intensive care unit that provides standardized high quality, comparative data to its participants [https://portal.myvps.org/]. All participants collect information on multiple parameters:
• patients and their stay in the hospital
• diagnoses
• interventions
• discharge
• various measures of outcome
• organ donation, and
• paediatric severity of illness scores.
Because of the standards of quality within the database, through customizable interfaces, the database can also be used for several applications:
• administrative purposes, such as assessing the utilization of resources and strategic planning
• multi-institutional research studies, and
• additional internal projects of quality improvement or research.
The Virtual Pediatric Intensive Care Unit Performance System database has over 190,000 unique records of patients from over sixty participating institutions. Institutions receive quarterly comparative reports, in addition to an annual customized comparative report of performance that incorporates statistical analyses of trends and a clinical interpretation of the analyses, derived in collaboration with the leadership team of the intensive care unit of the institution.
Currently, institutions that use the Virtual Pediatric Intensive Care Unit Performance System database maintain their data locally. This local storage of data will soon change with the transition to a web-enabled system of data entry, with data securely maintained by the Virtual Pediatric Intensive Care Unit Performance System. This new version will contain more robust cardiac features, including the utilization of the diagnostic and procedural nomenclature of the version of The International Pediatric and Congenital Cardiac Code (IPCCC) derived from the nomenclature of The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery [http://www.ipccc.net/].
Databases: United Kingdom – The Paediatric Intensive Care Audit Network (PICANet)
The Paediatric Intensive Care Audit Network is a database established in 2002 to record details of the treatment of all critically ill children in paediatric intensive care units of the National Health Service in England, Wales and Scotland. The Paediatric Intensive Care Audit Network was designed to develop and maintain a secure and confidential high quality clinical database of pediatric intensive care activity in order to meet the following objectives:
• identify best clinical practice
• monitor supply and demand
• monitor and review outcomes of treatment episodes
• facilitate strategic healthcare planning
• quantify resource requirements, and
• study the epidemiology of critical illness in children.
The core data set of demographic and clinical data on all admissions is the “Paediatric Critical Care Minimum Data Set”, and is collected by local paediatric intensive care units and collated by the Paediatric Intensive Care Audit Network, allowing comparison of paediatric intensive care unit activity at a local level with national benchmarks. An interesting development in the United Kingdom is that this database also is now providing the structure for reimbursement. This data set provides an important evidence base for assessment of structure, process, and outcome that permits planning for future practice, research, and interventions. The Paediatric Intensive Care Audit Network is standardised, risk adjusted and validated data, and also provides further analyses of data, such as by region, type of admission, and age of the patient.1 Mortality rates are standardized using the Paediatric Index of Mortality Scoring System 2 (PIM2), as discussed below. The national dataset is not cardiac specific, although individual pediatric intensive care units have flexibility for the collection of additional items.
Data verification in the Virtual Pediatric Intensive Care Unit Performance System
It is crucial to evaluate the quality of the data in order to ensure optimal decision making from a clinical and an administrative standpoint.2 The lack of adequate quality control of the data not only results in a decrease of the buy-in to the results of such analyses by various constituencies, but also poses significant ethical and legal issues. Data quality control includes two key concepts – “validity” and “reliability”. Validity relates to the concept that the measure is capturing the question being addressed by the measurement. Reliability relates to reproducibility of the measurement over time.Reference Sachdeva3
The Virtual Pediatric Intensive Care Unit Performance System [https://portal.myvps.org/] has developed proactive and affirmative steps to ensure a high level of validity and reliability of the data. Validity of the data has been maximised by standardization of the processes of collection of the data. These standardized processes include ‘a priori’ standardized definitions of data, and development of manuals about the collection of the data that are used by all of the sites participating in the Virtual Pediatric Intensive Care Unit Performance System. Perhaps not surprisingly, many measures do not have predefined definitions. This lack of predefined definitions requires operationalizing the clinical measure and developing a consensus to ensure operationally valid definition and collection of data. Reliability of the data in the Virtual Pediatric Intensive Care Unit Performance System has been ensured by focusing on inter-rater reliability, measured by the use of Kappa statistics and percent concordance. Achieving a high degree of inter-rater reliability ensures that multiple collectors of data are interpreting and abstracting clinical data in a consistent and similar manner, accurately.
A current limitation of the process of quality control in the Virtual Pediatric Intensive Care Unit Performance System relates to inter-site quality control. Although the Virtual Pediatric Intensive Care Unit Performance System has robust processes to ensure inter-site quality control with respect to validity, the inter-site quality control related to reliability is limited. Techniques to establish processes to ensure inter-site data reliability, including the development of standardized “mock charts”, are underway, and are expected to be implemented over the next several months. The Virtual Pediatric Intensive Care Unit Performance System continues to have a high level of intra-site data reliability, with the percent concordance rates in the 90% range consistently for the last five years.
Severity of illness adjustment – scoring systems
Risk adjustment can be performed at three levels:
• acuity classification
• severity risk stratification, and
• severity of risk adjustment.4
Acuity classification relates to the concept of stratifying patients based upon measures of acuity that are commonly used for staffing purposes. Acuity classification is exemplified by the Automated Nurse Scheduling Office System that may be used for staffing of nurses.4
Severity risk classification relates to the concept of stratifying groups of patients based upon severity categories. Severity risk classification is exemplified by the All Patient Refined – Diagnostic Related Groups system.4
A limitation of both acuity classification and severity risk stratification is that although they allow identification of patient groups based upon acuity or severity, they are limited in their ability to provide full risk adjustment to develop predictive models that can be used to compare and improve performance. The third level of risk adjustment, severity of risk adjustment, relates to the concept of developing a physiologic severity of risk assessment that can be used in statistical predictive models of outcomes of patients.4 A comparison of the predicted outcome to the observed outcome provides a measure of performance for units. Furthermore, severity of risk adjustment also allows statistical adjusting for differences in severity among patient groups for comparing outcomes within an intensive care unit over time and also across intensive care units.
Severity of illness risk adjustment models are evaluated by testing their “calibration” and “discrimination”. Calibration relates to model fit and is evaluated using goodness of fit tests, such as the Chi Square Goodness of Fit test and the Hosmer Lemeshow statistic. A statistically non-significant value, with a p greater than 0.05, supports no lack of fit of the model, that is, acceptable calibration. Discrimination properties of the model are tested using the Receiver Operator Characteristic (ROC) curve analysis. The Receiver Operator Characteristic curve plots sensitivity against ‘1-specificity’. A Receiver Operator Characteristic value of greater than 0.9 supports an acceptable level of discrimination properties of the model.
The Virtual Pediatric Intensive Care Unit Performance System utilizes two distinct physiologic risk adjustment methodologies:
• the Pediatric Risk of Mortality Scoring System (PRISM),Reference Pollack, Ruttimann and Getson5 and
• the Paediatric Index of Mortality Scoring System 2 (PIM 2).Reference Slater, Shann and Pearson6
The Pediatric Risk of Mortality (PRISM) scoring system currently included are Pediatric Risk of Mortality 2 (PRISM 2) and Pediatric Risk of Mortality 3 (PRISM 3) scoring systems.Reference Pollack, Ruttimann and Getson5, Reference Pollack, Patel and Ruttimann7 Both Pediatric Risk of Mortality (PRISM 2) and Pediatric Risk of Mortality (PRISM 3) are comprised of clinical variables that include physiological and laboratory measurements that are weighted on a logistic scale. The raw Pediatric Risk of Mortality (PRISM) score provides quantitative measures of severity of illness. The Pediatric Risk of Mortality (PRISM) score when used in a logistic regression model provides a probability of the predicted risk of mortality. This predicted risk of mortality can then be used along with the rates of observed mortality to provide a quantitative measurement of the Standardized Mortality Ratio (SMR). The Pediatric Risk of Mortality (PRISM) scoring system has been validated in the United States of America in several studies and has been shown to have good calibration and discrimination for use in paediatric intensive care units.
Similar to the Pediatric Risk of Mortality (PRISM) scoring system, the Paediatric Index of Mortality (PIM) score is comprised of physiological and laboratory values and provides a quantitative measurement to estimate the probability of death using a logistic regression model. The Pediatric Index of Mortality (PIM) score, although validated outside the United States, is in the process of validation in the paediatric population in the United States of America. A potential advantage of the Paediatric Index of Mortality (PIM) score over the Pediatric Risk of Mortality (PRISM) scoring system is that it is simpler and requires less resource for its collection of data and computation. Although the Pediatric Risk of Morality (PRISM) and Paediatric Index of Mortality (PIM) have been used in the multidisciplinary paediatric intensive care unit setting, further refinements may be needed to enhance these systems to fully capture the unique diagnoses and specific factors that impact outcomes for cardiac patients in the intensive care unit.
Mortality continues to become less common in paediatric intensive care units. As a result of this decline in mortality, increased efforts are underway to identify key markers and develop other scoring systems to assess morbidity and outcomes. For example, in addition to severity of risk scoring systems that predict mortality, the Virtual Pediatric Intensive Care Unit Performance System also includes measures for functional outcome after paediatric intensive care. The Pediatric Cerebral Performance Category (PCPC) scale and the Pediatric Overall Performance Categories (POPC) provide an estimate of the functional outcomes of children surviving the paediatric intensive care unit on an ordinal scale of one to six based upon their age appropriate functional outcomes.Reference Fiser8 The change in scores (Delta scores) reflects the change in functional outcomes after paediatric intensive care.
The Virtual Pediatric Intensive Care Unit Performance System is also in the process of incorporating the Pediatric Logistic Organ Dysfunction (PELOD) scoring system.Reference Leteurtre, Martinot and Duhamel9 The Pediatric Logistic Organ Dysfunction (PELOD) score is determined by allocating zero, one, ten, or twenty points across six organ systems:
• neurologic
• respiratory
• cardiovascular
• renal
• haematological, and
• hepatic.
Given that multiple organ system dysfunction syndrome is more common than mortality in critically ill children, estimation of the severity of multiple organ system dysfunction measured by the Pediatric Logistic Organ Dysfunction (PELOD) score provides another dimension of severity risk adjustment in the intensive care unit.Reference Leteurtre, Martinot and Duhamel9 The Pediatric Logistic Organ Dysfunction (PELOD) score and the Pediatric Logistic Organ Dysfunction (PELOD) score measured on a daily basis (dPELOD) have been shown to have good calibration and discrimination for use in paediatric intensive care units.Reference Leteurtre, Martinot and Duhamel9
The future
The primary use of national and international databases of patients with congenital cardiac disease should be to improve the quality of care for patients with congenitally malformed hearts. Although such technology may determine that the rates of risk adjusted mortality are within acceptable limits for any given centre and any given lesion, and potentially reassure the public that potential problems will be quickly identified and acted upon, such data should be interpreted with caution. In a hypothetical analysis of twenty centres performing complex paediatric cardiac surgery, statistically there will always be a top and bottom quartile. However this does not automatically infer that the top quartile is performing well or that the bottom quartile is performing badly. As stated by Robert Wachter in a recent article in the Journal of the American Medical Association, “The simple act of defining excellence, measuring it, and disseminating the results skews the system. Although diverse measures sometimes catalyze improvements in unmeasured areas through fundamental system redesign, this is an unusual outcome. More typically, individuals and institutions begin to focus on improving their performance on the variables measured, in doing so turning away from others. This “playing for the test” is not only expected; in some cases, it is the point of the whole exercise.”Reference Wachter10
As we ensure adequate levels of mortality, we should begin to develop key, meaningful markers of morbidity in order to improve outcomes further beyond measurements of mortality. Admissions to the hospital for congenital cardiac surgery with the presence of the diagnosis of a complication, as defined by the International Classification of Disease 9th Revision (ICD-9), had a substantially greater odds of death compared with admissions without a complication, with an odds ratio of 2.4, and a p value of less than 0.001, after adjusting for case-mix, gender, race, insurance, and surgical volume of the hospital. Admissions with the diagnosis of greater than one complication were found to be at incremental risk of mortality.Reference Benavidez, Gauvreau, Del Nido, Bacha and Jenkins11
The technology and expertise is now within cost effective reach to allow inter-institutional sharing of data for the creation of extended, multi-centre distributed knowledge bases. These databases can be available to either improve the quality of inferences from anecdotal data or to demolish many of the barriers to formal randomized controlled trials. National and international sharing of data is critical to driving improvements in the areas of safety and quality in intensive care, and is the infrastructure of prospective randomized clinical trials. It would be advantageous and desirable to eventually develop common national and international datasets, which facilitate comparison of individual centre outcomes and enable tracking within the databases, similar to the strategies already implemented by several databases including the following:
• the international Databases of The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery
• the Central Cardiac Audit Database of the United Kingdom
• the Paediatric Intensive Care Audit Network in the United Kingdom, and
• the Virtual Pediatric Intensive Care Unit Performance System database in the United States of America.
The utilization of common nomenclature and datasets by these subspecialty databases will facilitate the eventual linking of these databases and the creation of a comprehensive database that spans conventional geographic and subspecialty boundaries.
Acknowledgement
We thank The Children’s Heart Foundation (http://www.childrensheartfoundation.org/) for financial support of this research.
