Background

Evaluation of care in any medical or surgical field requires recognition that the outcomes of patients are influenced by the nature and severity of illness, as well as by effectiveness of treatment. Any assessment of effectiveness of treatment, therefore, must take into account the characteristics of both the patients and the procedures. These characteristics may be described alternatively as “risk factors” or “degrees of complexity”. In the discipline of cardiac surgery in adult patients with acquired cardiac disease, the overall process of evaluating and benchmarking quality of care has progressed over the past two decades to a point farther along than in virtually any other subspecialty in medicine. The impetus for this effort came in part as a response to the use by various regulatory agencies of unadjusted mortality data from hospitals for surgical coronary revascularization procedures, and the widespread publication and dissemination of this raw data. Led by organizations such as the Workforce on National Databases of The Society of Thoracic Surgeons, empirical clinical data, initially from tens of thousands, and more recently hundreds of thousands of surgical procedures, have been used to develop risk-models, to increase the accuracy with which the outcome of a given procedure on a given patient can be predicted, and to compare outcomes on non-identical patient groups between centres, surgeons and eras.Reference Shahian, Blackstone and Edwards¹ The process has involved the refinement of clinical sets of data, establishment of core variables, standardization of definitions, creation of quality controls, and the application of rigorous statistical methodology.Reference Ferguson, Hammill, Peterson, DeLong and Grover² Some early efforts relied on Bayesian models, since they are robust with regard to incomplete data, which was an important problem in the early experience of the database. At present, logistic regression models are the principle statistical technique for risk modelling for cardiac surgery for adults with acquired cardiac disease.

Initial risk modelling efforts were limited to data concerning patients undergoing isolated coronary artery bypass grafting. At the outset, measures of outcome were confined to mortality prior to discharge from the hospital. Over the past decade, these initial risk models have been recalculated and refined, using data from increasingly large patient populations. Risk models have been developed for isolated cardiac valve replacement procedures, and for combined coronary bypass and cardiac valve replacement procedures. More recently, measures of outcome have been expanded to include aggregate indices of important and durable morbidities, as well as short-term mortality.

The problem of evaluating quality of care in the management of pediatric patients with congenital cardiac diseases is very different, and in some ways a great deal more challenging. The minimal datasets that, by consensus, are used currently by the congenital databases of The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery, contain 164 individual diagnoses and 204 different procedures.Reference Mavroudis and Jacobs³ Many of these diagnoses, while unique and important, are relatively rare. Many of the procedures are performed in small numbers. Thus, the application of contemporary conventional risk modelling methodology would be applicable only to a fraction of the population of interest, and provide no objective means of evaluating the performance of centres or surgeons across a diverse and widely variable spectrum of diagnoses and procedures. Recognition of this problem led to the pursuit of new creative approaches to stratification of congenital cardiac patients and procedures with respect to complexity and risk of adverse outcomes. The first step was to develop a nomenclature that facilitated meaningful comparison across institutions. The next step was to develop a means of accounting for differences in “case-mix”. Nearly simultaneously, two groups undertook the development of tools or systems to address the heterogeneity of groups of patients with congenital cardiac diseases with respect to the challenge and likelihood of successful management, expressed as either the complexity of the operation or the relative risk of mortality. One group’s efforts led to the development of a methodology known as “Risk Adjustment for Congenital Heart Surgery – 1”, commonly named “RACHS-1”. The other led to the development of “The Aristotle Complexity Score.” Both of these tools are now utilized in the databases of The European Association for Cardio-Thoracic Surgery and The Society of Thoracic Surgeons.

The discrimination of tools for stratification of complexity as predictor of a given outcome such as mortality can be quantified by calculating the area under the receiver operating characteristic curveReference Shroyer, Coombs and Peterson⁴, or “C-statistic”, as determined by univariable logistic regression. The area under the receiver operating characteristic curve represents the probability that a randomly selected patient, who had the outcome of interest, such as mortality prior to hospital discharge, had a higher predicted risk of the outcome compared with a randomly selected patient who did not experience the outcome. The area under the receiver operating characteristic curve is generally 0.5 to 1.0, with 0.5 representing no discrimination, that is a coin-flip, and 1.0 representing perfect discrimination. For example, the model for risk-adjustment of the Adult Cardiac Surgery Database of The Society of Thoracic Surgeons, for predicting 30-day mortality after surgery to place coronary arterial bypass grafts, contains 28 clinical variables and has a C-statistic of 0.78.Reference Shroyer, Coombs and Peterson^4, Reference Welke, Shen and Ungerleider⁵

In this review article, we will discuss several aspects of the Risk Adjustment for Congenital Heart Surgery – 1 methodology and the Aristotle Complexity Score:

• • the rationale for the development of each system

• • the current use of each system

• • the plans for future enhancement of each system

• • the potential for unification of these two tools.

Risk Adjustment for Congenital Heart Surgery-1

The Risk Adjustment for Congenital Heart Surgery-1 methodReference Jenkins, Gauvreau, Newburger, Spray, Moller and Iezzoni^6–Reference Allen, Gauvreau, Bloom and Jenkins¹¹ was developed by a group of investigators at Children’s Hospital Boston under the leadership of Dr. Kathy Jenkins.Reference Jenkins, Gauvreau, Newburger, Spray, Moller and Iezzoni⁶ The goal was to adjust for baseline differences in case-mix and risk when comparing mortality prior to discharge from the hospital among groups of patients less than eighteen years of age undergoing surgery for congenital cardiac diseases. A nationally representative eleven-member panel of paediatric cardiologists and cardiac surgeons used clinical judgment to place 207 surgical procedures defined by the codes from the “Current Procedural Terminology 4” and the “International Classification of Diseases, Ninth Revision, Clinical Modification” into six groups judged to have similar risk for in-hospital mortality. These risk categories were then refined using empirical data, of mortality prior to discharge from the hospital, from the Pediatric Cardiac Care Consortium and three state-wide hospital discharge databases, from Illinois from 1994 and Massachusetts and California from 1995. The final method included six risk categories, with category 1 representing the lowest risk and category 6 the highest, as well as three additional clinical factors:

• • age at operation, with three categories of less than or equal to 30 days, 31 days to one year, and greater than or equal to one year;

• • prematurity; and

• • the presence of a major non-cardiac structural anomaly such as trachea-oesophageal fistula.

While some have identified the use of administrative data coded with the International Classification of Diseases as a potential short-coming, the developers of the Risk Adjustment for Congenital Heart Surgery-1 method point out that this administrative data has, up to now, been used to support a significant fraction of research about healthcare services pertinent to paediatric cardiology. Furthermore, algorithms, that use administrative data coded with the International Classification of Diseases to examine outcomes of surgery for congenital cardiac diseases, have been endorsed by the Agency for Healthcare Research and Quality of the federal government of the United States of America.Reference Jenkins⁸ It is recognized that administrative data coded with the International Classification of Diseases are frequently the only available data that are population-based, and can thus be a crucial source of information for certain types of research. The Risk Adjustment for Congenital Heart Surgery-1 method has shown excellent performance in a variety of settings, and has been used extensively in research about clinical outcomes in the United States and abroad.Reference Boethig, Jenkins, Hecker, Thies and Breymann^9–Reference Allen, Gauvreau, Bloom and Jenkins¹¹ The developers of the Risk Adjustment for Congenital Heart Surgery-1 method point out that a strength of their methodology is the ability to incorporate relatively rare procedures. This ability is especially important because surgery for congenital cardiac diseases in the paediatric age group is characterized by extreme diversity.

Utilizing several databases, this methodology has been evaluated with respect to the correlation between the level from the Risk Adjustment for Congenital Heart Surgery-1 tool and observed discharge mortality. In the administrative hospital discharge database used in the development of the Risk Adjustment for Congenital Heart Surgery-1 method, the C-statistic was 0.749 for risk-category alone and 0.814 when the additional clinical factors were incorporated. In the database of the Pediatric Cardiac Care Consortium, which is more similar to the STS congenital database, the C-statistic was 0.784 for risk-category alone and 0.811 with the additional factors.Reference Jenkins, Gauvreau, Newburger, Spray, Moller and Iezzoni⁶ Subsequent work in a variety of databases has resulted in C-statistics for the Risk Adjustment for Congenital Heart Surgery-1 method ranging from 0.74 to 0.85.

The congenital databases of The European Association for Cardio-Thoracic Surgery and The Society of Thoracic Surgeons have included the Risk Adjustment for Congenital Heart Surgery-1 system in their database reports since 2006. In the 2006 report of the congenital database of The Society of Thoracic Surgeons,Reference Jacobs, Jacobs, Mavroudis, Lacour-Gayet and Tchervenkov¹² 85.8%, or 27,202 out of 31,719 operations were eligible for analysis by the Risk Adjustment for Congenital Heart Surgery-1 system.

As with any risk adjustment tool, it is important that the intended use of the Risk Adjustment for Congenital Heart Surgery-1 method be understood. It was not created to predict the risk of death for individual patients, but rather to be a tool that allows meaningful comparisons across groups of patients. Objectives for the future include maximization of the utility and applicability of the Risk Adjustment for Congenital Heart Surgery methodology by applying it to all of the diagnosis and procedure codes of the International Pediatric and Congenital Cardiac Code, as used in the congenital cardiac surgery Databases of The European Association for Cardio-Thoracic Surgery and The Society of Thoracic Surgeons. It will thus incorporate newer operations that have been recently developed and will be applicable to virtually all procedures in these databases. As such, the universe of clinical data available for the ongoing validation of the Risk Adjustment for Congenital Heart Surgery methodology will be expanded considerably. These objectives also will be broadened to include development of measures of morbidity after surgery for congenital heart diseases and measures of performance.

The Aristotle Complexity Score

In 1999, under the leadership of Dr. Francois Lacour-Gayet, the Aristotle Committee was created to address the issue of stratification of complexity in surgery for congenital cardiac diseases. The developers of this tool recognized that standard methods of benchmarking in quality of care assessment were based upon stratification of risk, with nearly exclusive emphasis on the measurement of the outcome of operative mortality. They believed that to assess outcomes, including comparison of outcomes between centres, and to establish a platform for continuous improvement in quality, stratification based upon the risk of mortality alone is insufficient. The fundamental principle of the Aristotle Complexity ScoreReference Lacour-Gayet, Clarke and Jacobs^13–Reference Derby, Kolcz, Kerins, Duncan, Quezada and Pizarro²¹ is to define complexity as a constant for the challenge presented by a given surgical procedure.Reference Lacour-Gayet, Clarke and Jacobs¹³ The Aristotle committee postulated that the complexity of a given procedure in surgery for congenital cardiac diseases is the sum of three factors or indices: the potential for operative mortality, the potential for operative morbidity, and the technical difficulty of the operation. In the early stages of development of a tool for stratification of complexity embodying these three elements, the investigators confronted a dilemma: a new international system of nomenclature had been developed,Reference Mavroudis and Jacobs³ but surgical databases using this language for fields of data were in their infancy. There was a paucity of empirical data of consistent quality. Some congenital heart surgeons, including some of those dealing with the most complex patients, were reluctant to participate in registries. There was a feeling that centres dealing with the most challenging patients in large numbers, might have mortality rates above the average level. An obvious need existed for a fair benchmarking tool to relate outcome measures, including mortality, to complexity and case-mix. The proposed tool for stratification of complexity was named the Aristotle Complexity Score, following Aristotle’s belief in the importance of current opinion. In the year 350 before the Common Era, in Rhetoric, Book 1, Aristotle stated that “When there is no scientific answer available, the opinion, or “doxa”, perceived and admitted by the majority, has the value of truth.”

The Aristotle Basic Complexity Score is a tool that was developed by a panel of experts, made up of 50 surgeons repairing congenital cardiac defects in 23 countries and representing several major professional societies. The goal was to develop a tool for stratification of complexity that could be used to equitably evaluate and compare the performance of centres performing surgery for congenital cardiac diseases. The Aristotle Basic Complexity Score allocates a basic score to each operation, varying from 1.5 to 15, with 15 being the most complex, based on the primary procedure of a given operation as selected from the Procedure Short List of the International Nomenclature utilized by The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery. The Aristotle Basic Complexity Score represents the sum or aggregate of scores assigned to a given procedure for the three components of complexity:

• • potential for mortality, which varies from 0.5 to 5

• • potential for morbidity, which varies from 0.5 to 5 and

• • technical difficulty which varies from 0.5 to 5.

To facilitate analysis across large populations of patients, each procedure is then assigned an Aristotle Basic Complexity Level, which is an integer ranging from 1 through 4 based on the Aristotle Basic Complexity Score:

• • Aristotle Basic Complexity Level 1 is an Aristotle Basic Complexity Score of 1.5 to 5.9

• • Aristotle Basic Complexity Level 2 is an Aristotle Basic Complexity Score of 6.0 to 7.9

• • Aristotle Basic Complexity Level 3 is an Aristotle Basic Complexity Score of 8.0 to 9.9

• • Aristotle Basic Complexity Level 4 is an Aristotle Basic Complexity Score of 10.0 to 15.0.

Of 145 procedures from the original Procedure Short List of the International Nomenclature utilized by The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery,

• • 29 procedures are in Aristotle Basic Complexity Level 1

• • 46 procedures are in Aristotle Basic Complexity Level 2

• • 45 procedures are in Aristotle Basic Complexity Level 3 and

• • 25 procedures are in Aristotle Basic Complexity Level 4.

Both the Score and the Level are useful tools; the appropriate tool can be chosen to match the required analysis.Reference Lacour-Gayet, Clarke and Jacobs^14, Reference Jacobs, Lacour-Gayet and Jacobs¹⁵ Initial data from the multi-institutional databases of The European Association for Cardio-Thoracic Surgery and The Society of Thoracic Surgeons indicate that the Aristotle Basic Complexity Score and Aristotle Basic Complexity Level correlate well with mortality prior to discharge from the hospital after surgery for congenital cardiac diseases.Reference Jacobs, Lacour-Gayet and Jacobs^15–Reference O’Brien, Jacobs and Clarke¹⁷

In an analysis of data from the congenital databases of The European Association for Cardio-Thoracic Surgery and The Society of Thoracic Surgeons, the Aristotle Basic Complexity Score is associated with both mortality, with a C-statistic that equals 0.70, as well as prolonged post-operative length of hospital stay, defined as postoperative length of stay greater than 21 days, with a C-statistic that equals 0.67.Reference O’Brien, Jacobs and Clarke¹⁷ This implies that the Aristotle Basic Complexity Score generally discriminates between low-risk and high-risk operations.Reference O’Brien, Jacobs and Clarke¹⁷ In this study,Reference O’Brien, Jacobs and Clarke¹⁷ prolonged post-operative length of hospital stay was regarded as a very general proxy measure of morbidity. Additional concomitant procedures may alter the complexity of an operation. Discrimination was slightly greater when the analysis was restricted to operations consisting of a single procedure, and excluding operations with multiple component procedures, with a C-statistic that equals 0.73 for mortality and a with a C-statistic that equals 0.70 for prolonged post-operative length of hospital stay. When the mortality and morbidity components of the Aristotle Basic Complexity Score were examined separately, the mortality component predicted mortality with a C-statistic that equals 0.68, and the morbidity component predicted prolonged post-operative length of hospital stay with a C-statistic that equals 0.67. Finally, when fixed hospital-specific intercepts were added to the logistic regression models along with the Aristotle Basic Complexity Score, the C-statistic was 0.74 for mortality and 0.72 for prolonged post-operative length of hospital stay. For comparison, the C-statistics of the models containing hospital effects only, excluding Aristotle Basic Complexity Score, were 0.63 for mortality and 0.62 for prolonged post-operative length of hospital stay. Thus, adding the Aristotle Basic Complexity Score to a model containing hospital effects appears to improve its discrimination.Reference O’Brien, Jacobs and Clarke¹⁷

The congenital databases of The European Association for Cardio-Thoracic Surgery and The Society of Thoracic Surgeons have included the Aristotle Basic Complexity Score and Aristotle Basic Complexity Level in their database reports since 2002. In the 2006 report of the congenital database of The Society of Thoracic Surgeons,Reference Jacobs, Jacobs, Mavroudis, Lacour-Gayet and Tchervenkov¹² 94.0%, or 29,813 out of 31,719 operations were eligible for analysis by the Aristotle Basic Complexity Score and Level.

The Aristotle Comprehensive Complexity Score adds further discrimination to the Basic Score by incorporating two sorts of patient specific complexity modifiers:

• • procedure-dependent factors, including anatomical factors, associated procedures, and age at procedure and

• • procedure independent factors, including general factors such as weight and prematurity, clinical factors such as preoperative sepsis or renal failure, extracardiac factors such as duodenal atresia and imperforate anus, and surgical factors such as reoperative sternotomy.

As such, additional points, up to a maximum of 10, are added to the Basic Score to account for the added complexity and challenge imputed by these modifying factors. The Aristotle Comprehensive Complexity Score has been used by numerous investigators to analyze the outcomes from complex procedures.Reference Sinzobahamvya, Photiadis and Kumpikaite^18–Reference Derby, Kolcz, Kerins, Duncan, Quezada and Pizarro²¹ It should be understood that the primary objective of the Aristotle Complexity Score methodology is to evaluate and compare performance in the surgical management of congenital cardiac diseases. It was not intended as a method of predicting operative mortality for a given patient.

The Congenital Database Taskforce of The Society of Thoracic Surgeons is in the process of developing a new tool, the Aristotle Average Complexity Score, which will be based primarily on objective data from the congenital databases of The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery. Bayesian methods and subjective probability will be used where objective data are lacking. This tool will likely be incorporated into the analysis of the congenital databases of The Society of Thoracic Surgeons and The European Association for Cardio-Thoracic Surgery and will allow for the risk of mortality to be stratified into five levels.

Summary

In their initial forms, both the Risk Adjustment in Congenital Heart Surgery-1 methodology and Aristotle Complexity Score systems relied heavily upon expert opinion, and to varying degrees on empirical data. Certainly the inclusion of Technical Difficulty as one of three elements of the Aristotle Basic Complexity Score insures reliance, in part, on subjective probability. But, nearly ten years after the inception of these ideas for stratification of complexity and risk, a considerably larger mass of consistent and quality clinical data exists than was previously available. A goal, going forward, is to re-evaluate and further refine these tools so that, for example, the elements describing potential for operative mortality can be, to a greater extent, derived from empirical data. Ideally, the mortality elements of both the Aristotle Complexity Score and the Risk Adjustment in Congenital Heart Surgery-1 methodology will eventually be one and the same. But the big picture remains entirely different from the challenge of risk adjustment for coronary bypass surgery in adults, for example. Because of the diverse spectrum of cardiac congenital anomalies and surgical procedures, and the relative rarity of many of the anomalies and procedures, it seems likely that more refined methodologies for stratification of complexity, even when derived using larger empirical datasets, will continue to incorporate some element of subjective probability, acknowledging that there will remain some areas where limited empirical data exists for objective analysis.