New pharmaceutical products require a substantial amount of research and assessment before licensing: in contrast, the evidence available on efficacy and safety of new procedures is often inadequate for robust health technology assessment (HTA) in both its quantity and quality (12). Randomized controlled trials (RCTs), which have been the dominant framework for evidence-based medicine, are uncommon for new procedures. One reason for this is that RCTs of procedures are constrained by practical and technical difficulties such as blinding and variation in operators’ competency (Reference Britton, McKee and Black3). Observational studies are more common, despite their limitations, they can provide valid evidence, particularly on safety outcomes (Reference Glasziou, Chalmers, Rawlins and McCulloch8;Reference Potts, Prata, Walsh and Grossman19).
The Interventional Procedures (IP) Programme of the UK's National Institute for Health and Clinical Excellence (NICE), has published guidance about the efficacy and safety of over 300 procedures since 2002. Its assessments are based on the available peer reviewed publications, commentary from clinical experts and reports from patients of their experiences (13). When the evidence is inadequate, then occasionally a “use only in research” recommendation is made, but the practicality of this type of guidance is limited by the many obstacles to setting up new research, including bureaucratic and funding difficulties. More often, recommendations are made for a procedure to be used only with “special arrangements” for clinical governance, patient information and consent, and audit. Not infrequently, the committee drafting these recommendations sees a need for gathering data on particular safety and/or efficacy outcomes, which would inform review of its guidance.
The obvious mechanism for data collection is a register, including data on all patients treated by the procedure. In an ideal world, the scope of such a register would include all patients presenting with the particular disease or indication, and would incorporate very detailed information, including case mix data. Such ambitious registers are slow to set up and require intensive supervision. They depend on very good support for data input, and data analysis may be complex and time-consuming (Reference Newton and Garner15).
Commonly, new procedures are undertaken by clinicians in several different (unknown) hospitals, without dedicated support for data collection and input. All this argues for creation of registers with small numbers of data items, focused on the requirement for more information about safety and efficacy, with the potential for rapid setup and a relatively modest demand on clinicians regarding data input. Good coverage (i.e., inclusion of as many treated patients as possible) is a fundamental principle.
Pectus excavatum is the most common congenital deformity of the sternum and anterior chest wall. It occurs in one in 1,000 live births and is progressive, with the degree of chest deformity worsening with the child's growth and development. The main indication for treatment is usually cosmetic, but the deformity can occasionally cause respiratory problems. Traditional treatment is by major open surgery (commonly, the Ravitch procedure) (Reference Davis and Weinstein4). The Nuss procedure involves the surgical placement of metal bars underneath the sternum to elevate it and correct the deformity (Nuss 2005). This is called minimally invasive, but reported adverse events include cardiac and liver injury (Reference Gips, Zaitsev and Hiss7). When NICE assessed the evidence on the Nuss procedure in 2003, the published evidence was deemed inadequate for full assessment of safety and efficacy. Recommendations were therefore published which stipulated submission of data on all procedures to a register, which was specially adapted for this purpose. These data were used for review of the procedure, with an updated evidence base, in 2008.
This study reviews the coverage and usefulness of a short-term register, established specifically for the review of NICE guidance on safety and efficacy of minimally invasive repair of pectus excavatum (also known as the Nuss procedure) to provide empirical evidence to inform the general research question about the strengths and limitations of supporting data collection and using observational evidence for HTA processes.
METHODS
The register data set was drafted by interested clinicians immediately after NICE guidance for the procedure was published in 2003. With their collaboration, this was subsequently revised and adapted by an academic institution linked to NICE (the School of Health and Related Research in Sheffield—ScHARR).
Because the number of cases was expected to be small, it was decided that an on-line system for data submission was not necessary and that clinicians would be sent a pack offering them the options of (a) data submission using a CD-Rom containing a downloadable Access (Microsoft Corp) database or (b) a simple paper data form. Routine hospital statistics were used to identify all hospitals undertaking procedures for correction of pectus excavatum in England and lead clinicians in each hospital were asked to submit data on all Nuss procedures that they had undertaken, returning it to ScHARR. The pack was sent out in the summer of 2007. Over the next year, reminders were sent out every 3 months. Analysis of the data was undertaken by NICE staff with advice from the pediatric surgeon (fourth author, R.W.) who had been involved in the design of the register.
Although the Register collected details of cases undertaken between April 2000 and the end of 2008, coverage was assessed only for the procedures undertaken between 2004–07 inclusive. This period was chosen to enable comparison with nationally collected Hospital Episode Statistics (HES—England). These statistics offered only a procedure code (OPCS code T02.1 correction of pectus deformity of chest wall) and diagnostic code (ICD code Q676 Pectus Excavatum), which did not differentiate between the minimally invasive Nuss procedure and open surgical repair (14). To derive a figure for the number of true Nuss procedures undertaken at each hospital, a letter was sent to lead clinicians requesting information about local practice used to repair pectus excavatum. Lead clinicians in the hospitals were not contacted if the hospital was known to be too small to have a cardiothoracic unit or data records showed that only one case had been undertaken. If no response was received, neighboring clinicians were asked to provide information where they could. Lead clinicians in hospitals submitting data to the register were contacted if routine statistics identified more cases than had been submitted.
Data fields were selected for analysis if they contained data that provided insight into outcomes that previous NICE guidance had identified as important or were related to safety of the procedure. Ninety-five percent confidence intervals for adverse events rates were derived using the Wilson method (Reference Wilson24).
The literature was searched in October 2008 for any new publications relating to efficacy and safety of the procedure using the following key words: minimally invasive, repair, pectus excavatum, pectus bar, Nuss, thoracoscopy, endoscopic.
Data from the register, the published literature, advice from nominated specialists, and information from patients were reviewed by the NICE Interventional Procedures Advisory Committee (IPAC) in March 2009. After the meeting, members of the Committee were asked to complete a questionnaire evaluating the usefulness of the register data (see Table 1 for details) in terms of its impact on the recommendations made in the guidance.
Table 1. Summary of Evaluative Questions Put to HTA Committee
HTA, health technology assessment.
RESULTS
Details of 260 procedures were submitted to the register from thirteen hospitals (covering a period from April 2000 to the end of 2008, 8 years 9 months). Fifty-nine of these cases were submitted as a result of work done to validate the register and assess its coverage (These cases were added to the presentation of data for HTAbut were excluded from the validation exercise as the data was submitted after coverage assessment.)
During the period selected for validation of the register (2004–07), data on 156 procedures were submitted to the register from eleven hospitals. For the same period, HES statistics recorded data on an additional 246 procedures from another twenty-six hospitals that had not submitted data to the register. Correspondence with lead clinicians in those hospitals identified that a total of 3 cases were coding errors (three hospitals), 140 cases (from eleven hospitals) used traditional open rather than the Nuss technique, and 23 cases were previously unidentified Nuss procedures from two hospitals that were subsequently invited to contribute to the national register. It was impossible to find out whether the remaining eighty cases, undertaken in ten hospitals, were open, minimally invasive or coding errors.
The literature search revealed substantially more publications since the procedure was first assessed (approximately eighty-eight studies including 5,400 patients, compared with twenty-six studies of 1,760 patients in 2003).
Outcomes assessed in published literature were heterogenous making comparison difficult. The main efficacy outcome for this procedure is cosmetic appearance but the assessment of this outcome varied between parents, clinicians, and patients themselves. Some published studies did not provide details of the method of assessment of cosmetic appearance (for example, it was not clear whether it was assessed by the patient or clinician). Therefore, for the purposes of this study, we have specifically compared the register data only with the three largest case series that reported cosmetic outcomes. These three case series, of 322, 190, and 124 patients, reported “excellent” results in 91 percent, 78 percent, and 59 percent of patients, respectively. Only the latter study specified how the results were assessed—surgeon-rated on a scale from 1 (excellent) to 10 (poor).
By contrast, the register recorded patient-rated cosmetic appearance before the procedure, on a scale from 1 (“dislike”) to 10 (“like”), and at any subsequent follow-up appointments, on a scale from 1 (“the same as before the operation”) to 10 (“perfect”). Mean cosmetic appearance rating improved from 3.1 before the procedure to 8.4 after it (n = 119 patients who had both a preprocedure and postprocedure rating; median follow-up: 170 days).
The most common adverse events were the same in the register and the published literature: they were pleural effusion, bar displacement, pneumothorax, and wound infection. To present a comparison of the information from the register and published literature, data from the three largest case series (accounting for 1,162 cases), published since 2003 are presented in Table 2. The frequency of adverse events were broadly similar between the two sources, although one case series reported a higher rate of bar displacement than the register (50 of 668 cases [7.5 percent], as compared with 5 of 260 [2 percent] registered cases) and the register reported a slightly higher rate of wound infection than the literature. Definitions of adverse events differed between studies. The published literature reported the following significant but infrequent adverse events that were not identified on the register; pericardial effusion, pericarditis, cardiac, and liver injury. The register did not identify any additional adverse events that had not been reported in the published literature.
Table 2. Rates of Common Adverse Events in the Register and the Three Largest Published Studies
After reviewing the data presented from the register and the updated literature search, the Interventional Procedures Advisory Committee, decided to change its recommendation for use of the procedure, removing the requirement for “special arrangements” for clinical governance, consent and audit, because the new evidence on safety and efficacy was adequate.
Eleven of seventeen (65 percent) members of the Committee responded to the questionnaire about usefulness of the register data. All these respondents judged that the register data were consistent with published evidence. Eight (73 percent) respondents thought that the register had had an important or very important impact on the decision made by the Committee. Four (36 percent) respondents thought that the register raised issues that had not been identified by published literature; in particular they commented that the register identified a broader range of efficacy outcomes that included patients’ views and provided insight into outcomes achieved in a “real world” nonresearch setting.
DISCUSSION
The register examined by our study was created with the explicit objective to assess the coverage and usefulness of a short-term register, established specifically for the review of NICE guidance on safety and efficacy, but also to support its monitored introduction into clinical practice. Set against these objectives, the register was successful in augmenting the peer-reviewed evidence about the procedure, and decision makers indicated that they judged evidence from the register as useful. The register was less successful in monitoring dissemination, and good coverage was only achieved after the validation exercise and after contact with hospitals identified through routinely collected HES data.
Internationally, there are several examples of the use of registers to support HTA. For example, the Medical Services Advisory Committee (MSAC) of Australia have used information from the PillCam Capsule Endoscopy Register to validate assumptions used in their economic modeling and influence health technology appraisal decision making (Reference Malley, Selby and Jordan11). They concluded that, if interim funding coupled with data collection is to become an effective mechanism for bridging the evidence gap, registers need to be designed that cater for the unique characteristics of individual procedures. In their recent commentary “Registries for robust evidence,” Dreyer and Garner cited several published examples (Reference Dreyer and Garner5) including the use of Safety data from the acyclovir pregnancy register to change the Food and Drug Administration (FDA) pregnancy labeling categories (Reference Tilson, Doi and Covington22), and the use of observational data reflecting other therapies used at that time to compare outcomes from coronary artery bypass graft (CABG) surgery with those from percutaneous coronary intervention (Reference Bravata, McDonald and Gienger2).
In the United Kingdom, data linkage between the National Joint Register and the Department of Health's hospital episode statistics recently showed that the need for hip revision surgery is particularly high among women who received the new hip resurfacing procedure (3.7 percent within 3 years) (Reference Sibanda, Copley and Lewsey21).
In our study, the substantial “mismatch” of potentially relevant activity between the register and HES information was explained to a large extent by the coding in HES, which failed to differentiate between the procedure of interest (Nuss) and open surgery. The validity of data collected by registers compared with data captured by HES, and the critical importance of appropriate coding categories to enable this, have been debated before, in the context of cardiac procedures (Reference Aylin, Lees and Baker1;Reference Gibbs, Monro and Cunningham6). It would be ideal if the coding system used in the United Kingdom could be more responsive and timely in producing procedure codes of greater specificity for new interventional procedures. Steps have been taken toward that goal in recent years, including the production in quick succession of two updates of the Office for Population Census and Surveys (OPCS) IV classification system, partly prompted by the introduction of the “payment by results” system, to help measure (and pay for) hospital care more accurately (OPCS classification ref). In addition, the NICE Interventional Procedures Programme has now established regular dialog with those responsible for coding in the United Kingdom, with the aim of ensuring that in the future, new procedures assessed by the Programme receive specific codes in a more timely manner than in the past.
Neither the register nor HES data can be assumed a priori to represent a “gold standard.” The register ought to be superior for specificity (i.e., 100 percent of episodes of care captured by it will indeed truly relate to the Nuss procedures), but our data demonstrated an estimated 87 percent sensitivity (152/175 confirmed cases) rate during the validation period (i.e., not all Nuss procedures carried out in the United Kingdom during the study period had been submitted to it, even after the validation exercise and contact with clinicians based on its findings). Conversely, the HES dataset has manifestly low specificity (an estimated 61 percent based on 144 of 237 cases identified from the NUSS registered centers during the validation period), but it is likely that it may have had much higher sensitivity (i.e., capturing a greater proportion of Nuss procedures carried out during the study period). Linking of individual records from the register and the HES system would help to resolve these differences, and would, over time, also help improve the quality of each individual dataset. We would strongly support such record linkage of different observational datasets (such as registers and HES) as a method which could greatly enhance both the validity and usefulness of each contributory dataset (Reference Newton16;Reference Powell and Buchan20). Pragmatically, we recommend crude (i.e., “unlinked”) comparisons of activity volumes in audit/register datasets and in HES, to help identify potential under- or over-ascertainment of relevant activity (or both), in either data source until infrastructural, computational, and data protection obstacles preventing widespread use of record linkage processes are overcome.
In the context of health technology assessment, the primary aim of the register was to augment the published data on the Nuss procedure, which was considered inadequate when the procedure was first assessed in 2003. The quantity of evidence produced by the register could be considered a substantial addition to that original evidence base. In this particular case, however, several studies were published in the international literature between the first assessment and the review, providing a considerable increase in the number of treated patients. This could perhaps be seen to have reduced the usefulness of the evidence generated by the register. In our experience, this scale of increase in published literature is relatively unusual and it was impossible to predict. Foreseeing the speed and volume by which evidence about a specific new interventional procedure will accumulate in the near future is difficult, unless there is knowledge about studies in progress. We routinely ask our clinical experts about studies (and registers) known to them, but this enquiry tends to produce responses only about major multicenter trials or studies being done in units with which they have close contact.
In augmenting the evidence from published literature, registers are most often perceived to add value by providing information about use of procedures in a “real world” context (rather than the context of a controlled trial). This allows detection of uncommon adverse events and in the long term may fulfill a “surveillance” function. They are generally regarded as less useful for generating efficacy outcomes, but the register in this study provided subjective “quality of life” data which was arguably better than that from the published studies.
In our brief questionnaire survey, members of the NICE Committee who appraise the efficacy and safety of new interventional procedures judged the data from the register to be useful. It is worth considering how such data might influence decision making, when appraised alongside the published literature. Concordant evidence (between the register and published literature)—as in this case—provides reassurance in making decisions about safety and efficacy. Conversely, potential discordance between registers and peer-reviewed evidence might generate a skeptical attitude toward the validity or the relevance of published evidence. We were not able to examine such qualitative aspects of the potential impact of use of register data by appraisers empirically, but we would encourage research to examine these possibilities.
In principle, our evaluation of the Nuss interventional procedures register is an example of the potential usefulness to HTA processes of registers for new procedures. The main limitation of our study in this respect is that it was no more than a “case study” (essentially a study with a sample of “one”). Therefore, any attempts to generalize should be cautious. Richer observations and inferences about the usefulness of registers to policy makers could be made if register data collection were “routine” for most (or even all) new interventional procedures, as we have previously advocated (Reference Lyratzopoulos, Patrick and Campbell10). The development of registers specifically to inform HTA is relatively new—and this realization motivated the present study. Tunis and Chalkidou have suggested that funding for new procedures with a limited evidence base should be conditional on submission of data to registers (Reference Tunis and Chalkidou23). This in an attractive idea because the data collected could have other uses in addition to HTA, including provision of support for health service purchasing decisions; audit and quality assurance; and improvement of care (through feedback to clinicians and managers). However, resources required to support such developments are scarce, and there are also computational and data protection issues (Reference Newton and Garner15). In this context, it would have been useful to evaluate the cost of the NUSS register. Such costs relate both to the “central” support for data collection, collation, and analysis; as well as the time and resources required by clinicians and other frontline staff to enter data. This study was carried out over a 6-year period, and there are insufficient data available to produce meaningful estimates of such costs. The question about the cost of observational data collection is an important one, and further research would be useful, ideally collecting such cost data prospectively.
A potential solution that could help improve effective use of scarce resources, and also improve clinicians’ readiness to submit data, would be the creation of Web-based secure registers with small numbers of data items, that could subsequently be “record-linked” to data on specified patients included in “common good” datasets such as HES. Such systems would potentiate the value of evidence that can be generated by each individual data source. This could prove an effective and cost-efficient solution to the complexities of evidence generation in relation to surgical and interventional procedures innovation, informing patients, as well as different professional groups (clinicians, managers, and HTA or regulatory bodies).
CONTACT INFORMATION
Hannah E. Patrick, MRCP, MFPHM (Hannah.Patrick@NICE.org.uk), Consultant Clinical Adviser, Sally Gallaugher, MPH (Sally.Gallaugher@nice.org.uk), Technical Analyst, Interventional Procedures Programme, National Institute for Health and Clinical Excellence, MidCity Place | 71 High Holborn, London WC1V 6NA, UK
Carolyn Czoski-Murray, RGN, MA, MSc (c.murray@sheffield.ac.uk), Research Fellow, School of Health and Related Research University of Sheffield, Regent Street, Sheffield S1 4DA, UK
Robert Wheeler, MS FRCS (Robert.Wheeler@suht.swest.nhs.uk), Senior Lecturer and Consultant Paediatric Surgeon, Wessex Regional Centre for Paediatric Surgery, Southampton University Hospitals Trust, Tremona Road, Southampton, Hampshire SO16 6YD, UK
Marc Chattle, MA (m.chattle@sheffield.ac.uk), ReBIP Project Administrator, School of Health and Related Research University of Sheffield, Regent Street, Sheffield S1 4DA, UK
Mirella Marlow, MA MBA (Mirella.Marlow@NICE.org.uk), Associate Director, Interventional Procedures Programme, National Institute for Health and Clinical Excellence, MidCity Place 71 High Holborn, London WC1V 6NA, UK
Georgios Lyratzopoulos, MD, FFPH, MRCP, MPH, DTM&H, Ptychio Iatrikes (georgios.lyratzopoulos@nice.org.uk), Consultant Clinical Adviser, National Institute for Health and Clinical Excellence, MidCity Place, 71 High Holborn, London WC1V 6NA, UK; Clinical Senior Research Associate, Department of Public Health & Primary Care, University of Cambridge, Institute of Public Health, Forvie Site, Robinson Way, Camridge CB2 0SR, UK
Bruce Campbell, MS, FRCP, FRCS (bruce.campbell@nice.org.uk), Chair, Interventional Procedures Advisory Committee, National Institute for Health and Clinical Excellence, MidCity Place, 71 High Holborn, London WC1V 6NA, UK; Professor, Peninsula College of Medicine & Dentistry, Universities of Exeter and Plymouth, UK
