Decision making about the appropriate use and dissemination of new healthcare technologies needs to be supported by the provision of timely and high quality evidence. For pharmaceutical treatments, strict licensing and regulatory processes motivate the production of a reasonable evidence base, but new surgical and other interventional treatments may be adopted into clinical practice with a limited amount of evidence (Reference Campbell, Barnes and Kirby1). Although systems exist for the regulation of medical devices, there has been little demand for legislation on interventional procedures or for the production of guidance on their use. This has meant that the evidence base for new interventional procedures is typically constrained and that such procedures have often disseminated with a poor evidence base and in the absence of any kind of control or monitoring (Reference Wilson9).
In the United Kingdom, since 2002, the Interventional Procedures Programme (IPP) of the National Institute for Health and Clinical Excellence (NICE) has had responsibility for assessing the efficacy and safety of new interventional procedures and for producing guidance on their use (8). Given the paucity of similar systems for this purpose elsewhere in the world, there were almost no examples from which to draw, and the development of this program was, therefore, innovative. The IPP reviews evidence and consults widely to produce guidance on the efficacy and safety of procedures and the circumstances in which they should be used, including, when appropriate, stipulations about data collection and analysis. When the evidence about efficacy and/or safety is inadequate, guidance may recommend that a procedure should only be carried out in the context of research (Box 1, Example 1); or that special arrangements should be made to ensure that: (i) hospitals have given approval for the procedure to be performed within their local governance strategies; (ii) patients are advised in a full and explicit way during the process of consent; and (iii) details of all procedures are audited and reviewed (Box 1, Example 2). When the evidence on efficacy and safety is judged as adequate, the guidance recommends that normal arrangements should be in place for governance, consent, and audit (Box 1, Example 3). Where there is positive evidence of lack of efficacy and/or safety about a procedure, guidence may recommend that procedure should not be used. The guidance may also make recommendations about the type of specialist clinicians who should select patients and undertake procedures, the need for training, submission of data to specific registers, and the need for particular information from further research, so that guidance can be reviewed in the light of an improved evidence base.
Box 1. Examples of the type of ‘headline’ guidance on the efficacy and safety of interventional procedures provided by NICE's Interventional Procedures Programme.
From exchanges with Healthcare Technology Assessment (HTA) groups and with organizations planning healthcare around the world, it became clear to us that there was considerable interest in the issue of assessing and providing guidance on the use of interventional procedures—especially those that were new. However, there was no collated information about whether systems were in place in different countries and what form they took, to provide a basis for discussions. This study was set up to acquire and collate that information. Its aim was to provide a source of reference for those wishing to develop systems, and to stimulate debate about the most effective and efficient ways of guiding the uptake of new interventional procedures. Despite a previous international survey of HTA organizations (Reference Martelli, La Torre and Di Ghionno4), no similar study concentrating on interventional procedures currently exists.
METHODS
We aimed to identify key informers working for HTA organizations with a national remit rather than those with subnational (e.g., regional or federal) responsibility. Such key informers were identified by using three different strategies: First, Web site inspection of relevant HTA organizations that are members of the International Network of Agencies for Health Technology Assessment (www.inahta.org/); Health Technology Assessment International (www.htai.org/); and European Network for Health Technology Assessment (www.eunethta.net/)—. This strategy proved to be the most successful. The second strategy was by writing to relevant HTA organizations from which no key informer had been identified, in an effort to find such individuals. The third strategy was by directly contacting by e-mail several individuals personally known to the authors to identify suitable informers. When there was more than one HTA organization for a particular country, the organization(s) with a specific remit for interventional procedures was selected where this was known, and if there was more than one such organization, an attempt was made to identify a key informer in each one. In total, thirty-eight individuals working for thirty-eight different organizations in thirty countries were contacted during January 2008 (Table 1).
Table 1. Organizations (n = 38) in Thirty Countries Contacted to Identify Key Informers for Participation in the Survey
The e-mail sent to each identified key informer was signed by the senior author (B.C., Chair of NICE's Interventional Procedures Advisory Committee) and introduced and explained the purpose and aims of the survey. The e-mail also included (either in the cover note or as attachments) the following: a recent example of interventional procedures guidance published by NICE, to illustrate the background to the study; an invitation to participate in the survey; a hyperlink to the Web site where the survey questionnaire could be found, alongside an electronic copy of the questionnaire (sent as an attachment); and a prompt for suggestion of alternative/additional key informers within their organization or country when judged relevant. Those choosing to take part in the survey could either complete the survey on-line (through the hyperlinked Web site) or by using the survey questionnaire attached to the e-mail and returning it to the investigators (by e-mail or post). Nonresponders were sent up to two reminders. If there was no response, an alternative key informer in the same country was contacted if one could be identified by means of specialists known personally to the senior author (B.C.), or suggested by colleagues at NICE, or sourced from Web sites of relevant organizations.
The survey questionnaire (available form the authors, on request) enquired about nine different aspects of the process of assessment and production of guidance for interventional procedures: (i) presence or absence of a national system with explicit responsibility for assessment of interventional procedures; (ii) the type and funding of the organization(s) with remit for interventional procedures; (iii) the system of selection of interventional procedures for assessment; (iv) the types and sources of evidence used in the assessment of interventional procedures; (v) who appraises the evidence; (vi) consultation on draft assessment (or guidance); (vii) the format of recommendations or guidance; (viii) status of the recommendations or guidance (assessment); and (ix) use of interventional procedures guidance from other countries.
RESULTS
Twenty-eight key informers working for twenty-eight organizations in twenty-five countries provided information that could be used in subsequent analysis (representing an 83 percent response rate for the thirty countries with a contacted HTA organization). Table 2 lists the agencies and the countries from which responses were received. Information relating to different organizations in the same country (three in Sweden and two in The Netherlands) was amalgamated in subsequent analysis for each of those countries, to describe a national picture.
Table 2. HTA Organizations with a Key Informer Who Provided Information Included in Final Analysis
aResponse from a third Swedish organization reiterated information received from the other two organizations.
S&E, safety and efficacy; S,E&CE, safety, efficacy, and cost-effectiveness; FSR, full systematic review; Ad hoc, ad hoc Committee; SC, standing committee; RR, rapid review; V, E, M: (Implementation), voluntary, expected, mandatory.
Presence or Absence of a National System with Explicit Responsibility for Assessment of Interventional Procedures
Eight respondents (from eight different organizations in eight different countries) stated that there was a national program for providing guidance to healthcare providers on interventional procedures in their respective countries (Belgium, Brazil, France, Germany, Israel, The Netherlands, Sweden, and United Kingdom; Table 2). For sixteen countries, the respondents stated that there was no national system for assessment of interventional procedures. The latter group had been asked to describe any other systems that existed in their country and these are summarized in Box 2 (note, an invalid response to this question was received from one country). Despite respondents suggesting absence of a national system in sixteen countries, information in relation to subsequent questions was nevertheless provided for ten of these sixteen countries (Australia, Austria, Canada, Finland, Iceland, Ireland, Japan, Mexico, New Zealand, and Poland).
Box 2. Summary of responses received from organizations in sixteen countries without a national assessments system for interventional procedures
Type and Funding of Organization(s) with Remit for Interventional Procedures
In seven countries, the organizations concerned with assessment and production of guidance on interventional procedures are government departments or organizations, which receive direct government funding (Brazil, Iceland, Ireland, Israel, Mexico, Sweden, and the United Kingdom). Arrangements in other countries include the following: direct state funding of a semi-governmental organization (Belgium), a private not-for-profit organization (Canada), a public body funded by National Insurance fees (France) or Healthcare providers (Germany), state funding of academic university departments (New Zealand), and academic/research funding of a non–profit-making organization (Austria). Mixed funding models also exist: In The Netherlands, CVZ and CBO are funded by government but CBO also receives funding from professional organizations. ASERNIP-S (Australia) is funded by governmental, academic, and research organizations and forms part of the Australasian College of Surgeons.
System of Selection of Interventional Procedures for Assessment
Procedures are most commonly referred by government, professional organizations, or selected by the assessment organizations themselves, but most organizations also receive referrals for review of interventional procedures by different types of referring or mandating organizations (Table 3). Stated criteria for deciding which procedures should be assessed for production of guidance included the following:
• Health, public health, or clinical significance considerations (four countries: Australia, Belgium, Ireland, Netherlands)
• Demand, reimbursement, cost, or health budget considerations (six countries: Austria, Belgium, Brazil, Iceland, Ireland, Netherlands)
• Innovation or uncertainty (three countries: Austria, Iceland, Sweden)
• Health ministry involvement or healthcare policy considerations (four countries: Belgium, Brazil, Israel, Mexico)
Respondents from all but two countries (United Kingdom and Canada) referred to, or suggested, the existence of a prioritization process to decide which procedures to assess.
Table 3. Selection of Interventional Procedures for Assessment
Type and Sources of Evidence Used in the Assessment of Interventional Procedures
Full systematic reviews are used in five countries and rapid reviews of the literature in nine (Table 2). Published peer-reviewed articles are by far the most commonly used sources of evidence, but there is variation in the use of other sources, including material submitted for publication (four countries), conference abstracts (three countries), and data from both interventional procedure registers (six countries) and manufacturers (five countries) (Table 4).
Table 4. Source of Evidence Used in the Assessment of Interventional Procedures
FDA, Food and Drug Administration; EMEA, European Medicines Agency; HTA, healthcare technology assessment.
Who Appraises the Evidence
In five countries, a standing committee assesses the evidence to produce guidance, whereas in three countries, an ad hoc committee is specially convened for each individual procedure (Table 2). Seven countries provided an “other” response and described their arrangements, as follows:
• In Austria, the assessment is carried out by two reviewers, whereas decisions are made subsequently by a committee.
• In Finland, clinicians are selected according to the topic and HTA experts from FINOHTA act as a small working group to gather the evidence and write a summary of it.
• In Iceland, there is no committee; the clinical guidelines editor undertakes the assessment and seeks external expertise to review the conclusions or summary.
• In the Netherlands, the CVZ medical advisors appraise the evidence to produce guidance.
There is variation in the professional background and affiliations of people who appraise evidence and produce guidance, both as committee members and outside the context of committees. Clinical experts are the most frequent member category for committees. Manufacturers and their representatives are those most frequently involved as stakeholders who are asked to submit evidence. “Other” contributors include policy leads, healthcare providers, healthcare funding organizations, and regulators.
Consultation on Draft Assessment (or Guidance) (Table 2)
Wide consultation on draft guidance is undertaken in the United Kingdom (open Web-based public consultation, with additional targeting of professional bodies, patient groups, and manufacturers); Mexico (open Web-based consultation); Sweden (including regional seminars and involving decision makers and politicians); and Germany (involving healthcare providers, manufacturers, industry associations, and patient organizations). Selected experts are consulted (upon the draft assessment) in Belgium, Finland, Iceland, Israel, and the Netherlands. In Austria, there is consultation with industry. Three countries (Brazil, France, and Ireland) do not consult on their draft guidance.
The Format of Recommendations or Guidance
Organizations in six countries consider efficacy and safety, whereas, in addition to efficacy and safety, cost-effectiveness is considered by organizations in another nine countries (Table 2). Several countries use a standard range of categories (otherwise, a “typology”) to present their recommendations. In the Netherlands, the categorization is either for or against inclusion in the national health insurance package. In France, a recommendation is made about the whether the “benefit / risk” ratio is sufficient for “coverage” or not: if there is uncertainty, conditional coverage is recommended with additional data collection. In Mexico, a recommendation is made for normal use if safety and efficacy evidence is adequate, for use only in research if there is uncertainty, or for no funding of the intervention or procedure (advice is also given on cost-effectiveness). In the United Kingdom, NICE has four categories of recommendation: evidence adequate for normal use, inadequate evidence requiring special arrangements for clinical governance, consent and audit, use in research only, or the procedure should not be used. Similar categories are used in Austria and Israel, and are being developed in Finland. Australia's A-SERNIPs categorizes safety as “at least as safe as the comparator,” “less safe,” or “cannot be determined”; and efficacy is categorized similarly. It provides a rating of the evidence as good, average, or poor, to indicate its strength, quality, and magnitude: if further evidence is required the recommendations advise on appropriate methods to provide this. In Belgium, Brazil, Germany, Iceland, and Ireland, no standard categories are used for recommendations.
Status of the Guidance (or Assessment)
Implementation of guidance is mandatory in two countries (Brazil and Germany), “expected” in seven (Austria, France, Ireland, Israel, Netherlands, Sweden, and United Kingdom), and voluntary in five (Australia, Belgium, Finland, Iceland, and Mexico). The status of guidance is the same in the independently funded healthcare systems as it is in the state funded systems of all of these countries except Germany, Ireland, and the Netherlands (although HIQA in Ireland may have responsibility for the private sector in the future).
Table 5. “Who Assesses the Evidence” about Interventional Procedures
M, committee member; NM, non-committee member asked to submit evidence or commentary; HTA, healthcare technology assessment.
Four countries have mechanisms to monitor the uptake and use of procedures after guidance has been published (Austria, France, Iceland, and Sweden).
Use of Interventional Procedures Guidance from Other Countries
The survey asked about use of guidance from other countries, in the absence of that country's own. In Iceland, guidance from the United Kingdom, Scotland, Australia, New Zealand, Canada, United States, Spain, Sweden, and Denmark combined provides more than 95 percent of the information used in that country, with new information being sought only if required (e.g., about local use and cost data). Guidance from other countries is also used in Mexico to produce local or regional guidance on specific topics.
In France, there is a systematic search of other countries' guidance as part of the review of published data. In The Netherlands, guidance from the UK NICE (primarily), Germany (GBA), and U.S. insurance companies is used as part of the literature review, but a new, “own” decision is always taken.
In Austria, Brazil, Finland, Israel, and Poland, use is made of guidance from a range of countries but respondents did not specify the exact purpose. The United Kingdom and Australia were the most frequently mentioned sources of guidance used by others throughout the world. In six countries, there is no formal use of guidance from elsewhere (Australia, Belgium, Germany, Ireland, Sweden, and the United Kingdom).
DISCUSSION
Two key findings emerged from our study: First, several countries with established HTA infrastructures do not appear to have national programs for providing guidance to their healthcare systems on interventional procedures, whereas others have a variety of organizations (some still in development). This finding supports the notion of a developing and evolving field, which contrasts sharply with the established national and international frameworks for assessing pharmaceutical interventions and medical devices. This observation becomes even more significant if one considers that established HTA organizations and systems exist in only a relatively limited number of countries. Second, our study reveals substantial diversity in the methods, processes, extent, and comprehensiveness of existing HTA systems for interventional procedures in different countries.
Variations in organizational, procedural, and methodological aspects of healthcare technology assessment have previously been shown in a questionnaire-based survey of twenty-four different HTA organizations, but, unlike our study, that study did not focus specifically on interventional procedures (Reference Martelli, La Torre and Di Ghionno4). Another study has specifically examined the different guidance format and types of recommendations produced by different HTA organizations, using thematic analysis, and has identified substantial variations (Reference Draborg and Andersen2). We have observed similar variations in the specific area of interventional procedures, in relation to “mandatory,” “voluntary,” or “expected” nature of implementation of guidance. In addition we, too, are undertaking a thematic analysis of different pieces of interventional procedures guidance focused on the exact type of recommendations produced by different organizations and we aim to publish the findings in the near future.
The heterogeneity of organizations involved in assessing interventional procedures, the fact that many of them have only recently been established, as well as the resulting difficulty in identifying them are all reasons that made the collection of information about systems for producing interventional procedures guidance challenging. We tried hard to identify people who would have sound knowledge of the issues addressed by the questionnaire, using a multiplicity of methods for identification, and succeeded in obtaining informative responses from experts working in established HTA organizations with a national remit in different countries. However, respondents were taking part in a personal capacity; therefore, it is possible that some items of the information they provided might have been incomplete, or inaccurate, or seen as “subjective”' by other observers. It is also possible that we failed to obtain truly representative information about countries with substantial formal or informal “subnational systems” that perform healthcare technology appraisal at regional, or local organization level (for example, in the United States, Spain, and the United Kingdom), as we purposely focused on “national” systems alone. We recognize that, in reality, important components of the function of healthcare technology assessment for interventional procedures may also take place at a much more local (e.g., at hospital or healthcare provider economy) level.
Negative responses to the question about the existence of a national system need to be interpreted cautiously. Free text responses provided to qualify such “No” responses suggested that the terminology used in our questionnaire may have influenced how some participants provided information. Two organizations with established record of HTA for interventional procedures (CADTH in Canada, and ASERNIP-S in Australia) did not, in the opinion of our informers, provide “guidance” as such, nor did they constitute a single national system. Although these examples demonstrate the attention required in terminology use when formulating surveys about different national systems, they also reinforce how context-specific HTA can be. It should be noted that these informers did provide information in response to the other questions in the survey.
The fundamental criteria used in assessing interventional procedures in any country are safety and efficacy. However, according to the respondents, organizations in nine countries also consider the cost-effectiveness of procedures. Four of these organizations also consider factors such as demand, reimbursement, and health budgets cost when prioritizing procedures for assessment. Cost-effectiveness is assessed by all four of the organizations which use a “full systematic review” for identification and review of the published evidence.
There is wide variation in the ways in which evidence is selected, the extent of evidence reviews and the appraisal of the evidence. With regard to selection of evidence, peer-reviewed publications are universally used, but only a few organizations consider conference extracts (abstracts). More organizations undertake rapid reviews than full systematic reviews. Clinical experts help to appraise the evidence in fifteen countries, mainly by acting as committee members. Recent research advocates the usefulness of input from clinical experts who are not committee members, but also suggests a need to include experts both with and without operative experience of the specific procedure under appraisal in such processes (Reference Lyratzopoulos, Hoy, Veeramootoo, Shanmuganathan and Campbell3). Representatives of industry contribute to assessment of the evidence in seven countries, but only the United Kingdom's NICE has an industry representative as a committee member.
In conclusion, this survey has demonstrated the diversity of systems and practice for assessing and producing guidance on interventional procedures in countries around the world. Although the system for one healthcare organization or country is not necessarily appropriate for another, further comparison of methods used in different countries will be important to enable sharing of learning and good practice. Furthermore, given the current scarcity of HTA resources dedicated to the assessment of new interventional procedures, there is great potential for international collaboration. Such collaborations could maximize the efficiency and quality of the way healthcare systems internationally deal with innovation in surgery and interventional procedures. In particular, it would seem worthwhile to explore possibilities for collaborative data collection when the evidence for a procedure is inadequate.
CONTACT INFORMATION
Jonathan Plumb, RN, Dip.HE, BSc, MPH (jonathan.plumb@mhra.gsi.gov.uk), Honorary Research Fellow, Interventional Procedures Programme, W. Bruce Campbell, MS, FRCP, FRCS (bruce.campbell@nice.org.uk), Chair, Interventional Procedures Advisory Committee, Georgios Lyratzopoulos, MD, FFPH, MRCP, MPH, DTM&H (georgios.lyratzopoulos@nice.org.uk), Consultant Clinical Adviser, National Institute for Health and Clinical Excellence, MidCity Place, 71 High Holborn, London, WC1V 6NA, UK
