There is mixed evidence of the use of health technology assessment (HTA) in decision making around the world (Reference Borowski, Brehaut and Hailey2;Reference Buxton3;Reference Gallego, Melocco, Taylor and Brien6;Reference Jansson and Anell11;Reference McGregor and Brophy14). However, there is only a limited understanding of the evidence for the determinants of successful uptake of HTA. A small number of local studies have shown that providing the right type of HTA is an important determinant for success (Reference IJzerman, Reuzel and Severens10;Reference van Velden, Severens and Novak23). These studies confirm that the technical analysis provided through HTA has to be relevant, timely, and of high quality. However, several commentators have stated that the uptake of HTA is not solely dependent on the rigor and methodological soundness of the report. It has been suggested that more attention to the process of decision making needs to be given to improve the uptake of HTA (Reference Lehoux, Denis, Tailliez and Hivon12;Reference Martin and Singer13).
Several countries have established formal bodies that include a mandated requirement to appraise evidence from HTA. The Pharmaceutical Benefits Advisory Committee (PBAC) and the Medical Services Advisory Committee (MSAC) in Australia and the National Institute for Health and Clinical Excellence (NICE) in England and Wales, to name but a few, are now required to include HTA in support of their recommendations. However, these examples of formal decision making are of national bodies. There are fewer examples of such formal decision making at the meso (e.g., local area health services and public hospitals) level.
Nevertheless, in many healthcare systems, meso decision makers directly face budget constraints that determine their ability to fund health-related technologies (Reference Martin and Singer13;22). This means that every resource allocation decision has a direct opportunity cost that impacts on a facility's ability to treat patients effectively and equitably. By contrast, the PBAC makes recommendations about which drugs will be subsidized through the Pharmaceutical Benefits Scheme (PBS). This is an open-ended funding program and does not face such budget constraints.
If local institutions (such as hospitals or local health regions) face the consequences of a capped budget more directly (than, for example, national decision-making bodies), we would expect that, at this level of the health system, HTA and economic evidence can play a more valuable role in resource allocation decisions. This expectation is because, first, such evidence could guide decision makers on how to adopt new technologies without breaking budget caps and, second, it could help inform them about maximizing health outcomes within those budget constraints.
Although Australia was one of the first countries to formally incorporate HTA in decision making at a national level (Reference Mitchell15), it has recently been recognized that Australian HTA efforts are fragmented and uncoordinated. Most critically, there is a lack of assessment for a range of technologies used in the public hospital system, with procedural and coverage gaps, in particular around technologies frequently used in the public hospital sectors (1). Furthermore, there is very little information regarding the approval and introduction of new healthcare technologies at the local (Area/institutional) level (Reference Gallego, Melocco, Taylor and Brien6;Reference Spigelman20).
In Australia, both state and federal governments are responding to this situation by establishing new bodies to assess a broader range of technologies. For example, the Victorian Department of Human Services has established the Victorian Policy Advisory Committee on Clinical Practice and Technology (VPACT) to consider and make recommendations regarding the application of new and existing technologies and clinical practices in Victorian public health services (24). Southern Health, Melbourne, uses a form of mini-HTA in the work of its New Clinical Procedures Committee (NCPC) (19). The New South Wales (NSW) Department of Health issued a Model Policy for the introduction of new interventional procedures into clinical practice in 2005, which aimed to develop a more consistent approach to HTA across Area Health Services (AHS) (16).
Although, the main focus of these efforts has been to improve the scope of assessments in Australia, little attention has been given to gaining a better understanding of the local-level decision-making environment.
This study reports on the results of a survey of healthcare providers' and health service managers' opinions about local decision-making processes used for the approval and introduction of new health technologies at the local (Area/hospital) level. The survey was designed to provide a better understanding of staff perceptions of current process and to ascertain key characteristics that healthcare providers and health service managers considered important if a new health technology evaluation and approval process was to be implemented at the local level.
METHODS
Context, Setting, and Sample
The Australian state of New South Wales, with a population of around 6 million people, has a devolved system of healthcare delivery. Eight local health regions are responsible for delivering a range of health services to people residing in their catchment area (Reference Griffith7). This study took place in one of those regions; the Northern Sydney Central Coast Area Health Service (NSCCAHS). The NSCCAHS provides health care to a population of approximately 1.13 million people through its seven acute hospitals, and range of public health and community health services. Public hospitals in the Area range in size, the largest of which has 551 beds. All health service managers and healthcare providers working in the AHS were invited to participate.
Questionnaire Development and Data Collection
A previously developed questionnaire was used as the basis for this survey, with some questions specific to the purpose of the survey being added (Reference Hoffmann and Graf von der Schulenburg9;18;Reference Zwart-van Rijkom, Leufkens and Busschbach25). Further changes were done to include findings from face to face interviews conducted in an earlier study (Reference Gallego, Fowler and van Gool5). The survey was then pilot-tested in-house. Upon return of the pilot questionnaires, responses and respondents' comments were reviewed. As a result, some questions were rephrased to improve clarity. The questionnaire was divided into two main sections:
Section One. Health technology decisions in participants' own institution/facility (Qs 1–6). Participants were first asked the type of health technology they most commonly work with (e.g., interventional procedures, clinical devices). Participants were then asked to indicate their level of agreement with statements about the current AHS process for introducing a new health technology within their local Area/hospital, using a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree), with reference to the type of technology they most commonly work with. They were also asked to rate the influence that certain criteria have in the introduction and approval, using a 5-point Likert scale (1 = none to 5 = very strong). An open-ended question was also included to allow participants to comment on what the characteristics of a new health technology evaluation and approval process should be.
Section Two. Information about participants (Qs 7–11) sought details such as professional representation, work place of respondent, responsibility for managing a budget, and involvement in decisions about adoption or financing of health technology within the AHS.
For the purpose of this survey, health technologies were defined as all methods used by health professionals to improve health, prevent and manage disease, and rehabilitate and provide long-term care to patients, including procedures, programs, drugs, devices, and equipment.
Several distribution channels were used to maximize the response rate. The survey was placed on the World Wide Web between September and October 2006. An email was sent out with a link to the on-line survey to members of key committees (for example the senior and sector executive committee and the clinical redesign steering committee) within the AHS by means of their respective secretariats. This was followed by a broadcast email to all staff in the AHS. Because not all staff have a work email, departments were asked to print out copies for those who did not have an email. Hard copies of the survey were also distributed during an AHS cancer services workshop. The survey was handed out as participants came into the room, and participants were asked to drop it in a box at the end of the session.
Analysis
Responses were collated and analyzed using the Statistical Package for the Social Sciences (SPSS) for Windows Version 10 (SPSS Inc., Chicago, IL). Descriptive statistics were used to summarize data. To test the associations between responses and participants' characteristics (e.g., involved in decision making, profession, and so on), responses were dichotomized (e.g., disagree/strongly disagree = 1 and agree/strongly agree = 2), and a chi-square (χ2) test for linear distribution was conducted. Significance was set at the 5 percent level.
Open-ended responses were analyzed using a modified thematic analysis, which involved an open coding technique. Data were segmented and coded with labels. Parts of the text were identified and related to a concept or idea (Reference Strauss and Corbin21).
Ethics
This study was approved by the University of Technology, Sydney (UTS) Human Research Ethics Committee (HREC) and endorsed by the Medical Research Ethics Committee of the Royal North Shore Hospital.
RESULTS
The total number of people participating in the survey was 174. The survey was partially completed by 8 percent (n = 13) of the participants; however, all responses were included in the final data set. Forty-nine percent (n = 84) of the survey participants completed the survey online, 32 percent (n = 57) used a hard copy, and 19 percent (n = 33) completed it during a cancer workshop.
Information About Participants
Fifty-six percent (n = 90/160) of the participants currently have or ever had responsibility for managing a budget. Sixty-three percent (n = 101/159) were involved in making decisions about the adoption or financing of health technologies or treatments at their institution. The characteristics of the participants are shown in Table 1. As at June 2006, nursing staff in the NSCCAHS accounted for 43 percent and medical 9 percent as a proportion of all staff (17). The proportion of medical staff in our survey is higher than that of the staff profile.
Table 1. Characteristics of Participants
a For example, Consumer advocate, division manager, general practitioner, or health promotion officer.
b For example, General practice, private hospital.
Types of Technologies with Which Survey Participants Most Commonly Work
Survey participants work with a wide variety of health technologies, ensuring that results are not dominated by any particular decision-making process or type of technology. The types of health technologies that most survey participants worked with are described in Figure 1.
Figure 1. Types of technologies that participants most commonly work with (n = 172). *Other (n = 9) includes survey participants who chose more than one option (e.g., interventional procedures and clinical devices).
Current AHS Process for Introducing the Type of Technology (Selected in Question 1) Within Survey Participants' Institution
When participants were given statements about current AHS processes for introducing new health technologies, only around a quarter of participants agree/strongly agree that the process was appropriate (n = 46; 27 percent), easy to understand (n = 40; 23 percent), evidence-based (n = 48; 28 percent), or consistently applied (n = 16; 26 percent). Barely 20 percent of survey respondents agreed with the notion that the decision-making process was fair or an efficient use of their time.
More than half of respondents reported that they had some budgetary responsibilities. Of those, 66 percent agree with the statement “One of the barriers for taking up new health technologies is the difficulty of moving resources from one sector (budget) to another”. Seventy-eight percent agree/strongly agree that budgets are tight and resources cannot be freed to adopt new technologies. Thirty-eight percent considered that their opinion was influential in decisions to introduce a new health technology in their area of specialty (Supplementary Tables 1 and 2, which can be viewed online at www.journals.cambridge.org/thc).
To determine whether there was an association between participants' characteristics (e.g., involvement in decision making) and their awareness of current processes to introduce a new health technology, proportions were compared (see Table 2). No statistical differences were found between the subgroups, with the exception of a greater awareness of current processes amongst those who are involvement in decision making.
Table 2. Association between Survey Participants' Characteristics and Awareness of Current Process to Introduce a New Health Technology
a df = 1; p < .05.
Perceived Influence of the Main Stakeholders in the Current Process for Introducing and Approving the Type of Health Technology (Previously Selected) in Survey Participants' Institution
The State Government/Department of Health (67 percent) and the AHS executive (66 percent) were considered to have the most influence for the introduction and approval of new health technologies at survey participants' institution. Those with the least influence were private health insurance funds (15 percent) and patients (10 percent).
Main Factors Perceived as Currently Influencing the Introduction and Approval of Health Technology (Previously Selected) in Survey Participants' Institution
Participants considered that “total costs impact to the Area/hospital/department” (70 percent) and evidence on safety/quality (62 percent) are factors that have a very strong/strong influence in the current process for introduction and approval of health technology. Patient preferences (12 percent) and equity (19 percent) were considered to be least influential (see Table 3).
Table 3. Influence of Factors in the Current Process for the Introduction and Approval of Health Technology at Participants' Institution
a Other included key clinicians, type of equipment, clinical champion, actual budget, professional association, institution funding it, impact on staff, and medical surgical colleagues.
Characteristics and Concerns Over Future Processes
Forty-two percent of participants (n = 70) provided comments about the key characteristics that the new Area Health Technology Evaluation and Approvals process should have. Major themes that emerged from these comments are presented below.
Participants wanted a fair and transparent (n = 8), evidence based (n = 8), timely (n = 2) decision-making process. Some considered that the process should focus on patients' needs (n = 5). Others commented that the process should be efficient (not time consuming) (n = 4), centralized and consistent (n = 2) throughout the Area. Ensuring there are good communication systems in place was considered a key feature by some of the respondents (n = 6). Participants considered it was important to have appropriate representation from key stakeholders in the committee (n = 15). Some (n = 12) mentioned it was essential to include clinicians in the decision-making process and some (n = 7) also mentioned that patients/consumers should “have a say.”
One of the concerns expressed by participants was bureaucracy, for example “I am worried that bureaucracy will stifle the process.” Some participants expressed their skepticism about the new process (n = 8). This group of participants described the process as “irrelevant” and seem to dislike the idea of “another committee.” Some participants (n = 4) also expressed their discontent and lack of confidence in the current AHS governance. “Governance is so chaotic and patient centric that it's hard even to service the urgent needs of surgical patients let alone expect from our managers the vision of new techniques.”
DISCUSSION
Studies such as these have several limitations. First, the wording of questions and the way they are presented influences responses. This is why in designing this survey we have followed previously tested and published instruments (Reference Hoffmann and Graf von der Schulenburg9;18;Reference Zwart-van Rijkom, Leufkens and Busschbach25). Second, the sample is not representative of the overall staff in the AHS and is likely to be over-representative of staff with budgetary responsibilities. However, it could be argued that, although the perceptions of such staff may not be representative, they are likely to be most informed of current decision-making processes. Furthermore, they might be more likely to be involved or affected by the new processes. Finally, we are reporting on the perceptions and opinions of staff in only one of eight AHS. Thus, the generalizability of these results can be questioned. However, from our knowledge of other NSW AHS, we understand that they too are currently developing and implementing new HTA initiatives and are grappling with similar issues to those faced by the NSCCAHS as are local health authorities around the world.
Notwithstanding the comments above, the results of this survey could allow hypothesizing that health providers and health service managers have a low perception of existing processes to introduce and approve new health technologies. Such a perception is consistent with what Hailey et al. described nearly 2 decades ago as a lack of coherent mechanisms in Australia to control investment in medical technologies (Reference Hailey and Roseman8).
It seems that, at the time of the survey, processes for the approval and introduction of new health technologies at the local (Area/hospital) level had a lot of flaws and there is plenty of room for improvement; however, it is important to consider the barriers and apprehension expressed by participants.
According to respondents, it is important for the process to be transparent and explicit. As stated by the Department of Surgery, Calgary health region, “the more clinicians and decision makers are aware of a systematic but simple process to evaluate the impact of new technologies, the more efficiently HTA will be undertaken and the greater the likelihood that it will be embedded within clinical practice” (4).
This survey provides some evidence that local decision-making processes have not always been the most conducive to HTA. Thus, any new HTA initiative needs to be accompanied by a reform of decision-making processes. These, it would seem, are minimum requirements for local health authorities to develop more coherent mechanisms for the rational investment in healthcare technologies.
The NSCCAHS has used the results of this survey to aid the design of their new HTA and decision-making process. It has established a committee composed of consumer representatives, a health economist, clinicians, as well as members of the Area and State bureaucracy. The role of the committee is to make recommendations to the NSCCAHS Executive based on their appraisal of the application and assessment. The NSCCAHS has established a clear and simple application process and has devoted resources to conduct an assessment of relevant evidence. As part of this initiative, a communication strategy was implemented to improve knowledge of the process within the Area and to try and ensure that all significant new technologies are assessed and appraised through the new process.
Although these efforts are clearly in line with the results from the survey, there is a need to monitor the initiative and to continually develop our understanding of how HTA can be undertaken efficiently and effectively at the local level.
CONTACT INFORMATION
Gisselle Gallego, BPharm, PhD (gisselle.gallego@chere.uts.edu.au), Research Fellow, Kees Van Gool, BEc, BA, MEc (kees.vangool@chere.uts.edu.au), Senior Research Officer, Centre for Health Economics Research and Evaluation, University of Technology Sydney, City campus, PO Box 123, Broadway, NSW 2007, Australia
Dianne Kelleher, BSc, DipDiet, DipEd, MHPM (dkellehe@nsccahs.health.nsw.gov.au), Manager, Health Technology Assessment, Clinical Governance Unit, Northern Sydney Central Coast Area Health Service, Pacific Highway, St. Leonards, Sydney, NSW 2065, Australia
