Health technology assessment (HTA) agencies must set research priorities when the resources required for assessing technologies eligible for evaluation outweighs the available resources to conduct them. Both theoretical and applied methods of priority setting in HTA (Reference Eddy7;Reference Lara and Goodman11;Reference Noorani, Husereau, Boudreau and Skidmore14;Reference Phelps and Parente15) and health services research (Reference Doyle, Waters and Yach5;8;Reference Townsend, Buxton and Harper18) have been published. Guidance for priority setting in HTA is also available (Reference Donaldson and Sox4;Reference Henshall, Oortwijn, Stevens, Granados and Banta9) and suggests the method used be transparent, explicit, systematic, pragmatic, efficient, and consistent with the mission or goals of the HTA program that is setting priorities.
The Canadian Agency for Drugs and Technologies in Health (CADTH, formerly the Canadian Coordinating Office for Health Technology Assessment) is an independent organization that is funded by and serves the federal, provincial, and territorial ministries of health and their constituents. Rather than one single health system, Canada provides health services through thirteen provincial and territorial governments (Reference Battista, Côté, Hodge and Husereau1). These healthcare systems are publicly funded with universal coverage for hospital and physician services contingent on provincial residency requirements. Private insurance companies are not able to provide coverage for services provided through the public funding. Drug costs for outpatients and long-term care is not included in this, although all provinces offer some public coverage (Reference Battista, Côté, Hodge and Husereau1).
The priority setting process for HTA at the Canadian Agency for Drugs and Technologies in Health has evolved over the past decade; significant changes to this process have occurred over the past 3 years. This study describes the process for priority setting large-scale assessments (i.e., with original systematic reviews and economic evaluations) which uses a multicriteria decision-analytic approach (MCDA) called the analytic hierarchy process (AHP) together with a deliberative process for identifying pan-Canadian research priorities for HTA.
METHODS
Criteria Selection and Weighting
We identified fifty-nine unique priority-setting criteria across the eleven agencies through a previously published systematic review (Reference Noorani, Husereau, Boudreau and Skidmore14). While the description of prioritization criteria differed across agencies, they could be grouped under eleven categories as shown in Table 1. The eleven categories of criteria were sent to members of two CADTH advisory committees responsible for priority setting CADTH HTA research. Committee members consist of representatives from the health authorities that are responsible for the provision of health services in their jurisdictions across Canada. One committee consists of fourteen representatives from federal, provincial, and territorial publicly funded drug plans (the CADTH Advisory Committee for Pharmaceuticals), while the other consisted of twelve representatives serving various roles in the federal, provincial, and territorial ministries of health (the CADTH Devices and Systems Advisory Committee). Most of the drug advisory committee are pharmacists working for health ministries while the nondrug advisory committee had varied backgrounds. Committee members were asked by email to review the list and provide additional criteria that they believed should be considered when prioritizing HTA research.
Table 1. Categories of Prioritization Criteria Identified from A Systematic Review (Reference Neumann, Rosen and Greenberg13)
HTA, health technology assessment.
We used the AHP (Reference Saaty and Vargas16) to assign weights to identified criteria. In August 2005, we sent a survey to committee members by email asking them to perform fifty-five pair-wise comparisons of eleven criteria categories. The survey questions asked committee members to judge the relative importance of one priority setting criterion versus another using a nine-item ordinal scale. A sample question from the survey appears in Figure 1. Comparisons between criteria from consistent responders (AHP consistency index <0.2) of the survey were translated into numerical relationships that allowed weights to be assigned to criteria. More detailed methods for this approach have been published elsewhere (Reference Dodgson, Spackman, Pearman and Phillips3;Reference Saaty and Vargas16).
Figure 1. Proposed scenario.
In November 2005, we shared the ranking of criteria at a face-to-face meeting of the same committee members to arrive at consensus (i.e., majority) regarding which core criteria would be used in the future for ranking topic proposals. Criteria weights for the core criteria were then recalculated so that they again added to one. Subcriteria were then developed for each core criterion identified and priority weights were assigned using consistent pair-wise comparisons for the purpose of scoring.
Topic Selection Process
Potential technology assessment topics are identified through several sources, including informal surveys of advisory committees and other Canadian healthcare stakeholders, a review of findings from our horizon scanning program and inquiries made to our rapid review program. Potential topics are also identified through proposals received by the CADTH Web site. Less frequently, more formal surveys have been undertaken to identify topics of interest. Upon receipt of any proposal for assessment, requestors are contacted for clarification of the policy or practice issue and the context in which decisions will be made along with the timing and type of information required to inform a decision. Proposals are validated by further consultation with researchers, policy makers, and clinical subject experts. Proposals that appear to have pan-Canadian relevance and are likely to be chosen based on core criteria are then submitted to advisory committee members for prioritization.
Advisory Mechanism
For each proposed technology assessment decision, information relating to each criterion is identified and synthesized into a committee briefing paper by a CADTH researcher working with an information specialist. The briefing paper also contains contextual information about the technology, its availability, and the policy decision problem. The information relating to each criterion in the briefing papers is then rated by two researchers independently. In case of disagreement, a neutral third party is called to achieve consensus (unanimity minus one). The ratings are then translated into a score and rank using subcriteria, and criteria weights.
Additional information related to the proposed topic's expected level of interest in decision making, timeliness of the project, the current access and reimbursement policies relating to the technology, variation in rates of use, and ongoing research initiatives in Canadian health jurisdictions is also synthesized for committee consideration. This additional contextual and colloquial information is intended to help inform deliberation by further describing the pan-Canadian relevance of assessment for decision making. Information gathering is facilitated through the use of CADTH jurisdictional representatives and an on-line form.
Committee members are given the briefing papers, their ranking, and summaries of additional information, and the available research capacity of the HTA program and asked to deliberate at a face-to-face meeting for the purpose of advising the HTA program regarding research priorities.
RESULTS
The pair-wise comparison of eleven criteria led to similar ranking of criteria across the two CADTH advisory committees (see Table 2). Both committees made choices that gave the most weight to knowing the potential clinical impact of the decision and the least weight to knowing the controversial nature of the decision or its ethical and legal implications. The largest difference between committees was the perceived importance of understanding the disease burden associated with a technology to set priorities.
Table 2. Criteria Weights Assigned by CADTH Advisory Committees for Topic Selection through Pairwise Comparison
CADTH, Canadian Agency for Drugs and Technologies in Health.
Through face-to-face discussion with both committees, six common core criteria were selected by a majority vote: disease burden, potential clinical impact, available alternatives, potential budget impact, potential economic impact, and available evidence. These criteria were considered the most influential in creating priorities for HTA research, representing 73 percent of the weight behind committee members’ decisions to embark on an assessment of a health technology. A hierarchical representation of the criteria, criteria weights, and subcriteria for one of the advisory committees can be seen in Figure 2.
Figure 2. Hierarchical representation of goals and criteria for prioritizing technology assessment research at CADTH.
Since May 2006, we have reviewed 102 topic proposals at sixteen biannual in-person advisory committee meetings held between spring 2006 and fall 2009. These have resulted in twenty-nine selected research priorities for the CADTH HTA program. Of sixteen proposal cycles, number one-ranked proposals have been selected by committee members five times; 62 percent (n = 18) of selected topics had been a top three rank during the scoring and ranking process (five topics ranked 1; six topics ranked 2; seven topics ranked 3). Rejected proposals have been recycled or dealt with through less-extensive HTA reports. Acceptable proposals become published CADTH Reports an average of 12 months following approval. Individual Canadian health jurisdictions are then free to use this information to inform healthcare decision making.
Feedback from advisory committee members has been positive. Members have indicated the process has been helpful and beneficial and that ranking is of more importance when topics have similar scores. Our experience suggests that this process has led to better engagement of advisory committee members. We continue to enhance the process considerably through feedback provided by our advisory committees and CADTH staff. A more detailed description of the process can be made available by contacting the authors.
DISCUSSION
The authors were able to successfully develop a practical approach for HTA priority setting that incorporates a multicriteria decision analytic (MCDA) approach and a deliberative process intended to identify health technology policy decisions that will have the greatest relevance to the Canadian health system and analysis that will be of the most relevance to Canadian healthcare decision makers. Previous recommendations have suggested that priority setting be a key principle for the conduct of HTA (Reference Drummond, Schwartz and Jönsson6). Some recent descriptive reviews have suggested there is still considerable variation across HTA agencies in terms of how priorities are set (Reference Noorani, Husereau, Boudreau and Skidmore14;Reference Schwarzer and Siebert17). Given some lack of awareness regarding our current process (Reference Menon and Stafinski12), we believed it was important to share our process and experience with others.
The strength of using the AHP for priority setting is that it can combine both quantitative and qualitative criteria in a manner that is consistent with the most recent guidance for priority setting in HTA (Reference Henshall, Oortwijn, Stevens, Granados and Banta9). Specifically, the EUR-ASSESS Priority Setting Subgroup recommendation suggested priority setting processes must be transparent, with an explicit agreed-upon process, systematic, pragmatic, efficient, consistent with the goals of the program and shared. By its nature, MCDA is goal-oriented and introduces transparency into decision making by declaring criteria for decision making along with their relative importance. The AHP approach is a widely used MCDA technique that is additionally shared, systematic, and pragmatic (Reference Dodgson, Spackman, Pearman and Phillips3).
In addition, MCDA approaches can lend themselves to considering both the costs and benefits of conducting research by looking at the trade-off of research resources versus a criteria-weighted score. The importance of considering this trade-off has been most recently highlighted by Townsend (Reference Townsend, Buxton and Harper18). Although we have not explicitly incorporated research costs into our current process, we believe our approach has led to gains in efficiency because our HTA projects typically consume similar amounts of resources.
We believe a further strength to our approach is the incorporation of a deliberative process involving key government officials and users of HTA information. This approach allows for factors beyond the significance of conducting research to be discussed when making final decisions about what projects to undertake. This could partly explain why number-one ranked proposals were only selected in five out of sixteen cycles. Although we have not analyzed the reasons for not proceeding with research with number one-ranked proposals, our experience suggests these were often due to committee perception that research would not enjoy uptake or that more rapid research was required. In more than one case, research proposals that were originally rejected were given priority after appearing at a subsequent committee meeting. This reflects the ever-changing context in which HTA and healthcare issues evolve. Our deliberative process also increases the interaction between researchers and policy makers and ensures timeliness, factors that have been empirically shown to enhance the use of research in practice (Reference Innvaer, Vist, Trommald and Oxman10).
Our approach may have potential limitations. Although MCDA approaches are goal-oriented and descriptive, they are not normative in that they do not formally account for value trade-offs across criteria and prescribe what topic should be selected. Rather, they provide an analysis of the factors that are important to consider when making these choices. Although approaches grounded in multiattribute utility theory have a stronger theoretical basis for helping decision makers achieve their goals, they are considerably less pragmatic to implement and require considerable more up front work. Because deliberative processes are inherently normative and value-based (Reference Culyer and Lomas2), we believed using this approach would lead to similar results without the additional time, expertise, and resources required.
The five most common criteria used by other HTA agencies, identified in a previous review (Reference Noorani, Husereau, Boudreau and Skidmore14), were also independently selected by our committees through pair-wise comparison suggesting a high degree of external validity. One additional criterion, “knowledge of availability alternatives” was highly ranked by advisory committee members but is not commonly considered by other agencies. Similarly, a previous review of criteria used by United States-based organizations conducted in the 1980s (Reference Eddy7), described only one of six organizations considering availability of alternatives. The reason for differences in preference between our committees and other committees internationally is not known.
Similarly, both of our advisory committees assigned the highest weight to the potential clinical impact of the policy decision, a criterion which has been identified in all other current, practical approaches by HTA agencies. Nevertheless, disease burden was inconsistently rated across committees, achieving a score reflecting almost half (52 percent) of that of clinical impact by one committee and 84 percent of that by the other. This might be explained by the different professional backgrounds of committee members. Ultimately the CADTH committees elected to use disease burden as a priority-setting criterion. This is consistent with Neumann and colleagues, who suggested “comparing rankings of diseases by disease burden and research dollars can provide a useful starting point for discussions about priorities for research and the need for better methods for determining them” (Reference Neumann, Rosen and Greenberg13).
One limitation of our approach is the potential for preferences of committee members to change over time with changes in committee membership. Although the original survey capturing preferences of committee members was conducted in 2005, the terms of reference, and general composition of CADTH committee members has remained consistent since that time, so we believe that the criteria weighting has remained robust.
In line with the EUR-ASSESS recommendations, we have also conducted a preliminary evaluation of the impact of our priority setting framework. A survey of our advisory committees indicated 67 percent of respondents believed the relevance of HTA topic proposals increased during the time that this process was implemented. Additionally, 88 percent of respondents believed that changes to the HTA topic development process resulted in an increase in the level of impact of HTA reports. We hope this portrait of our current priority setting framework will invite further feedback and discussion across HTA and similar healthcare research agencies and will provide some insights into the application of tools from decision science and policy making for the purpose of setting research priorities.
CONCLUSIONS
A priority-setting process for HTA projects at CADTH was developed. This process involves multicriteria decision analysis and a deliberative process. The approach works well and was easy to implement. Feedback from CADTH advisory committee members involved in priority setting has been positive. This approach may assist HTA and other research agencies in better priority setting by informing the selection of the most important and policy-relevant topics in the presence of a wide variety of research proposals. This may in turn lead to efficiently allocating resources available for HTA research.
CONTACT INFORMATION
Donald R. Husereau, BSc Pharm, MSc (donh@cadth.ca), Director, Project Development, Michel Boucher, B Pharm, MSc, RPh (michelb@cadth.ca), Lead, HTA Development, Hussein Z. Noorani, MSc (husseinn@cadth.ca), Lead, HTA Impact, CADTH, Canadian Agency for Drugs and Technologies in Health, 600-865 Carling Avenue, Ottawa, Ontario K1S 5S8, Canada
CONFLICT OF INTEREST
All authors have received grants to their institution from the Canadian federal, provincial, and territorial Ministries of Health.
