What Types of Products and Outputs?

Table 1 shows that the number of documents published by the agencies peaked in 1997–1998, then slightly decreased but remained steady until 2000–2001. Because the agencies differed in annual budget and number of employees, this table should be interpreted with caution. Its purpose is not to rank agencies according to their outputs. The populations of Ontario (11,894,000) and Quebec (7,417,000) are fairly large compared with British Columbia (4,101,000), Alberta (3,059,000), and Saskatchewan (1,017,000), while the budget allocated to HTA by each provincial government also varies considerably. The national-level agency receives funding from the provinces because its mandate includes facilitating coordination across the country. In addition, as stressed by several CEOs, agencies face significant challenges in recruiting researchers specialized in HTA. The data show that decision-makers and practitioners across Canada, including those in provinces and territories devoid of an HTA agency, had access to, on average, over thirty HTA products during the study period.

Table 2 provides details on the type of products delivered by the HTA agencies during the study period. While four agencies were still emphasizing the “traditional,” full HTA report (representing 81-100 percent of their production), two of them diversified their products by publishing joint reports (47 percent for A6), a result of partnerships with key health organizations (e.g., medical associations, workers' compensation boards), and several short documents such as peer-reviewed bulletins, rapid HTAs, and technical briefs (43 percent for A3). Among the non-peer-reviewed documents that were excluded from our more detailed analyses, it is worth mentioning that Agency 4 produced up to ten different publications. In addition, three agencies did not have a newsletter during the period under study, while the other three issued an electronic and/or paper newsletter on a quarterly basis. The format and appearance of reports has substantially improved since 1995, a change that may have improved reader-friendliness. Overall, while the bulk of HTA documents produced in Canada were full HTA reports (76 percent), we can see that a few agencies were diversifying their products in an attempt to meet the informational needs of a range of users.

We observed substantial variation in the number of traditional scientific outputs such as scientific publications and conferences. Table 3 shows that three agencies actively supported the publication of results in scientific journals (A1; A4; A5), while the majority encouraged presentations at conferences. Agency 1, which relies on several university-affiliated researchers, stands out in this respect. The CEO of this agency also confirmed that researchers are encouraged to apply for external funding to cover expenses for attending scientific meetings. As discussed below, this opportunity has a direct impact on this agency's portfolio. Overall, as recognized by most CEOs, while scientific journals may not be the most appropriate vehicle for reaching decision-makers, it is clear that such outputs do contribute to the agency's visibility and prestige. Indeed, establishing scientific credibility is often viewed as essential for convincing certain audiences. This is even more so in the case of physicians who are involved or interested in clinical research and who value peer-reviewed publications. Given that the resources of the six agencies differ significantly, the disparity in terms of scientific outputs is not entirely surprising. It nonetheless shows that conflicting legitimacies are at play—emphasizing the production of traditional scientific knowledge versus user-oriented knowledge. Because the credibility of HTA is intimately linked to its scientific rigor, this tension may impact the agencies' ability to recruit and retain evaluators seeking to build a career according to academic standards. Several interviewees favored a balanced approach—one that encourages peer-reviewed publications yet does not ignore target users of HTA who do not read scientific journals.

Assessing What Kinds of Technologies? Addressing What Issues? And Providing What Types of Conclusions?

Table 4 presents the types of technology assessed by the Canadian agencies. Two agencies clearly focused on health services (76 percent for A1; 56 percent for A6), while Agency 3 focused on drugs (74 percent). Four agencies were only slightly or not at all involved in assessing drugs (A1; A4; A5; A6), perhaps because several provinces had university groups and/or governmental bodies that produced cost-effectiveness analyses of pharmaceuticals. Diagnostic technology and screening tests did not constitute a very large proportion of HTA reports (less than 31 percent), and devices were only infrequently evaluated (less than 19 percent). A limited but significant portion of the production of five agencies included essays and publications concerning research tools. This kind of output, which does not directly respond to users' requests, is nonetheless useful in consolidating the practice and promotion of HTA. Excluding drugs, most agencies presented a fairly balanced portfolio. Overall, only a few agencies focused on certain topics, several did not do drug assessment, and most assessed several different types of technology. The interviewees believed this was the result of a certain institutional specialization of the agencies as a result of their specific jurisdictional contexts. As mentioned by a few CEOs, a critical mass of researchers may be required before engaging in a new area of research, and there is a willingness to limit overlap between groups of knowledge producers.

Extending the analysis to examine the content of HTA reports reveals further distinctions between the agencies. HTA should examine not only the efficacy, safety, and costs of health technology, but also the ethical, social, and legal issues. We were curious to see whether this full range of topics, in fact, was being addressed. Table 5 shows that the most frequently issues addressed were costs and effectiveness. Around a third of the documents of all agencies included a cost component. As one might expect, the vast majority of the production of four agencies (71 percent for A2; 41 percent for A3; 78 percent for A4; 69 percent for A6) examined effectiveness. However, very few agencies actually related costs to effectiveness by doing full-fledged cost-effectiveness analyses. Only Agency 3 did so in close to half of its reports. We were surprised to find that, despite the growing prevalence of chronic diseases, quality of life was seldom part of assessments. Safety was addressed in less than a third of the agencies' reports.

As expected, ethical and social issues were not frequently addressed. Agency 1 seemed to discuss such issues most frequently (in 40 percent of its reports), while Agencies 3 and 4 rarely provided information on this subject. That the official definition of HTA was not put into practice to its fullest extent is even more striking in the case of legal issues. According to interviews, securing access to staff specialized in ethical and legal issues when resources are limited and the demand great was problematic for several agencies. Of interest, we observed that the state of current practices was documented fairly often by two agencies (68 percent for A1; 69 percent for A6), which is compatible with the “contextualization” of HTA findings for regional or provincial decision-makers. By providing information about the current levels of service utilization, surgical rates, or prescription patterns, perhaps evaluators were aiming to better inform decision-makers about the potential gap between evidence and practice and, therefore, to support decisional processes. Similarly, four agencies fairly frequently included other outcomes, perhaps in response to specific requests, such as findings about organizational aspects (e.g., collaboration between midwives and obstetricians, funding arrangements) or data about the impact of a recent provincial policy (e.g., hospital length-of-stays, waiting lists) in their assessments.

In summary, close to 50 percent of the HTA reports available in Canada from 1995 to 2001 addressed effectiveness, 37 percent examined costs, and less than 25 percent discussed cost-effectiveness, ethical and social issues, safety, and quality of life. Up to 27 percent of documents conveyed information about current practices and up to 43 percent reported other outcomes. Table 5 shows that, overall, Canadian agencies were not entirely adhering to the concept of HTA. First, ethical, social, and legal issues were not dealt with very often. Second, the majority of agencies addressed current practices and other outcomes that, while relevant to practice and policy, are not key components of HTA guidelines. Perhaps this could be interpreted as a redefining of HTA—providing a more contextualized informational base for decision-makers—even though several issues are not being addressed.

We also thought it relevant to consider the conclusions reached in these HTA documents. In general, the documents were clearly written and contained three types of conclusions: (i) negative (24 of 197=13 percent); (ii) neutral (124 of 187=66 percent); and (iii) positive (39 of 187=21 percent; see examples in Table 6). Negative messages included reporting a technology had not been proven effective or emphasizing the risks associated with its use. Neutral conclusions did not stipulate the actions that should be followed based on the assessment, emphasized the weakness of the evidence, or remained somewhat evasive, stressing that further research was needed. Positive conclusions underscored that a given technology was both effective and safe, or cost-effectiveness reasonable. Formal recommendations were less often found. Except for A5 and A6, only a small proportion of HTAs contained recommendations: A1, 6 of 25=24 percent; A2, 3 of 17=18 percent; A3, 6 of 61=10 percent; A4, 1 of 32=3 percent; A5, 16 of 36=44 percent; A6, 9 of 16=56 percent. This is compatible with how interviewees explained the fragile balance between drawing conclusions about the strength of the evidence and defining what should be done with findings. If the former is definitely part of evaluators' task, the latter is seen as the responsibility of decision-makers and clinicians.

A Summary: Six Agencies With Different Production Lines?

Table 7 shows how the six agencies, even though they are all assessing health technology and services, are organizational entities with rather different “production lines.” First, some have engaged in product diversification by issuing shorter documents. Second, some have tended to focus more on certain types of topics (drugs, health services), issues, and outcomes. Product diversification may be seen as a response to the imperatives of decision and policy making (timing, readability). The specialization of agencies may result from attempts to attain a critical mass of researchers with similar expertise and to complement the work of other groups (or to avoid stepping on another organization's turf). Agencies, thus, are responding to the expectations of specific users, while at the same time trying to build their own unique mandate or vision statement.

Agency 1 has diversified its products to a limited extent, supports scientific publications, and has specialized in health services assessment. Its high percentage of contextualized documents addressing current practices and other outcomes is highly compatible with the aim of providing “information about utilization of health-care services in the […] population. And this may be information to assess the equity or access to services, it may be information about the quality of care, or about the outcomes” (Researcher). All interviewees from this agency confirmed that, even though the Ministry of Health “informs the agency agenda,” most projects were investigator-initiated. Because several of these projects may be funded through national and provincial research funding bodies, the portfolio of this agency is partly shaped by the funding criteria set by these bodies and its investigators can maintain a high level of scientific autonomy.

Agency 2 has diversified its products, weakly supports scientific publications and has a balanced portfolio. Its clearly stated mandate is to “explore ethical and social issues raised by the accelerating demand for technological interventions” and suggests an interest in developing the “theoretical foundations for using large data sets to understand the determinants of health in population groups, and to derive better evidence-based approaches in health care” (Web site). This is compatible with the finding that 24 percent of its documents fitted in the “Essay and Research Tool” category. Even though we observed limited contextualization of findings, Agency 2 is explicit about its role in providing decision-makers with guidance: “Research findings are published in detailed reports, serving a principal objective of disseminating results in a form useful to decision-makers in a position to use them. By identifying and communicating how limited health-care resources can be most effectively applied, [the agency] is able to assist the public sector in its policy development and planning efforts” (Web site). In fact, input from several sources is considered when the agency decides on what topics to assess. According to one researcher from this agency, the most important ones are prioritized because they raise significant, province-wide issues—“issues that have been raised from several different perspectives simultaneously.” In other words, this agency's portfolio has been shaped by an academic orientation to HTA, and by recurring and convergent suggestions for assessments.

Agency 3 has also diversified its products, weakly supports scientific publications, and specializes in assessing drugs. This specialization is consistent with the finding that it is the agency with the largest proportion of assessments examining cost-effectiveness (51 percent). Its priority-setting process is quite formalized and is significantly influenced by program-specific committees composed of representatives from various regions. As explained by an executive from this agency, a cumulative list of topics is built up throughout the year and then presented to the members: “We had a list of almost fifty possible topics for them and some were clearly not appropriate and some were very appropriate. But we asked them to vote and in the end, we got ten that received multiple votes.” Consequently, the agency's portfolio is the result of a structured process and reflects topics that matter to a wide set of stakeholders.

Agency 4 acts according to a broad mandate: “Support a community of researchers who generate knowledge that improves the health and quality of life of people […] throughout the world (Web site).” This agency has diversified its production to a limited extent, supports scientific publications and shows a balanced portfolio. Its findings are contextualized as well, but with the goal of “servicing” its population, from decision-makers to members of the public. In the words of one researcher: “We don't select topics, we answer everything that comes to us. What differs is the level of analysis or the comprehensiveness of the assessment.” This view was reinforced by the President of the organization of which this agency is part, stressing that the agency should never decline a request. As a result, according to the agency's head, only between 5 and 10 percent of the projects are investigator-initiated. Overall, this agency's portfolio is oriented by a public service “pull,” rather than a scientific “push.”

Agency 5 has not diversified its products, supports scientific publications, and has a balanced portfolio. Its findings are contextualized and recommendations are frequently formulated. Input for priority setting is sought from several sources, and criteria are used to decide which topics to investigate. These criteria include the burden of illness, the number of individuals concerned, the costs, the available evidence, and so on. While it believes requests formulated by the Ministry of Health or officials from the health-care system should not be declined, the depth of the response is based on available resources and the extent to which the request fits the criteria. Up to a third of this agency's documents are investigator-initiated. Thus, the final portfolio may reflect an interest in the production of both scientific and user-oriented knowledge. Such a balanced approach helps investigators maintain a certain level of scientific autonomy.

Agency 6 has diversified its products to a limited extent and specialized in health services assessment. Contextualization of its findings is important, and recommendations are frequently formulated. This is compatible with its mandate: To support the “development of strategic health policy by providing evidence on researchable issues” (Web site). The agency has been engaged in an environmental scanning process that informs its priorities. This process facilitates frequent interactions between researchers and decision-makers. Of interest, because this process relies on bringing together the opinions of diverse stakeholders, one researcher notes that “it's difficult to say how much of that [the topics chosen] is us and how much of it them, there's an interplay there.” In other words, the resulting portfolio is seen as responding to both supply and demand for HTA.

Paradox 1. Diversification of Products Versus Streamlining

It is interesting to observe that, after recent efforts to standardize or “streamline” the content of HTA reports (INAHTA), two agencies have diversified their products. This finding is not entirely surprising given the recent emphasis, especially in Canada, on increasing the uptake of research (23;28;29). Full HTA reports may be seen as too lengthy, technical or detailed to be digestible for certain categories of decision-makers, or in situations where decisions need to be made within a very short time frame. Shorter documents may take the form of detailed, stand-alone summaries of full HTA reports designed to inform busy decision-makers, or rapid technology assessments designed to provide brief and perhaps provisional answers to specific questions. Nonetheless, producing shorter documents requires a systematic and rigorous approach to summarizing and interpreting the evidence in a concise way. Standards and guidelines for developing shorter documents should be developed, not only to facilitate the work of the supply side, but also to help the demand side distinguish the different types of HTA products (16;32).

Paradox 2. Contextualization Versus “One-Size-Fits-All” Assessments

Our results show that Canadian agencies often contextualize their results by adapting the content of their assessments to the jurisdictional context. This contextualization of findings is recognized as key to facilitating the use of research. Jacob and Battista (21) observed that assessments had a greater impact when they were tailored to local conditions (issues, actors, decision processes). However, one methodological assumption in HTA is that all systematic syntheses of the available evidence based on the same methodology should yield similar conclusions (8). Therefore, in principle, any HTA should be applicable to any context. A paradox, thus, lies in the willingness to make explicit the relevance of the findings to a local audience, while still adhering to a universal way of assessing health technologies. We believe that only rarely will “one-size-fits-all” assessments fulfill the needs of decision-makers. Because some interviewees explained how they could use a HTA produced by another agency and develop a specific set of recommendations adapted to the decisional issues of their own jurisdiction, further research should examine how sharing work between agencies could help increase both access to and uptake of HTAs.

Paradox 3. Addressing a Large Scope of Issues Versus One Well-Delineated Question?

HTA is often promoted as being an interdisciplinary inquiry that addresses a wide spectrum of issues, from effectiveness to ethics (24). Our findings suggest that, even though this may be true in principle, it is not the case in practice. From the producers' perspective, there may be some obvious reasons why ethical, social, and legal issues are rarely addressed. The staff of HTA agencies are usually trained in epidemiology and may feel less equipped to analyze legal, social, and ethical issues. From the users' perspective, the need to obtain, in concise, reader-friendly language, an answer to a well-defined policy question may preclude a sophisticated analysis of such inherently complex issues (5). Nonetheless, health policy and clinical practice are often shaped by legal incentives (4). Cookson and Maynard (8) also stressed that HTA should examine equity and instill the socially responsible allocation of resources. Thus, collaboration between HTA practitioners and ethicists, sociologists, and jurists should be increased to respond to the very specific requests of decision-makers and maintain a broad interdisciplinary perspective (22).

Paradox 4. Doing More But Producing Less Measurable Outputs?

Despite that the budgets of most Canadian agencies have modestly but steadily increased since the late 1990s, the number of HTA documents published is not increasing drastically. Perhaps the staff of HTA agencies are being mobilized in a range of activities that do not translate automatically into measurable outputs. In fact, facilitating the uptake of HTA requires promoting and explaining what HTA is and how it can be used to inform policy and practice. The need for such groundwork is confirmed by a study on technology acquisition in Canadian hospitals that found: “Respondents had accepted the concept of technology assessment, although they may not have instituted the appropriate organizational mechanisms to put this concept into practice” (10, page 26). Indeed, HTA is at risk of remaining a poor instrument for introducing knowledge-based change if the ability of providers and decision-makers to understand HTA principles and methods is not increased (27). In other words, HTA agencies have to deploy these less measurable but necessary efforts, and they will pay off in the long term.

Coordination and Collaboration

While perhaps not a complete surprise to some, these four paradoxes clearly highlight how the HTA community has set a very broad mandate for itself while operating with limited resources. Could coordination and collaboration amongst agencies within large countries or across European States facilitate the sharing of expertise and assessments and decrease some of the tensions inherent in the HTA production process? To explore this issue, we suggest adopting a global perspective that defines HTA production capacity as depending upon a collective of agencies. The five provincial agencies each receive funding from one major source (usually the provincial Ministry of Health), and the agencies each turn to this one source when faced with an increased demand. Nonetheless, it is possible to frame things differently by recognizing there exists a pool of qualified resources, able to produce a pool of HTA documents that can be used by decision-makers in any jurisdiction. Perhaps we could build on the existing strengths and expertise of each agency to better meet the needs of decision-makers across the country by actively supporting access to a range of HTA products covering different kinds of issues. Incentives for collaboration among agencies could be organized according to a broader, countrywide knowledge-brokering strategy. A similar reasoning could apply to collaboration among European states.

Given the four tensions outlined above, collaborative efforts could aim to (i) support a certain diversification of HTA products through the development of common guidelines for both short and full HTA reports that would orient not only the assessment process but also how decision-makers appraise and use such documents; (ii) build a network of evaluators and knowledge-brokers located in each agency who could share their know-how about contextualization and adapt/translate HTA findings produced in other jurisdictions to the concerns of local decision-makers; (iii) facilitate access to experts in ethics, sociology, and law who are familiar with HTA and who can contribute directly to assessments and support a certain specialization of teams of evaluators; (iv) develop a joint strategy to promote HTA through a series of capacity-building activities targeting providers and decision-makers across the country.

The overall goal of such efforts would be to consolidate assessments, boost production capacity, and increase HTA uptake. Because five of the six Canadian agencies are provincial initiatives, the question of encouraging specialization in agencies has never been analyzed in a systematic manner, although the mandate of the national-level agency includes, in principle, facilitating coordination. According to McDaid, such a coordinating role remains both limited and challenging because “there have been disagreements over the future role of both federal and provincial bodies given the existence of local agencies” (30). However, given that agencies are operating under serious constraints and there is an increasing demand for HTA, it seems more important than ever to fully develop this role.

By combining qualitative and quantitative analyses, the objective of this study was to more fully understand the constraints affecting HTA production. Of course, our findings and analyses should be interpreted with caution. The purpose of this study was not to compare the respective performance of the agencies. Because their mandate, structure, and level of financial resources vary greatly, the data we analyzed says more about their individual particularities than about what should be a “gold standard” in HTA production. In other words, our intent was not to define the one best way for producing HTA, but rather to gauge how HTA is being practiced and applied in different Canadian jurisdictions. Despite this caveat, this study is valuable in that it extends previous work by Menon and Topfer (31) by documenting three additional years of production and by including the work of two other agencies in the analyses. It is also a response to an overall lack of studies on the links between organizational structure, HTA content, and HTA production.