Concerns about the nonevaluated benefits and high costs of healthcare interventions have led to the institutionalization of health technology assessment (HTA), a policy-oriented, interdisciplinary research process aimed at informing decision making (Reference Banta and Oortwijn2;Reference Menon and Marshall64;Reference Vallesco-Garrido and Busse84). Around the world, programs for HTA have been established through either the creation of agencies, institutes, or HTA units in universities, hospitals, or in existing governmental and nongovernmental bodies (hereafter HTA organizations: HTAOs) (45;47;Reference Vallesco-Garrido and Busse84). In parallel, the object of HTA has evolved from the capital-intensive technologies to clinical outcomes and delivery models (3;4;5;21), which led HTAOs to diversify their production and expand their interactions with stakeholders at the political, organizational and services delivery levels (Reference Lehoux, Tailliez and Denis56).Footnote 1 In light of growing demands for public expenditures accountability, this increasing role of HTAOs in decision making underscores the importance for them to demonstrate their performance.
Organizational performance refers to a judgment reached through the interaction of stakeholders on the overall and specific qualities that characterize the relative worth of an organization (Reference Sicotte, Champagne and Contandriopoulos80). This definition is consistent with the view that organizational performance has many dimensions, for which seemingly contradictory conclusions can be reached. Indeed, an organization performing well according to one criterion or in the perspective of one group of stakeholders may possess attributes that make its performance worst in terms of other criteria (Reference Cameron17;Reference Sicotte, Champagne and Contandriopoulos80). There is much to learn about HTAOs' performance from available evaluations and the accumulating evidence of HTA impact. Yet, what constitutes a performing HTAO will remain an elusive and fragmented concept unless we develop a shared language to integrate its various dimensions and the complementary perspectives of stakeholders. In this study, we propose a theoretically grounded model of HTAO performance that aims to do just that.
Based on an exploratory review of evaluations of HTAOs, Wanke et al. proposed a generic evaluation framework that highlights the design, product, and implementation considerations for planning evaluations (Reference Wanke, Juzwishin and Thornley85). Of interest, their framework presents dimensions of performance that relate HTAOs to the broader system in which they operate. Hailey (Reference Hailey42) presents elements of effectiveness for HTAOs and structures his discussion around four dominant models of organizational performance: goal (Reference Price72), constituency (83), open system (9;86), and competing value (Reference Bennis8) models. Representing different approaches to organizational analysis, these models each identify a set of performance dimensions. However, none of them is entirely applicable to HTAOs (Reference Hailey42): they do not allow for the process of arbitration among competing perspectives nor do they integrate the complementary dimensions necessary to judge the social value of an HTAO. This problem of fragmented perspectives is recurrent in organizational performance studies. In reviewing that literature, it became apparent to Sicotte et al. that Parson's social action theory (Reference Parsons69;Reference Parsons70) could provide a comprehensive view of how organizations function, and thus, provide a framework that accounts for the multidimensional nature of performance (Reference Sicotte, Champagne and Contandriopoulos80;Reference Sicotte, Champagne and Contandriopoulos81). Inspired by Parson, they developed a conceptual framework of healthcare organizations' performance that is based on four fundamental functions that an organization needs to align and balance to perform well: goal attainment, production, adaptation to the environment, and culture and values maintenance. The framework has been used to assess the performance of hospitals (20;36;40;59) but needs to be adapted for HTAOs.
The “goal attainment” function concerns the strategic aptitudes of the organization to effectively and efficiently meet its set objectives. The “production” function is the technical core of the HTAO. It includes the volume of production, coordination mechanisms that ensure the smooth running of operations, and the quality of outputs. The “adaptation” function refers to the ability of the HTAO to attract required resources, including the capacity to mobilize external support. Furthermore, this function represents the ability to respond to population needs in accordance with social values, and the capacity to learn and innovate. Finally, the “culture and values maintenance” function provides meaning and cohesion among members of the organization, consensus on fundamental values and norms, and contributes to organizational climate. From the Parsonian perspective, performance is determined by the dynamic equilibrium resulting from the interactions among the functions. These interactions translate into strategic, tactical, operational, contextual, allocation, and legitimization alignments between functions, which work in synergy or in opposition to influence performance (Reference Sicotte, Champagne and Contandriopoulos80).
The objective of this study is to present a model of HTAO performance that stems from the work by Sicotte et al. For each function, we describe the dimensions of performance identified in the HTA literature and test them against the available empirical evidence of HTAOs’ performance. Using examples, we then show how the alignments between these functions provide additional insights into the nature of HTAOs’ performance. The model can serve as learning tool to optimize HTAOs’ performance and ensure their enduring role as key players in healthcare systems.
METHOD
The approach consisted in reinterpreting the literature pertaining to HTAOs’ activities and performance in light of the four core organizational functions just described (Reference Parsons69;Reference Parsons70;Reference Sicotte, Champagne and Contandriopoulos80). For each function, we first sought to develop an exhaustive list of dimensions that have been used or claimed to be important in the evaluation of HTAOs’ performance. A thorough search was undertaken of the literature on the performance of HTA in general and of HTAO in particular using the databases of the OVID platform (MEDLINE, EMBASE, Healthstar) and the NHS Centre for Reviews and Dissemination (NHS EED, HTA, DARE). This approach was supplemented by a review of the publications and Web sites of HTAOs and HTA-related resources, such as evidence-based policy and practice, and knowledge transfer resources. Finally, references listed in identified articles provided an indirect route to additional relevant references. The search strategy targeted references published in English or French (January 1992–March 2007). The scientific literature consisted of quantitative and qualitative studies and conceptual and opinion papers. The gray literature consisted of external evaluation reports, last available annual reports, technical reports, opinion surveys as well as strategic and operational information presented on the Web site of organizations that are member of the International Network of Agencies for Health Technology Assessment (47).
The first author performed a thorough analysis of the literature and extracted the information pertaining to performance and related concepts. All identified dimensions and subdimensions of performance were tabulated and discussed and confirmed through several meetings involving the authors. Using the results of this first step as our framework, a second reading of the literature was undertaken to specifically identify those dimensions of HTAOs’ performance that have been used in evaluations and extract representative examples for each of the six alignments.
RESULTS
Our analysis reveals a rich body of literature pertaining to the performance of HTAOs (n = 47 documents). Whereas relatively few HTAOs underwent a formal evaluation, externally commissioned (n = 7), we found numerous partial evaluations (n = 32) and comparative studies (n = 8) that appraised some aspects of their performance. Published on the Web site of HTAOs or in the scientific literature, partial evaluations consist of annual reports (n = 8), survey reports and formal presentations (n = 6), or scientific articles (n = 18) that assess one or few selected aspects of an organization's activities. Comparative studies (n = 8) consist of scientific articles describing either the production or the impact of HTAOs. Combined with opinion and conceptual papers rich in best-practices ideas, these sources served to identify the performance dimensions of our model (Figure 1). For each function, we first explain what the conceptual dimensions of performance mean. Then, we describe the extent to which they have been used to assess the performance of HTAOs. Finally, we provide evaluation examples to illustrate how the alignments between the functions represent additional dimensions of performance.
Goal Attainment Function
This function has to do with the strategic choices that an HTAO must make to achieve its long-term goals of effectiveness and efficiency. Effectiveness is the degree to which the organization achieves its objectives. Efficiency refers to the ability of the organization to manage its activities and resources such that it reaches its goals at the lowest costs. Thus, appraisal of the goal attainment function should capture the ability of the HTAO to produce and disseminate information (i.e., outputs) that in turn affects the way decisions are made (i.e., expected impacts).
The exploratory review of evaluations of HTAOs by Wanke et al. (Reference Wanke, Juzwishin and Thornley85) and the Research Impact Framework, developed by Kuruvilla et al. (Reference Kuruvilla, Mays and Pleasant51), were particularly useful to identify the performance dimensions for this function. Building on these references, Figure 1 lists the most relevant dimensions to evaluate HTAOs’ effectiveness. It provides a basis to develop a coherent narrative of actual and potential impacts of HTA activities that facilitates comparison and learning across time, space, and methods (Reference Kuruvilla, Mays and Pleasant51).
Decision-Making Impact
The impact of HTAOs on decision making at the policy, governance, management, and clinical levels can be assessed for different groups of stakeholders from international, national, regional, and local perspectives (Reference Kuruvilla, Mays and Pleasant51;Reference Black11). According to Wanke et al., “at the first level of impact, stakeholders are aware of the existence of the HTA agency or products. Awareness may lead to an attitude change regarding HTA, ideally representing acceptance of the agency or HTA products” (Reference Wanke, Juzwishin and Thornley85). At the second level, the goal is to assess whether and how HTAOs influence decision making. Here, we draw from the literature on the use of science in public policy (53;Reference Pelz, Yiner and Cutler71), which describes three types of utilization. Symbolic use of HTA outputs serves to strategically justify or legitimize existing policies or positions. Conceptual use refers to change in awareness, thinking, or understanding of specific issues and can lead to new ideas and language that influence the nature and the substance of policy discourse (Reference Kuruvilla, Mays and Pleasant51). Instrumental use refers to actual change in policy or practice.
Organization and Service Impact
Given their object of analysis, HTA activities can have a range of impact on quality of care, on the way information systems are developed and used, on the way health systems and services are managed, and on how resources are allocated and used (Reference Kuruvilla, Mays and Pleasant51).
Figure 1. Organizational functions and dimensions of performance. Numbers in parentheses represent times a function, dimension, or a subdimension was used to evaluate the performance of the health technology assessment organization (HTAO) in the 47 reports consulted. Adapted from Sicotte et al. (Reference Senge79).
Societal Impact
Described as “ultimate HTA outcomes” by Wanke et al., this level looks beyond the impact of individual HTA outputs and explores the impact of HTAOs’ mandate on the health status of the population or the health system in general (Reference Wanke, Juzwishin and Thornley85). Furthermore, as a result of the dissemination and capacity building activities of HTAOs, various impacts on knowledge, attitudes, and behaviors can be expected (Reference Davies, Drummond and Papanikolaou25;Reference Kuruvilla, Mays and Pleasant51). These activities can improve what we call “HTA literacy” and foster a culture of evidence. Akin to health literacy, HTA literacy encompasses the wide range of “knowledge-uptake” skills and competencies required to seek out, comprehend, evaluate, and use HTA information to make informed decisions (53;Reference Zarcadoolas, Pleasant and Greer87). As such, it builds receptor capacity and allows for an interested and informed involvement of stakeholders in the public debate about health, scientific knowledge, and technology. Paraphrasing Lavis et al., the creation of an HTA-attuned culture among users and a decision-relevant culture among HTA researchers can create cultural shifts that would facilitate the on-going use of HTA knowledge in decision making. This constitutes the most appropriate generic measure of the future impact of HTAOs (53).
Research Impact
HTA activities can have impacts in the field of HTA itself and in the specific fields of the technologies evaluated in terms of: type of skills and knowledge generated, research methods, technology development, research networks and user involvement, and research leadership (Reference Kuruvilla, Mays and Pleasant51;Reference Wanke, Juzwishin and Thornley85).
Efficiency
Whether the additional costs of providing information are justified or commensurate with the gains in favorable outcomes attributable to the HTAO's activities is difficult to measure because of the absence of the counterfactual and the methodological—and political—challenges associated with the attribution of impact (Reference Hailey42;Reference Jacob and McGregor49;Reference Wanke, Juzwishin and Thornley85). In the literature, evaluations of the effectiveness tend to focus on processes or intermediate outcomes such as impact on decision making. Whereas some evaluations allude to the efficiency of HTAOs (6;Reference Davies, Drummond and Papanikolaou25), we found no example of methodologically based efficiency evaluations. To date, authors have discussed efficiency in terms of the economies of efforts that can be realized through the closer collaboration of HTAOs, and by means of priority setting mechanisms that consider value for money or payback in making decisions about priorities for assessment (Reference Cookson and Maynard21;Reference Davies, Drummond and Papanikolaou25).
Production Function
As the technical core of the organization, the production function can be assessed according to four interlinked dimensions of performance.
Quantity
This dimension describes the production volume of an organization per type of output (Reference Jacob and McGregor49). Whereas HTAs reports, early warning briefs, and so on, are the most evident products of an HTAO, diffusion and dissemination activities transform HTAs into knowledge for potential users. In turn, capacity building activities aim at increasing the availability of individuals in the policy making and end-user communities who can effectively use HTA results. They can also aim at changing the attitudes of decision makers toward HTA and the bureaucratic processes by proposing ways to integrate research evidence into decisions (45). From a performance perspective, this dimension can be used to track trends in production and compare it with set targets. We note that HTAs are by far the primary focus of interest in the performance literature, with relatively few evaluations of dissemination and capacity building activities.
Table 1. Processes at Work in an HTA Organization
HTA, health technology assessment.
Productivity
This dimension relates the quantity of outputs to the resources invested. While productivity assessment is widespread for healthcare organizations, few examples exist in the HTA literature (6;27;49;73). Productivity indicators can serve as internal references to assess trends of productivity or measure the effects of strategic or operational change.
Coordination Mechanisms
Central to the performance of HTAOs, coordination mechanisms refer to core HTA and management processes (Reference Busse, Orvain and Velasco14;Reference Hailey42;Reference Wanke, Juzwishin and Thornley85), which are the formalized methods to produce some outcomes, generally involving a series of ordered and/or interdependent goal-directed steps. In fact, processes ensure the continuity and the coherence between types of activities. They are pivotal to the production and the life of the organization (Table 1).
Quality
Owing to its multidimensional and contingent character, the meaning of quality cannot be dissociated from the intentions and roles of different stakeholders and the particular environments in which HTA outputs are used (Reference Haddad, Roberge and Pineault41). Notwithstanding, we group our findings into three subdimensions of quality.
Accessibility
This concept describes how easy it is for stakeholders to get to, understand, and use HTA outputs. Timeliness is imperative to increase the prospects for use of HTA outputs (31;39;49;65;66;67;74). HTA activities are to be undertaken quickly, early, and/or routinely in the life cycle of health technologies to ensure the information is available when decisions have to be made (Reference Davies, Drummond and Papanikolaou25;Reference Sculpher, Drummond and Buxton78). Readability, channels of dissemination and geographical availability constitute additional characteristics of HTA outputs that determine whether obstacles exist between the output and the user (Reference Drummond and Weatherly31;Reference Ross74).
Comprehensiveness
This subdimension refers to the scope of HTA. Comprehensiveness can be assessed in terms of the types of outputs produced, technology evaluated, dimensions considered, studies conducted, assessors and users targeted, the types of evidence considered, and purpose (4;21;23;30;38;56;57;63) (Supplementary Table 1; available at http://www.journals.cambridge.org/jid_thc). Some have argued that a broader scope for evaluation is warranted as decision makers increasingly tackle complex issues (30).
Technical Quality
Technical quality in the HTA literature refers to the quality of HTA reports, and the methodological quality of HTA, in terms of the appropriateness of the methodology selected and the competency of execution (14;32;35;39;43;56;60). Numerous benchmarks exist for the technical quality of HTA (Reference Busse, Orvain and Velasco14;43). For dissemination and capacity building activities, Lavis et al.'s five questions’ framework for knowledge transfer strategies provides a practical base to build from: what, to whom, by whom, how and with what effects evaluation knowledge should be transferred to decision makers (53)?
Adaptation Function
The adaptation function deals with the relationship between the HTAO and its environment. To maintain flexibility and competence in the face of rapid changes, an HTAO must seek to develop practices and draw resources that are responsive and adjustable. Hence, its ability to acquire and maintain resources, gain external support, respond to needs, learn, and innovate become valued performance dimensions to assess the success with which an organization judiciously manipulates the opportunities and threats present in the environment (Reference Sicotte, Champagne and Contandriopoulos80;Reference Hailey and Juzwishin44).
Capacity to Acquire and Maintain Resources
This dimension pertains to the ability to attract human, technological, financial, or symbolic resources. Furthermore, the extent to which an HTAO is involved in networking, through communication, collaboration, or coordination activities, is an important aspect of its performance (Reference Maynard and McDaid60). The underlying idea is to mobilize resources by avoiding duplication, and by sharing resources, expertise, and information (47;56;61;64;85).
Ability to Gain External Support
The capacity to involve the community and mobilize its support can be evaluated in terms of stakeholders’ commitment (13;22;23;24;55;60;65). The social interaction theory of knowledge transfer posits that a greater degree of interaction between researchers and end-users results in greater short- and long-term impacts (Reference Landry and Amara52;Reference Maynard and McDaid60). The importance “for HTA agencies to interact with partners . . . may depend to some extent on the technology being evaluated and on the nature of the policy question being addressed, but it is mainly a function of how the organization perceives its role” (12). In practice, “appeal to stakeholders and involving them in decisions and the process through which decisions are made are becoming touch stones of best practices” (23).
Two other performance subdimensions pertain to external support: visibility and credibility. Visibility refers to the potential sphere of influence of the HTAO and determines its reputation. If associated with positive perceptions, visibility can convey the necessary legitimacy for the organization to exercise its influence on decision making. Whether we talk about knowledge of the organization's mission or of its specific activities, the idea is to create a pull for the results by educating stakeholders on the value of HTA (Reference Dixon, Coleman and Nicholl27). Finally, HTAOs must acquire and maintain credibility, both scientific and political (Reference Jacob and Battista50). Scientific credibility rests on recognized expertise and rigorous standards of methodology, and conveys authority in scientific judgment as perceived by peers. Political credibility refers to the trustworthiness of an organization and conveys authority in its social value judgment as perceived by stakeholders (23;27;50;62).
Responsiveness to Needs
This dimension refers to the ability to identify and address the changing, and sometimes conflicting, needs and objectives of HTAO's stakeholders. It calls for an ability to anticipate issues, contextualize information, and guarantee local relevance (Reference Menon, Fung and Harstall65). Early warning and horizon scanning activities undertaken by HTAOs are indicators of their anticipatory capacity (Reference Carlsson and Jorgensen18;Reference Douw, Vondeling and Oortwijn29;Reference Mowatt, Bower and Brebner66). Responding to needs requires knowledge of the context, change agents, barriers to HTA utilization, and availability of resources to use HTA outputs and implement changes (Reference Drummond and Weatherly31;Reference Hivon, Lehoux and Denis46;54).
Innovation and Learning
This dimension points to the capacity to innovate and change the field of HTA. Innovation refers to the development of new processes or activities to adapt to the changing needs of HTAO's environment (Reference Menon and Marshall64). Learning refers to the way organizations build and organize knowledge and use the broad skills of their workforce to pursue organizational goals (28;Reference Nutley and Davies68). It also pertains to the ability to learn from experiences in a systematic manner. Such a “collective learning” involves at minimum enhancing the competencies of at least some individuals and the mechanisms to mobilize this knowledge within the HTAO (23;26;Reference Nutley and Davies68). Yet, competencies and processes per se do not make an organization adaptable and flexible. As stated by Davies et al., “the key feature of a learning organization relates . . . to the ways in which people within the organization think about the nature of, and the relationship between, the outside world, their organization, their colleagues and themselves” (26;Reference Senge79). In the Parsonian perspective, these “ways of thinking” are intrinsically linked to the culture and value maintenance function of an organization.
Culture and Values Maintenance Function
This function captures the shared principles and beliefs that produce meaning and cohesion among members of an organization. Some authors view organizational culture as an “attribute” or a defining quality of an organization. Others regard it more globally as defining the whole character and experience of organizational life (Reference Scott, Mannion, Davies and Marshall75;82). Given the lack of consensus on the definition, we will, per Scott et al., treat an “organization's culture” as an emergent property of its status as a social institution and assume that its main characteristics can be described (Reference Scott, Mannion, Davies and Marshall75). Culture is concerned with the common and accepted ways of doing things within an organization as well as with the shared ways of thinking about and making sense of an organization (Reference Scott, Mannion, Davies and Marshall76). These shared ways are embedded in fundamental values, which provide the basis for how members of the organization aspire to interact. The relevant dimensions of performance thus relate to whether the way an HTAO functions is congruent with its core values and aspired organizational climate (Reference Sicotte, Champagne and Contandriopoulos80).
Our literature review revealed three core values that can influence the performance of a HTAO.
Independence
This value refers to the scientific and financial self-determination of an HTAO, which include the ability to independently select assessment methods, draw conclusions, publish results, and control the budget (Reference Eskola, Höckerstedt and Mäkäräinen34). HTAOs must remain independent of particular interests groups, maintain an arm's length relationship with the funding stakeholders, and maintain an agenda that is driven by a requirement for excellence (Reference Drummond and Weatherly31). Such a position has the advantages to provide legitimacy, strengths to recommendations, and political independence (24).
Transparency
Identified as a key element of a fair and trustworthy process, “informational transparency” means that individuals involved in the assessment and appraisal process know the issues being considered and the content of the deliberations (i.e., internal transparency) and that the processes, deliberations, decisions, and reasoning of the decisions making body are made available to stakeholders (i.e., external transparency) (Reference Martin, Giacomini and Singer58). Practically, transparency is believed to improve the usefulness, consistency, and the generalizability of HTA reports (14;31;32;35;39;43;56;60). Furthermore, participatory transparency denotes the democratic ability to take part in the assessment (Reference Brown and Micheal16). Brown argues the demonstration of transparency by an HTAO is always dependent on the contexts in which it is evoked and practiced. It will vary according to the form of disclosure desired, the means by which disclosure is achieved and any number of other limiting conditions. Acknowledging this fact protects against the “naive” assumption that a state of absolute transparency is attainable (Reference Brown and Micheal16).
Accountability
Accountability refers to the responsibility of an HTAO to have mechanisms in place to monitor the appropriateness of the results generated for decision making (53;Reference Black10). This responsibility depends on the level of scientific and economic independence the organization has and, as such, must be socially and politically negotiated (19;Reference Eskola, Höckerstedt and Mäkäräinen34).
These values regulate the relationships between the producers and users of HTA. They contribute to the sociopolitical positioning of health technologies and the legitimization of HTAOs. Internally, these values confer to the members the ability to reflect critically on the way they tackle tasks and relate to one another. Hence, they affect organizational climate and collaboration. Relevant subdimensions to assess organizational climate include: leadership, communication, teamwork, commitment to innovation, and attitudes to change, motivation, and sense of belonging (23;Reference Scott, Mannion, Davies and Marshall76). These qualities are central to the culture of organizations and conducive to a better performance.
Performance Alignments: The Interchanges Between the Organizational Functions
According to Sicotte et al. “a performing organization is one that manages to maintain a dynamic equilibrium among its various functions” (Reference Sicotte, Champagne and Contandriopoulos80). The analysis of the reciprocal relationships between functions makes explicit alignment mechanisms that influence organizational performance and affect its equilibrium. We provide brief descriptions of the alignments and present how they can translate into performance questions (Supplementary Table 2; available at http://www.journals.cambridge.org/jid_thc).
Strategic Alignment
The interchange between the adaptation and the goal attainment functions allows the HTAO to adjust how it acquires resources, gains external support, responds to needs, and innovates according to its prioritized goals. It also questions the relevance of the goals and how they are prioritized, in light of the changing environment and the adaptation capability of the organization.
Allocation Alignment
This alignment refers to the tight balance between the characteristics of the HTAO production and its adaptation mechanisms. In one direction, the focus is on the appropriateness of resource allocation among the various means of production. In the other, the focus is on whether the characteristics of the production are optimal given the way the organization interacts with its context.
Contextual Alignment
This alignment deals with the fit between the culture and values of the HTAO and its adaptation to the environment. The idea is to understand the mobilizing effects of culture and values on the adaptation mechanisms of the organization and the effects of these mechanisms on the culture and organizational climate of the HTAO.
Tactical Alignment
This alignment pertains to the conformity of the production system with the goal priorities and the relevance of the goals given the characteristics of the production system.
Operational Alignment
Achieving operational equilibrium means there is a fit between the culture and values of the HTAO and the production function. The performance questions have to assess the mobilizing effects of culture; values and organizational climate on the productivity and the volume, coordination, and quality of the production; and the influence of the production system on organizational values and climate.
Legitimization Alignment
This alignment bridges goal attainment with the culture and values maintenance function. Evaluation questions should reveal how the values and the organizational climate affect its effectiveness and efficiency. They also shed light on the organization's shared principles, beliefs, and patterns of interactions in response to goal achievements. Combined with all or a selected number of dimensions for each function, these alignments can provide crucial information to assess and influence the performance of an HTAO.
Initial Validation of the Conceptual Framework
As a first validation test, we went back to the forty-seven reports consulted to identify what has actually been used to evaluate HTAOs. We counted an event when a function, a dimension, or subdimension was specifically evaluated or when it was referred to in relation to the performance of an HTAO (Figure 1). We were able to fit in the model all the dimensions of organizational performance reported in the empirical literature. Of the forty-seven reports, 74 percent (n = 35) report on the production, 57 percent (n = 27) on adaptation and goal attainment, and 38 percent (n = 18) on the culture and values maintenance functions. Five reports (all formal evaluations) cover the four functions, but none of them cover all dimensions.
When a function is identified in evaluations of HTAOs performance, the number of evaluated dimensions and subdimensions and the specificity with which the authors define what they are evaluating vary greatly. For example, of the seven reports that discuss organizational climate as an important dimension of performance, only one reported on six subdimensions, most referred specifically and only to teamwork. Often, dimensions are discussed with no reference to subdimensions, and vice versa. On the whole, dimensions are referred to in general terms, whereas subdimensions are precisely defined.
DISCUSSION
We developed a theoretically grounded model of HTAO performance. Interpreted from a performance perspective (Reference Sicotte, Champagne and Contandriopoulos80), Parson's theory of social action (Reference Parsons69;Reference Parsons70) provided the integrative framework to achieve conceptual clarity, comprehensiveness, and flexibility. The appeal of Parson's framework lies in its ability to subsume diverse conceptualizations of performance and, thus, make possible the simultaneous consideration of several dimensions and perspectives (Reference Sicotte, Champagne and Contandriopoulos80). In fact, the four functions encompass the four dominant models of organizational performance discussed by Hailey (Reference Hailey42). Furthermore, the alignments combine these functions in a coherent system and as such reveal additional aspects of performance dealing with their dynamic equilibrium. In practice, this translates in an ability to evaluate organizational models of HTA, understand how these have worked in their context, and how these can continue to work in the future (Reference Banta and Jonsson3). This flexibility of the model is particularly important in light of the trend in several nations to realign healthcare policy to pursue equity instead of efficiency goals (Reference Cookson and Maynard21), a trend that may radically alter the nature of HTA and challenge HTAOs performance.
The literature revealed a rich body of evidence on the qualities that characterize the relative worth of an HTAO. It also highlighted that the very concept of HTAO performance is elusive and fragmented. These findings corroborate those of Wanke et al., which showed great heterogeneity not only in the way performance is approached but also in the characteristics considered (Reference Wanke, Juzwishin and Thornley85). By reinterpreting the literature in light of Parson's functions, we were able to derive an articulated list of performance dimensions. The analysis revealed the dimensions most often assessed and other important ones that, hitherto, remain unexplored. The absence of evaluation on the research impact of HTAOs is noteworthy given the roles of research networks in the conduct and dissemination of HTA outputs and the pivotal role of skills and methods in the life of an HTAO. Furthermore, we were able to structure the narratives on HTAOs performance along the six alignments even though no functions or dimensions of performance were, a priori, explicit. This means that each alignment can serve as a dimension of performance that captures the mutual influences of the four organizational functions. This capacity of the model to encompass the scattered literature on HTAOs’ performance points to its construct validity. Indeed, our findings demonstrate the comprehensiveness and flexibility of the model by its capacity to accommodate various evaluation designs, contexts of evaluation, organization models, and perspectives.
Although our literature review was wide, we did not request unpublished evaluation reports from HTAOs. Had we done so, we might have found that some of the unexplored aspects of organizational performance have indeed been studied. We doubt it would have changed our model. The dearth of published HTAO’ evaluations may be due to the fact that many organizations are too early in their life cycle to have undergone an evaluation. Arguably, it is likely due to the politically sensitive nature of the exercise, practical time, resources constraints, and problems relating to evaluation designs (3;6;37;85). Our model provides a shared language that can help break some of these barriers down. In particular, the alignments can help make sense of the external forces that affect the equilibrium of the HTAO, namely the complexity of organizational milieu, the changing nature of research evidence on new health technologies, and the value conflicts between stakeholders (25;42;49;85).
In our model, “the culture and values maintenance function aids and constrains the three other functions to the extent that values and norms are reflected in the production processes, that organizational goals are determined and judged in accordance with these values and that adaptation mechanisms are consistent with the values system” (Reference Sicotte, Champagne and Contandriopoulos80). Also, the model recognizes that HTAO’ performance is contingent on the specific characteristics, functioning, and regulating mechanisms of the healthcare system in which it operates (Reference Jacob and McGregor49;Reference Jacob and Battista50;Reference Wanke, Juzwishin and Thornley85). This is consistent with the argument by Scott et al. that, in health care, culture and performance are likely created together in a reciprocal and mutually reinforcing manner that is thoroughly dependent on wider contexts and influences (Reference Scott, Mannion, Marshall and Davies77). To date, the social role of HTAOs has been to infuse a culture of evaluation in the healthcare decision-making process. What remains to be seen is whether the culture of HTAOs, that is, through their independence, transparency, and accountability values, will be a synergistic force for their performance.
The next steps in this research endeavour are to validate the model with stakeholders and develop indicators upon which performance judgements can be based. In light of the heterogeneity in the structure and functioning HTAOs, our ambition is to provide a tool applicable to a wide range of organizational models. Built on theoretical foundations, this tool will serve to evaluate HTAOs’ performance against their stated goals or explicit standards. Furthermore, we hope it will provide the basis for a more radical form of learning: one that contributes to the development of scientifically and socially grounded HTAOs.
CONTACT INFORMATION
Louise Lafortune, MSc., PhD Candidate (louise.lafortune.1@umontreal.ca), Coordinator, Canadian Research Chair in Health Technology Assessment, Lambert Farand, MD, PhD (lambert.farand@umontreal.ca), Associate Professor, Isabelle Mondou, MD (isabelle.mondou@umontreal.ca), Coordinator of IDEES (Institut D'Évaluation En Santé), Claude Sicotte, PhD (claude.sicotte@umontreal.ca), Professor, Renaldo Battista, MD, PhD (renaldo.battista@umontreal.ca), Professor, Director, Department of Health Administration, Faculty of Medicine, University of Montreal, C. P. 6128, succ. Centre-villle, Montréal, Québec H3C 3J7, Canada
The authors acknowledge Pascale Lehoux for her insights and comments. This project is funded through the Canada Research Chair in Health Technology Assessment (HTA) held by Renaldo Battista, at the University of Montreal.
