On behalf of society, healthcare budget holders have to make choices regarding the use and diffusion of healthcare interventions and reimbursement for their costs. While the number of available health technologies is increasing, as budgets are limited, policy makers in health care need to know whether the societal benefits of a particular technology or treatment are worth the investment that must be made to offer it (Reference Park, McDaid and Weiser1). Consequently, health technology assessment (HTA) is increasingly important, as it informs decision making on how to obtain the best value for money (Reference Nicod and Kanavos2). HTA is a policy-oriented form of research designed to inform decision makers on the relative effectiveness or cost-effectiveness of a new technology in comparison with current or best practice (Reference Drummond, Schwartz and Jönsson3). HTA is a relatively young science and the uptake of healthcare policy makers is too recent to be fully understood. Due to the huge amount of studies and developments in healthcare-related products, the number of HTA studies has increased over the past decades. Although stakeholders’ uptake of these studies is increasing (Reference van Velden, Severens and Novak4;Reference Oliver, Innvar and Lorenc5), the extent of the use of HTA information in policy making varies and a better matching of HTA and user needs is needed to strengthen the use of HTA (Reference Oliver, Innvar and Lorenc5;Reference Macintyre, Chalmers and Horton6).
In recent years, different studies, including systematic reviews, have been conducted to identify multiple barriers to and facilitators for the uptake of HTA information (Reference van Velden, Severens and Novak4;Reference Garrido7–Reference Prosser, Koplan and Neumann11). For instance, collaboration and relationships with policy makers were identified as facilitators (Reference Oliver, Innvar and Lorenc5), and lengthy reports and lack of credibility of the HTA research findings (industry-sponsored studies) were identified as barriers (Reference van Velden, Severens and Novak4). However, these studies did not evaluate the relative importance of each of these factors. To deepen our understanding of these factors, it is important to investigate which barriers and facilitators are most influential in relation to the uptake of HTA studies.
Moreover, it is important to investigate differences in perception between policy makers and HTA researchers about these barriers and facilitators to bridge the gap between HTA and informed decision making. To the best of our knowledge, no studies have made a quantitative assessment of the relative importance of barriers and facilitators regarding the use of HTA and compared the views of policy makers and HTA experts. Accordingly, the aim of this study was to identify among policy makers and HTA experts, using best-worst scaling (BWS), the most important barriers and facilitators and their relative weight in the uptake of HTA studies for policy makers from different levels in the Netherlands. A secondary aim was to explore whether there are any differences between policy makers and HTA experts.
METHODS
Best-Worst Scaling
As we aimed to identify the relative importance of a large list of barriers and facilitators, we used the BWS method. The BWS method is an increasingly popular conjoint analysis technique (Reference Mühlbacher, Kaczynski and Zweifel12;Reference Cheung, Wijnen and Hollin13), devised by Finn and Louviere (Reference Finn and Louviere14). In comparison with the more widely used and traditional discrete-choice experiments (DCEs) [Reference Train15], BWS surveys differ mainly in that they elicit additional information on the least preferred option and are capable of incorporating a larger set of items or factors for eliciting preferences (Reference Flynn, Louviere and Peters16). In addition, the BWS tasks may require less cognitive burden on behalf of the participants and may, therefore, be easier to complete than the DCE (Reference Marley and Louviere17).
In this study, we used the BWS object case (also named BWS Case 1) to quantify the relative importance of barriers and facilitators regarding the use of HTA in policy decisions (identified from a literature search and validated by experts). Two BWS surveys were used (Reference Flynn, Louviere and Peters16): one addressed the facilitators and one addressed the barriers. In the BWS, participants were asked to complete a series of choice tasks in which they had to choose the most and least important factor (i.e., barrier or facilitator) from a list of five factors, derived from a master list (explained below). There are multiple tasks containing a set of factors in which the combination and ordering of factors differ. A BWS was suitable for the purpose of this study as it simplifies ranking tasks for participants, is robust for scale-related biases, and effectively discriminates between the ratings of different factors involved in complex decisions (Reference Mühlbacher, Kaczynski and Zweifel12;Reference Flynn, Louviere and Peters16).
Participants
A convenience sampling strategy was used to recruit policy makers (i.e., decision makers and advisors on various levels) and HTA experts (i.e., PhD students and senior researchers in HTA) in the Netherlands. Policy makers with various roles were included, for example, representatives of the National Healthcare Institute, and the Dutch Ministry of Health, Welfare, and Sport. Using a convenience sampling strategy, researchers created a list of potential stakeholders based on personal networks. For HTA experts, a mailing list from the last congress of the Dutch Association for Health Economics was used, enriched by a search for the relevant Dutch HTA research departments of several universities. Potential participants (n = 222) were then approached by means of e-mail with a request to participate in our survey. The initial e-mail included a link to the BWS survey which then randomly assigned the participant to one of the four versions of the questionnaire for each survey. If there was no response, a reminder was sent within 2 weeks. All participants were recruited between November and December 2015.
Identification of Barriers and Facilitators
A literature review using PubMed and Web of Science was conducted to identify barriers to and facilitators for the uptake of HTA. In these databases, we relied on the following keywords: “economic evaluation AND facilitators AND review”, “economic evaluation AND barriers AND uptake AND review”, “economic evaluation AND facilitators AND uptake”, “economic evaluation AND barriers AND uptake”, “HTA AND facilitators AND review”, “HTA AND barriers AND review”, “HTA AND facilitators”, and “HTA AND barriers AND uptake” for articles published until January 7, 2015. This search led to a total of 113 studies.
Titles and abstracts were scanned for their relevance, which led to sixteen articles. The full texts of these studies were scanned and seven were excluded, as they were not reporting on the facilitators or barriers to HTA uptake. In additional, based on personal knowledge of studies and by checking bibliographies, we manually identified seven more relevant articles. Consequently, the list of barriers and facilitators was drafted from literature and internal discussion (Reference van Velden, Severens and Novak4;Reference Oliver, Innvar and Lorenc5;Reference Garrido7–Reference Prosser, Koplan and Neumann11;Reference Merlo, Page and Ratcliffe18–Reference Hoffmann, Stoykova and Nixon26). As factors are categorized inconsistently across reviews, it was necessary to define and validate the list of factors for inclusion in this study. The list of factors and facilitators was extensively discussed among the working group, and external experts (two HTA experts and one policy maker) were asked to provide feedback on the list regarding (i) completeness, (ii) overlap, and (iii) wording. This led to a master list of twenty-two barriers and a master list of nineteen facilitators for the BWS survey (see Supplementary Table 1).
The BWS Survey
The Sawtooth Software's SSI Web platform was used to design the two surveys. Fractional, efficient designs were used, characterized by: orthogonality (factors were shown and paired an approximately equal number of times), minimal overlap (minimizing the number of times each factor appears within the same set across the design), positional balance (factors appear approximately an equal number of times in each position), connectivity (factors are directly or indirectly linked), and stability (for each survey, four different versions of the questionnaire were used to increase variation). Four versions were designed for each BWS, and the numbers of choice sets were, respectively, fourteen for the barriers and twelve for the facilitators surveys.
The questionnaire began with some demographic questions (i.e., age, gender, and about their role (policy maker, HTA expert)). Then, the online survey randomly assigned participants to one version of the barriers questionnaire and one version of the facilitators questionnaire. The order of the surveys (barriers/facilitators) was also random. For the barriers, participants were asked to choose the most important and least important barrier to the uptake of an HTA, answering between fourteen choice sets, each composed of a set of five factors from the master list of twenty-two barriers. For the facilitators, participants were asked to choose the most important and least important facilitator for the uptake of an HTA, using twelve choice sets, each composed of a set of five factors from the master list of nineteen facilitators.
In addition, in each survey, participants were asked to rate the difficulty of the choice task using a Likert-scale (0 = very easy to 7 = very difficult). Qualtrics was used to administer the online survey. Supplementary Tables 2 and 3 show representative examples of a BWS question, including the introductory text. The survey was piloted with three HTA experts from Maastricht University. Their feedback was used to revise the instructions and better clarify potentially confusing aspects of the survey without changing the scientific methodology and content.
Analyses
Completed surveys were included in the analyses. Descriptive statistics were calculated to present the demographic characteristics of the respondents. Analyses of BWS questionnaires have been studied extensively and have well-validated psychometric properties (Reference Marley and Louviere17;Reference Johnson27). Hierarchical Bayes analysis (Reference Mühlbacher, Kaczynski and Zweifel12;Reference Johnson27;Reference Orme28) was conducted to analyze preference data, using Sawtooth Software's SSI Web platform. Hierarchical Bayes estimation has frequently been used to analyze DCE and BWS studies (Reference Cheung, Wijnen and Hollin13). It fits a priori distributions of the parameters to the sample data. To derive a posteriori distributions, individual-specific data are used (Reference Mühlbacher, Kaczynski and Zweifel12).
Using the logit rule, crude importance scores per factor were determined for each participant. For each factor, the mean relative importance score (RIS) was calculated with its 95 percent confidence interval. Based on the raw coefficient of the preference function, rescaled scores were estimated, which represent the relative importance of the factors. The RIS of all factors combined for each individual sum up to 100, in which a higher score indicates higher importance of the factor (Reference Orme28). Based on the mean RIS, factors were ranked from most to least important for the uptake of HTA in policy making.
To check the quality of the responses, the individual's fit statistic was used. A fit statistic lower than 0.25 indicates purely random responses to the choice tasks, and that these responses should be excluded from analyses (29). One-way analysis of variance (ANOVA) analyses were used for subgroup analyses on gender, age, and role, exploring whether the perceptions of the importance of these factors differ according to these characteristics. For the subgroup-analyses, these variables were dichotomized, with gender (male, female), role (policy maker, HTA expert), and the mean being the cut-off score for age (younger, older).
RESULTS
Descriptive Statistics
A total of 84 stakeholders started the survey, resulting in forty-nine (58.3 percent) completed surveys; thus, there were forty-nine completed responses regarding the demographic characteristics. Twenty-five incomplete responses (approximately 71 percent of total incomplete responses) contained information on the demographic characteristics (i.e., gender, age, and role). No significant differences were found between participants who completed the survey and participants who dropped out. The mean age was 43.2 years (SD 12.0) and 53.1 percent were male. There were sixteen policy makers (32.7 percent) and thirty-three HTA experts (67.3 percent), with six PhD students and twenty-seven senior researchers in HTA. Stakeholders rated both BWS experiments as moderately difficult on a 7-point Likert scale (barriers, Mean = 4.74, SD = 1.41; facilitators Mean = 4.51, SD = 1.50).
Relative Importance of the Barriers
Table 1 depicts the RIS of the barriers, described in Supplementary Table 1. The overall fit statistic (0.46) was considered good, and all respondents had a fit statistic higher than 0.25 and were, therefore, all included in the analysis (29).
Table 1. Relative Importance Scores of Barriers to the Uptake of HTA, Overall and by Role
Note. Data given as relative importance scores with confidence intervals in brackets.
The five most important barriers (with an RIS score higher than 6.00) were “No explicit framework for decision-making process” (that uses HTA evidence), “Insufficient support by stakeholders” (policy-makers’ perception of insufficient support by end-users), “Lack of support” (within the organization for the use of HTA), “Limited generalizability” (of HTA studies to the policy-maker's context), and “Absence of appropriate incentives” (for the implementation of cost-effective interventions).
Subgroup analyses showed six significant differences between policy makers and HTA experts (see Figure 1). “Absence of policy networks” (or observatory agencies that promote the joint efforts of researchers and policy makers), (Mean = 4.36, SD = 4.52) versus (Mean = 1.67, SD = 2.70), F(1,47) = 6.80, p < .05; and “Lack of contact and interaction” (among policy makers, HTA researchers, and other stakeholders), (Mean = 7.35, SD = 6.06) versus (Mean = 3.95, SD = 4.98), F(1,47) = 4.36, p < .05 were more important for policy makers than for HTA experts.
Figure 1. Relative importance scores of the barriers: Policy makers (n = 16) versus HTA experts (n = 33), ordered according to ranking of the overall analysis. The x-axis indicates the relative importance score per barrier. Asterisks indicate significant difference between the roles.
In addition, “Absence of appropriate incentives” (for the implementation of cost-effective interventions), (Mean = 4.87, SD = 2.03) versus (Mean = 6.89, SD = 2.87), F(1,47) = 6.38, p < .05; “Lack of support, within the organization” (for the use of HTA), (Mean = 4.42, SD = 3.27) versus (Mean = 7.78, SD = 3.70), F(1,47) = 9.53, p < .05; “No explicit framework for decision-making process” (that uses HTA evidence), (Mean = 6.27, SD = 4.36) versus (Mean = 9.65, SD = 5.02), F(1,47) = 5.30, p < .05; and “Lack of financial resources” (to conduct relevant HTA research), (Mean = 2.27, SD = 2.19) versus (Mean = 4.42, SD = 3.44), F(1,47) = 5.22, p < .05 were more important for HTA experts than for policy makers.
Furthermore, there were no differences between male and female participants, except for “Insufficient quality” (of HTA research findings: quality not in line with scientific requirements). “Insufficient quality” was clearly perceived as more important to male participants (Mean = 7.77, SD = 4.63) than female participants (Mean = 2.32, SD = 2.35), F(1,47) = 5.24, p < .05.
In addition, there were some differences between the younger and older age groups: “Lack of timeliness” (gap between HTA research and policy making, regarding the timeliness of the research), (Mean = 3.44, SD = 3.01) versus (Mean = 5.71, SD = 4.40), F(1,47) = 4.44, p < .05; “Uncertainty” (surrounding the HTA results), (Mean = 2.13, SD = 3.04) versus (Mean = 5.42, SD = 4.16), F(1,47) = 9.90, p < .05; and “Insufficient support by stakeholders” (policy-makers’ perception of insufficient support by end-users), (Mean = 5.05, SD = 4.20) versus (Mean = 8.66, SD = 4.38), F(1,47) = 8.63, p < .05 were deemed more important by the older age group than by the younger age group.
Relative Importance of the Facilitators
Table 2 depicts the RIS of the facilitators, described in Supplementary Table 1. All respondents were included in the analysis as the overall fit statistic (0.46) was considered good, and all respondents had a fit statistic higher than 0.247.
Table 2. Relative Importance Scores of Facilitators to the Uptake of HTA, Overall and by Role
Note. Data given as relative importance scores with confidence intervals in brackets.
Six facilitators had an RIS higher than 6.00, including “Availability of explicit framework for the decision-making process” (using HTA evidence), “Sufficient support by stakeholders” (policy-makers’ perception of support by end-users), “Appropriate incentives” (for the implementation of cost-effective interventions), “Sufficient quality” (of HTA research findings: quality in line with scientific requirements), “Sufficient awareness” (within the organization regarding the relevance of HTA), “Sufficient support within the organization” (for the use of HTA).
Subgroup analyses showed six differences between policy makers and HTA experts (see Figure 2): “Creation of policy networks” (or observatory agencies that promote the joint efforts of researchers and policy makers), (Mean = 7.64, SD = 5.68) versus (Mean = 3.48, SD = 5.19), F(1,47) = 6.50, p < .05; “More personal contact and interaction” (among HTA researchers, policy makers, and other stakeholders), (Mean = 7.87, SD = 6.74) versus (Mean = 4.14, SD = 5.55), F(1,47) = 4.35, p < .05 were more important for policy makers than for HTA experts. “Sufficient credibility” (of the HTA research findings (fewer industry-sponsored studies), (Mean = 2.21, SD = 1.60) versus (Mean = 4.38, SD = 3.33), F(1,47) = 6.05, p < .05; “Sufficient financial resources” (for conducting relevant HTA research), (Mean = 2.14, SD = 2.34) versus (Mean = 5.07, SD = 4.19), F(1,47) = 6.72, p < .05; “Sufficient quality” (of HTA research findings: quality in line with scientific requirements), (Mean = 5.18, SD = 4.56) versus (Mean = 7.83, SD = 4.15), F(1,47) = 4.10, p < .05; and “Availability of explicit framework for the decision-making process” (using HTA evidence), (Mean = 4.70, SD = 4.04) versus (Mean = 9.54, SD = 5.57), F(1,47) = 9.62, p < .05 were more important for HTA experts than for policy makers. Furthermore, there was a difference between male and female participants: “Availability of explicit framework for the decision-making process” (that uses HTA evidence), (Mean = 9.53, SD = 4.73) versus (Mean = 6.19, SD = 6.01), F(1,47) = 4.72, p < .05 was more important for men than for women.
Figure 2. Relative importance scores of the facilitators: Policy makers (n = 16) versus HTA experts (n = 33), ordered according to ranking of the overall analysis. The x-axis indicates the relative importance score per facilitator. Asterisks indicate significant difference between the roles.
Moreover, there were two differences between the younger and older age groups: “Appropriate timing” (between HTA research and policy making), (Mean = 2.27, SD = 3.42) versus (Mean = 4.76, SD = 4.15), F(1,47) = 5.23, p < .05; and “Sufficient support by stakeholders” (policy-makers’ perception of support by end-users), (Mean = 5.55, SD = 4.85) versus (Mean = 9.94, SD = 5.76), F(1,47) = 8.29, p < .05 were more important for the older age group than for the younger age group.
DISCUSSION
This study provides information on the relative importance of barriers and facilitators regarding the use of HTA, and the differences between policy makers and HTA experts in the Netherlands. Not surprisingly, many of the most important barriers and facilitators were opposites, meaning that the most important facilitators were the factors that reduced the most important barriers to the use of HTA by policy makers. For instance, no explicit framework for the decision-making process (that uses HTA evidence) was an important barrier, while the availability of an explicit framework for the decision-making process (that uses HTA evidence) was perceived as the most important facilitator.
This was also true for support by stakeholders, support within the organization for the use of HTA, and the availability of appropriate incentives (for the implementation of cost-effective interventions). Interestingly, these most important factors seem also somewhat tangible (e.g., strategies to enhance awareness and support within the organization), making it possible to reduce the research/policy gap. In a systematic review by Oliver et al. (Reference Oliver, Innvar and Lorenc5), which identified barriers and facilitators regarding the use of evidence in public policy, all factors were categorized into six themes: “Organizations and resources,” “Contact and collaboration,” “Research and researcher characteristics,” “Policy-maker characteristics,” “Policy characteristics,” and “Other.”
Categorizing the most important barriers and facilitators for these themes showed that “Organizations and resources” and “Policy characteristics” factors were the most important. Generally in line with Oliver et al. (Reference Oliver, Innvar and Lorenc5), attention should thus be drawn to enhance support and create an explicit framework (policy guideline, or policy statement) for the decision-making process, in which HTA is more transparently used in setting priorities (Reference Oliver, Innvar and Lorenc5). However, surprisingly, policy characteristics were reported only moderately often in the literature, while this study found that the creation of an explicit framework for the decision-making process is both the most important barrier and most important facilitator.
In addition, in line with Oliver et al., the generalizability and quality of research findings are important factors (Reference Oliver, Innvar and Lorenc5). Interestingly, the literature suggests that the factors of “Contact and collaboration” (e.g., appropriate timing between HTA research and policy making, improving longstanding relations between professionals/policy makers and HTA researchers, and more collaboration between the groups) (Reference Oliver, Innvar and Lorenc5;Reference Garrido7) may be important, as these factors were among the most frequently reported barriers and facilitators regarding the use of evidence in literature (Reference Oliver, Innvar and Lorenc5). Our findings, however, suggest that this frequency of reporting may not reflect the relative importance of the factors.
Despite being among the important factors (as it is frequently reported), policy makers and HTA experts deem other factors more important. Accordingly, findings suggest that it is highly important to increase awareness of the need for creating an explicit framework for the decision-making process, stressing the need to include HTA evidence. Policy statements on the importance of including HTA in decision making may facilitate this process. This is also in line with a study in Denmark that identified the need for a political or managerial decision to use (mini-) HTA (Reference Ehlers and Jensen30). In addition, when processes are explicit, deliberative and formalized, HTA tends to have a higher profile in the decision-making process (Reference Oortwijn, Broos and Vondeling31).
On the organizational level as well, It is important to create awareness and motivate managers to support the inclusion of HTA evidence in setting priorities and to place incentives in line with this. Here as well, the Danish study reflects the importance of a managerial decision to systematically use HTA in policy making. One interpretation of the study's finding is that HTA researchers should engage more directly with decision makers and create awareness and motivation to systematically incorporate HTA evidence in decision making (Reference Ehlers and Jensen30). This may also explain in part why findings suggest that contact and collaboration were relatively less important to participants in our study, as these factors may be the mediators of political and managerial decisions (and thus opinions which are reflected only in the outcome factors).
Subgroup analyses showed some differences between gender, age groups, and roles. Differences between policy makers and HTA experts in perceptions are relevant as they may indicate how to address the research/policy gap. Although HTA experts mostly recognized the relative importance of barriers and facilitators regarding the use of HTA by policy makers, they underestimated the importance of the absence of policy networks or observatory agencies that might promote the joint efforts of researchers and policy makers, and the lack of contact and interaction among policy makers, HTA researchers, and other stakeholders.
On the other hand, HTA experts seem to overestimate the relative importance of the absence of appropriate incentives for the implementation of cost-effective interventions, the lack of support within the organization for using HTA, the lack of an explicit framework for a decision-making process that uses HTA evidence, and lack of financial resources for conducting relevant HTA research. Moreover, HTA experts perceived the quality and credibility of the research to be more important than did the policy makers. This means that HTA experts may need to enhance efforts to involve policy makers in the early stages of research, reinforcing contact and collaboration. Policy makers consider this to be one of the most important factors, while HTA experts may have failed to recognize this issue.
Another explanation related to the noted inconsistency that, in the literature, contact and collaboration were not found to be one of the most important factors overall (Reference Oliver, Innvar and Lorenc5), may be that HTA experts perceive contact and collaboration to be relatively unimportant, or that expert opinion is focused more on the outcomes (i.e., the need for an explicit framework in decision making or putting a policy statement together) rather than on the mediator (contact and collaboration).
To better understand the most important factors related to the uptake of HTA by policy makers, a framework can be used to show the underlying mechanism for using HTA. The I-Change Model may help in understanding the position of the most important factors regarding the uptake of HTA in policy making (Reference Cheung, Evers and Hiligsmann32). The I-Change Model integrates concepts of various health behaviors, communication and promotion models, to explain behavior (Reference de Vries, Mudde and Leijs33), and has been used widely in identifying the determinants of adoption behavior (Reference Cheung, Evers and Hiligsmann32;Reference de Vries, Eggers and Bolman34;Reference De Vries, Eggers and Jinabhai35). According to the I-Change Model, behavior is the result of intentions, which are influenced by motivational and awareness factors. Motivational factors are attitude (perceived advantages and disadvantages of the behavior), social influence (perceived social influence on the behavior), and self-efficacy (the perceived capability to conduct a certain behavior).
First, it is important to increase awareness within the organization regarding the relevance of HTA. Then motivational factors need to be addressed; it is important to conduct high quality, relevant research that will enhance positive attitudes toward the use HTA. In addition, it is important to align HTA usage with appropriate incentives. To improve perceptions of social support, efforts need to be made to establish policy networks that promote the joint efforts of researchers and policy makers, to create personal contact and interaction between policy makers and researchers, and to involve policy makers in the research process. To improve the policy maker's self-efficacy, it would be beneficial to create an explicit guideline regarding the use of HTA in decision making.
Our study could have several limitations. One limitation is the potential lack of generalizability of the results, as the study was conducted with Dutch policy makers and HTA experts only. It is feasible that factors perceived as important for the use of HTA in the Netherlands may be different in other countries, especially in emerging countries, due to differences in the decision-making context (Reference Hoffmann and von der Schulenburg36). Middle-income countries (e.g., India and Russia) are less advanced in the use of HTA than more industrialized countries (e.g., United Kingdom), although the community that uses HTAis growing in middle-income countries (Reference Oortwijn, Broos and Vondeling31). Our findings may, therefore, not be transferable to all other countries, particularly to middle-income countries. Future research should investigate the relative importance of barriers and facilitators regarding the use of HTA in different countries, and compare results. In addition, despite recruiting decision makers on all levels within the health system, it is unknown whether certain levels were overrepresented in the responses. This may be a potential limitation regarding the generalizability of our findings.
Another limitation is that participants were not familiar with the BWS tasks (as indicated by the difficulty scores), which may have led some to drop out. As 58.3 percent of the participants who started the survey actually completed it, our sample may be biased. However, analyses showed no differences in demographic characteristics between participants who did not complete the surveys and those who did. Moreover, in searching for barriers and facilitators to draft the initial list of relevant factors, a scoping review rather than a full systematic review was conducted, as systematic reviews on the barriers and facilitators for HTA and evidence use have been conducted recently (e.g., Oliver et al.) (Reference Oliver, Innvar and Lorenc5). Therefore, this study potentially excluded some relevant articles regarding barriers and facilitators. However, it is not expected that many additional barriers and facilitators would have been identified, as data saturation was observed during the draft of the initial list of factors. In addition, the initial list was validated by experts.
In conclusion, this is, to our current knowledge, the first study that has assessed and quantified the relative importance of barriers and facilitators regarding the use of HTA and compared the views of policy makers and HTA experts in the Netherlands. This resulted in a list of factors that are most important in bridging the gap between policy and research. Barriers and facilitators related to organization, resources, and policy characteristics were particularly important. It is of utmost importance to create a pulse on the national level of the Netherlands to make a political statement and create an explicit framework for the decision-making context to include HTA evidence. Due to some differences in the perceptions between policy makers and HTA experts, it is necessary to create a framework for closer collaboration and informed decision making.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: https://doi.org/10.1017/S0266462317000290
Supplementary Table 2: https://doi.org/10.1017/S0266462317000290
Supplementary Table 3: https://doi.org/10.1017/S0266462317000290
CONFLICTS OF INTEREST
The authors have nothing to disclose.
