This study presents the results of a review of the Alberta Heritage Foundation for Medical Research (AHFMR) experience in delivering a skills development program over the past 7 years. The review is placed within a comparative context with other similar programs in other international settings. AHFMR considered it important to conduct the review with a view toward:
The results of the review are expected to inform subsequent work in which the opportunities for exchange or improvement through collaborative efforts with universities, other research organizations and regional health authorities are explored in greater depth.
BACKGROUND
The AHFMR is an independent research funding agency established in 1980 to support health research in Alberta. The HTA unit was transferred from the Alberta government to the AHFMR in 1995 under the terms of a 5-year health research collaboration agreement, which was renewed in 2000. The purpose of the HTA unit is to conduct health technology assessments to inform policy questions at Alberta Health and Wellness (government), regional health authorities, and other health-care providers in the province. The unit is composed of a director, assistant director, administrative assistant, three research associates, two part-time information specialists, and a training position for skills development. The AHFMR skills development program, spanning 6 months, was established in 1996 and has the following objectives for program participants:
- To acquire an introduction to HTA, the mechanisms behind it, agencies associated with it, and how it can be used to enable effective health-care policy and decision making.
- To acquire and polish skills involved in doing HTA activities, including systematically and scientifically evaluating clinical studies for proven effectiveness; increasing writing, critical evaluation, data synthesis, and presentation skills; networking with HTA and policy-makers; and becoming aware of other resources.
The director and assistant director provide guidance to the participants who work alongside the research associates. The intended program participants may be candidates from both the research and policy-making arenas—individuals working in regional health authorities, Ministry of Health, or as graduate students, or academic faculty. Priority is given to candidates who have a PhD or MD degree. As well, the program gives preference to candidates who are able to arrange a secondment from their current place of employment, as this fosters the transfer of knowledge and skills to their work setting, contributing to AHFMR's broader capacity building objective. Preference is given first to Albertans, then Canadians, and then international applicants. AHFMR provides a basic remuneration for participants.
During their stay at the HTA Unit, individuals are expected to complete at least one Health Technology Assessment, involving:
- Outlining the project and project goals in collaboration with HTA staff;
- Completing a literature search in cooperation with an information specialist;
- Scanning abstracts from the literature results;
- Choosing appropriate references to be retrieved; and
- Completing the project.
Four of the eight individuals in the Skills Development Program have come from the international community—Cameroon, Columbia, China, and Romania. The other four were from Alberta. Four of the eight participants were physicians or have subsequently become a physician. Three had their PhD, and one was a Master's prepared psychologist. Four had previous experience in or exposure to HTA or the Cochrane methodology for conducting systematic reviews.
METHODS
The review involved a qualitative study design that included the following:
- Survey of the Skills Development Program participants' experiences and perceptions;
- Semistructured interviews with program stakeholders; and
- Written survey of HTA agencies/programs in other Canadian and international jurisdictions.
Participants
Information from participants was obtained through two data sources. First, completed exit interview questionnaires were available for three of the seven individuals who had completed the program. These questionnaires addressed the participant's objectives, perceptions of most and least valuable experiences, environment, supervision and support, program length, recommendations for improvement and intended use of knowledge and expertise gained. Second, a questionnaire was designed to solicit information on the rationale for choosing the AHFMR program; the participant's objectives and expectations, and whether these had been met; satisfaction with the program; perceived value; perceptions of program strengths; and suggested areas of improvement.
Consent for participation in this review was obtained by telephone or e-mail inquiry. A copy of the interview questionnaire was forwarded to the participants, either attached to the electronic message or subsequent to the telephone inquiry. In-person interviews were held with four individuals who resided locally. Telephone interviews were conducted with two participants who lived out of province or country. One individual could not be contacted, and one declined to participate in the review without explanation.
An exit interview was available for the participant who declined the interview; thus, between the review and exit interviews, information was available from seven of the eight individuals who had participated in the program.
Stakeholders
Program stakeholders were defined as AHFMR management with an interest in the HTA Skills Development Program, leaders from academic programs and research organizations with an interest in HTA, and initiators or users of specific HTA projects undertaken by the Skills Development Program participants. Semistructured interviews were conducted with four representatives of AHFMR, including the Director and Assistant Director, the Vice President to whom the HTA Unit is accountable, and the Director of a related program. The questions addressed the history and rationale for the program; perceptions of program value; strengths, weaknesses, and opportunities for improvement; and potential enhancements to collaborative efforts with other programs and organizations. Questions were forwarded to each individual in advance of the interview.
Two semistructured interviews were held with representatives from two universities with programs that include HTA or related courses. Questions addressed their perceptions of program value, strengths, weaknesses, opportunities for improvement, and current and potential future linkages with the Skills Development Program.
Case studies of two participants' experiences were undertaken to determine and illustrate the tangible benefits and possible impacts of the participants' products. The two placements in 2000 and 2002 were selected as they represented work that was relatively recent yet completed with sufficient time for impact to be realized. The case studies involved telephone interviews with key individuals representing the initiators and recipients of the HTA products and included information obtained from the program participants. Stakeholder interview questions addressed the reason behind the project request, perceived value, and uses of the completed product. The case studies are summarized in the review report to provide the reader with a picture of the major project work undertaken by the program participants, from conception to completion to subsequent utilization and impact.
Other HTA Agencies/Programs
An electronic message introducing the study was sent to the 39 members of the International Network of Agencies for Health Technology Assessment (INAHTA). The members were asked for assistance in identifying HTA agencies that offer a professional development program that provides mentorship and/or practical application of HTA skills. The reviewer then contacted the responding INAHTA programs to provide them with the interview questions. In some instances, the initial communication resulted in additional follow-up with individuals recommended by the respondent. An e-mail and questionnaire was sent to each of these sites. The written questionnaire solicited descriptive information about the organization's practical or mentorship program and its participants, relationship with academic programs, perceived success factors, and issues.
Responses were received from eleven of the thirty-nine INAHTA member agencies contacted. Of these, six completed the questionnaire. The remaining five indicated they did not have a comparable program. In addition to and resulting from the INAHTA inquiries, responses were received from the International Master's Program in Health Technology Assessment and Management.
Content analysis of all qualitative data received from each stakeholder group included identification of themes within and across respondent groups. A full report of the methods, instruments, and results can be obtained at the AHFMR's Web site (www.ahfmr.ab.ca/hta/hta-publications/initiatives/HTA-FR12.pdf).
RESULTS
Comparison of HTA Programs Offering Professional Development
Eleven agencies providing or planning to provide some form of HTA practical training or mentorship were identified in Austria, Canada, Italy, Israel, Spain, and Sweden (Table 1). Seven of these do so as part of the International Master's Program in Health Technology Assessment and Management (Ulysses Project). Six, including two involved in the Ulysses Project, reported offering other professional development opportunities.
The Ulysses Project is a multidisciplinary program, organized by a consortium of five academic and seven HTA agency partners located in Canada, Spain, and Italy (1). The program targets two types of students—evaluators who will produce HTA reports and decision-makers who will use HTA reports. The 3-year Master's program involves four modules of 2 weeks each, held in four different cities. Each module is composed of two courses.
Some variation in requirements for the Master's program exists among the participating universities with respect to thesis, internship, and major project requirements. The internship is mandatory in some programs and optional in others. In the first cohort of twenty-three students, ten are reported to have completed an internship at one of the designated HTA agencies/sites. The internships varied in length at 4, 6, 8, 12, and 24 months.
The evaluation of the internship program, based on interviews conducted with the supervisors, found that all perceived the internship to be a valuable complement to the academic program. All training sites provided a computer and workspace; some provided a stipend.
Three programs, AHFMR, the Swedish Council on Technology Assessment in Health Care (SBU), and the Vienna Institute of Demography (VID) are not affiliated with academic programs. They provide a training opportunity for individuals wishing to gain practical skills in HTA. These programs range in length from 6 weeks to 4 months in Sweden, to 5–6 months in Vienna, and 6 months in AHFMR. The average number of individuals accommodated per year ranges from one at AHFMR to two–three in Sweden.
In addition to the Ulysses Project, three programs at the Canadian Coordinating Office for Health Technology Assessment (CCOHTA), Institute for Clinical Evaluation Sciences (ICES), and Israeli Center for Technology Assessment in Health Care (ICTAHC) describe offering practical components associated with Masters or Doctoral programs in HTA, pharmacy, or other graduate programs. These generally involve course projects or residency programs to teach and/or develop skills in HTA methods.
Value of AHFMR's HTA Skills Development Program
Collectively, the six participants interviewed reported completion of six full-length health technology assessments, six TechNotes (shorter analyses), and three briefs or information papers. Although a comprehensive impact analysis was not undertaken, the case studies identified the following results of the HTA projects undertaken by two participants:
- In both instances, the HTA reports were perceived to have influenced funding decisions at the provincial government or regional health authority level.
- The HTA initiators made use of the products in subsequent planning, evaluation, and/or teaching activities in their organizations.
- Products from both projects were broadly circulated or published in the peer reviewed literature. One became the basis for a subsequent published report, which compared the results of the HTA with national survey data on the same topic.
- Both projects resulted in HTA knowledge transfer activities in the participant's or initiator's work place.
Five of six participants interviewed perceived their skill set to be enhanced through participation in the program and perceived a greater evaluative consciousness as a result of participation in the program. All participants interviewed stated their time in the program was well spent. Although they had different expectations and objectives, these were generally reported to have been met. One international participant who had expected to implement an HTA unit in her home country had not met this expectation and was unlikely to do so in the future.
Those indicating their skill set was enhanced indicated it was improved in the area of research, specifically undertaking critical analysis and in the practical application of HTA. Whereas some had previous theoretical knowledge of HTA, they expressed appreciation for the direct experience in applying this knowledge through their project work. Four of five respondents stated they feel confident in applying their knowledge of HTA.
Of the six participants who have completed the program and for whom data are available, one returned to her employment position, three pursued further educational opportunities, and two returned to clinical practice. Three indicated they had applied the HTA skill set in their subsequent employment or professional activities by:
- Helping others understand HTA;
- Conducting seminars and publishing articles on HTA in their own discipline;
- Being asked to participate in conferences and/or meetings related to HTA;
- Being more knowledgeable and effective in a graduate program, including responding to requests for assistance from classmates; and
- Conducting HTAs.
Three of five participants interviewed who have completed the program indicated the HTA experience was not instrumental in helping them secure their present position. However, five of six interviewed thought the experience in the HTA Unit increased value to their employers or would increase their value to future employers.
Four of six participants indicated that they actively network with other individuals in HTA. The remaining two indicated they had some contact with HTA professionals, either through continued contact with the HTA Unit staff or through their own research community.
While several program participants from other countries expressed the objective of returning to their country of origin to establish or enhance HTA units, this value at the international level has not yet been realized. Two of the four international participants are still completing their learning experiences in Canada; thus, it is too early to determine the impact their experience may have in their own country. One international participant chose not to return to her country and is presently pursuing further postsecondary education in Canada.
A wide range of responses was received when participants were asked what was most valuable about the experience. Two themes emerged: practical experience in conducting and writing HTAs; and exposure to the world of HTA and how it differs from other systematic evaluations. Other aspects that were described as valuable by individual respondents included the opportunity to conduct extensive literature searches and critical appraisals, undertake HTA in a topic area of interest, and following the HTA process through from beginning to the final product, including publication.
When asked what was least valuable, all participants interviewed indicated there was nothing in the experience that was not valuable in some way.
AHFMR respondents offered the following perceptions of the value of the program:
- Contribution to the resources of the HTA Unit through the production of publishable reports;
- Contribution to advancing the knowledge of HTA at the local, national, and international stage;
- Enhanced success and visibility on the international stage—“by bringing in people from other countries, we are then raising AHFMR on the world stage, increasing quality and status”;
- Exposure of the Foundation to new ideas—learning about other jurisdictions and from the knowledge of best practices they bring to the Unit through their project work; and
- Increased ability for the Foundation to interact with government policy-makers, relating to one participant's secondment from the provincial government's health ministry to the HTA Unit. This participant acted as liaison between the Foundation and the provincial health ministry in subsequent endeavors.
Strengths and Opportunities for Improvement
When representatives from the universities and other jurisdictions with similar programs were asked what they believed constituted the successful elements of their own program or, more generally, of any skill development program, the following characteristics were suggested:
- Practical hands-on experience—the possibility of learning while conducting an actual assessment, or conducting, under mentorship, a systematic review on a limited question;
- Offering a combination of academic rigor and practical experience;
- Mentorship—through a close one-on-one relationship with experienced HTA staff;
- Support from their program or organization, so that individuals feel what they are learning is relevant and the skills they bring back will be valued;
- Raising awareness—for example, of how industry shapes clinical studies and how marginal the real effects of some medical interventions are;
- Flexibility to respond to specific needs;
- Having an international component, which appeals to professional students and facilitates excellence through access to international experts and enables students to gain insight from a broad mix of students from different countries;
- Expertise of the program directors and collaborators;
- Ability to attract the target audience;
- Trainee satisfaction with activities and abilities acquired;
- Demonstrated acquisition of the knowledge and skills; and
- Integration of students into the HTA workforce or contribution to further HTA development.
The strengths of AHFMR's HTA Skills Development Program as perceived by interviewees and considered against the suggested success factors were as follows:
- Affiliation with AHFMR, an internationally recognized organization with a strong reputation for excellence in research and demonstrated commitment to HTA. These aspects were factors in attracting some of the participants to the program.
- Program characteristics including its hands-on, practical approach, flexibility in structuring objectives to match the needs of individuals and ability to attract individuals with strong pre-existing skills.
- Level of experience the mentors/supervisors have in conducting HTAs, something respondents suggested academic organizations do not always have.
- Skill level of individuals attracted to the program.
- Opportunities for linking and networking with the academic and research community in the same city or province and to undertake collaborative arrangements with regional health authorities.
Two fundamental themes emerged in the nonparticipant interviews regarding opportunities for improvement—enhanced linkages and increased program formalization. AHFMR and academic respondents perceived an opportunity to “take the program to the next level” by integrating it with a university curriculum. Suggested strategies included formal agreements between AHFMR and one or more academic programs regarding an HTA residency placement program, alignment in the timing of HTA courses and Skills Development Program placements, joint faculty appointments between AHFMR and an academic program, and provision of joint seminars.
Although specific mechanisms were not identified, several respondents indicated a desire for the HTA Unit to explore enhanced linkages with regional health authorities and with an AHFMR program designed to enhance research capacity in regional health authorities. By developing HTA skills in individuals employed in health regions or service delivery organizations, the Skills Development Program would contribute to building HTA capacity in the policy and service delivery environments. To accomplish this, it is considered necessary that the HTA Unit ensures relevance by having program participants work on HTAs that represent regional health authority, rather than AHFMR or government policy priorities.
The second theme, consistent with taking the program to the next level, involved suggestions for enhancing the formalization of the program through more formal and competitive application and program entry processes, and more clearly defined end processes. At present, candidates who apply must provide three letters of reference, a description of their learning objectives and a description of how they will use their newly acquired knowledge. The program is mainly advertised through an invitation on the AHFMR Web site. Approximately two to three requests are received every 4 months. Most are turned down due to lack of capacity or insufficient qualifications. Decisions for acceptance or rejections of applications are made by the HTA Unit Director.
Length of time in the program has varied. Extensions to the 6-month placement may occur at the request of the participant or the Unit, usually based on the need for continued work on the participant's HTA project. It was suggested that, if such extensions are desired, the participant continue the work under a different role description, such as in the position of research assistant.
Several interviewees suggested a need to revisit the PhD or MD entry requirement. Others, however, argued for retaining these requirements. The rationale provided for targeting individuals with their PhD or MD related to efficiency and potential to influence broadly. Because of their advanced research skills, individuals with their PhD will have less of a learning curve and stand to benefit more than someone with less research experience. As well, doctoral-level participants are likely to have a broad influence through academic teaching positions. Targeting physicians is considered important as a means of influencing clinician behavior.
The arguments presented for reconsidering the qualifications were as follows:
- If one desires broad dissemination of HTA knowledge and skills in the nonacademic setting, for example, in regional health authorities, it is not feasible or necessarily desirable to require a PhD. It is students going through the course-based Master's program at the universities who are the most likely candidates for the Skills Development Program and are the most likely to “go to an organization and apply their skills/knowledge.”
- The skills learned for PhD are very different than what is needed for HTA, in that the PhD student learns to generate data rather than assess data. As well, individuals with their PhDs were noted to often come through the biosciences and have limited experience outside academia.
- Physicians are not necessarily better qualified to learn HTA skills than are other health service providers, thus having an MD should not necessarily be a priority. An individual's specific background and experience was suggested to be more relevant.
Perceptions on the optimal length of program were divided. Half of the participants interviewed stated they thought a 6-month placement was sufficient. The other half indicated the program was not long enough. Optimal program length was reportedly linked to the individual's objectives and Unit's project expectations. For example, several participants suggested 6 months was sufficient for the production of several TechNotes (shorter analyses) or one complete assessment and one TechNote. However, a 9- to 12-month experience was considered necessary if a participant intends to make HTA a career and wants a full range of experiences, including seeing a project through from beginning to end, interacting sufficiently with policy-makers and researchers, and attending conferences and meetings.
Skills Development Program Model and Role Description
In addition to the explicit opportunities for improvement suggested by respondents (as described above), the reviewer identified an underlying theme in the various interviews with participants and stakeholders—a potential for differentiating the role description for program participants based on fundamentally different needs and objectives. Several interviewees discussed their perception that the needs with respect to HTA skills development may be placed on a continuum from those requiring limited exposure to those desiring the highest standard of skills with respect to production of HTA reports.
The program has attracted and will likely continue to attract two types of participants—those choosing HTA as a career who will become HTA “producers” and those with an interest in the application of HTA concepts and skills in their subsequent employment and who will hopefully “carry the torch” of evidence-based decision making broadly. The former may ultimately work in an academic or HTA agency environment. The latter are more likely to work in government or a health-care delivery organization.
This review finding is consistent with the International Master's Program in HTA and Management (Ulysses Project), which differentiates students into two fundamental streams—the HTA producer stream and the HTA user (or policy) stream. Internship placements vary in length from 4 weeks to 24 months, partially determined by the chosen stream.
In the AHFMR review, the Ulysses Project model was adapted, resulting in a recommendation to differentiate the HTA Skills Development Program offerings for prospective candidates. Without affecting the fundamental mandate of the program, it was suggested that the objectives of the HTA Skills Development Program could be reviewed and revised to more clearly differentiate between the needs of (a) individuals intending to become HTA producers, and (b) individuals who intend to apply HTA in their employment capacity as policy or clinical decision-makers. Two separate program streams could be established (Table 2).
Such conceptualization of the program results in alternate considerations for the future role description, including the target program participants, objectives, length of program, entry qualifications, product expectations, type of relationship/linkage arrangements with collaborating agencies, credentialing, and marketing strategies.
Terminology associated with the different streams may be established to assist in differentiating the types of opportunities offered—for example, “internship,” “residency,” and/or “fellowship.” The length of program and entrance criteria would be adjusted according to the expectations associated with the streams.
The reviewer expects that, if the program is differentiated by need, the proportion of local versus international participants will vary based on these needs. For example, it is likely that the program will continue to draw international interest from those desiring to make HTA their career and those who are looking to establish an HTA agency in their jurisdiction. On the other hand, individuals who are interested in a general knowledge of HTA for application in subsequent employment (clinical, regional health authorities, government) will likely be Alberta or Canadian residents already employed in Canada and/or taking Master's level courses through an academic program.
Candidate qualifications and criteria could vary according to program stream. For individuals pursuing an HTA career, it will be important that they have a solid grounding in research methods and, ideally, have taken courses in HTA and/or critical analysis. The PhD qualification would be applicable, although the program could consider equivalencies. For individuals pursuing a clinical or policy career, the entrance qualifications may be linked to specify university course prerequisites, or equivalencies based on background or experience. In consideration of the feedback received during this review, it is suggested that physicians not be given priority over other health disciplines.
The reviewer considered the current activities expected of participants in the Skills Development Program to be appropriate, based on the feedback received from past participants and the program's mandate. Some modification to the job description to accommodate the expectations associated with each of the suggested streams may be required. For those with a career interest in HTA, it was suggested the experience include completion of one full HTA, from project identification through report production, and including the full spectrum of experiences associated with this task, for example, participation in project steering committee meetings and conducting the external review process.
For individuals in the second stream, the experience may more appropriately focus on the production of TechNotes or in the future, other products such as field evaluations or appraisals. This stream option would provide a greater emphasis on carrying HTA work through the decision-making process. As well, more opportunities for participants to interact with regional health authority and government policy-makers could be incorporated into the experience.
Regardless of stream chosen by the prospective Skills Development Program candidate, the reviewer recommended the HTA Unit be flexible to accommodate each individual's unique objectives as much as is practicable.
DISCUSSION
This review involved an assessment of AHFMR's HTA Skills Development Program from the perspectives of program participants, AHFMR representatives and other individuals with exposure to the program. The number of informants was limited as the program is relatively small, with eight participants since its inception in 1996. However, it was considered important to attempt to gain an understanding of the strengths, weaknesses, and impacts of the program, as AHFMR considers future enhancements. Despite small numbers, several themes emerged which offered considerable information upon which to base recommendations for these enhancements.
The results were based on the opinions and perspectives of key stakeholders who were readily available. These stakeholders were asked to state their thoughts on the value and benefit of the program from their own perspectives and the possible perspective of others. Undertaking a comprehensive analysis of the participants' products and their impact was beyond the scope of the review. As not all program participants were interviewed, a complete picture of the participants' perceptions was not achieved. Because of the small number of participants surveyed, it is possible that the missing responses would have resulted in somewhat different findings and conclusions.
The provision of a practical, skills-based HTA experience under a mentorship model, while not unique to Alberta, appears to be in the developmental stages for most jurisdictions who responded to the questionnaire. With one exception, AHFMR's Skills Development Program was the only jurisdiction in which the skills development opportunity was described as a “program.” Requests for formal descriptions and evaluative information from INAHTA members with practical skills development opportunities yielded limited written documentation, suggesting that HTA agencies are still in the early stages in the process of formalizing these programs.
