Kazakhstan emerged as an independent country in Central Asia in 1991 following the collapse of the Soviet Union. In health care, legacies of the past include a strong emphasis on hospital care, inefficient resource allocation, over-medicalization, and the use of procedures and medications that are not evidence-based (Reference Rechel, Roberts and Richardson1). The Government of Kazakhstan demonstrated leadership in moving the country forward by building its economy and making substantial efforts to modernize. A major priority is to accelerate advancement of the healthcare sector. Fortunately, the reforms are supported by a steady economic growth over the past decade (Reference Katsaga, Kulzhanov, Karanikolos and Rechel2).
In 2008, the Government of Kazakhstan entered into an agreement with the World Bank to launch a Health Technology Transfer and Institutional Reform Project to extend to 2013. The objective was to accelerate reforms by introducing international standards and building long-term institutional capacity in the MOH and affiliated health institutions. The project consists of several components, including reform of the pharmaceutical, blood bank, laboratory, and medical education sectors. It also included the introduction of evidence-based medicine (EBM), clinical practice guidelines, hospital accreditation, and HTA. The aim of the HTA component was to strengthen the capacity of the Kazakhstan health sector for better healthcare and policy decisions. These include determining the medical services, interventions and drugs to be included in the State Benefits Package
The broad initiative required the creation of new organizational structures and access to capacity building delivered by international experts. For health technology assessment (HTA), in 2009, the MOH established the Scientific and Practical Centre for Standardization and Health Technology Assessment (CS) in the Kazakhstan capital, Astana.
Through a competitive World Bank process, the MOH contracted with the Canadian Society for International Heath (CSIH) to provide expert technical assistance and advice. The contract stipulated that a twinning arrangement form the basis for education and mentoring to encourage sustained institutionalization of the new skills and knowledge. Beginning in September 2010, a seven-person CSIH team assisted the MOH partners in developing an HTA strategy and building the capacity of the CS HTA Unit.
In this case study, we describe our approaches and experiences in mentoring the emerging Kazakhstan HTA initiative, the outcomes of the process, and some of the challenges we encountered in the hope that this may assist other emerging HTA programs.
METHODS
Mentoring Approach
The seven-person CSIH team had broad experience and training in HTA, EBM, medical practice, pharmacy, information science, health economics, and health policy. The consulting activities were supported by the CSIH office in Ottawa, Canada, and coordinated by an eight-person local office in Astana. Extensive in-country consultations (fifteen over 2.5 years) ensured relevance and continuous connection with the Kazakhstan twinning partners.
Approaches used in mentoring included (a) developing a series of guidance documents for the MOH and encouraging their dissemination throughout Kazakhstan; (b) teaching courses and workshops on various practical aspects of HTA to a variety of participants including staff in the HTA Unit, the MOH, representatives of national healthcare and research institutions, and interested clinicians; and (c) one-on-one mentoring of HTA Unit staff in the development of a series of assessment reports.
Guidance Documents
Twelve HTA guidance documents were prepared by members of the CSIH team to address specific topics identified by the MOH. These included methodologies and approaches to guide the fledgling Kazakhstan HTA unit and the MOH in sustaining future HTA development. The documents included HTA guidelines, lists of suggested HTA topics, a capacity building plan, and suggestions for financial planning and sustainability. The guidance documents drew on CSIH team experience in HTA programs and source material from INAHTA, ISPOR and EUnetHTA. Close interactions with the staff at the MOH and the HTA Unit were essential to incorporate their feedback and to take account of the local context.
Workshops
The target audience included the HTA staff at CS, MOH policy makers, and several stakeholders from the research institutes and universities. Education sessions ranged from half-day seminars to week-long workshops with attendance by two to three dozen learners. Topics included an introduction to medical literature searching/critical appraisal, HTA methods and tools, health economics, special considerations in HTA, and dissemination. Instructors used interactive methods during the sessions including hands-on group exercises, case studies and practical examples. Access to online resources was generally available to allow instruction about seeking sources of evidence. The participants were asked to fill out evaluations at the end of several of the courses. The HTA instruction meshed well with courses in EBM and clinical practice guideline development that were undertaken to support and sustain other aspects of the CSIH project as many of the same Kazakhstani participants were involved.
The HTAs
The most extensive and challenging element of the project was the production of five HTA reports by staff members at the HTA unit with one staff member taking the lead on each topic. This hands-on exercise served to provide a sustained learning experience for the unit staff, and to address practical MOH requirements for advice on current issues on use of health technologies in Kazakhstan. Topics were identified and selected by MOH staff, all relating to relevant and timely issues of importance to Kazakhstan health care and policy.
The HTA Unit leads developed study protocols and timetables for the HTAs. A CSIH consultant was assigned to each HTA unit lead as a mentor to provide guidance and feedback through each step of the HTA process. Mentoring was carried out by face-to-face meetings during visits to Kazakhstan and extensively by e-mail or Skype, often with weekly conference calls to monitor progress and respond to queries for assistance. Research questions (using the standard PICOS format) and search strategies were developed by the CS leads in partnership with CSIH mentors. Clinical input from Kazakhstani experts was sought to as great an extent as possible.
RESULTS
Guidance documents
Methodologies and strategies
The aim of these documents was to build a foundation to advance HTA in Kazakhstan. An HTA guideline described the main steps in developing an HTA report, including choosing HTA topics; clarifying fundamental tasks for HTAs (i.e., developing the research questions); finding and assessing the evidence; preparing a report; and disseminating findings and conclusions.
With respect to choosing topics for the HTA unit to begin their HTA journey, CSIH recommended that the ideal topic choices would: (a) be feasible in a short period of time with limited literature (e.g., fewer than approximately six studies); (b) be of interest to the writer; and (c) be relevant and timely for the Kazakhstani health system.
Management of the HTA process was emphasized, for example, develop a work schedule and milestones and integrate quality assurance in the processes. The checklist for HTA reports issued by the International Network of Agencies for Health Technology Assessment (INAHTA) was suggested as a set of widely accepted criteria to check the quality of HTA products (Reference Hailey3). A “Road Map” was developed to identify required actions and proposed milestones to move forward with HTA in Kazakhstan, proposing a phased approach to accomplishing the activities.
One document discussed building a strong organizational foundation for the HTA Unit including developing an HTA strategic plan as an essential early activity. Sound and resilient internal processes and standards are required, initially focusing on a narrow set of activities and types of HTA products, building a team of specialists with the right mix of skills, and investing in information retrieval including high-speed Internet with access to relevant databases and publications. Skills in the English language are key to allowing Kazakhstan staff members to understand the literature, much of which is in English.
Another document explored possible ways of funding HTA in Kazakhstan and described strategic planning for financial sustainability and risk management in HTA. A review of HTA agencies internationally revealed that most are primarily or totally publicly funded (Reference Garrido, Kristensen, Nielsen and Busse4). The HTA Unit is expected to inform the decision-making processes of the MOH. A framework for tracking and recording information related to HTA impact in a systematic way was proposed, drawing on the experience of HTA programs in other countries (5).
Capacity building and training materials
These documents addressed management issues of interest to the MOH such as developing capacity in HTA in Kazakhstan, determining the optimal organizational structure of the HTA unit(s), assessing HTA impact on health care, and developing an HTA database. Training materials were prepared to prepare “trainers” to lead HTA educational activities for broader audiences in Kazakhstan.
The capacity building exercise started by means of a competency assessment to identify gaps in the level of skills and education among CS HTA staff. Recommendations on skill sets needed and capacity enhancement were then developed. CSIH suggested expanding the HTA unit's staff and strengthening capacity beyond CS by creating centers of HTA-related expertise in organizations and facilities throughout the country.
Workshops
CSIH consultants conducted several seminars and workshops covering a wide range of HTA topics. The most intense was a 7.5-day “Master Class” in HTA including 2.5 days of economic instruction (Table 1). The Master Class was planned as a “train-the-trainers” course as some participants (HTA Unit staff) were to provide subsequent training, using a “cascade” model, for audiences that might include decision makers in local health authorities and physicians. The participants were engaged in learning: they participated attentively, posed thoughtful questions and interacted enthusiastically in class work. In particular, they responded well to hands-on practical exercises and discussions of real life HTA examples. Feedback from the economic workshop participants showed that a large majority found it to be helpful and that the presentations made were clear to them. More examples and practical work were suggested as possible improvements.
Table 1. Content of HTA Workshops
HTA Reports
Undertaking the five required HTA reports over the past 6 months of the project was an interactive process between the twinning partners.
The CSIH team proposed that the HTA reports be “rapid reviews” to accommodate the skills and time available at the HTA Unit. Such exercises are most suitable for learning purposes and provide rapid results for the learner and those seeking the evidence. Each assessment included sections on safety, clinical effectiveness, and cost and/or budget impact analysis. Detailed cost-effectiveness analyses were not performed although all reports included an abbreviated review of published cost-effectiveness studies. Some HTAs also included a section on health system impact.
The finalization of HTA topics (Table 2) and accompanying research questions involved numerous consultations and iterations with HTA unit leads and MOH decision makers. Simplified templates and tools were developed and used for preparing the HTA reports. These included a clearly specified thirteen-step process for preparing the reports. The review of published cost-effectiveness studies included a standard five-step process, study selection and evidence table templates, and a set of questions for determining study transferability and interpreting their results. The search for evidence allowed HTA unit leads to practice newly acquired skills in literature searching but there was a major reliance on CSIH to develop the search strategies, run the searches and acquire the literature.
Table 2. HTA Topics for the Initial KZ HTA Reports
The HTA unit leads worked with a timetable for submission of reports. Drafts were reviewed by CSIH mentors who provided comments in writing or by means of conference calls. Clinical experts in Kazakhstan were involved throughout the process to a varying extent. Before submission to the MOH, three reports underwent external review by local specialists not previously involved with the HTA reports.
The HTA topics reflected important current issues in Kazakhstan health care related to decisions on possible extension of existing services and appropriate clinical practice. All reports were submitted to the MOH and have been considered by decision makers.
DISCUSSION
This experience in developing a fledgling HTA presence in Kazakhstan provided important lessons. As international momentum in the adoption and advancement of HTA gains momentum, it is worth reflecting on what worked and what could have been improved, bearing in mind that international experiences in one location must be context sensitive.
Successes
Improving understanding and credibility
The mentoring process using “twinning” strategies enhanced the general understanding and raised awareness of HTA among Kazakhstan target audiences. The approach was successful in communicating and embedding the purpose, concepts, methods, and technical realities of preparing HTA reports and also provided HTA an appreciation of the global community of HTA and the importance of linking into international best practices. Producing HTAs reports provided confidence to HTA unit leads as they worked to master the required skill set and established credibility because the projects produced advice for MOH decision makers.
Methodological foundation
Building a solid foundation is important in scientific endeavors. Based on the adaptation of international best practices, useful source documents for initial and future HTA activities in Kazakhstan were developed. The materials have been successfully used by the CS HTA unit to provide HTA information to other centers in Kazakhstan. Training took place at medical universities in six regions of the country and involved over 150 people (Reference Kosherbayeva, Tabarov and Gizatullina6).
Educational activities
Many aspects were deemed to be successful, for example, (a) engaging with participants as aspiring peers in HTA, (b) exposing participants to international HTA experiences and methods, (c) using practical examples and experience from other countries, (d) using case studies and group exercises and interactions in the offerings, (e) accommodating audience needs and preferences. Enthusiastic interactive dialogue encouraging discussion revealed a high level of curiosity and interest. Productive interactions involved working with researchers one-on-one or in small groups, in-person or remotely. Further training is needed to ensure sustainability.
International networking
Participation at the HTAi meetings in Bilbao in 2012 and Seoul in 2013 by CS staff members was the beginning of international involvement. The project allowed the Scientific and Practical Centre for Standardization and Health Technology Assessment to qualify for INAHTA membership, a path to networking, mentorship, and other opportunities. Kazakhstan participants have also been actively involved at ISPOR meetings.
Challenges
Language challenges
The working language in Kazakhstan is Russian (with Kazakh increasing in prominence) and the twinning partners had varying expertise with the English language, both spoken and written. The working language for the CSIH consultants was English with the as-needed support of Russian-English speaking translators used by CSIH in Astana. In addition, the bulk of HTA and medical literature internationally is in English. Subtleties in language interpretation, combined with “niche” terminology and concepts in HTA and health care proved to be an ongoing challenge. Fortunately, the CSIH staff members in Astana were competent in both languages and provided priceless insights, as did a Russian-speaking member of the CSIH consulting team.
MOH and Stakeholder Involvement
Communication with MOH officials improved over time. Increasing numbers of MOH officials participating in HTA training increased commitment to the concept of HTA and therefore increased support of the utility of HTA at a high level. However, in the context of the broader healthcare system, deeper appreciation for HTA and its applications is yet to evolve.
Identifying Learning Needs
Ensuring a clear understanding of gaps in the knowledge and skills of the target audience was essential; however, participants presented a wide range of knowledge and skills. An important example was a low level of exposure to clinical epidemiology and EBM. This required an investment of time and effort to instill a paradigm of questioning and to review skills such as the pursuit of and use of evidence, methods of critical appraisal, and interpretation of economic results. Many participants also lacked experience in analytic writing, in particular use of clear crisp language as required in HTA. There was also a need for more development of business skills in areas such as project management, logistics and critical analysis.
As with all training where acquisition of a fundamental knowledge base requires a steep learning curve, it was challenging to strike a balance between introducing basic concepts and honoring the knowledge and expertise of the learners. It proved to be important to take the time to clarify actual needs and focus participants’ attention.
Turnover and Availability of Staff
Due to the dynamic nature of health system transformation in Kazakhstan, turnover of staff members challenged continuity in ongoing relationships. This was true at the level of HTA Unit researchers but also true in senior reporting structures within CS and the MOH. During the project, staff within the HTA Unit were under some pressure as they had other, non-HTA, tasks for the MOH. This limited the time that was available for them to work on their HTA reports.
Health System Data
The contextualization of HTA to local settings is important, particularly for MOH decision makers. A challenge in Kazakhstan was access to relevant and timely data. The quality of local data on epidemiology of diseases, resource usage, and costs of health care was often non-existent or insufficient for HTA purposes. Without this information, it was difficult to gauge the transferability of studies to the Kazakhstan context and to perform analysis such as budget impact analysis. This underlined the importance of an MOH commitment to resourcing the development and linking of necessary databases.
Learning Characteristics
The culture of EBM and evidence-based health policy is in its early days in Kazakhstan; the Soviet legacy of eminence-based practice and decision making in the health system conflicts with the emerging principles and practices of EBM. As knowledge and experience are gained, the opportunity for HTA to become more relevant and credible will improve.
It became apparent that the education objectives of the program and the learning preferences of participants required alignment. For example, CS HTA staff members were keen to conduct sophisticated economic modeling without a sound grasp of essentials. A more careful and detailed educational needs gap analysis at the project's outset may have prevented this mismatch.
Identifying the best teaching styles and communication methods was another challenge. The experts often shared information using a didactic approach, allowing for questions or clarification. However, participants were most engaged when teaching was case-based with practical examples driving delivery of content.
Lessons for Future HTA Mentoring
The HTA project met its prime objectives in that the CSIH team was able to begin to advance the understanding and application of HTA among Kazakhstani participants and establish a credible foundation for HTA activities. It was important for the HTA experts to “walk side by side” with the HTA unit researchers throughout the HTA process, addressing key practical aspects as a team. It is hoped that documenting this case study will help other potential mentors of young HTA programs (Table 3).
Table 3. Recommendations for mentors of HTA programs
POLICY IMPLICATIONS
As a result of the CSIH mentoring process in this 2.5-year MOH/World Bank funded project, there is an increased opportunity to inform local healthcare policy through in-country evidence-based advice. MOH officials appear to demonstrate a commitment and promising capabilities of HTA in health policy making. It is our hope that this case study may serve as a basis for discussions about further development of HTA in Kazakhstan and other developing countries.
Success in implementing an HTA program will depend on essential factors such as political commitment, financial support, and leading-by-example. With the emerging presence of a national HTA institutional commitment, Kazakhstan is well positioned to become a leader in HTA in Central Asia.
CONTACT INFORMATION
Sergei Muratov, MD, MPH (sergei.muratov@rogers.com), Canadian Society for International Health, One Nicholas, Suite 1105, Ottawa, ON, Canada K1N 7B7
David Hailey, MSc, PhD, Grad RIC, University of Wollongong 22 Sinclair Street, ACT 2902, Australia
Vicki Foerster, MD, MSc, Canadian Society for International Health, One Nicholas, Suite 1105, Ottawa, ON, Canada K1N 7B7
Bruce Brady, MA, MSc, Ministry of Health 7-1, 1515 Blanshard Street, Victoria, BC, V8W 3C8, Canada
Don Juzwishin, PhD FCCHL, Director, Health Technology Assessment & Innovation, Alberta Health Services 12 Floor - 129, North Tower, Seventh Street Plaza, 10030 - 107 Street Edmonton, AB, Canada T5J 3E4
Philip la Fleur, BScPharm, MSc, RPh, Evidence Based Medicine Research and Consulting Corp, 43 Rice Dr, Whitby, ON, Canada L1N7Z3
Jessie McGowan, MLIS, PhD, AHIP, Canadian Society for International Health, One Nicholas, Suite 1105, Ottawa, ON, Canada K1N 7B7
CONFLICTS OF INTEREST
S. Muratov reports Consultancy to author from Canadian Society for International Health, Work on project described in the article, and Travel/accommodations expenses covered or reimbursed to author from Canadian Society for International Health, Work on project described in the article. D. Hailey reports Consultancy to author from Canadian Society for International Health, Work on project described in the article, Travel/accommodations expenses covered or reimbursed to author from Canadian Society for International Health, Work on project described in the article.V. Foerster reports Consultancy to author from Canadian Society for International Health, Work on project described in the article, Travel/accommodations expenses covered or reimbursed to author from Canadian Society for International Health, Work on project described in the article. B. Brady reports Consultancy to author from Canadian Society for International Health, Work on project described in the article, Travel/accommodations expenses covered or reimbursed to author from Canadian Society for International Health, Work on project described in the article. D. Juzwishin reports Consultancy to author from Canadian Society for International Health, Work described in the manuscript, Travel/accommodations expenses covered or reimbursed to author from Canadian Society for International Health, Work described in the manuscript. P. la Fleur reports Consultancy to author from Canadian Society for International Health, Work on project described in the article.
J. McGowan reports Consultancy to author from Canadian Society for International Health, Work on project described in the article, Travel/accommodations expenses covered or reimbursed to author from Canadian Society for International Health, Work on project described in the article.
