Introduction
In the pursuit of universal health coverage (UHC), economic evaluations are increasingly being recognized as one of the important means to inform decisions for resource allocation and priority setting (1). In the recent past, the world has witnessed a rapid upsurge in the publication of economic evaluations; however, there have been concerns regarding their methodological quality and reporting standards (Reference Pitt, Goodman and Hanson2–Reference Griffiths, Legood and Pitt4). Numerous systematic reviews have evaluated the quality of economic evaluation undertaken across countries and regions (Reference Hutter, Rodríguez-Ibeas and Antonanzas5). The unavailability of methodological guidance, limited researcher capacity, and data availability are some of the often-cited plausible reasons limiting the conduct of quality economic evaluations (Reference Santatiwongchai, Chantarastapornchit, Wilkinson, Thiboonboon, Rattanavipapong and Walker6–Reference Adeagbo, Rattanavipapong, Guinness and Teerawattananon10). Additionally, studies with low quality are less likely to support the effective decision-making process (Reference Drummond and Sculpher11).
Addressing these gaps, several countries as well as organizations have formulated specific guidelines to standardize the basic principles of economic evaluations (12–Reference Wilkinson, Sculpher, Claxton, Revill, Briggs and Cairns14). While a large number of high-income countries (HICs) and an increasing number of middle-income countries (MICs) have developed country-specific guidelines and reference cases, such a guidance is largely lacking in low-income countries (LICs) (12). The International Decision Support Initiative (iDSI) developed a reference case, which was designed for use by low- and middle-income countries (LMICs); however, a recent systematic review reported limited citation of the iDSI reference case within published economic evaluations, possibly due to the iDSI reference case dissemination focussing on policy makers to a greater extent than the research community (Reference Emerson, Panzer, Cohen, Chalkidou, Teerawattananon and Sculpher15). Another study reviewing the impact of Panel Reference Case concluded inconsistent compliance to basic recommendations (Reference Phillips and Chen16). It is expected that researchers are more likely to adopt and apply country-specific guidance with local relevance, which will contribute to improvement in the quality of economic evaluations being undertaken.
Although a number of methodological guidelines exist (12), their impact on practice has not been widely evaluated. While a study to review the adherence to the iDSI reference case has been recently published (Reference Emerson, Panzer, Cohen, Chalkidou, Teerawattananon and Sculpher15); however, few similar studies for country-specific guidelines have been undertaken. Reviewing the adherence to existing guidelines can highlight the disparities between recommendations and actual practice and provide useful implications for countries targeting formulation of guidelines in the near future. In addition, it can provide insights on the need of the analysts and aid revision of already formulated guidelines.
The current systematic review aims to quantify the applicability of the guidelines for economic evaluations measured in terms of adherence to best practices as outlined in the country-specific guidelines. Furthermore, to have broader insights, we assessed adherence for three countries across different income groups, one each from high-income (Canada), upper-middle income (South Africa), and lower-middle income (Egypt) countries. For the purpose of selection, a list of countries having health technology assessment (HTA) or pharmacoeconomic guidelines was prepared, stratified by income groups, using information from Guide to Economic Analysis and Research (GEAR) and International Society for Pharmacoeconomic and Outcomes Research (ISPOR) repositories (12;17). Subsequently, the list was refined to include only those countries for which the guidelines were available in English language. Finally, one country from each income group was selected randomly using a lottery system. As of 2020, no LIC has developed a guideline, therefore, none of them could be included (12).
The three selected countries have widely varying healthcare systems and are at different stages of using HTA evidence. The healthcare system in Canada is publicly funded with independent health insurance schemes at the provincial level (Reference Marchildon18). The origin of guidelines for economic evaluation dates to 1994 when the first version of the guidelines was published (19). The guidelines development process initiated when Ontario (a province of Canada) released draft guidelines for conduct of economic evaluations in 1991, and concerns regarding the development of multiple provincial guidelines were raised. Subsequently, a national level steering committee was commissioned to develop a common set of guidelines which were then endorsed by the Canadian Coordinating Office for HTA (CCOHTA) and thirteen provincial departments of health (Reference Torrance, Blaker, Detsky, Kennedy, Schubert and Menon20). The purpose of the guidelines was to provide guidance to both doers and users of economic evaluation on key methodological aspects and promote consistency and comparability across evaluations aiding rational decision making. Currently, the Canada Agency for Drugs and Technologies in Health (CADTH) has published the fourth version of the guidelines in March 2017 (21). The CADTH is an independent, not-for-profit organization funded by the federal, provincial, and territorial governments in Canada. CADTH undertakes common drug reviews (CDRs) and pan-Canadian Oncology drug reviews (pCODRs) to provide recommendations on new and existing technologies to healthcare decision makers. The guidelines here are applicable for technologies beyond pharmaceuticals, including medical devices, procedures, diagnostics and curative, preventive, and promotive healthcare services. The guidelines are not mandatory in nature; however, they are widely used by HTA researchers including health economists.
The healthcare system in South Africa is mixed in nature with both public and private provisioning of healthcare services. Inequity in insurance coverage, reliance on out-of-pocket (OOP) expenditure, and persisting disparities in health system design inherited from the Apartheid era have impeded progress toward UHC (Reference Mills, Ataguba, Akazili, Borghi, Garshong and Makawia22;Reference Ataguba23). To address this problem, the country is proposing major health system reforms under a National Health Insurance system that will institute a specific role for HTA (24)a. The first set of pharmacoeconomic guidelines was developed in December 2012 to inform the pricing of pharmaceuticals in the private sector (25). The application of these guidelines is voluntary, though regulations allow for a mandatory request of a pharmacoeconomic evaluation for a particular medicine to be made.
Egypt is the only LMIC besides Indonesia and Philippines to have developed guidelines for economic evaluation (12). Egypt witnessed a continuous increase in OOP expenditure coupled with limited public healthcare spending, demanding the need for the optimization of available healthcare resources. In this context, the Pharmacoeconomic unit was institutionalized under the Ministry of Health and Population in 2011 with an objective to support and inform pricing and reimbursement decisions (Reference Elsisi, Kaló, Eldessouki, Elmahdawy, Saad and Ragab26). Shortly thereafter, in 2013, the pharmacoeconomic guidelines were developed to standardize the process of pharmacoeconomic evaluation through a transparent and uniform approach, ultimately improving the quality of submissions over time (27). The guidelines aim to assist in price negotiations and drug reimbursement decision making; however, the application of these guidelines is not mandatory.
Methodology
Literature Search
A systematic search was carried out in PubMed, Embase, and York Centre for Reviews and Dissemination (CRD) databases to identify health economic evaluations conducted in the context of the three selected countries. These searches were conducted from 1 year after the publication of the most recent edition of the country-specific guidelines, that is, 1 March 2017, 1 December 2012, and 1 August 2013 for Canada, South Africa, and Egypt, respectively, to 31 December 2018. In addition, bibliographic search was done from primary papers to identify additional studies that could be of relevance. The systematic review followed the CRD guidelines and Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines (28) (PRISMA checklist, Appendix 1, Supplementary File 1). The study protocol is published in the International Journal of Health Systems and Implementation Research (Reference Sharma, Aggarwal, Chauhan and Prinja29).
The search strategy combined terms for two keywords, first related to economic evaluations and its types and the second related to the selected countries. The search terms included “cost-effectiveness analysis,” “cost-benefit analysis,” “cost-utility analysis,” “cost-minimization analysis, “pharmacoeconomic evaluation,” “health technology assessment,” “economic evaluation,” “incremental cost-effectiveness ratio,” and “Canada,” “Egypt,” and “South Africa”. The detailed search strategy used is given in Appendix 2, Supplementary File 1. Furthermore, since the guidelines for the three countries were published at different times—March 2017, December 2012, and August 2013 for Canada, South Africa, and Egypt, respectively; therefore, the time period for inclusion of studies for the three countries varied. This difference in time period obligated the need to run the searches for the three countries separately.
Eligibility Criteria
A study was included in the review if it met all the four pre-established inclusion criteria. Firstly, it should be a full economic evaluation—a study that compares both the costs as well as the consequences of at least two alternatives (Reference Drummond, Sculpher, Claxton, Stoddart and Torrance30;Reference Briggs and O'Brien31). Secondly, the study setting should be in at least one of the three specified countries. Thirdly, the study should have been published after 1 March 2018 for Canada, 1 August 2014 for Egypt, and 1 December 2013 for South Africa (1 year after the publication dates of the most recent editions of the respective country-specific guidelines). It is highly likely that the studies published within 1 year of the guidelines’ release might have not considered the latest version of the guidelines, given the duration of publication process which usually extends up to 1 year. Finally, economic evaluations should relate to a health sector intervention, technology, or program. Only peer-reviewed publications were searched and grey literature was not included. Furthermore, there was no restriction on the language of publication.
Study Screening
First, duplicates were removed using EndNote. A two-stage screening process was employed, title and abstract screening was done independently by two reviewers (DS, ASC). The full text of papers, shortlisted after first screening, were reviewed to assess whether the studies met the inclusion criteria. Any disagreements between reviewers were resolved after discussions with the third reviewer (AKA). Studies meeting all the four inclusion criteria were included in the review.
Quality Appraisal
Drummond's checklist for economic evaluations was used to assess the quality of the included studies (Reference Drummond, Sculpher, Claxton, Stoddart and Torrance30). Drummond's checklist has ten key criteria and twenty-eight sub-criteria for critical assessment of economic evaluations. Unlike other checklists such as the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement and British Medical Journal (BMJ) checklist, the focus of Drummond's checklist is beyond reporting quality. The checklists aim to answer two important questions: Is the methodology employed in the economic evaluation appropriate and are the results valid? (Reference Gonzalez-Perez32). Furthermore, it is by far the most commonly used checklist while undertaking systematic reviews for health economic evaluations (Reference Watts and Li33). Quality appraisal was undertaken independently by two reviewers (DS, ASC) with a third reviewer (AKA) being the arbitrator. Inter-reviewer agreement on quality appraisal of studies was high with the kappa statistic being .82.
Data Extraction
Data extraction was undertaken by one reviewer (DS). A specifically designed template that was pilot-tested using ten studies was used for extracting data. The data extraction template was divided into two sections: general characteristics and methodological details (Supplementary File 2). Under general characteristics, we collected details on study title, year of publication, funding source (international, national, no funding, not reported), author affiliation (national or foreign), author background (clinical, public health, health economics—authors having a masters or doctorate degree in health economics or those affiliated with an institute/organization at the post of a health economist), disease category (according to the International classification of disease, ICD-10), and intervention type (diagnostic, pharmaceutical, vaccine, device, programme, surgical). Under methodological specifications, details were collected related to the principles of economic evaluation. The details included the type of economic evaluation (cost-effectiveness analysis—consequences measured in natural units, cost-utility analysis—consequences measured using quality adjusted life-years or disability adjusted life-years cost-benefit analysis—consequences measures in monetary terms, cost-minimization analysis—consequences of the two alternatives being compared are considered to be equal), details of target population (demographic and disease-specific details), comparator (do nothing, standard of care: treatment option enlisted in the standard treatment guidelines, most commonly used therapy: by far the most commonly used therapy in clinical practice, multiple comparators), study perspective (payer, societal, both), time horizon (less than 1 year, 1–5 years, more than 10 years, lifetime), types of costs included (direct health system costs, out-of-pocket costs, indirect costs), measure of health outcome (quality adjusted life-years, disability adjusted life-years, life-years, clinical end points), discount rate (1.5%, 3%, 3.5%, 5%, not applicable, not reported), and methodology for sensitivity analysis (deterministic sensitivity analysis, probabilistic sensitivity analysis, both, none).
Data Analysis
Descriptive statistics were calculated to report absolute frequencies (N) and relative frequencies (%) for each of the general and methodological characteristics. The methodological specification and reporting standards listed in the country-specific guidelines were converted into discrete binary variables (Appendixes 3 and 4, Supplementary File 1). No weighting was applied to individual variables and all were scored equally as either 0 or 1, where 0 indicated non-adherence to the guidelines and 1 represented adherence to the guidelines. This scoring translated to a total of 30 points which could be adhered to for Canada, 45 for South Africa, and 48 for Egypt (Supplementary File 2). In order to have a comparative analysis of adherence across the countries, the absolute scores were converted into percentages.
The quality appraisal score was also calculated by converting the absolute score into percentage. Drummond's checklist used for quality appraisal does not provide an inbuilt scoring system. The checklist recommends opting the responses for each of the questions as either yes, no, not clear, or not applicable. However, to quantitatively measure the quality and generate a summary score, all questions were valued equally and the number of positive responses was calculated out of total applicable responses.
Correlation between adherence scores and quality appraisal scores was evaluated using Pearson correlation analysis. Stratified analysis was undertaken to assess the adherence to country-specific guidelines by categories. Various independent categorical variables such as type of disease, intervention, economic evaluation, lead author background, affiliation, and source of funding that could possibly impact the adherence to the country-specific guidelines were identified. The difference between the mean adherence score for these variables was tested using one-way analysis of variance (ANOVA). The level of significance was set at .05.
Results
The systematic literature search yielded a total of 3,571, 2,847, and 1,154 non-duplicate articles for Canada, South Africa, and Egypt, respectively. The majority of the articles (3,144, 2,519, and 1,055 each for Canada, South Africa, and Egypt, respectively) were discarded at the stage of title and abstract screening as these did not meet the inclusion criteria and were either letters, editorials, reviews, or non-economic studies. The full text was reviewed for the remaining articles, out of which 383, 249, and 83 from Canada, South Africa, and Egypt, respectively, were excluded mainly because these were either partial economic evaluations, economic evaluations undertaken in other countries, or systematic review of economic evaluations. For systematic reviews, manual screening of included studies was undertaken to identify any additional economic evaluations. Finally, 44, 79, and 16 studies for Canada, South Africa, and Egypt were included (Figure 1; see Supplementary File 3, References of included studies).
Figure 1. PRISMA flowchart of included studies.
General Characteristics of Included Studies
The majority of the studies in Canada and Egypt were cost-utility analysis (73 and 63%), while cost-effectiveness analysis (60%) was predominantly undertaken in South Africa. The involvement of foreign authors was considerably higher in South Africa (87%) who represented the lead author in 65 percent of the studies. In addition, a clinician was the first author in more than half of the economic evaluations undertaken in Canada (65%) and Egypt (56%), while in 52 percent of the South African studies, the lead author was a public health professional. Furthermore, funding from an international source (United States Agency for International Development, Bill & Melinda Gates Foundation, World Health Organization, President's Emergency Plan for AIDS Relief) was relatively more prevalent in South Africa (73%) in comparison to Egypt (12%) and Canada (11%) (Table 1).
Table 1. General characteristics of the included study
CBA, cost-benefit analysis; CCA, cost-consequence analysis; CMA, cost-minimization analysis.
Interventions for infectious diseases were more commonly evaluated in the two middle-income countries (72% in South Africa and 44% in Egypt), whereas economic evaluations for noncommunicable diseases (NCDs) were more frequent in Canada (59%). Furthermore, while the majority of the South African economic evaluations were on human immunodeficiency virus (45%) and tuberculosis (23%), studies in Egypt were targeted around interventions for hepatitis C (56%) and pneumonia (19%).
Curative interventions comprised about 59–69 percent of the total studies in these countries, however, the studies differed in terms of the type of intervention being evaluated. Economic evaluations of medications were most common in Egypt (57%), while about 55 percent of the evaluations in South Africa assessed public health programmes. Canadian economic evaluations assessed both diagnostics (25%) and pharmaceuticals (27%) almost equally (Table 1).
Methodological Characteristics of Included Studies
Table 2 summarizes the methodological characteristics of the economic evaluations.
Table 2. Methodological characteristics of the included study
Analytic Technique
The majority of the economic evaluations undertaken in Canada (82%) and South Africa (80%) were model-based, whereas 63 percent of economic evaluations in Egypt were conducted alongside randomized controlled trials (RCTs) or observational studies as shown in Table 2.
Study Perspective
Payer perspective specified as either third-party payer or healthcare payer was the most commonly used perspective across the three countries. However, a large number (44%) of Egyptian economic evaluations did not specify the perspective of analysis (Table 2).
Comparator
Almost half of the studies (46–52%) in all the three countries used standard of care for comparison against the new intervention. In addition, several studies in Canada compared the new intervention to the most commonly or routinely used therapy (41%).
Measure of Health Outcome
Quality adjusted life-year (QALY) was the most frequently used outcome measure in Canada (75%) and Egypt (50%). However, in South Africa, clinical end points (38%) and life-years (22%) were more widely used for calculating differences in outcomes of the alternate interventions. A minor proportion of economic evaluations undertaken in South Africa (28%) and Egypt (12%) reported using disability adjusted life-years (DALYs), whereas none of the Canadian studies reported DALYs.
Source of Effectiveness Data
Almost half of the economic evaluations in South Africa (50%) and Egypt (50%), and around 41 percent in Canada used effectiveness evidence from observational studies. Even though systematic reviews of RCTs are considered as the gold standard; however, these were rarely used for obtaining effectiveness evidence Canada (14%), South Africa (11%), and Egypt (12%).
Time Horizon
Approximately, 73 percent of the evaluations in Canada used either a time horizon of more than 10 years or a lifetime time horizon. This number was comparatively lower in South Africa (49%) and Egypt (24%). Furthermore, around 31 percent of the Egyptian economic evaluations did not specify the time period of analysis.
Costs
A large majority (66–80%) of evaluations across the three countries used secondary data source for measuring costs. About 93 percent of evaluations undertaken in Canada included direct health system costs only. A large number of studies in South Africa (62%) also included direct health system costs only. OOP expenditure was included in only 2, 34, and 1 percent of the studies in Canada, South Africa, and Egypt, respectively. The inclusion of indirect costs due to productivity losses was rarely done. Only two studies undertaken in Canada and Egypt and three studies in South Africa reported to have included indirect costs in their overall cost calculations. Furthermore, almost one-fourth of the Egyptian studies did not provide details on the type of costs included.
Discount Rate
Universally, wherever discounting was done, both costs and outcomes were discounted using a common rate, varying between 1.5 and 5 percent. Almost half of the studies in Canada and 58 percent of the studies in South Africa used a discount rate of 1.5 and 3 percent, respectively. Around 9, 16, and 25 percent of the economic evaluations in Canada, South Africa, and Egypt, which had a time horizon of more than 1 year, did not discount future costs and outcomes. Additionally, a couple of South African studies reported that discounting was done; however, the rate used was not specified. Furthermore, although the source of applied rate was cited in the majority of Canadian (89%) and Egyptian (100%) evaluations, however, a high proportion (45%) of South African studies did not justify the basis for choosing a particular rate.
Uncertainty Analysis
In Canada, South Africa, and Egypt, 71, 49, and 31 percent of studies, respectively, performed both deterministic as well as probabilistic sensitivity analysis and the detailed findings on the type of sensitivity analyses are presented in Table 2. A complete absence of any form of sensitivity analysis was seen in approximately one-fourth of the Egyptian economic evaluations.
Quality of Economic Evaluations and Adherence to the National Guidelines
The mean adherence of the published studies to the country-specific recommendations on methodological principles of economic evaluation was the highest for Canada (71%), followed by South Africa (65%) and the lowest for Egypt (60%). Adherence scores were higher for all key principles for Canadian evaluations. The principles having highest adherence were study comparator (>90%), followed by study perspective and costs to be included. Recommendations on the discount rate to be used and time horizon to be adopted were less commonly complied with. Furthermore, although Canadian guidelines gave clear recommendations on using cost-utility analysis, however, the guidelines for the other two countries stated that any type of economic evaluation might be undertaken subject to justifications. The latter recommendation resulted in poor adherence (18%) in South Africa, where around 68 percent of the studies did specify the type of economic evaluation (cost-utility, cost-effectiveness, or cost-benefit analysis), however, they failed to justify the same. Similarly, in Egypt, around 38 percent of the studies did not provide justifications for the type of economic evaluation undertaken. Additionally, while recommendations on uncertainty analysis were mostly abided by economic evaluations undertaken in Canada (88%) and South Africa (81%), the corresponding figure was lower for Egypt (56%), as shown in Supplementary Figure 1.
Quality of studies was also the highest for Canada (82%), followed by South Africa (78%) and Egypt (55%). Adherence to the guidelines was positively correlated (r = .72, p < .001) with quality of studies. Economic evaluations which had a higher adherence score were found to have higher quality (Supplementary Figure 2).
The mean adherence score was significantly higher for studies using a cost-utility analysis design (72%, p-value < .001), those with a local/national funding aid (72%, p-value .02), undertaken by a health economist (71%, p-value .05) and pharmacoeconomic evaluations (70%, p-value .04) in comparison to their counterparts. No significant difference in the mean adherence score was observed according to disease type (Table 3).
Table 3. Factors influencing adherence to national guidelines
a Others: not reported, no funding, pharmaceutical companies.
*p-value <.05.
Discussion
This study illustrates the applicability of country-specific guidelines for the conduct of economic evaluations, measured in terms of adherence. We found that there were significant differences in the methodological conduct, adherence to the guidelines, and quality of economic evaluations across the three countries. The methodological differences reflect the contextual heterogeneity in terms of health system design and financing, researcher capacity, and data availability. The trends in diseases studied largely mirror the disease burden profile of these countries (Reference James, Abate, Abate, Abay, Abbafati and Abbasi34). The pattern of methodological differences between Canada, South Africa, and Egypt for analytic technique, study perspective, outcome measure observed by us is similar to what has been reported earlier by reviews comparing economic evaluations undertaken in HIC and LMIC settings (Reference Hutter, Rodríguez-Ibeas and Antonanzas5;Reference Santatiwongchai, Chantarastapornchit, Wilkinson, Thiboonboon, Rattanavipapong and Walker6).
The adherence to the national guidelines was highest for economic evaluations undertaken in Canada, followed by South Africa and lowest for Egypt. This finding is in line with the stage of maturity for HTA systems in the three countries. Neumann et al. reported in their systematic review that while Canada published close to 300 cost-utility analysis during 1990−2012, the corresponding number was significantly lower for the entire African region-29 (Reference Neumann, Thorat, Shi, Saret and Cohen3). Furthermore, the guidelines for the conduct of economic evaluation have been in place in Canada since 1994 (19). Over the years, these guidelines have been continuously refined with revised versions being released in 1997, 2006, and 2017, respectively (21;35;36). On the other hand, South Africa and Egypt have published the first set of their guidelines more recently in 2012 and 2013, respectively (25;27). Secondly, local expertise and capacity to lead the conduct of economic evaluations is strongest in Canada (Reference Pitt, Goodman and Hanson2). Overall, it may be concluded that while Canada has higher capacity and experience in conduct of economic evaluations, the other two countries are still in their initial phases. Moreover, only three studies each in South Africa and Egypt were found to explicitly cite the national guidelines against thirty-five studies in Canada, which is likely to be linked to the level of “enforcement” or implications for a study that does not explicitly align to a methods guidance. In the South African context, we are not aware of a single pharmaceutical pricing decision in the public domain that has explicitly utilized the Pharmacoeconomic guidelines. Additionally, the Canadian guidelines were more detailed and extensive in comparison to the South African and Egyptian guidelines. Despite this fact, better adherence was observed in case of Canada which may have two plausible explanations. First, the more detailed the guidelines are, the better is the adherence, or alternately, there is a general higher level of capacity among HTA researchers in Canada resulting in better adherence. However, further research is needed to conclude the exact reason with certainty.
Adherence to national guidelines in certain situations may incur additional technical, monetary, or temporal resources for HTA researchers. For example, in countries where the existing repositories of secondary data for costs, effectiveness, quality-of-life data value sets, etc. are not available, adherence to the guidelines may require additional primary data collection for researchers which may impact the overall cost, duration, and technical capacity to conduct the study. Thus, it would be useful to inculcate monetary and nonmonetary incentives to HTA researchers for adherence to the national guidelines. For example, studies requiring additional primary data collection should receive more than the routine funding. Similarly, the acceptance of the study by the technical appraisal committees of HTA agencies could be made contingent upon authors’ self-reporting of adherence of the HTA studies to the national guidelines.
The ultimate aim of any guidelines for conduct of economic evaluations is to increase the methodological quality and comparability (Reference Sanders, Neumann, Basu, Brock, Feeny and Krahn13;Reference Wilkinson, Sculpher, Claxton, Revill, Briggs and Cairns14). Following a predefined set of methodological principles allows for transparency in conduct and is a potential way to enhance the study quality (Reference Wilkinson, Sculpher, Claxton, Revill, Briggs and Cairns14). Given this fact, it was unsurprising to note that higher adherence to the national guidelines resulted in better quality economic evaluations.
A lower rate of adherence was observed for programmatic economic evaluations in comparison to studies evaluating pharmaceuticals. While the methods for evaluating the cost-effectiveness of clinical interventions, especially involving pharmaceuticals, are more extensively specified in the national guidelines, however, they do not incorporate specific guidance for the evaluation of public health programmes (Reference Action, Heintz, Gerber-Grote, Ghabri, Hamers and Rupel37;Reference Edwards, Charles and Lloyd-Williams38). Even in case of our review, two out of the three included guidelines were explicitly formulated for pharmacoeconomic evaluations (25;27). Moreover, a number of distinguishing features of programmatic evaluations limit the adherence to the common set of recommendations outlined in the national guidelines. The national guidelines recommend using the health system or the government perspective; however, both the resources used and in particular, the benefits accrued of a public health programme extend outside the gamut of the healthcare system. Thus, it becomes essential to contemplate specific considerations such as who funds the program (one or multiple funders including patients and public/private organizations), multiple comparator scenarios, duration of the programme (from planning to rollout to occurrence of health consequences), and valuation of consequences (proximal end points or long-term changes in quality of life), while undertaking programmatic evaluations (Reference Weatherly, Drummond, Claxton, Cookson, Ferguson and Godfrey39;Reference Drummond, Weatherly, Claxton, Cookson, Ferguson and Godfrey40). The distinct nature of healthcare programs calls for concerted efforts for adapting guidelines such that they better address specific issues pertaining to economic evaluations of programmatic interventions.
Another independent predictor of adherence to the national guidelines identified in our review was the background or formal training of the lead author. In studies where a health economist was the lead author, we observed better adherence to the guidelines. Previous systematic reviews also report similar findings of better quality of studies with health economists as the primary authors. It is reported that health economists have advanced training for undertaking economic evaluations. They also understand the principles of economic evaluations better and are thus more likely to adhere to them (Reference Philips, Bojke, Sculpher, Claxton and Golder41;Reference Pandey, Pandey and Shah42). It was also observed that studies which received international funding had lower adherence rates as compared to studies with national funders. A plausible justification of this could be that internationally funded studies have more non-native authors who we expect to be less sensitive to use national guidelines, probably due to the lack of awareness or the lack of accountability. Additionally, the existence of alternative guidance from international institutes such as Panel reference case, iDSI reference case, and/or World Health Organization guide to cost-effectiveness analysis may be more appealing to the international authors than the local guidelines (Reference Sanders, Neumann, Basu, Brock, Feeny and Krahn13;Reference Wilkinson, Sculpher, Claxton, Revill, Briggs and Cairns14). The finding that adherence is higher in locally funded studies and those with health economists as lead authors provides evidence for creation of country-level institutional frameworks for funding HTAs to ensure higher quality and standardization of methods. Nevertheless, it is acknowledged that the stage of development of the guidelines, the capacity of local HTA researchers, the availability of routine data sources to support HTA, and the extent of institutional frameworks that mandate HTA evidence for policy making are all important factors that could affect adherence of the HTA assessments to the national guidelines. Some of these factors were beyond the purview of the present study and, hence, should be assessed in any similar future assessments.
Strengths and Limitations
One of the major strengths of this systematic review is that it was not targeted to a specific disease or technology but adopted a holistic approach wherein all forms of economic evaluation for any disease and any intervention type were evaluated. Second, the review evaluates both adherence and quality for three countries across different income groups.
Nevertheless, our findings are subject to a few limitations. First, the time period for inclusion of studies was relatively short, approximately 5 years for South Africa and Egypt and only 1 year for Canada (based on the publication of the most recent version of the national guidelines). This criterion limited the study from commenting on time trends in quality and adherence of economic evaluations. Second, we included economic evaluations up to December 2018, and subsequently were engaged in data abstraction and analysis. Although we could capture a significant number of studies (139), we cannot preclude the possibility that we missed out on more recent economic evaluations published in 2019 and 2020. Third, we identified HTA reports published by country-specific HTA agencies, however we could not retrieve full HTA reports for Egypt. Therefore, in order to have uniformity across the three countries, we included only those studies which have been published in peer-reviewed journals. This may have led to omission of certain HTA evaluations that have been carried out but for which either reports/abstracts are available at the Web sites of the respective HTA agencies. However, as reported in a previous study, only 19 percent of the HTA studies undertaken by Canadian HTA agencies included a full economic evaluation (Reference Tarride, McCarron, Lim, Bowen, Blackhouse and Hopkins43). Additionally, since we included only peer-reviewed articles, our study estimates may be viewed as a more optimistic scenario of both adherence to the guidelines as well as quality of the HTA studies. Fourth, we relied on what has been reported by the authors for measuring adherence. Since the majority of the peer-reviewed journals have restrictions on word-count, there could be instances where the authors might have complied with a certain set of recommendations but failed to report the same due to word-count constraints. It is indeed very difficult to conclude what was actually done in the study and, therefore, we used what has been reported as a proxy for what has been done. Additionally, we also reviewed all available Supplementary Material of the included studies that provides elaborate details of the methodology adopted to conclude for adherence. Lastly, we were not able to establish a clear distinction between “adherence” with guidance and “alignment” of an economic evaluation with generalized good practice principles for economic evaluation. Furthermore, we acknowledge that various reporting checklists such as Drummond's checklist or the BMJ checklist cited in the national guidelines may act as a potential confounder, affecting the quality of studies. However, only 14 percent (Reference Torrance, Blaker, Detsky, Kennedy, Schubert and Menon20) of the studies included in our review reported having used any of these checklists.
Conclusion and Policy Implication
While there has been a rapid increase in both the publication of economic evaluations and guidelines for methodological conduct, this review highlights the usage and adherence to such guidelines. Among the three countries, the adherence to the national guidelines was found to be highest for economic evaluations undertaken in Canada, followed by South Africa and lowest for Egypt, a finding in line with the stage of maturity for HTA systems in the three countries. Furthermore, the quality of economic evaluations improved with adherence to the country-specific guidelines. Additionally, we also identified several key factors affecting adherence such as source of funding, lead author background, and type of intervention. It is recommended that locally funded and health-economist led HTAs should be encouraged for greater adherence to the guidelines. Moreover, concerted efforts for developing guidelines addressing the specific issues pertaining to economic evaluations of programmatic interventions would be a useful resource for HTA researchers to improve the quality of such evaluations. Furthermore, in addition to overall adherence, we also identified individual principles wherein adherence was relatively poor. Our findings can facilitate revision of existing guidelines and the development of new guidelines that are more context-specific and applicable. The above points, in turn, would enhance the potential to generate quality economic evaluation and subsequently promote better policy decisions.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321000404.
Acknowledgment
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Funding
The research was undertaken as part of doctoral thesis of Sharma D, which is supported by the Indian Council of Medical Research, New Delhi, India.
Conflict of Interest
The authors declare that they have no conflicts of interest.
