Introduction
Peripheral artery disease (PAD) is a major health problem. In 2010, it was estimated that PAD affected 200 million people worldwide (Reference Fowkes, Rudan, Rudan, Aboyans, Denenberg and McDermott1). Patients with PAD are at high risk of developing major adverse cardiovascular events (MACEs) and major adverse limb events (MALEs). Among patients with symptomatic PAD, annual rates of MACEs are 4–5 percent and rates of MALEs are 1–2 percent (Reference Hiatt, Fowkes, Heizer, Berger, Baumgartner and Held2). MACEs include myocardial infarction, ischemic stroke, and cardiovascular death, and MALEs include major amputations and acute limb ischemia. PAD is associated with significantly elevated all-cause mortality, cardiovascular disease mortality (CVD), and combined CVD morbidity/mortality at 3 and 6 years after the diagnosis (Reference Criqui, Ninomiya, Wingard, Ji and Fronek3).
First-line treatment is based on lifestyle management that improves functional outcomes and reduces MACE (Reference Parvar, Fitridge, Dawson and Nicholls4). Lifestyle management, including smoking cessation and exercise therapy, can modify important risk factors. In addition, several pharmacological interventions can be used, such as antiplatelet agents, anticoagulant agents, and statins (Reference Parvar, Fitridge, Dawson and Nicholls4). For symptomatic patients with PAD with intermittent claudication who have not responded to medical treatment, limb revascularization is recommended. Patients with critical limb ischemia require more urgent revascularization because of an increased risk of tissue loss and amputation, as well as an extremely high risk of cardiovascular events (Reference Beckman and Creager5). Either of the two strategies can be used: endovascular surgery as the first choice or open surgery (Reference Olin, White, Armstrong, Kadian-Dodov and Hiatt6). For endovascular surgery, plain old balloon angioplasty (POBA), drug-coated balloon (DCB) angioplasty, stent placement (bare-metal stent, drug-eluting stent [DES], or covered stent), and atherectomy may all be reasonable options in specific circumstances and for specific lesion anatomy (Reference Conte, Bradbury, Kolh, White, Dick and Fitridge7). DCBs and DESs contain medications that inhibit vessel restenosis and have been shown to result in notable improvements in clinical outcomes in several studies (Reference Dake, Ansel, Jaff, Ohki, Saxon and Smouse8–Reference Shanmugasundaram, Murugapandian, Truong, Lotun and Banerjee10). However, this point is controversial, as a recent study found an increased risk of death following the use of paclitaxel-coated balloons and stents in PAD (Reference Katsanos, Spiliopoulos, Kitrou, Krokidis and Karnabatidis11).
In addition, these medical devices are expensive, and the question of their cost-effectiveness—and of the quality of economic studies on this topic—remains. Furthermore, this information is valuable in health technology assessments (HTAs), which, in turn, support decision-making processes, for instance at a hospital level. Knowing the quality of available economic studies would seem to be essential for making knowledge-based decisions. The purpose of the present study was to perform a systematic review of the literature on economic evaluations of DCBs and DESs in PAD in order to assess the methodological and reporting quality of the currently available publications on the topic. We also aimed to provide valuable information for HTA analysts and policy makers dealing with DCBs and DESs in PAD.
Materials and Methods
In July 2020, we performed a systematic literature review to identify economic evaluation studies relating to DCBs and DESs. To do so, we followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (see Supplementary File 1, PRISMA Checklist).
Search Strategy
The articles have been selected from four electronic databases, namely, PubMed, Embase, the National Health Service Economic Evaluation Database (NHS EED), and the Cost-Effectiveness Analysis (CEA) Registry. To identify all relevant studies published in the last 10 years (from 3 July 2010 to 3 July 2020), the search strategy was first developed in the PubMed database and then was applied to the other databases. This 10-year period was chosen because the technology has remained the same over this time span with no significant evolution (Reference Shanmugasundaram, Murugapandian, Truong, Lotun and Banerjee10). The search term combined a descriptor of the device (MeSH or not) and a term related to economics (MeSH or not) (Supplementary File 2, Study Protocol).
Study Selection
First, duplicate articles were removed. Then, titles and abstracts were screened by two reviewers (AE and LN) to select relevant articles based on inclusion and exclusion criteria according to the PICOS format (population, intervention, comparators, outcomes, and study design) (Table 1) (Reference Schardt, Adams, Owens, Keitz and Fontelo12). Articles written in languages other than English or French, studies in which DCBs or DESs were not the sole topic, and other studies such as reviews, editorials, congress communications, letters, and noneconomic studies were excluded. Discrepancies were resolved by a third reviewer (NM).
Table 1. Study eligibility criteria in PICOS format
PICOS, population, intervention, comparators, outcomes, and study design.
Quality Assessment
To evaluate the overall quality of articles screened, two reviewers (AE and LN) used three established checklists to appraise the reporting and methodological quality of the economic evaluations. These three tools are all qualitative instruments. If a discordant classification appeared, the two reviewers discussed discrepancies until a consensus was reached.
First, general characteristics were extracted using international guidelines published by Drummond et al. (Reference Drummond, Sculpher, Claxton, Stoddart and Torrance13). These recommendations provide general guidance about the way in which the results of economic evaluations should be reported. Authors must provide information about the type of evaluation conducted, perspective chosen, and costs considered (direct, indirect, and both). They also need to characterize the time horizon, comparators chosen, incremental cost-effectiveness ratio (ICER), and source of funding.
Next, a checklist developed by Cooper et al. was utilized to assess the quality of the sources of evidence used in the studies (Reference Cooper, Coyle, Abrams, Mugford and Sutton14). This tool, which ranks evidence used in studies on a scale of 1–6, evaluated the quality of sources for the main clinical effect sizes, baseline clinical data, cost data, and utility data. If the information was not clearly stated, the scale awarded a rank of 9. We then joined the rankings into three quality categories defined by Cooper et al. (Reference Cooper, Coyle, Abrams, Mugford and Sutton14). Level A corresponded to the highest level of evidence quality, covering ranks 1 and 2. Level B corresponded to an intermediate level of evidence quality, covering a ranking of 3. Ranks 4, 5, 6, and 9 were grouped together into level C, corresponding to the lowest level of evidence quality.
Finally, we used the CHEERS (Consolidated Health Economic Evaluation Reporting Standards) checklist, which is a tool used for the reporting of economic studies (Reference Husereau, Drummond, Petrou, Carswell, Moher and Greenberg15). This checklist includes twenty-four items in six categories (title and abstract, introduction, methods, results, discussions, and others). All twenty-four items were checked per article by two reviewers (AE and LN). In the event of disagreement, a consensus was reached through discussion with the third reviewer (NM).
The above-mentioned methodology of combining several checklists was based on previous work on the quality of economic evaluations (Reference Fontenay, Catarino, Snoussi, van den Brink, Pineau and Prognon16;Reference Nédellec, Pineau, Prognon and Martelli17). To our knowledge, no single checklist exists that is able to measure both the reporting and the methodological quality of economic evaluations. For this reason, we chose to consolidate these three qualitative instruments that explore different aspects of an economic evaluation: the global methodology assessed by the Drummond guidelines, the quality of the sources of evidence by the Cooper checklist, and the reporting quality by the CHEERS checklist. This strategy enabled us to be more exhaustive in evaluating the screened articles and to offset the inherent weaknesses of a single tool.
Results
Selected Studies
Figure 1 summarizes the details of study identification and reasons for inclusion/exclusion. The initial electronic literature search identified 1,728 studies. A total of 1,498 articles were obtained after the removal of duplicates. After screening titles and abstracts, we excluded 1,488 articles as they did not meet the selection criteria: 1,453 did not report on DCBs or DESs, 13 did not have accepted designs (letters and congress abstracts), and 22 were not economic evaluations. We read the full text of the remaining ten articles and excluded one article because of its design and a further three because they were not economic evaluations. Our literature search resulted in six studies that fulfilled inclusion criteria (Reference Kearns, Michaels, Stevenson and Thomas18–Reference Kearns and Thomas23).
Figure 1. PRISMA flowchart for study selection. CEA, Cost-Effectiveness Analysis; DCB, drug-coated balloon; DES, drug-eluting stent; NHS EED, National Health Service Economic Evaluation Database.
Characteristics of the Studies
The majority of studies were conducted in Europe (66.7 percent); two studies (33.3 percent) were conducted in North America. The selected articles were published between 2012 and 2018. Four studies (66.7 percent) were conducted in the last 5 years. According to the criteria outlined in the Drummond guidelines, three studies (50 percent) were cost-effectiveness analyses and three (50 percent) were cost–utility analyses (Table 2). A payer perspective was retained for five studies (83.3 percent) and one (16.7 percent) did not state a perspective for the analysis. Additionally, direct costs were used for all studies. Authors considered a lifetime horizon in three publications (50 percent) and a two-year horizon in three studies (50 percent).
Table 2. Details of the Drummond checklist criteria for each included study
BMS, bare-metal stent; CEA, cost-effectiveness analysis; CLI, critical limb ischemia; CUA, cost–utility analysis; DCB, drug-coated balloon; DES, drug-eluting-stent; EES, everolimus-eluting stent; IC, intermittent claudication; PAD, peripheral artery disease; PAOD, peripheral artery occlusive disease; POBA, plain old balloon angioplasty; PTA, percutaneous transluminal balloon angioplasty; QALY, quality-adjusted life-year; SES, sirolimus-eluting stent; TLR, target limb revascularization.
Two studies (33.3 percent) provided both deterministic and probabilistic analyses, two (33.3 percent) provided only deterministic analyses, and two studies (33.3 percent) provided only probabilistic analyses.
Quality of the Sources
We used the Cooper scale to evaluate the quality of the study sources. The results of the Cooper scale are presented for each study in Table 3. The results of the hierarchy of data sources are given in Supplementary Table 1. The data sources for the clinical effect size and safety were of a high quality (level A, 100 percent). Most studies were based on randomized controlled trials (RCTs) with direct comparison between comparator therapies. For instance, the CONSEQUENT trial, which is a prospective, multicenter, two-armed, randomized controlled trial, was the main source used for documenting the clinical effect size in the study by Albrecht et al. (Reference Albrecht, Waliszewski, Roca, Redlich, Tautenhahn and Pech20). This was also observed for other studies using RCTs such as IN.PACT SFA I and IN.PACT SFA II as a source (Reference Salisbury, Li, Vilain, Jaff, Schneider and Laird19–Reference Sridharan, Boitet, Smith, Noorbakhsh, Avgerinos and Eslami21).
Table 3. Summary of the main results of the Cooper and CHEERS checklists
NA, not applicable; RCT, randomized controlled trial.
In most studies (83.3 percent), cost data were based on recently published cost calculations from reliable databases (in the same jurisdiction as the study performed). In the remaining study (16.7 percent), the sources of cost data were of medium quality, and the origin of the source was not clearly stated.
Only one study (16.7 percent) used data from direct utility assessments for the specific study. In this study, health utilities were assessed using the EuroQol questionnaire (Reference Salisbury, Li, Vilain, Jaff, Schneider and Laird19). Two studies (33.3 percent) used utility data that were direct estimates from a previous study on patients with the diseases of interest.
Quality of Reporting
We used the CHEERS checklist to establish the quality of reporting; Table 3 presents the main results of each study. Most studies (83.3 percent) did not state the aspects of the system in which the decision needed to be made or for which the economic evaluation had been designed (Reference Kearns, Michaels, Stevenson and Thomas18–Reference Katsanos, Karnabatidis, Diamantopoulos, Spiliopoulos and Siablis22). In addition, very few studies (33.3 percent) reported the choice of discount rate used for costs and outcomes and explained why it was appropriate (Reference Kearns, Michaels, Stevenson and Thomas18;Reference Kearns and Thomas23). We noticed that the authors correctly stated the time horizon over which costs and consequences had been evaluated, but they rarely explained why the time period chosen was appropriate (33.3 percent) (Reference Albrecht, Waliszewski, Roca, Redlich, Tautenhahn and Pech20–Reference Kearns and Thomas23). Finally, very few studies (33.3 percent) provided a figure to illustrate the model structure (Reference Salisbury, Li, Vilain, Jaff, Schneider and Laird19;Reference Sridharan, Boitet, Smith, Noorbakhsh, Avgerinos and Eslami21–Reference Kearns and Thomas23).
However, all studies (100 percent) reported information properly for the abstract, background, and objectives, measurement of effectiveness, estimation of resources and costs, and characterization of uncertainty.
Discussion
To our knowledge, this study is the first systematic review to analyze the quality of economic studies on drug-eluting medical devices used for treating PAD and to use three formal checklists. Although it is a blossoming field, we did not identify many economic studies on the topic. Indeed, only six studies in 10 years seem a rather small number for a widespread pathology such as PAD. However, the aim of this systematic review was not to analyze these medical devices or their efficiency but to assess the level of evidence used to inform economic evaluations. With the present work, we wish to contribute to the HTA process, which is not only designed to review and summarize outcomes but also to assess the quality of the evidence itself. In addition, we would like to share our experience with this three-tool approach and discuss its strengths and flaws.
Our examination of the studies first revealed that most of the studies retrieved fully complied with the Drummond guidelines. This represents a strength of the included studies, which seemed, in general, to provide the essential information about the economic evaluation performed (type of study, perspective, and evaluated costs). One of the flaws of the studies highlighted with this checklist was a lack of detail about the sources of funding. Studies sponsored by industry are more likely to reach positive conclusions than similar studies funded by not-for-profit organizations (Reference Krimsky24). Nevertheless, the Drummond guidelines only give a general overview of an economic study because they focus solely on key elements. For example, this tool does not guarantee that the data sources used are of a good quality or that all information needed to understand the study is reported properly. For this reason, the Drummond guidelines must be complemented by the use of the Cooper and CHEERS checklists.
The Cooper checklist is a useful tool for assessing the level of evidence of the sources used to perform an economic evaluation. Here, its use revealed that most studies used good-quality clinical effect size and safety data sources. However, the Cooper checklist does not allow an in-depth analysis of the quality of the clinical data used, especially for retrospective clinical data that are at a high risk of bias. Exploring this point, a recent study suggested using two checklists, the Joanna Briggs Institute (JBI) Checklist for Prevalence Studies and a modified version of Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I) (Reference Glasgow, Edlin and Harding25); it would be relevant to combine the Cooper checklist with these two checklists. However, it must also be kept in mind that the more tools used for a single study, the more complicated and longer the assessment is. For information, we found in our study that the average time spent for grading an article with the three tools was reasonable and after a first full reading of the article did not exceed 1 h for a trained scorer.
Finally, we chose to use the CHEERS checklist to assess the quality of reporting. We would like to underline here how poor-quality reporting may raise doubts on the general quality of the economic study itself. In fact, we observed that important information was not clearly stated in most of the studies we retrieved. This could have skewed our perception of the studies and prevented us from concluding on their actual quality.
Additional items could be taken into consideration when evaluating medical devices, such as the learning curve or the organizational impact. As stated by Craig et al., there is a need to adopt new modeling approaches to incorporate and/or assess certain unique characteristics of medical devices that are often unaddressed (Reference Craig, Carr, Hutton, Glanville, Iglesias and Sims26). When new models are developed for the economic evaluation of medical devices, we should keep in mind that verification tools such as the checklists used in the present work will need to be adapted and will need to integrate new items such as the learning curve.
Some limitations of our study need to be highlighted. First, the search was performed in scientific journals only and we did not include economic evaluations from gray literature. In addition, during data collection, we found many articles related to the use of DESs and DCBs in coronary arteries, and this complicated screening for articles only on PAD. For this reason, it is possible that some articles about DES and DCB treatment of PAD may have been discarded in the screening step due to the large number of articles referring to DES/DCB use in coronary arteries.
Conclusion
All identified studies relied on the high-quality level of clinical evidence to inform the respective economic evaluations of DCBs and DESs in PAD, but the quality of reporting in these economic evaluation studies was low. Nevertheless, it is worth mentioning the argument that currently exists about the safety of these devices: the divergent analyses conducted by Katsanos et al. and Nordanstig et al. merit consideration (Reference Katsanos, Spiliopoulos, Kitrou, Krokidis and Karnabatidis11;Reference Nordanstig, James, Andersson, Andersson, Danielsson and Gillgren27). It could be that the apparent increased mortality associated with these devices is actually connected to confounding factors unrelated to paclitaxel. Finally, our evaluation of the reporting quality suggests that essential information was not present in these studies. Without well-reported data, readers are not able to critically assess whether the results provide reliable information or whether the conclusions are valid. This is of concern in the interpretation of economic studies, especially in an HTA process.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321000532.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Conflict of Interest
The authors declare no conflict of interest.
