Bayesian methods provide an interesting and innovative alternative approach to standard statistical methods, with the specific advantage of explicitly incorporating previous information into current data when evaluating a health technology. Such approaches have been of particular interest in the context of evaluating implantable medical devices (IMDs), and approximately 10 percent of FDA approvals for medical and radiological devices have been based on such analyses (Reference Campbell1;Reference Pibouleau and Chevret2). Indeed, unlike drugs, most IMDs are updated by means of successive introductions of technical changes into new versions, so that Bayes approaches that allow for the accumulation of data on inferences appear appropriate.
This information on previous versions may be useful in the decision process, when data documenting the clinical impact of technical changes are scarce and noncomparative. Indeed, decision makers are often faced with the issue of assessing the latest version of an IMD when only a small amount of clinical data are available on this last version while genuine information on previous versions exists. The example of the neuroform intracranial stents that are used in the treatment of wide-necked aneurysms in conjunction with embolic coil implants is particularly illustrative of such a context. Three successive versions of this IMD had been proposed over 5 years, with accumulated data based on eighteen independent evaluations including 548 patients, when a new, fourth version of the neuroform was proposed (Reference Pibouleau and Chevret3). In this context, a Bayesian hierarchical meta-analysis model allowed for the assessment of the overall success rate of the IMD by incorporating all of these data, and it also permitted testing of the heterogeneity of effects across the versions. Based on this model, with noninformative priors for the model parameters, a marked benefit of the version #2 over the version #1 was estimated, while an absence of improvement was found for the version #3 with respect to the version #2 (Reference Pibouleau and Chevret3). More recently, a single trial on the version #4 of the neuroform stent was published, based on twenty-one patients (Reference Mangubat, Johnson, Keigher and Lopes4). Decision makers may wish to analyze the overall benefit of the neuroform stent and evaluate the potential benefit achieved by this latest version. One may wonder whether experts’ beliefs about the benefits brought by the last versions (version #3 and version #4) of the neuroform stent would provide more helpful insights than those provided by all of the past information that has been modeled. Indeed, in Bayesian inferencing, informative distributions have been recommended for use wherever substantive prior information exists (Reference O’Hagan5).
The integration of experts’ beliefs into a Bayesian model requires the formal incorporation of their opinions into a mathematical prior distribution. This process, referred to as the “elicitation” process, presupposes some understanding of how a person assesses the probability of an event, mostly from heuristics (Reference O’Hagan, Buck and Daneshkhah6). To increase acceptance of the use of Bayesian inferencing in clinical research, a sound methodology of eliciting experts’ beliefs is required. Numerous elicitation procedures have been reported in the literature, though their measurement properties have rarely been evaluated (Reference Johnson, Tomlinson, Hawker, Granton and Feldman7). Computer-based surveys, through interactive and graphical interfaces, may improve elicitation process (Reference Low-Choy8). First, it may allow to achieve elicitation in a transparent, repeatable and robust manner; second, it may give access to a broad range of experts geographically dispersed and at last, the use of interactive tools may help the expert to assess her/his own probability distribution without requiring the presence of someone with statistical and psychological knowledge, namely a “facilitator”.
Thus, we developed a computer-based tool to elicit experts’ beliefs about the success rate of the neuroform stent. Given the marked lower efficacy of version #1, it was not considered in this study. We showed how experts’ beliefs differ from the available data, and we assessed the potential impact of these beliefs on the assessed benefit of the latest version of the device.
MATERIALS AND METHODS
Data
From a total of eleven previously published evaluations of the new versions (#2, #3, and #4) of the neuroform stent, we selected fourteen datasets of complete evaluations, including nine on version #2, four on version #3, and one on version #4. Observed success rates, defined as the placement of the stent in the target artery and an aneurysm occlusion ≥ 95 percent on the immediate postprocedural angiography, are reported in Table 1 for each dataset.
Table 1. Estimations of the neuroform Success Rate for Each Study
Note. The fourth and fifth columns indicate independent estimations and the last two columns reports the results obtained with the hierarchical Bayesian meta-analysis (MA) model, either with a non-informative prior or with an informative expert-derived prior (results reported on the bottom). Non-informative priors were Norm(0, var = 106) for the logit of the overall success rate, μ, and LN(0,1) for the inter-version and inter-study standard deviations, ν and σ; informative priors were Norm(1.6, var = 2.3) for μ,and Gamma(1,4) for ν; MLE: maximum likelihood estimate y, number of successes; n, number of treated aneurysms;
Experts
We elicited experts’ beliefs on the success rates of the latest version of the neuroform stent (version #4) and on the success rate of the previous version (version #3) to determine their beliefs about the benefits brought by the latest version.
A group of nineteen experts was selected from the set of corresponding authors who had written articles on intracranial aneurysms that were referenced in PubMed in the last 6 years. The characteristics of the participants are reported in Supplementary Table 1, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000403. The experts were an international group with practices in Europe (n = 7), North America (n = 5), South America (n = 3), and Asia (n = 3). They were all clinicians who were equally distributed among three specialities: neuroradiology, neurosurgery, and neurology. The majority of them were highly active in intracranial aneurysm treatment, and 11 of 19 (58 percent) used the neuroform stent. Concerning normative goodness, previous statistical training, considered as a proxy measure for knowledge in statistics (Reference Johnson, Tomlinson and Hawker9), was reported by 9 of 19 (47 percent) of the participants.
Elicitation Tool
We developed a Web-based tool elicitation tool that enabled expert elicitation of probabilities, as previously done in many other settings when face-to-face elicitation is not an option, for instance due to time or budget constraints, and to allow to contact experts worldwide on topics about which little or no knowledge is available. A review of existing tools in the field of environment and health has been previously published (Reference Devilee and Knol10).
The quantity to be elicited was the success rate of a stenting procedure. The elicitation question was formulated as follows: “if 100 patients were to be operated using this stent, what would be in your opinion the number of successes”. Indeed, it has been shown that the assessments made using relative frequency, that is, how many successes there would be in a sample of a given size, exhibit less scatter and express complete certainty less often compared with those made with direct probability (Reference O’Hagan11). A brief description of the version of the stent in question was first provided to the experts. The whole questionnaire is available in Supplementary Table 2, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000403.
Various techniques for eliciting a success rate have been described in the literature (Reference O’Hagan, Buck and Daneshkhah12;Reference Winkler13), while others focused on a regression coefficient (Reference Denham and Mengersen14;Reference Garthwaite, Al-Awadhi, Elfadaly and Jenkinson15). Because the elicitation was conducted on a group of nonstatistician experts without the assistance of a facilitator, we used the fixed-interval approach of Leal et al. (Reference Leal, Wordsworth, Legood and Blair16) that appeared more appropriate for nonstatistician experts compared with the variable-interval method (Reference Abbas, Budescu, Yu and Haggerty17).
Let X denotes the number of successes among 100 patients, under this fixed-interval method, the expert is asked about the range of plausible values of X and the mode, the most likely value. If any inconsistency occurred, the expert was automatically informed. The distance between each extreme value and the most likely value was then equally divided by 2, and the expert was asked to give a probability (expressed as a “weight of belief”) for both resulting intervals (in principle, assessed probabilities must sum to one otherwise they are standardized to one). A histogram was then derived from these estimates (Supplementary Figure 1, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000403), and the expert was asked to confirm whether it represented her/his beliefs or to correct her/his previous estimates if it did not. Once validated, this histogram constituted the expert's distribution of success rate for the considered version of stent.
As recommended (Reference Johnson, Tomlinson, Hawker, Granton and Feldman7), we used strategies to minimizing the effect of biases on the elicited beliefs (Table 2). First, to encourage the experts to consider possible sources of uncertainty explicitly (Reference O’Hagan11), they were asked about predictive factors of success. Second, a training exercise was built on the same model as the elicitation exercise. To make them more aware of aleatory uncertainty (Reference Hora18), this exercise focused on a future event which had nothing to do with their domain of expertise. Third, a feedback mechanism with a graphical aid and opportunity for revision was provided.
Table 2. Sources of Bias and Methodological Strategies Used to Control Them
Note. Based on Johnson et al. (Reference Johnson, Tomlinson, Hawker, Granton and Feldman7).
The questionnaire was implemented in php/html and was initially tested by four departmental colleagues and three experts in neuroradiology and neurology. The definition of success was reformulated to eliminate any ambiguity. Modifications in the design were implemented to make the tool clear, easy and attractive to use.
The methods used to assess the measurement properties of the questionnaire are summarized in Table 3. The belief elicitation procedure demonstrated good agreement for validity and good feasibility (data available upon request).
Table 3. Methods of Assessment of Measurement Properties of the Elicitation Questionnaire
Construction of Expert Priors
Each expert elicitation resulted in a histogram that constituted the expert's individual distribution of the success rate of the stent version considered. Based on this histogram, the probability of the success rate in each 5 percent interval from 0 to 100 percent was computed.
The experts’ individual distributions were then pooled to obtain a collective prior distribution using the linear opinion pool method proposed by Stone (Reference Stone19). Assuming the equivalence of experts, this provided a simple average of the experts’ distributions. This method resulted in a so-called “group prior” for each version of the stent. A parametric distribution of the experts’ data was then fitted using the parametric family of distributions suggested by the group prior on a logit scale.
Incorporation of the Experts’ Beliefs into the Statistical Model
To take into account the heterogeneity of the IMD versions in addition to the inter-study and intra-study heterogeneity, a Bayesian three-level hierarchical model was fitted. A detailed report of this method is published elsewhere (Reference Pibouleau and Chevret3). Such a model allows to estimating not only the overall success rate of the IMD but also the success rate of each IMD version. This model requires the specification of prior distributions for three parameters (on a logit scale): the overall success rate (µ), the inter-version variability (ν) and the inter-study variability (σ).
We first considered noninformative prior distributions which were: a normal distribution with a mean of 0 and a large variance, N(0, 106) for µ, and a log normal LogN(0,1) distribution for σ and ν (Reference Pibouleau and Chevret3).
Then, to integrate the experts’ opinions into the above model, we used their beliefs on the success rate of the version #3, P3, and that of the version #4, P4, to inform the priors of μ and ν. The group prior distribution of the success rates of both versions combined on a logit scale was used as the informative prior for m. The prior for ν was obtained through a sampling approach. Details of the model as well as its specification in BUGS code are provided in Supplementary Table 3, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000403.
We generated posterior distributions by combining the elicited prior distribution with the trial data using OpenBUGS software version 3.2.2. A total of 40,000 iterations following 10,000 burn-in iterations were used after checking for convergence. Posterior mean and median estimates of the success rates were computed for each study and for each version of the stent. Last, the posterior probability that the success rate of a new version was higher than that of the previous one was computed.
To detect a version effect, we compared the Deviance Information Criterion (DIC) obtained after fitting the data either with a Bayesian random-effects model or with a hierarchical one. A better fit of the hierarchical model would indicate a version effect, as previously published (Reference Pibouleau and Chevret3).
All statistical analyses were performed using the R software package, version 2.13.2 (R Foundation for Statistical Computing, Vienna, Austria, URL http://www.R-project.org).
RESULTS
Table 4 summarizes all of the information available on the success rates of version #3 and #4 according to the source of information or model.
Table 4. Information Available on the Success Rates of the Two Last Versions of the neuroform Stent
a Non-informative priors were Norm(0, var = 106) for the logit of the overall success rate, μ, and LogN(0,1) for the inter-version and inter-study standard deviations, ν and σ.
aInformative priors were Norm(1.6, var = 2.3) for μ, and Gamma(1,4) for ν.
Elicitation of Experts’ Beliefs
The median of the most likely values of the success rates were 75 percent (IQR: 70–88 percent) for version #3 and 90 percent (IQR: 65–93 percent) for version #4, with a pairwise difference across the two versions of 0.5 percent (IQR: 0–5 percent). Only one expert believed that the success rate of the version #4 was lower than that of the version #3.
Based on the elicited ‘group’ prior, the median of P3 was 75.9 percent (IQR: 62.2–87.8 percent) versus 80.8 percent (IQR: 61.5–91.3 percent) for P4. The resulting “group” priors on a logit scale for each version and for pooled versions are provided in Supplementary Figure 2, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000403. Based on these distributions, a Normal prior N(1.6, 2.3) was chosen for μ and a Gamma prior γ(Reference Campbell1, Reference Mangubat, Johnson, Keigher and Lopes4) was chosen for ν.
Estimation of the Success Rates of the Device Versions
The hierarchical meta-analysis model was first applied to the complete neuroform stent data using noninformative priors (Table 1). The median posterior success rates of version #2 and version #3 were close, 86.3 percent (IQR: 83.3–88.8 percent) and 86.0 percent (IQR: 81.9–89.4 percent), respectively, whilst that of version #4 was lower, 83.1 percent (IQR: 76.2–87.9 percent) (Table 4).
This result was also illustrated by the 47.9 percent estimated posterior probability that version #3 had a better success rate than the former. This probability dropped to 35.6 percent when considering the superiority in efficacy of version #4 over #3. Actually, based on the model DIC, the random-effect meta-analysis model was the best-fitting model, highlighting the absence of a version effect in efficacy. With this model, the median success rate of the pooled versions was estimated at 85.7 percent (IQR: 83.2–88.2 percent).
The hierarchical meta-analysis model was then applied with the informative ‘group’ priors (Table 4). The resulting posterior distributions of success rates were close for all versions: the median of P2 was 85.7 percent (IQR: 83.1–88.1 percent), that of P3 was 85.6 percent (IQR: 82.6–88.3 percent) and that of P4 was 85.1 percent (IQR: 81.7–88.0 percent). The posterior probability that P3 was above P2 was 48.3 percent, and the posterior probability that P4 was above P3 was 44.5 percent.
Compared with the estimate derived from the noninformative priors, the success rate of P4 had a higher mean estimate (84.5 percent versus 80.5 percent) and a reduced uncertainty (standard deviation of 6.2 percent versus 11.2 percent). Otherwise, the DICs were close regardless of the model.
DISCUSSION
This article describes a case study in the evaluation of an implantable medical device assessing how feasible it is to conduct an elicitation session in a structured manner using a Web questionnaire and to form a probability distribution. An original computer-assisted method of eliciting prior distributions for Bayesian models using expert knowledge was first proposed. Then, we elicited expert opinions using cumulative probabilities to model the imprecision of individual experts as well as the variability between experts. We then presented an estimation method for using this source of elicitation data.
For eliciting the prior probability distribution of the success rate, we used a fixed-interval method though other methods implemented in the various available computer-based tools (Reference Devilee and Knol10) may have been used. Note that some of these methods were directed toward a regression coefficient as the summary to elicit (Reference Denham and Mengersen14;Reference Garthwaite, Al-Awadhi, Elfadaly and Jenkinson15). Otherwise, the variable-interval method in which experts are asked about specified percentiles, mostly the median and the quartiles, of their subjective distribution, could appear an alternate candidate. However, the fixed-interval method was reported faster and slightly superior over the variable-interval method along several dimensions such as monotonicity, accuracy, and precision (Reference Abbas, Budescu, Yu and Haggerty17). Moreover, a clear-cut preference by most participants for the fixed-interval method was also reported and explained by convenience and ease of use (Reference Abbas, Budescu, Yu and Haggerty17). This was confirmed in our survey with 79 percent of the participants who found the questions to be clear and the response options easy to use. Regarding the number of intervals of the fixed-interval approach, we used four intervals whereas such a number varies across studies (Reference O’Hagan11;Reference Leal, Wordsworth, Legood and Blair16;Reference Phillips and Wisbey20;Reference Soares, Bojke and Dumville21). Nevertheless, a higher number of intervals has been reported to be time-consuming and difficult to undertake by respondents (Reference Leal, Wordsworth, Legood and Blair16).
Among the available computer-based tools, the SHELF framework developed by Oakley and O’Hagan (Reference Oakley and O’Hagan22) is of particular interest as it proposes five techniques of elicitation with a R package to fit parametric distributions to experts’ answers. However, this framework relies on a face-to-face interview, thus could not apply directly in our setting. Nonetheless, exploring the possibilities offered by the SHELF framework is an interesting research perspective for distance elicitation which was recently adopted by Sperber et al. (Reference Sperber, Mortimer, Lorgelly and Berlowitz23).
Once the experts have reported their own answers to the four intervals probabilities, the elicitation task was completed by converting these into a probability distribution, using a mathematical aggregation. Other approaches such as behavioural approaches for reaching consensus could have been used to combine individual expert distributions. Nevertheless, no clear benefit of using interactive approaches over individual elicitation methods has been reported (Reference Clemen and Winkler24). Moreover, guidelines for making healthcare decisions recommend not enforcing consensus in expert opinions (e.g., using standard Delphi methods) to appropriately assess uncertainty in parameters (Reference Philips, Ginnelly and Sculpher25). First eliciting the individual beliefs of the experts and then gathering them to gain additional benefits from the exchange of information among them appears promising (Reference Oakley and O’Hagan22).
When using noninformative priors, success rates were found to be similar between versions #2 and #3 but lower for version #4. By contrast, when using informative priors derived from experts’ beliefs, success rates were similar whatever the version. Indeed, as experts were very confident about the success rate of version #4 compared with that of version #3, this translated into a low inter-version variability, bringing the hierarchical model closer to a standard random-effects one. These results suggest that the experts were overly optimistic, so inferences about the IMD based solely on the opinions of experts could be questionable. Moreover, one cannot exclude some conflict of interest with industry, even though only one participant endorsed, that may explain the over-optimistic opinions of the experts regarding the improvement of version #4 over #3. In all cases, respondents are likely a biased sample of the population and this limits the generalizability of our findings. Although the number of participants was greater than the median of the reported sample sizes (n = 11) (Reference Johnson, Tomlinson, Hawker, Granton and Feldman7), the low response rate was the major limitation of our study. Of the 341 contacted authors, thirty-four (10 percent) connected to the Web site. Our recruitment method, that is, asking by e-mail all authors of articles published on intracranial aneurysms, appears inappropriate to the complexity of the task requiring a strong implication of the participants. Three reminder e-mails were sent to all authors but difficulty was probably to gain attention from very busy people. Methods allowing to further motivate the experts, for example by contacting members of learned societies, participants in conferences related to the specialty, or clinicians from all of the neurosurgery departments in Paris hospitals, could have increased the response rate.
Of the thirty-four participating authors, fifteen (44 percent) dropped out at the training exercise level and only nineteen fulfilled the elicitation questionnaire. The length of the questionnaire, as well as the training exercise seemed to have discouraged some of the participants, possibly indicating their difficulty of understanding. However, experts of the “dropout group” did not significantly differ from those of the “elicitation group,” except with respect to their English skills (first language for 42 percent in the elicitation group versus 27 percent in the dropout group) and statistical training (47 percent versus 20 percent, respectively). The remote process of elicitation may thus have selected experts already at ease with probability calculus, illustrating the need for some help file in the elicitation. Besides, such a normative goodness is actually one expertise required from the participants in addition to substantive goodness, so that, although uncommon, it is likely that the selected participants were actual experts.
Among the nineteen experts of the elicitation group, only eleven of them had already used the neuroform stent. This difference in expertise questioned the assumption of the equivalence of experts we made when building the group prior. Indeed, users had less uncertainty about the success rates of both versions and were also more optimistic than nonusers regarding the benefit brought by the latest version. To take into account the difference in expertise, we derived a new “weighted” group prior with the linear opinion pool method by attributing double weight to users compared with nonusers. The resulting weighted group prior did not markedly differ from the unweighted prior and posterior estimates were not affected by the weighting (data available upon request).
In conclusion, we proposed a practical method of eliciting experts’ beliefs about the success rate of the latest version of an IMD with the use of an Internet-based elicitation tool allowing for the estimation of the efficacy of successive versions of the IMD. The proposed elicitation method was easy to implement, with elicitation at a distance that did not require the intervention of a facilitator. A key feature of this work is that all available evidence was included in the evaluation in a transparent way. This method could be applied to many other settings of evaluation, when data are sparse and experts are asked for their opinion.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: http://dx.doi.org/10.1017/S0266462314000403
Supplementary Table 2: http://dx.doi.org/10.1017/S0266462314000403
Supplementary Figure 1: http://dx.doi.org/10.1017/S0266462314000403
Supplementary Table 3: http://dx.doi.org/10.1017/S0266462314000403
Supplementary Figure 2: http://dx.doi.org/10.1017/S0266462314000403
CONTACT INFORMATION
Leslie Pibouleau, MSc, PhD, (l.pibouleau@has-sante.fr), Service Evaluation Economique et Santé Publique, Haute Autorité de Santé, 2 avenue du Stade de France, 93 218 Saint-Denis La Plaine, France.
Sylvie Chevret, MD, PhD, Professor of Public Health, Service de Biostatistique et Information Médicale, Hôpital Saint-Louis, 1 avenue Claude Vellefaux, 75 475 Paris Cedex 10, France.
CONFLICTS OF INTEREST
All authors report they have no potential conflicts of interest.
