Interviews Regarding the PET HTA

Members of the team that produced the PET HTA (n = 11) and six stakeholders—nuclear medicine physicians (n = 2), health economist (n = 1), MOHLTC representative (n = 1), and internists (n = 2)—were interviewed in person or over the phone approximately two months after the completion of the report. All interviews were conducted by one individual and were taped. An interview guide was followed, focusing on the following general areas related to the ICES PET report: (i) general impressions, (ii) motivation for the report, (iii) methodology used, (iv) interpretation of available information, and (v) implementation of the findings of the report. All topics in the interview guide were completed by all interviewees. Tapes of the interview were transcribed in point form, and responses were tabulated into thematic categories.

Choosing Articles for Inclusion—Methodological Purity Versus Clinical Conviction

Results of Interviews

A majority of those interviewed (fourteen of seventeen) labeled the general quality of the literature about PET as poor. Small sample sizes, few Canadian studies, and a dearth of good quality cardiology and cost-effectiveness studies were identified as major drawbacks of the literature. A minority (three of seventeen) believed that poor literature had been inappropriately included in the HTA. Concerns were also expressed that different standards of evaluation had been applied across the different clinical areas. Four respondents believed that cardiology had been held to a higher standard, and two suggested that the report's recommendations about the use of PET for intractable seizures incorporated a lower standard of evidence than was used for oncology and cardiology.

Comments

It is universally agreed that methodologically poor studies often overestimate the benefits of a therapy or the value of a diagnostic test. Many methodologists argue that only Grade A studies should be included in a systematic review (9), because to do otherwise would expose the unsuspecting public to unproven technologies, perhaps at the expense of other technologies for which there is better evidence. On the other hand, clinicians and manufacturers of technologies often argue that it is difficult to conduct high-quality studies early in the life of a new technology and that ignoring a large body of literature suggesting a technology is beneficial unethically deprives the public of access to that technology.

Although no Grade A or B studies were found for patients with intractable seizures, this shortcoming was regarded as understandable because of the small number of patients with such disorders, making it difficult to conduct studies with large sample sizes. Thus, some studies of poorer methodological quality in patients with seizures were considered in the HTA. This decision was also likely influenced by the authors' empathy for patients with this condition—many of whom are very young, face a difficult decision about risky surgery, and have few alternatives. The report concluded that PET scanning should have a limited role in the investigation of patients with intractable seizures being considered for surgery. Most authors felt comfortable with this conclusion because there are few such patients, and the use of PET for this indication would not have large overall resource implications.

At the beginning of the study there were no Grade A or B studies for cardiologic indications. Unlike patients with intractable seizures, there are many patients with severe heart failure, so the paucity of good evidence could not be justified by the lack of patients to be studied. Furthermore, the resource implications of using PET for patients with heart failure are potentially enormous. Therefore, the authors believed that evidence from Grade A or B studies was required before PET could be recommended for patients with heart failure. However, there was considerable pressure from some cardiologists to introduce PET based upon the existing literature. It seemed unwise to dismiss these views with one sentence in the report declaring that the literature was of insufficient quality to be considered seriously. Thus, it was decided to review those Grade C studies (all retrospective) that were believed to be of highest quality, but not to endorse PET scanning for patients with heart failure because of the lack of good-quality studies. While the report was being written, a randomized trial in patients with heart failure was published that found no effect of PET upon mortality, compared with conventional imaging (10). Although the trial had some shortcomings, particularly related to patient selection, this study reinforced the authors' decision not to recommend PET on the basis of the Grade C evidence.

Repeating Previous Studies—Necessity or Waste of Time?

Results of Interviews

Some of the interviewed authors of the ICES report were concerned that performing a new HTA led to unnecessary duplication of previous work. Two of the stakeholders argued that the ICES report did not adequately acknowledge the contribution of the OMA study. Conversely, some of the authors believed that the OMA study was so methodologically flawed and inferior to other HTAs that it would have been a mistake to feature it prominently in the ICES report, despite its being an Ontario-based study.

Comments

At first blush, it seems a great waste of effort for multiple groups to use similar methods to review the same literature about the same technology. However, the ICES authors believed that their conclusions about PET would inevitably be challenged no matter what their findings—by the government if the report suggested that PET was an established technology and should be considered as an insured service and by physicians and patients if the evidence for PET's beneficial impact upon patient care was not yet established. In both circumstances, it would be crucial for the authors to have a thorough grasp of the literature and, in particular, of the merits of individual studies. The authors believed that they could only attain the appropriate level of understanding if they had reviewed the individual studies themselves and that thorough knowledge of an HTA report produced by others would be insufficient. This approach is quite different from that of a “pure” systematic review, which does not need to be repeated unless the previous review was poorly done, or important new information needs to be added to a previous review.

Choosing the Topics to Study

Results of Interviews

Some respondents were concerned that not evaluating PET for conditions for which the literature was poor could delay the future clinical use of PET (six individuals) and curtail future research opportunities (seven individuals). On the other hand, four individuals contended that the report would actually encourage new research into PET.

Comments

The criteria used to delineate the scope of the study are often different for HTAs and traditional systematic reviews. For systematic reviews, the most important features tend to be the amount and quality of the literature and the interest of the reviewer. For an HTA, these factors are modified by the need for a timely report and the need to address issues of importance to policy-makers who can be pressured by industry, patients, and clinicians to introduce a technology in the absence of good-quality evidence. At the same time, the HTA authors must be wary of the financial interest of payers in minimizing the evidence in favor of a technology, if that technology is expected to add considerably to their budget. The pressure to introduce PET, combined with the ICES authors' desire to include some of the topics considered in the OMA report, led the authors to include intractable seizures, dementia, and heart failure in the ICES HTA, despite the overall poor quality of that literature. On the other hand, the need to complete the report in approximately six months contributed to the authors' decision not to review PET for the evaluation of coronary artery disease.

It is interesting that several interviewees believed that, by not including potential indications for which the literature was poor, the report might inhibit the future use of PET for those indications. This finding seems to reflect a lack of confidence in the degree to which evidence is used in decision making. In the ideal world, the literature would be constantly updated, and once evidence for the usefulness of PET for a “new” indication is available, PET would be introduced for that indication. However, this rarely occurs. Some of those interviewed suggested that they were worried that decision-makers might approve PET for the few indications mentioned in the report and then use the report in the future to deny PET for other indications. Their concerns may be valid, but it is not clear that including poor-quality evidence in the report would remedy this problem. However, there can be a “Catch-22” associated with the evaluation and introduction of an expensive technology such as PET. Payers would like high-quality evidence before PET scanning diffuses into clinical use, yet high-quality evidence cannot be generated without PET scanners! If the government only funds PET scanners for indications that have been established, Ontario researchers will never be able to contribute meaningfully to the “cutting edge” PET literature.

Should Content Experts and Policy-Makers Be Included as Authors?

Results of Interviews

Those interviewed appeared to be satisfied that the report was unbiased—thirteen respondents indicated that there was no or little bias in the selection of the literature. Nine interviewees approved of the decision not to include nuclear physicians as authors, commenting that their omission decreased the potential that the report would be biased in favor of PET. On the other hand, two argued that nuclear physicians should have been more involved. One believed that the authors had been “creative” in their use of terms describing PET, a feature that might lead informed readers to question the authors' expertise in PET. The other person suggested that there was no inherent disadvantage to an advocate's potential contributions:

Some of the authors indicated that their lack of knowledge about PET had necessitated a “steep learning curve” when writing the report. The majority of those interviewed believed that not including decision-makers as authors had been reasonable. Two individuals believed that policy-makers should have been more available for feedback during the preparation of the report. One worried that having no MOHLTC representation on the committee might decrease the Ministry's commitment to implement the recommendations. Another individual suggested that hospital administrators should have been invited to the stakeholders meeting.

Comments

Systematic reviews often include content experts as authors but rarely include policy-makers. HTAs include none, one, or both. It is important to recognize that all authors are biased. Some consistently favor methodological purity over clinical common sense, whereas others appear driven by a desire to introduce a new technology no matter what the evidence or cost. Most lie between these two extremes, and there can be no universal rule about who to include as the authors of an HTA, aside from attempting to have a team of authors that can provide a balanced summary and interpretation of the literature. By including neither decision-makers nor nuclear medicine physicians as authors, ICES reasoned that the two groups with the largest potential biases would be treated equitably. An HTA of PET from Quebec took a different approach to ICES by including both nuclear medicine physicians and policy-makers on its authorship team (3).

In retrospect, hospital administrators represent an important constituency that was not included in the stakeholders meeting, because hospitals will have to provide considerable funds to support the capital and operating expenses of a PET scan. It is also worth noting that the Ontario and federal governments wish to attract high-technology industries to stimulate economic growth. Their interest in PET technology extends far beyond the issues of diagnostic accuracy, impact upon outcomes, and cost-effectiveness considered in the ICES PET report, to issues of societal innovation and economic growth. No government or industry persons representing these views were asked to comment on the ICES report, which illustrates how HTA can still function in a “health-care silo.”

When to Perform an Economic Evaluation

Results of Interviews

The vast majority of those interviewed believed that the ICES report had dealt with the existing cost-effectiveness literature appropriately, although one stakeholder believed that the authors should have conducted their own economic evaluation.

Comments

An estimate of cost-effectiveness and overall budget impact is important for decision-makers and, thus, is included in most HTAs. However, this is usually not a focus of systematic reviews. It is generally agreed that evidence about diagnostic accuracy is likely to be generalizable across countries but that clinical practice and the costs of resources are not. However, the major limitation of the published economic evaluations of PET was not the lack of information about Ontario resource use, but the poor quality of the information about PET's effect upon patient outcomes, without which it is impossible to accurately estimate cost-effectiveness.

The authors decided not to conduct their own economic evaluations, because their systematic review of the literature found no convincing evidence about PET's effect upon outcomes. They believed that the results of economic evaluations based upon poor clinical evidence would not be worth the considerable effort needed to produce them.

Should the Authors of an HTA Make Policy Recommendations?

Results of Interviews

Most interviewees believed that the ICES approach was reasonable, although one individual worried that the lack of clear policy recommendations might have diminished the potential influence of the report.

Comments

It is likely that policy-makers pay more attention to an HTA report if it actually suggests a course of action (e.g., “Ontario should introduce eight PET scanners during the next two years”) than if it simply provides a systematic review of the literature. The problem with the former approach is that the authors of an HTA rarely, if ever, are aware of the evidence about all the other technologies or programs that could be funded instead of the one they have evaluated. At the time the ICES PET HTA was being written, there was considerable concern about excessive waiting lists for radiotherapy and elective hip and knee arthroplasty, over-crowded emergency rooms, and lack of adequate home care in Ontario. Thus, it seemed inappropriate for the ICES authors to suggest that PET should be introduced, when they had not compared PET with other competing demands on scarce resources. In addition, the PET authors were not in a position to balance the “medical” aspects of PET with the possible impact of investing in PET technology on the economy of Ontario. The authors hoped that describing the number of patients who would be eligible for PET for various clinical indications would provide decision-makers with useful information about the amount of resources needed if PET would be introduced for those indications.

Dissemination of an HTA Report

Results of Interviews

Many of those initially interviewed were pessimistic about the report's likely impact—eight individuals indicated that they were unsure or did not think that the report would have any influence on policy. However, during the year after publication, individuals within the Ministry of Health have used the report to push for the implementation of PET in a limited context, accompanied by research and evaluation (7). Two decision-makers in the Ministry of Health were interviewed approximately a year after the publication of the report to determine whether the HTA had influenced Ministry policy. In general, they approved of ICES's hands-off approach to advocacy, although one thought that ICES might have used even weaker language in its report about the usefulness of PET.

Comments

Because simply publishing a report with no active dissemination often has little impact upon policy, some may view publication without any dissemination efforts as an abdication of the authors' responsibility to have their findings incorporated into decision making. This criticism is more commonly aimed at HTAs than systematic reviews, because HTAs are expected to influence policy. On the other hand, an HTA team that is too aggressive in pushing its findings may alienate decision-makers, which can be counterproductive. As well, it is not always clear who the true decision-makers are. Individual decision-makers change frequently, they can be hard to reach, and active dissemination is time consuming and expensive. Thus, the appropriate approach to dissemination varies markedly, depending on the circumstances. Despite the most passive approach to dissemination possible, the ICES report still appeared to have considerable influence (7). This experience illustrates the importance of a report reaching the right decision-makers at the right time.

FINAL WORDS

Unlike systematic reviews, HTAs are not solely concerned with evaluation of the scientific evidence. By using an actual HTA as an example, this study has explored seven key differences between an HTA and a systematic review (Table 1). These relate to methodological standards, repeating of previous studies, breadth versus depth, inclusion of content experts and policy-makers, performance of economic evaluations, making policy recommendations, and active dissemination.

HTAs cross the ideological divide between scientific investigation and political decision making, and authors of HTAs should recognize that policy-makers consider “impartial” investigators as political stakeholders too—stakeholders in the scientific community, which has its own set of values and priorities. Just as a court of law weighs the words of an “impartial” expert witness with the testimony of those more emotionally entangled with the details of the case, so too the verdict on the dissemination of a health technology ultimately encompasses scientific evidence, competing social and professional priorities, and the public's views.