Health technology assessment (HTA) is now 35 years old (Reference Banta and Jonsson1). During this time, the science and use of HTA has evolved enormously, adapting to changing healthcare and policy environments. Today, the HTA community has strong methods and procedures to produce HTA (Reference Kristensen and Sigmund5–Reference Lampe and Pasternack7;Reference Simpson, Hiller and Gutierrez-Ibarluzea10), and its results are increasingly used worldwide for decision making. It seems therefore that the time has arrived to set up best practices and to benchmark HTA processes and organizations. The work initiated by Drummond et al. is highly valued, because it opens a new area of needed work following the HTA community's previous emphasis on methodological and process development.

However, one of the challenges in trying to define a set of homogeneous audit criteria across HTA organizations is that one of its unique features is that HTA is context specific. The issue of remit is already pointed out in the study in considering the limitations for compliance with some of the principles. However, it should be a matter of deeper consideration. For example, in Principle 2, the audit question “are the recommendations of the HTA organization made by an independent expert advisory committee?” is not always applicable for some HTA programs, for example hospital based programs. They generally function as components of a hospital's management structure, and face unique demands for timeliness and direct applicability that are not conducive to the use of external advisory bodies. The main challenge here is, as the authors already suggest, finding “a common set of principles and associated audit questions that are common across all jurisdictions.”

Context is also a matter of the question to be answered by the assessment. Reading the audit questions, it seems to me that they are a little biased toward just one of the methods used in HTA, that is, economic evaluation. The main basis of the assessment of any health technology (HT) is, and should continue to be, clinical, showing that the HT produces added clinical benefits (compared with standard of care) without unacceptable harms, ideally in everyday clinical practice. Therefore, an audit criterion dealing with the quality of the clinical information and the process for analyzing it in conducting an HTA (Reference Lampe and Mäkelä6;Reference Lampe and Pasternack7;11) should also be included. Moreover, the performance of a health economic evaluation in an assessment, independently of the remit of the organization, is not always requested by decision makers. There are good examples of HTA produced by National/Regional Agencies that do not include a primary comprehensive health economic evaluation, and which often rely on a review of published studies (3). Therefore, the audit criteria under Principle 5 (“is a full systematic review of clinical evidence required as a basis of economic modeling?”) or under Principle 8 (“does HTA include a sensitivity analysis?”) are not always relevant if no economic evaluation is requested in the analysis. The economic focus observed in the development of audit criteria is also seen in Principle 7 (i.e., a full societal perspective should be considered when undertaking HTAs), wherein the societal perspective is seen from the health economic evaluation point of view and, therefore, all criteria are related to costs. A societal perspective in HTA also considers a more comprehensive set of items including ethical and legal aspects and citizen and patient preferences/perspectives; all very relevant parameters specially for innovative HTs (e.g., personalized medicine, rare disease treatments). Therefore, the audit criteria should be more balanced toward the inclusion of all the components of a comprehensive HTA product (Reference Liberati, Sheldon and Banta8) and process, albeit acknowledging that the applicability of each element may vary according to context.

In developing the audit criteria, the authors argue that “focused questions can be answered in a reasonably unambiguous manner.” However, the audit criteria still leave room for significant ambiguity and need more concrete work. For example, under the audit criteria for Principle 2 (“is the HTA organization independent of the body making the reimbursement or coverage decision?”). Does independency require that the organization assessing the HT has no governmental funding for its work if it will inform reimbursement decisions in a public health system? Notice that, most of the HTA agencies in Europe (and around the world) are publicly funded. Another source of ambiguity is related to the appropriate answer to each criterion. For example, under Principle 2 with the audit criterion “does the organization normally commission outside groups to undertake HTA?” Whether this is desirable may depend largely on the political context in which the HTA organization was created. Therefore, while these audit criteria are descriptively useful, as they are now, provide a limited basis for meaningfully benchmarking HTA processes across national lines.

The authors do acknowledge the importance of context when they point out that “the weights to audit criteria should be given by a representative sample of general population in the jurisdiction concerned.” This is right for public HTA programs, where general public is their ultimate stakeholder, but this would apply much less readily to other HTA bodies, which may not have the general public as their ultimate stakeholder. Furthermore, different weights given by different jurisdictions will probably lead to different scores (i.e., the same HTA organization rated in different jurisdictions would probably have a different overall scores), and this may limit comparability across organizations in different countries. These are more arguments for suggesting that while the development of international principles is a valuable project, the feasibility of their application to some kind of audit or scoring that works across international lines needs more work.

Finally, much as a good HTA process itself requires wide stakeholder involvement, so too does the development of principles and audit criteria to benchmark HTA organizations. The authors already mention the need to better understand differences among HTA organizations in their acceptance and use of principles to revise or reject individual principles. This is not just a matter of understanding, however, and is also a matter of involvement in the definition of principles and audit criteria of those who conduct HTA in real life (i.e., HTA institutions that have to react to questions posed by policy decision makers) and, as already pointed out by experts in the HTA field, a matter of considering previous work done regarding principles by real life doers (Reference Banta2). Furthermore, inputs for elaborating a credible set of principles and audit criteria should also come from representatives of all those who will decide on access to technologies (policy decision makers, payers, managers, and clinicians) and those who will be directly affected by these decisions (patients, citizens and industry) (Reference Hivon, Lehoux, Denis and Tailliez4;Reference McGregor9). A balanced participation of a representative sample of stakeholders would probably minimize any biases and non-ambiguity in audit criteria.

In summary, the work done by Drummond et al. is a good starting point. However, the next step in this work should seek to be both more comprehensive in its approach to HTA and more inclusive (i.e., involving representation from all stakeholders). This broad approach could lead to a set of different principles and audit criteria much according to the real landscape of HTA bodies around the world. It may be that for this next step, a global and neutral space in which all interested and concerned parties in the world of HTs and HTA are represented will be the best place to develop recognized and credible principles and audit criteria. From my point of view, the international Society for Health Technology Assessment (HTAi) should be this place.

CONFLCTS OF INTEREST

The author reports she has no potential conflicts of interest.