The health technology assessment (HTA) Core Model, hereafter referred to as the Model, is a framework and tool for HTA production. It was originally developed within the EUnetHTA project in 2006–2008 and further built up during EUnetHTA Joint Action (JA) 2010–2012 (Lampe et al. in this issue). The Model provides potentially relevant research questions to guide the content of an HTA, guidance on assessment methods to help answer the research questions and reporting templates (Reference Lampe, Mäkelä and Velasco Garrido1). The content of an HTA is divided in nine domains: the Health problem and current use of the technology, Description and technical characteristics of the technology, Safety, Clinical effectiveness, Economical, Ethical, Organizational, Social, and Legal aspects domains. Each domain is further divided into a set of generic questions, which are the assessment elements of the Model. When using the Model, the generic questions are transformed into actual topic-related research questions. For each assessment element there is a reporting template called a result card. Result cards present the question, the methods used to find the information, the analysis and synthesis of information for answering the question, the answer itself, and references. The result cards can be stored in an online repository and form a collection of HTA information also referred to as Core HTA information.
The aim of splitting the production of HTA into a set of questions and answers is to facilitate easy sharing of information and work across individuals and organizations, which, in turn, may reduce duplication of efforts in HTA production. When preparing a national HTA report one could easily locate and use relevant pieces of information in the Core HTA information database. Clear and transparent reporting of methods in the result cards enables evaluation of the validity and relevance of the information for one's own setting. The cards are in English and the authors are instructed to report the results in such a way that they have relevance beyond their own national or local context. Core HTA information contains no judgments or recommendations, but only the factual information required to make a judgment or recommendation. Functionalities of the Model have been tested in several pilot HTA projects and the feedback is used to further develop the model (Reference Pasternack, Anttila and Mäkelä2). Comparing the Model content and structure with a traditional national HTA report has not been reported before.
In order for an HTA agency to make best use of the Core HTA information, the content of the Model should, as much as possible, correspond to the content of a national HTA report. In a perfect match the result cards cover all relevant aspects required in a national report and one result card can be adapted as one section or paragraph in a national report. If a certain section in the national report requires information from several result cards, adaptation requires more effort. This also occurs if some information is missing in the result cards, and the authors need to perform additional work to complete the national report. Furthermore, the list with assessment elements of the Model can help assessors to point out relevant research questions which otherwise may not have been considered for the national report.
The aim of this single case study was to examine the applicability of the assessment elements of the HTA Core Model for national HTA report production by evaluating the correspondence of content and distribution of information. By comparing the distribution of information we mean how the information is ordered and grouped in the Model compared with the national HTA report: that is, is the information belonging to one assessment element in the Model presented in one paragraph, or is it distributed in several paragraphs in the national report. The work was done as part of the development and testing of the Model during EUnetHTA JA.
METHODS
Information from a national report and the HTA Core Model were gathered for this case study in 2012. At that moment a draft HTA report of the National Health Care Institute – ZIN on endovascular repair (EVAR) of abdominal aortic aneurysm was available and selected as the national HTA report (Reference Groot de, Ligtenberg and Vijgen3). An ongoing instead of a completed report was selected, because it was foreseen that the report authors would better be able to evaluate the relevance of the Model in respect to their report, than authors who had completed the assessment some time ago. Moreover, the check list function of the Model could be tested in a real life situation, that is, whether the list of assessment elements helped the authors to include all relevant themes in their final report.
The titles of the main sections of the EVAR report are presented in Figure 1. The structure is typical for a ZIN report belonging to the series named “Rapporten en Standpunten.” The series includes regularly published reports (20–40 per year) which aim to advise the Ministry of Health, Welfare, and Sports and/or health insurers about the impact of including medical technologies in the universal healthcare insurance package. ZIN uses standard scientific methods and procedures for producing HTA reports. A reporting template with defined sections and defined subtitles for paragraphs within those sections is used to guide the report authors: some of the subtitles are obligatory and some modifiable.
Figure 1. Content relations of the EUnetHTA Core Model and a Dutch national HTA report (endovascular repair of aortic aneurysm, EVAR, by ZIN).
The 113 assessment elements of the Model (as of March 2012), presented by generic questions (4) were evaluated for relevance by the author of the EVAR report who had brief experience of preparing ZIN reports and no previous experience with the Model. The content of the draft EVAR report was analyzed by an experienced Model user who annotated the paragraphs and sentences in EVAR report with corresponding assessment elements. Paragraphs with no annotations were tagged as “national information.” Both analyses were discussed by all the authors of this article.
RESULTS
Comparison of the Content
Of 113 generic questions presented in the assessment elements of the Model 45 (40 percent) were considered of interest for the national assessment of EVAR. The economics domain contained the highest percentage of assessment elements (100 percent) that are of interest and the domain on ethical aspects the lowest (6 percent). Of the forty-five questions that were considered of interest, twenty-nine (64 percent) were already covered in the draft EVAR report, at least partly (Table 1). The importance of two assessment elements of the Model, one from ethical and other from organizational domains were interpreted differently in the Model and the EVAR report: the author of the EVAR report had marked the assessment elements as “not-of- interest,” but still included respective information in the EVAR report.
Table 1. Relevance of the Assessment Elements of the HTA Core Model for the EVAR Report
EVAR, Endovascular repair of aortic aneurysm.
There were sixteen generic questions in the Model that did not have the corresponding content in the EVAR report, but which the authors of EVAR report regarded of interest. These were mostly from the organizational and economics domains, including questions about material investments, training requirements, centralization requirements, indirect costs, and budget impact. Two issues that were missing in the first draft (development of the technologies and information of patient acceptance) were added to the subsequent versions of the EVAR report. There were four content issues in the EVAR report which had no clearly corresponding assessment elements in the Model: information about re-interventions (e.g., laparotomies performed due to bleeding or hernia after EVAR), secondary prevention, identification of the subpopulation who benefit most, and the length of the hospital stay.
There were some policy related sections and general methodological sections in the EVAR report which could not be annotated to correspond with any of the assessment elements of the Model. For example, the HTA reports of ZIN typically define the type of evidence required for the assessment (e.g., acceptable study designs of the original studies), the stakeholders’ comments, and the final conclusion with advice or recommendation.
Comparison of the Distribution of Information
An overall picture of the content relations of the HTA Core Model and the EVAR report is presented in Figure 1. There were five instances where the content of one single paragraph in the EVAR report corresponded with one single assessment element of the Model. They were: incidence and prevalence of the disease, symptoms of the disease, burden of disease to the patient, and unit costs of implementing EVAR. In the EVAR report, twelve paragraphs were identified that contained information from multiple (two to seven) assessment elements. In six of these cases, information from various domains were combined. For example, information from a section called “endovascular treatment” contained information that could be traced back to assessment elements from four domains. Vice versa, two assessment elements contained information that was dispersed over different sections in the EVAR report.
DISCUSSION
Forty percent of the assessment elements in the Model were considered of interest for the national EVAR report. Four issues that were covered in the EVAR report were not included in the assessment elements of the Model. Furthermore, the distribution of information between the Model and the national report differed. As this was the first case study of this kind we do not know if these percentages reflect the HTA reports in general, however, as ZIN uses standardized templates we think the results can be considered a good indication for this type of ZIN reports.
The list with assessment elements from the HTA Model seems rather comprehensive as only a few issues from the national report were missing. On the other hand, the percentage of relevant assessment elements for the national EVAR report was rather low in most domains. For the safety and effectiveness domain the percentages of assessment elements considered relevant was quite low, 31 percent and 50 percent, respectively, although safety and effectiveness are typically considered very relevant for a national HTA report. This may hint over-representation and one could question whether all assessment elements in these domains are really necessary to be included in the HTA Model. On the other hand, the list with assessment elements is intended to be exhaustive for different kind of assessment, so downsizing the number may compromise this aim. Comparison with reports on other topics and from other countries would help identifying the real necessity of the assessment elements in these domains.
A general notion was that the Model does not yet clearly distinguish the specificities of assessing devices. For example, for EVAR it would have been relevant to highlight the difference of the safety of the stent alone versus the full endoscopic intervention.
Four relevant issues in the EVAR report had no clearly corresponding assessment elements in the Model. The persistence of the health benefit after operation and the amount of re-interventions required were important outcomes to be compared between EVAR and the comparator, open repair of abdominal aneurysm. These were not clearly covered in any of the assessment elements of the Model. We think they should have their own assessment element. Re-interventions probably are relevant to the two first descriptive domains, as well as to the effectiveness and organizational domains, as they affect treatment pathway, disease progression, and resource usage. As it is possible to include one assessment element in several domains we propose to add an element in both the clinical effectiveness and organizational aspects domain, with a generic question asking “How permanent is the result and what is the rate of re-interventions using the technology versus the comparator?” Length of hospital stay is an important outcome when comparing the operative treatments of abdominal aortic aneurysm which could not be attributed to one of the existing assessment elements. It can be considered from different points of view (e.g. patients, organization or society). We propose that the current assessment elements in organizational and social domains could be rephrased to address more specifically the work flow and patient flow related to hospital stay. Also “what kind of information should be given to the patients to prevent the progress of and the rupture of the aneurysm of the aorta” could not be attributed to one of the assessment elements. We think that adding an assessment element such as “What kind of patient counseling is needed to prevent the progression or relapse of the disease?” would be worthwhile, considering that there are interventions with low effectiveness where much could be achieved by secondary prevention. Similarly, we propose adding an assessment element “How to prevent the condition” to the Model. Finally, information about specific subpopulations, in which the intervention works better or worse, seemed to be missing in the Model. A new element such as “Are there subgroups that benefit more/less from the intervention?” would remind the HTA doers to consider not only age, co-morbidities and disease severity, but also how genetic disposition might affect differences in effect.
The level of detail in the assessment elements, that is, the “grain size” or granularity of information, is crucial for easy adaptation of HTA information. If the generic questions of the assessment elements of the Model are very broad the whole idea of structuring HTA and splitting its content into single questions and answers is not valuable anymore. On the other hand, if the assessment elements are too specific and detailed, it may be difficult to identify, and also produce, all the important result cards. The issue of granularity, or level of detail of the assessment elements, has been discussed by the Model developers repeatedly as there are pros and cons for both small and large grain size. It seems like the grain size of information is larger in the EVAR report than in Model: there were more instances where a paragraphs of EVAR report merged information from several assessment elements of the Model, than the opposite where one assessment element is distributed into several paragraphs of the EVAR report. An example of the latter is the risk factors issue. Instead of a single assessment element called “What are the known risk factors for the condition?” the author of the EVAR report proposed having two separate elements for the risk factors for acquiring the aneurysm and the risk factors for worsening the condition or rupture to better show their importance for primary and secondary prevention.
Differences in reporting standards affect the adaptability of HTA information too. The EVAR report presented summaries of original studies and did not intend to analyze the results per outcome across individual studies. It is, therefore, obvious that each paragraph describing one trial contained information from several result cards, as different endpoints such as mortality or quality of life are organized per results card in the Model. However, with Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology (Reference Guyatt, Oxman and Vist5) becoming more frequently used by HTA agencies this mismatch may be less likely in the future, as GRADE also presents results per outcome.
Some of the assessment elements would require further refinement to make them more explicit and understandable. The author of the EVAR report deemed some of the assessment elements of the Model as not of interest for the report although the corresponding content was in the report already. This occurred in questions related to ethical and organizational issues, and may reflect the fact that the generic questions and their clarifications are not formulated clearly enough and that these areas are less familiar to the assessors.
The assessment reports of ZIN contain a value judgment of the available evidence and advice for decision making. These sections are by default missing in the Model: Core HTA information should not include judgments, and definitely no recommendations on the use of the technology (Reference Lampe, Mäkelä and Velasco Garrido1). Advice for decision making is considered a national task which cannot be adapted directly from the repository of Core HTA information.
A limitation of this study is that the results represent the status of the moment of the Model and the EVAR report. Both have evolved since, but the changes made in the subsequent versions of both are not tracked and reported in this study. This may lead to recommendations by the authors that already have been implemented. Another limitation is that this study focuses only on the content and distribution of information in the report. However, other aspects, such as methodological approach can affect the adaptability as well.
As the observations rise from one case only, the results presented are for illustrative purposes only, and not for suggesting actual changes in the Model. For the same reason, the results are not generalizable to HTA reports from other agencies, or maybe even to other HTA reports of ZIN. The aim was rather to explore potential “red flags” regarding missing elements, or inappropriate sequence and granularity of information, which could be used directly in the Model development. Spreading knowledge of the HTA Core Model in national HTA agencies and collecting early feedback of the adaptability of structured HTA information was considered an additional benefit to motivate this study.
We conclude that most issues that are relevant for a ZIN report are covered by the assessment elements of the HTA Core Model. The Model served as a check list for identifying and reminding of relevant questions for a national report. Matching the level of detail and sequence of information in Model with national reports could further improve the adaptability of the HTA information produced by the HTA Core Model.
CONTACT INFORMATION
Iris Pasternack, MD, founder of Summaryx Ltd., Untuvaisentie 2 A 19, 00820, Helsinki, Finland
Ingrid de Groot, PhD, assessor Zorginstituut Nederland, Eekholt 4 Diemen, 1112 XH, The Netherlands
Sarah Kleijnen, MSc, project coordinator, Zorginstituut Nederland, Eekholt 4 Diemen, 1112 XH, The Netherlands
Paula Polman, MSc, MPH, manager Zorginstituut Nederland, Eekholt 4 Diemen, 1112 XH, The Netherlands
CONFLICTS OF INTEREST
Iris Pasternack and Sarah Kleijnen report grants from European Commission to their institutions. Iris Pasternack also reports travel expenses covered by ISPOR The other authors report they have nothing to disclose.
