HTA has been defined as “a multidisciplinary activity that systematically examines the technical performance, safety, clinical efficacy and effectiveness, cost, cost-effectiveness, organizational implications, social consequences, legal and ethical considerations of the applications of a health technology” (Reference Granados, Jonsson and Banta5).
Recognizing the need for comprehensive HTAs that can support decision making, different suggestions has been developed on how best practice, in undertaking and reporting of HTA, can be achieved (Reference Kristensen and Sigmund8;Reference Velasco, Perleth and Drummond15). Five main, and equally relevant, types of outcome have been recommended: safety, efficacy/effectiveness, psychological/social/ethical, organizational/professional, and economic (Reference Velasco, Perleth and Drummond15). In Denmark, there is a tradition for conducting HTAs within a framework of four key elements: the technology (clinical evidence), the patient, the organization, and the economy, each with several underlying areas (Reference Kristensen and Sigmund8).
The assessment of organizational and patient-related preconditions for and consequences of a technology can be a significant factor in ensuring that the HTA is useful as a policy tool (Reference Johri and Lehoux7;Reference Lehoux and Blume10) as: (i) Health technologies often do influence and are influenced by current organizational structures, daily staff routines and work practices, educational requirements, and job satisfaction (Reference Banta and Luce1;Reference Liberati, Sheldon and Banta12). (ii) Patients’ attitudes and experiences with a health technology can be highly relevant for the implementation and effects of the technology (Reference Hansen, Kristensen and Sigmund6).
Studies of HTA practice indicate, however, that this comprehensiveness is seldom present, with many HTAs failing to include assessment of issues which can be classified as organizational and patient-related (Reference Draborg, Gyrd-Hansen, Poulsen and Hørder2;Reference García-Alrés, Ondategui-Parra and Neumann4;Reference Lehoux, Tailliez, Denis and Hivon11;Reference Poulsen14) The development of guidelines for the inclusion of organizational and patient-related issues in HTAs would be assisted by the existence of a general view of how these issues are currently assessed in HTAs (Reference Fulop, Allen, Clarke and Black3;Reference Perry, Gardner and Thamer13;Reference Poulsen14). A study was thus initiated to examine current practice for including and managing organizational and patient-related assessments in HTAs, and to review comprehensive assessments that could assist in developing guidelines for “best practice” (Reference Lee and Sinding9).
The objective of this study is to discuss the content and managing of the organizational and patient-related assessments included in full HTAs.
DESIGN AND METHODOLOGY
An approach was chosen in which organizational and patient-related issues were considered to be distinct from each other and from technological/clinical and economic issues.
Organizational issues are here seen as the organizational dimensions which are affected by (and which themselves affect) the implementation and use of a technology. The study focus is on organizational dimensions and the organizational context of a technology rather than a policy analysis of a technology and its policy processes. The approach comprises an analysis of the management and structural dimensions of the technology and of the social and behavioral processes related to the use of the technology, as well as the preconditions for these. These various organizational dimensions are both dynamic and often interlinked. The organizational issues have thus been reviewed under four main categories: structure, process, control/evaluation, and culture/environment, each which has then been divided into several subcategories.
Patient-related issues are here seen as patients’ perspectives on the use of a technology. The term “patient” encompasses all potential end-users of the technology in question and covers a continuum from the individual patient, to a group of patients or members of society. The patient's combination of experiences, perceptions, expectations, and actions in relation to a technology is a bodily experience interrelated with the individual's history and life situation, and is, therefore, different from the professional's perspective. The patient-related issues have thus been reviewed under six main categories: patient information, psychological aspects, effect of the technology (e.g., on quality of life), social issues, ethical issues, and impact on the patient's financial situation. Some categories were broader than others and were thus divided into subcategories.
Data Generation
HTA reports was identified from INAHTA members’ Web sites where a total of forty-one agencies from twenty-one different countries were listed, including a profile of each member agency and a link to its Web site. Due to language barriers, it was only possible to include agencies that had English or Scandinavian language versions for their Web sites and publications. On this basis, seventeen agencies (ASERNIP-S, MSAC, AETMIS, AHFMR, CCOHTA, DACEHTA, DSI, GR, NZHTA, SMM, CMT, SBU, CRD, NCCHTA, NHS QIS, AHRQ, and VATAP) from ten countries (Australia, Canada, Denmark, England, the Netherlands, New Zealand, Norway, Scotland, Sweden, and the United States) were included in the initial list for review. Each of these Web sites was searched by the two reviewers to identify full HTA reports (excluding early warnings and rapid assessments) published in the period January 2000 to July 2005.
A total of 382 full HTA reports were identified. Of these reports, 146 (38 percent) included any organizational and/or patient-related issue identifiable in the abstract, the table of contents or the description of the purpose of the study. Due to limited resources fifty of these reports were selected, by simple random selection in the software program SPSS, as the final sample for the literature review. A list of the fifty reports is available by request to the lead author.
Data Assessment
Quantitative and qualitative analysis were performed based on a thorough review of organizational and patient-related assessments included in a random sample of 50 full HTA reports. To ensure a broad and uniform assessment of the reports over time, each report was assessed using a checklist specially developed for the purpose (Reference Lee and Sinding9). The checklist consisted of questions relating to (i) the general description of the HTA and the technology in question, (ii) the content of organizational issues, (iii) the content of patient-related issues, and (iv) the methodology of the HTA. Each question required both a quantitative answer (yes/no) and a qualitative answer in which the reviewers elaborated on the content of the issue, the data source and the way in which the issue was assessed (managed).
The two reviewers independently read and reviewed each of the fifty HTA reports. Apart from the common sections of a report (e.g., summary, introduction, methodology, discussion, conclusion, recommendations, synthesis, and appendix), attention was paid only to sections that related to organizational and/or patient-related issues. Any findings that clearly related to technological/clinical or economic issues, apart from the sections mentioned above, were ignored. After the independent assessments, the two reviewers discussed each report and an electronic checklist was filled in consensus.
RESULTS
A search among seventeen HTA agencies from ten countries showed that 38 percent of the published full HTA reports included any organizational and/or patient-related issue. A random sample of fifty reports including organizational and/or patient-related issues showed that all the reports included technology issues, while forty-three (86 percent) included patient-related issues and forty-two (84 percent) included economic and organizational issues, respectively. Thirty-one (62 percent) of the reports included all four elements. Of the eight reports not including any organizational issue, two stated a reason for not doing so. Of the seven reports not including any patient-related issue, two stated a reason for not doing so. Approximately half of the fifty reports included for the review were undertaken by Scandinavian INAHTA members. Whereas this may reflect a greater tendency of Scandinavian HTAs to include organizational and patient-related assessments, it may also be a result of biases in generating the study sample. However, a statistical comparison (Chi goodness-of-fit-test) of the distribution of countries in the net list of 146 reports, including organizational and/or patient-related issues, and in the study sample of 50 reports, showed no significant difference. The relatively large Scandinavian representation in the sample thus appears to be partly a result of the selection process, but may also be due to differences in HTA practice in different countries. Table 1 provides an overview of countries represented in the study sample and in the net list.
Table 1. INAHTA Member Countries Represented in Study Sample (n = 50 reports) and in Net List (n = 146)
Methodology of the Assessments
The extent to which the HTAs reported on key methodological issues for the assessments of organizational and patient-related issues differed. In half of the fifty HTA reports, the generalizability of the results to other contexts was discussed, although typically for the report as a whole rather than for the organizational or patient-related assessments separately.
In terms of study design, 80 percent of the HTAs were based on a review of existing literature, either alone or in combination with other designs, mainly case studies but also comparative studies. As presented in Table 2, secondary data were used in eighty-one assessments, that is: in forty-one of the forty-two HTA reports including organizational assessment and in forty of the forty-three HTA reports including patient-related assessment. For thirteen of the assessments, there was no description on how secondary data were generated. A systematic literature review was used in sixty-four of the sixty-eight assessments where the method of data generation was described. In more than half of the sixty-eight assessments, the analytical method was described, whereas two-thirds of the assessments included a critical discussion of the relevance and validity of the included data.
Table 2. Generation of Data in HTA Assessments of Organizational and Patient-Related Issues
Primary data were used in forty-four assessments: in twenty-six of the forty-two HTA reports including organizational assessment and in eighteen of the forty-three HTA reports including patient-related assessment. A quantitative approach was used in twenty-nine of the forty-four assessments using primary data, twenty-six generating data from questionnaires, eight from registry, and one from preference instruments. In more than half of the assessments, the analytical method was described as half of the assessments included a critical discussion of the quality and interpretation of data.
A qualitative approach was used in thirty-one of the forty-four assessments. Although there was no description on data generation for five of the assessments, thirteen comprised individual interviews, eleven text documents, nine focus group interviews, three prospective methods, and two direct observations. In more than half of the assessments, the analytical method was described, whereas one-third of the assessments included a critical discussion of the quality and interpretation of data.
Organizational Issues Included
Of the forty-two reports that assessed organizational issues, 95 percent stated a purpose for doing so and 33 percent included one or more research questions to be addressed in the assessment. The variety of organizational issues included in the assessments is presented in Table 3. All forty-two reports assessed issues related to process: these were predominantly related to various actor group and organizations involved, as well as to staff numbers and skills, and to work flow. Issues related to interaction/communication and potential barriers/bottlenecks were included in more than half of the reports. Of the forty-two reports, 93 percent assessed issues related to structure: these were mainly assessments of physical, resource, and legislative structures, and of diffusion of the technology. A smaller number of reports assessed economic consequences and incentive structures. Of the forty-two reports, 81 percent assessed issues related to control and evaluation of the technology: these related mainly to control and evaluation systems, with fewer reports including issues related to the responsibility for these systems. A little over half the forty-two reports assessed issues related to culture and environment: these related mainly to cultural factors. Issues related to physical and psychological working environment were less often included, while assessment of impact on the outer environment was absent.
Table 3. Organizational Issues Assessed in HTA Reports (n = 42)
Patient-Related Issues
Of the forty-three reports that assessed patient-related issues, 93 percent stated a purpose for doing so and 40 percent included one or more research questions to be addressed in the assessment. The variety of patient-related issues included in the assessments is presented in Table 4. Of the forty-three reports, 77 percent assessed psychological issues: these related mainly to patient worry and discomfort while a smaller amount of reports included issues of patient satisfaction and patient involvement in decisions related to the use of the technology. A total of 72 percent of the forty-three reports assessed ethical issues: these were related to patient acceptance and, to a much lesser extent, general public acceptance of a technology. A small number of reports assessed specific ethical considerations. Seventy percent of the reports assessed social issues: these related mainly to the technology's impact on the patient's daily life. A smaller number of reports assessed implications for the patient's significant others and for the patient's ability to work. Of the forty-three reports, 70 percent assessed patients’ perceptions of the technology's effect on their health, mainly as quality of life assessment. A total of 67 percent of the forty-three reports assessed issues related to patient information, and 28 percent assessed issues related to the patient's financial circumstances in relation to the use of the technology.
Table 4. Patient-Related Categories Assessed in HTA Reports (n = 43)
Managing the Included Issues
As types of issues included in the organizational and patient-related assessments varied, so did the way in which the issues were managed. The assessments were in most reports restricted to some description of the included issue(s) while the issue in a few reports just was mentioned. Other reports included comprehensive assessments with in-depth descriptions and discussions of the included issues, of which three examples are as follows:
(i) The first is an HTA on Diabetic retinopathy screening. The HTA included considerations of a specific training program for the professionals including purpose and expected outcome of the program, structure of the program and participants as well as supervision and quality assurance. Also the patient population and goals for different kinds of information and education were considered. Patient views and experiences were sought and discussed in relation to preferences and satisfaction with information. Proposals for information form, content, informant, and timing were given and barriers to the provision and receipt of information discussed.
(ii) The second is an HTA on Perinatal intensive care. The HTA included a discussion of the expected future demand for the technology in relation to the level of supply, and what consequences this might have for access to the technology, patient care, and required capacity of intensive care cots and staff. Decision processes in situations involving uncertainty and value judgment differences between health professionals and patients were discussed. How decisions can be supported, benefits for and involvement of the patients (in this case parents), as well as international differences in patient involvement in decision making were discussed and policies and guidelines for optimizing this practice outlined.
(iii) The third is an HTA on Home tests for Chlamydia. In the HTA, the purpose of screening and its future organization, for example, changes which should be implemented were discussed. A comprehensive description of all the different elements of the work processes related to the use of the technology was presented. For each element in turn, there was consideration of why the element was important, how and by whom it could be conducted, and how much time/resources were needed for that part of the process. Preconditions for general acceptance of the technology were discussed including the nature of the specific disease and the trustworthiness of the test results. Also ethical concerns in relation to effects on cultural norms, and possible approaches to reduce religious barriers to the use of the technology were discussed.
DISCUSSION
For a full HTA to function as a decision-making tool, it needs to be comprehensive and to take a broad perspective relevant to the policy question. It is important, therefore, to consider if inclusion of an assessment of the potential organizational and patient-related issues is relevant to the specific technology under study. Eventual inclusion of organizational and/or patient-related assessment in an HTA, choice of issues, the way they should be managed and which methodology to use in their assessment, therefore, depends on conditions such as the purpose of the HTA, the research questions and the economic basis for performing the HTA.
The present review finds, in according with former studies, that inclusion of organizational and patient-related assessments in HTAs is less common than inclusion of technological/clinical and economic assessments.
The review also finds that only in a minority of the reviewed reports the assessments are based on stated research questions to be addressed in the organizational or patient-related assessment.
The review shows that when organizational assessments are included they mainly are concerned with issues such as; which actors and organizations are involved in the use of the technology; work flow; staff numbers and skills; and physical, resource, and legislative structures. Issues related to organizational culture, communication, and the physical and psychological working environment are less often included. When patient-related issues are included, these are mainly concerned with issues, such as worry and discomfort, impact on the patient's daily life and quality of life, patient acceptance of the technology, and patient information. Issues related to patient involvement in decision making and impact of a technology on the patient's personal economy and on their significant others are less often included.
Some of the organizational and patient-related assessments were broad, including a variety of issues, but often these issues were managed in a rather restricted and superficial way and the usefulness to decision making can, therefore, be questioned. More comprehensive assessment of the included issues was less often performed but providing knowledge that can be useful in deciding whether or not to implement a technology, and in planning the implementation strategy of a technology.
There is also room for improvement in relation to reporting on the methodology applied in the assessments. In some reports, the methodological choices were both described and discussed allowing the reader not only to follow the methodological steps taken but also why and what consequences they had for the study findings, including the generalizability of the results.
Many of the HTA reports included a literature review as the only study design and some did, regardless of the at times limited result, not explicitly discuss this choice in relation to the to the purpose and perspective of the study. The choice of study design thus appeared in some cases to follow a generally accepted approach rather than involve consideration of the most appropriate design tailored to the individual assessment.
Absence of description of the considerations made when determining whether to include organizational and patient-related assessments as well as absence of considerations related to content and methods limits the usefulness of an HTA. The reader is left uncertain of the relevance and validity of the organizational and patient-related assessments—was the relevant perspective chosen, were the relevant issues included and were they assessed using a relevant methodology?
LIMITATIONS TO THE STUDY
While the approach chosen for the study provides a useful insight into current practice in relation to organizational and patient-related assessment in HTA, there are several limitations: The study is descriptive in nature and no comparative analysis has been conducted. The review only includes a sample of fifty HTA reports published by INAHTA members and in Scandinavian or English languages. A disadvantage of these choices is the resulting lack of information about HTA practice outside the INAHTA network and among non-English oriented INAHTA members.
Due to varying structure of the HTA, reports identifying organizational and patient-related issues is not straight forward. To not overlook issues, a broad approach including any mentioning of issues related to one or the other was chosen. The advantage of this approach was that any inclusion of organizational and patient-related issues were credited, the disadvantages being that reports merely raising points but not assessing such an issue were included.
Some areas still need to be examined and developed further. First, the general quality of organizational and patient-related assessments would be enhanced considerably if systematic and relevant analytical models or frameworks were developed. Such models could be targeted at both specific types of policy questions and specific technologies. Second, further investigation is needed to identify issues in relation to, and useful ways of reporting on the generalizability of results from organizational and patient-related assessments.
CONTACT INFORMATION
Anne Lee, MSc, RN (ale@cast.sdu.dk), Consultant, Centre for Applied Health Services, Research and Technology Assessment; Line Sinding Skött, MSc (lsd@kmd.dk), Consultant, Centre for Applied Health Services Research and Technology Assessment; Helle Ploug Hansen, PhD, MSc (HPHansen@health.sdu.dk), Professor, Health, Man and Society, Institute of Public Health, University of Southern Denmark, 9 J. B. Winsloews Vej, DK-5000 Odense C
