Health technology assessment (HTA) is a rapidly growing discipline, which has expanded massively both in its scope of operation and its level of sophistication. In response, a wide range of methods have emerged to address the needs of decision makers working in this area of healthcare policy (15). However, to date, the main focus of methodological research in HTA has been in areas of systematic review of clinical evidence, for example meta-analytical techniques such as multiple treatment comparison, and in approaches to the economic modeling of cost-effectiveness (e.g., probabilistic sensitivity analysis for exploration of decision uncertainty) (Reference Ades, Sculpher, Sutton, Abrams, Cooper, Welton and Lu1;Reference Claxton, Sculpher and McCabe4). Given the undoubted role of HTA in policy making, it is perhaps surprising that more systematic research has not focused on how information is presented and understood by its key stakeholders in addition to what this information is and how it is derived. There is now significant research in management science that demonstrates the importance of methods of information presentation in decision making (Reference Bell2;Reference Benbasat and Dexter3;Reference Irani and Ware10;Reference Jarvenpaa11;Reference Remus21. It seems curious, therefore, that in HTA, where evidence-based methods are paramount, relatively little attention has so far been given to the evidence that demonstrates the importance of effective presentational and graphical design in communicating key outputs.
THE ROLE OF INFORMATION GRAPHICS
It has long been understood that the way in which information is presented, as well as its content, can be critical in decision-making processes. This has been illustrated across a range of differing research fields (Reference Lurie and Mason13). In clinical decision making for instance, Elting et al. (Reference Elting, Martin, Cantor and Rubinstein6), used tables, pie charts, bar graphs, and icon displays to display equivalent information to physicians and showed decision accuracy was affected by the mode of information presentation. Pieczkiewicz et al. (Reference Pieczkiewicz, Finkelstein and Hertz18) demonstrated that graphical displays provide quicker responses and were preferred by clinicians engaged in the detection of infections and graft rejection for transplant surgery. Perera et al. (Reference Perera, Heneghan and Yudkin16) outlines a graphical method for depicting the processes involved in complex clinical interventions and concludes that the graphical depiction of an entire intervention allows its structure to be quickly understood. Powsner and Tufte (Reference Powsner and Tufte20) describe the use of graphical displays to present clinical information to general practitioners. A wide variety of empirical studies in management and cognitive science have supported these conclusions demonstrating the impact of different modes of presentation in decision making (Reference Bell2;Reference Benbasat and Dexter3;Reference Jarvenpaa11;Reference Remus21). These studies show that the effectiveness of information graphics is commonly task dependant and typically a function of the level of expertise for different subject groups and individuals.
In areas associated with HTA, several graphical tools have been developed to support the reporting process. In meta-analysis of clinical evidence and randomized-controlled trial design, for instance, we have seen the emergence of bespoke graphical forms such as Forest plots (Reference Lewis and Clarke12) and CONSORT diagrams (Reference Piaggio, Elbourne, Altman, Pocock and Evans17). Likewise, in the presentation of outputs from economic modeling the use of Cost-Effectiveness Acceptability Curves (CEACs) has become commonplace (Reference Claxton, Sculpher and McCabe4;Reference Fenwick, Claxton and Sculpher7). Although there is little, if any, empirical evaluation of the effectiveness of these tools, it seems reasonable to assume that their development and continued use in the literature points to perceived benefits in communicating key concepts to readers and policy makers.
Information visualization and information design are growing disciplines which have evolved in response to the recognition that graphical methods have an important role in communicating information (Reference Spence22–Reference Ware25). These developments have responded to advances in information technology and new media capabilities which provide ever-increasing possibilities for the organization and presentation of data (e.g., the use of user-interaction, animation, and 3D graphics). Typically research in this area has explored how graphical tools can enable the presentation of complex datasets in accessible formats to enhance understanding and improve decision support (Reference Thomas and Cook23). In addition, a growing body of research now demonstrates that methods of information presentation can be critical in promoting understanding among users (Reference Lurie and Mason13), and this may be especially relevant where a range of different groups with differing perspectives are involved, and where potentially complex data sets are presented—conditions particularly relevant to the field of HTA.
There is considerable scope, therefore, to investigate the use of information visualization as a tool in HTA decision support. Such an investigation, however, needs to integrate several key elements to ensure that the conclusions are coherent, useful, and relevant to context. It should also be noted that the empirical testing of information graphics is a complex multivariate task and experimental designs in this field need to be carefully controlled (Reference Plaisant, Dragecevic and Fekete19).
EXISTING GUIDANCE FOR THE USE OF GRAPHICS IN HTA
Within HTA, there are many differing sources of methodological guidance. One of the most comprehensive set of guidelines in the United Kingdom is issued by the National Institute for Health and Clinical Excellence (NICE) for practitioners of HTA (15). This guidance is substantial, currently running to eighty pages, and is periodically updated in response to an extensive consultative process among researchers and users. Despite this, however, little can be found in this document relating to the use of information graphics. Although the guidance offers considerable space to a discussion of the importance of transparency and clear presentation in HTA reports, there are few specific recommendations for the design and use of graphics. This is in stark contrast to the level of detail contained in the guidelines about other aspects of methodology. Of interest, however, the importance of information presentation may be gaining some traction within the NICE context. In its latest draft, methods guidance for assessing and synthesizing evidence in public health, for instance, NICE devotes more than three pages to the usefulness of graphical tools to present key outputs (14). However, this is still a long way away from advocating a comprehensive and systematic approach to their use and agreement about common standards in the field.
It is now over 10 years since the EUR-ASSESS report detailed the available presentation methods for HTA in Europe (Reference Granados, Jonsson and Banta8). Since that time, the Internet has become an increasingly media-rich platform, capable of delivering animation, sound, and interactive graphics. Computers have become faster and cheaper, and there now exist more possibilities for visual communication than ever before. While these potentials have been embraced for example in business and finance, very little subject-specific guidance on visual presentation is available at the time of writing for those producing HTA reports in the United Kingdom.
The growing prominence of modeling in the NICE policy framework has increased the need for economic models to be made more accessible, transparent, and directly interpretable by the policy makers. Arguably, model users as well as model developers now need to understand the dynamics of the models, as well as the outputs. For example, policy makers now often need to respond to specific queries such as how outputs differ for different subgroups of patients or under alternative treatment regimens. In addition, there is a recognized need to be able to properly account for levels of parameter uncertainty typically associated with model inputs to properly interpret the outputs. Drummond et al. (Reference Drummond, Schwartz and Jönsson5) advocate the use of interactive models to allow different stakeholders to enter the costs and benefits relevant to their setting. They also point out that many international organizations involved in HTA activities issue nontechnical versions of their reports for the lay public highlighting the importance of nonexpert as well as expert understanding of the scientific evidence used to guide policy.
A SURVEY OF CURRENT USE OF GRAPHICS IN HTA
To understand how information graphics are currently deployed in HTA, we conducted a comprehensive content analysis of the fifty most recent “Multiple Technology” assessment reports commissioned by NICE as part of the UK HTA appraisal process and published between October 2003 and November 2007. Single Technology assessment reports were excluded from our study because they were too few in number to yield meaningful outputs.
An initial scoping review was carried out to delineate the key categories and parameters of interest for our analysis, and this was further refined during the full systematic analysis.
The two major dimensions of classification for information graphics were, first, report positioning/section and, second, the type and physical form of graphic. From this, it was possible to construct a two-dimensional matrix charting the relationship between the frequency and type of graphic used in each report against its relative position in the report structure. In addition, further information was recorded about the use of tables within each report, the length of the report, and the identity of the authoring research team.
The analysis used the following sectional division for HTA reports: (i) Introduction/Background, (ii) Systematic Review: Methods, (iii) Systematic Review: Results, (iv) Economic Review and Model Critique, (v) Economic Modeling: Methods, (vi) Economic Modeling: Results, (vii) Conclusion, and (viii) Appendices.
All figures in the HTA reports were classified according to a two-layered taxonomy based on four generic categories, which were further divided into subtypes as outlined here: 1. Line Graphs: (i) Time Series graphs, (ii) Cost-Effectiveness Acceptability Curves (CEACs), (iii) Threshold Diagrams, and (iv) Other. 2. Flow Diagrams: (i) State Transition, (ii) Decision Tree, and (iii) Other. 3. Area/Position Charts: (i) Bar Charts/Histograms, (ii) Scatter Plots, and (iii) Other. 4. Other Graphics: (i) Forest Plots, and (ii) Other.
For each HTA report analyzed, the number of graphics of each type used in each report section was recorded. Other data recorded was the number of tables used in each section of the report as well as the report size. In addition, qualitative information was recorded, for example, where there was interesting features or use of graphics.
RESULTS
Summary results from the survey are shown below (a more detailed description of this analysis is available on our Web site) (9).
Graphics Used
In all, a total of 965 graphics were counted within the fifty reviewed reports. When broken down by graphical category, we found 301 Line graphs (comprising; 102 time series graphs, 124 CEACs, 37 threshold analysis graphs, and 38 others), 124 Flow charts (78 state transition diagrams, 41 decision trees, 5 others), 187 Area/Position charts (88 bar charts, 55 scatter plots, 44 other), and 353 Others (of which the vast majority, 331, were Forest plots). This demonstrates a significant and fairly wide-spread use of graphical methods. Although the raw count data shows Forest plots to be the most numerous, a large number of these were contained in just a very few reports where they were used repeatedly. Over 230 of the Forest Plots, for instance, were contained in just six reports.
Report Positioning
Figure 1 below shows the number of reports that make use of a major graphical type in our categorization, analyzed by report section.
Figure 1. Counts of reports which use the major graphical types in each report section.
This bubble chart demonstrates a wide variation in the types of graphics used across different report sections. The use of graphical tools is particularly common within the economic sections of the reports, where flow diagrams are extensively used in the modeling methods sections and line graphs predominate in the results section. It is also clear, however, that graphics are extensively used in the systematic review chapters and in the appendices of reports.
In this analysis, we have chosen to present the incidence of reports using a graphic at least once in each section rather than raw counts. This is because the reports (which sometimes compare as many as sixteen different treatments) typically use similar graphics repeatedly, thus distorting aggregated results if raw counts are used. In addition, certain graphics (e.g., Forest Plots) appear almost exclusively in only one section of the report.
From our analysis, it can be seen that there are certain established graphic tools that are used commonly in HTA reports and stand out clearly. Many reports, for instance, include “study selection” decision tree diagrams in the systematic review section, as well as Forest plots to display an overview of the evidence of clinical effectiveness of interventions.
In the modeling section, there was frequent use of state transition diagrams, usually demonstrating patient flow in a Markov model. The results of the modeling were often presented as a cost-effectiveness acceptability curve (CEAC), bar charts were commonly used to display the results of one-way sensitivity analyses, and scatter plots to display probabilistic analyses outputs. However, even the most common forms of graphic were only used in less than two-thirds of the reports (state transition in 60 percent of the model methods sections, and CEACs in 62 percent of the model results sections). This is surprisingly low considering that the CEAC is one of the three graphic presentation techniques specifically recommended in the NICE methods guidance.
A large proportion of the counted graphics fit within our typology, with surprisingly few counted in “other” categories. Given that the typology is restricted, containing only eight specific kinds of diagram, this suggests that the current palette of information graphics used in HTA in the United Kingdom is quite limited.
Use of Graphics Versus Tables
The relative benefits of tabular versus graphical presentation of data has been considered in several studies (Reference Spence22). Tables present data in numerical form which is more accessible and precise for calculations, whereas graphics are often better for showing patterns within the data. For these reasons combined displays have often been advocated. In our review, we compared the extent of use of tables versus the use of graphics in HTA reports.
This review has found that tables are used, on average, nearly three times more than graphics in NICE-commissioned HTA reports. The mean number of graphics used per page was 0.2, and the mean number of tables, 0.58. These results indicate a preference in HTA reports to use numerical data in tables rather than graphical presentation. There may be many reasons for this, however it might also indicate that there are opportunities for the use of information graphics (often in addition to rather than instead of tables) to augment the communication of research data.
Graphics Used, by Place of Authorship
Figure 2 below uses a radar chart to show the relative number and distribution of graphics of different types used by six of the research groups which produced the HTA reports reviewed (only two of the reviewed reports were authored by seventh center, Liverpool, so this is not included in the diagram).
Figure 2. Radar chart showing the average number and types of graphics used per health technology assessment report by each authoring research team.
These results show there is considerable variability between research teams both in the number of graphics used (shown by the overall area size of each radar plot) and in the types of graphics used (shown by the shape of the shaded area). Whereas some teams hardly use certain graphics at all, other teams use the same techniques frequently. While NICE provides some guidance on the production of HTA reports (15), the observed variation in the use of graphics suggests there is uncertainty and a lack of agreement as to the best methods for presenting data in HTA in the United Kingdom, as well as scope for greater communication between the authoring research teams. Design standards for many visually presented media, such as books and Web sites, are constantly discussed, imitated and improved by those who use them. These results of our survey suggest that within HTA in the United Kingdom, researchers do not typically have the expertise in visual communication or access to specialists in this area. There may also be time constraints on the generation of graphical presentations of data.
A STRUCTURED APPROACH TO INFORMATION GRAPHICS IN HTA
For information graphics in HTA decision making to be addressed seriously a structured approach is required. The key components of a proposed framework for such an approach are outlined in Figure 3.
Figure 3. Framework for development of effective information graphics in health technology assessment.
Identification of Specific Application Areas Within HTA
There are several distinct activities inherent in most HTA reporting process that could potentially benefit from improved information graphics. Each of these has different characteristics and is, therefore, likely to have differing requirements for the role that graphics can perform. In our analysis of HTA reports, we distinguish the following four areas of application: (i) Systematic Review and Meta-Analysis (e.g., Flow charts for Inclusion/exclusion in literature searches, Forest Plots), (ii) Data sourcing (e.g., visual analogue scales for utility measurement), (iii) Model Development (e.g., Graphical development aids, debugging, and error detection tools), (iv) Model Outputs (e.g., Presentation of model structure and design (influence diagrams), primary outputs from model, sensitivity analysis, etc).
User-Group/Requirements Analysis
HTA engages a range of different user groups each with its own level of expertise, responsibilities, and needs. HTA researchers and developers, policy makers and advisors, patient interest groups, industry stakeholders, and the general public, all have an interest in HTA outputs. A thorough information requirements analysis is needed for each of these stakeholder groups. Policy makers, for instance, may need to better understand the dynamics and sensitivity analyses of cost-effectiveness models to inform choices about targeting treatments. It may be necessary to prioritize particular user-group requirements in driving the design of specific graphical tools. It may also be important to recognize that a range of graphical tools are required to respond to the differing needs of separate stakeholder groups.
Incorporating Constraints
Key limitations exist as to what is possible with regard to the use of graphics in HTA. Broadly these can be divided into (a) constraints which relate to human aspects of understanding (e.g., the extent to which HTA report users likely to understand the specifics of a particular graphical form) and (b) physical constraints, such as limitations in the media available for communication. Paper-based reports reproduced in black and white clearly place more fundamental limitations on design than the interactive color displays possible on computers. Information graphics need to be designed which account for these limitations, although at the same time seeking to extend the boundaries of existing practice through demonstration of the potential benefits of moving beyond these constraints (e.g., if color reproduction could be shown to provide significant benefits then maybe there would be good arguments for its incorporation within reports).
Potentials
There is now a wide ranging literature which investigates the uses of information graphics and data visualization in management and decision making (Reference Lurie and Mason13;Reference Thomas and Cook23). In addition, the growing sophistication and accessibility of computer graphics lends itself to greater use of these technologies to support decision processes. A structured approach to the use of this research and technology in the field of HTA needs to embrace the potentials inherent and look at ways in which it can be applied specifically in response to the needs of the key user groups. As previously noted, it is over 10 years since the EUR-ASSESS guidelines highlighted the graphical potential of the Internet in HTA (Reference Granados, Jonsson and Banta8). It would appear that much of this potential has yet to be exploited.
Standards and Guidelines
A critical and sometimes neglected area in management support is the implementation of standards and guidelines. This is especially important in the field of HTA where many diverse interest groups with differing expertise and needs interact. For the implementation of information graphics to become effective and widely accepted in the HTA community, therefore, agreed guidelines and standards are likely to be essential.
Prototypes and Exemplars
Exemplars and case studies can provide an important means of exploring and demonstrating new frontiers in information graphics. In health technology assessment, prototypes of innovative graphical and visualization tools can be crucial for developing ideas and eliciting feedback from users.
One such prototype system dubbed “The Sensitivity Mixing Desk” was developed by the authors as part of an HTA study. This computer-based tool presents a series of interactive slider controls on the screen, which allows users to adjust a range of selected parameters within a spreadsheet-based Markov Model, and view the corresponding effect on model outputs. The tool illustrates the potential of interactive graphics in enhancing the understanding of the parameter sensitivities for an economic modeling analysis. The interface was designed to address the perceived need to improve understanding of the dynamics of the models used for cost-effectiveness analysis in HTA to a wider range of users (Reference Drummond, Schwartz and Jönsson5).
Another exemplar is shown in Figure 4. This uses a Sankey diagram, a technique often used in engineering to display resource and energy flows, to depict patient flow in a randomized trial of two interventions, including all the information recommended in a CONSORT-style flow diagram (http://www.consort-statement.org/). The width of the lines corresponds to quantities of flow (i.e., patients) in the trial and enables a clear overview of the proportions of patients excluded and allocated to the arms of the trial. If this technique were applied to a series of trials in an HTA context, it could enable the viewer to compare different trials at a glance, and quickly absorb the necessary study design and sample information, enabling a fast but accurate comparison to be made by a range of users.
Figure 4. Sankey representation of a CONSORT diagram for a randomized control trial.
CONCLUDING REMARKS
We have argued here that the development of information graphics in HTA needs greater recognition and a more systematic approach. Graphical presentation can play a key role in the processes of communication central to effective decision making, and we need to explore and acknowledge this role. A research framework for future investigation and development of information graphics in HTA is proposed. This places the needs of users as central to the development of improved graphical methods of communication and identifies the key dimensions which need to be considered in future research.
CONTACT INFORMATION
Martin Pitt, EngD (martin.pitt@pms.ac.uk), Senior Research Fellow Will Stahl-Timmins, MA (will.stahl-timmins@pms.ac.uk), PhD Student, Rob Anderson, PhD (rob.anderson@pms.ac.uk), Senior Lecturer in Health Economics; Ken Stein, MD, FFPH (ken.stein@pms.ac.uk), Professor of Public Health, Director, Peninsula Technology Assessment Group, Peninsula College of Medicine and Dentistry, Universities of Exeter & Plymouth, Noy Scott House, RD&E Hospital, Barrack Road, Exeter, EX2 5DW, UK
