The telemedicine evaluation literature has grown substantially from the advancement of a specific framework for assessing telemedicine by the Institute of Medicine (IOM) (16). This early report identified five dimensions important to evaluating telemedicine: quality, access, cost, patient perceptions, and clinician perceptions. However, a more recent report by The Lewin Group (27) both confirmed and extended these evaluation dimensions. It considered the properties of these dimensions, in terms of measures and their impacts, but also the methodology issues involved in evaluation. This directly responded to some disquiet expressed in the literature over rigor and consistency that limited the generalizability of some studies' findings. In light of these concerns, we discuss two possible improvements for telemedicine evaluation.
First, we argue that the focus of the evaluation itself should be widened to look at telemedicine in the context of the patient pathway (also known in the literature as the clinical pathway) to understand its place along the patient's journey through the health service. Second, we put forward simulation as a tool for evaluating telemedicine through its representation of the patient pathway. Simulation will be discussed as a viable methodology for addressing some of the weaknesses documented in telemedicine evaluation, through a review of the measures and methodologies used in the assessment of telemedicine. The discussion of the potential benefits of simulating patient pathways is supported by an illustration—using leg ulcer sufferers as a case example—to contribute to an understanding of care delivery by traditional and telemedicine processes.
TELEMEDICINE EVALUATION ISSUES
Measures
Reviews of telemedicine evaluation are limited (1;12;25;27), but those that do exist provide important overviews as to the status of evaluations in terms of the measures and methodology used to assess telemedicine. Most evaluations have sought to assess various quantitative measures of effectiveness (e.g., diagnostic accuracy), efficiency (e.g., cost), and engagement (e.g., patient satisfaction) to determine its success. However, these studies have tended to focus on single clinical contexts, specialties, and measures. To highlight the problematic issues around the measures currently assessed in telemedicine evaluation, we will briefly focus on three key evaluation measures: (i) Diagnostic accuracy—this measure has tended to overly dominate many studies' outcomes (1); (ii) Cost (and its associated variables, e.g., benefit, utility, and so on)—many studies have equated a cost-saving as a benefit, but with no reference to how it affects clinical outcomes (28); and (iii) Patient satisfaction—this measure is the most common evaluation undertaken, tends to produce consistently positive results, but can often be misleading, as patient satisfaction measures often fail to go beyond first impressions (19).
A major reason for some of the limitations cited in the research is that few high-quality studies exist (15). Despite the usefulness of the clinical and economic data that have been produced, the methodological paucity in the research has somewhat undermined the value of the assessments undertaken. This finding has resulted in calls for patients and practitioners alike to remain skeptical over the professed benefits of telemedicine compared with traditional face-to-face patient care (12). In an attempt to manage some of the methodological issues in telemedicine evaluation, simulation is proposed as a potentially useful tool for producing more robust findings.
Methodology
As a methodological approach, simulation revolves around creating computer models of social structures and processes. These models are subject to “simulation” that is experimentation through the manipulation of variables (e.g., time and cost) to understand the behavior of the model and evaluate the extent to which it provides an accurate account of the behavior of the observed system (13). In health care, simulation has achieved some success as a problem-solving tool (3). Moreover, as well as being able to incorporate the aforementioned evaluation measures, it will be posited that simulation modeling offers a systematic approach for addressing key evaluation methodology issues that have been usefully summarized by the seminal Lewin Group report (27). These issues have been identified as technological maturity, focus of evaluation, perspective of evaluation, comparator, randomization, and time horizon. Table 1 defines each of these issues in turn, along with an example of how they are addressed in current telemedicine evaluation studies and how simulation can offer a potential solution to these challenging issues.
From Table 1, it can be seen that current telemedicine evaluations largely fail to address the broader organizational, clinical, and social processes that new technology impacts upon. Some evidence that exists suggests that this is a critical issue and that the focus of the evaluation may be too narrow. For example, Lehoux et al. (17) found that the use of telemedicine did not fit into clinicians' communication routines of consultation and referral. We suggest that a more fruitful direction for the evaluation of telemedicine is to focus on the patient pathway. The patient pathway includes all the clinical routines (or processes) into which telemedicine is placed, so that it can be assessed on how it sustains or supports variation in clinical practice.
SIMULATING THE PATIENT PATHWAY
Patient pathways are tools that assist in providing general guidelines of care for dealing with individuals and groups of patients suffering from a wide variety of diseases. However, the majority of studies focused on traditional patient pathways (23). The introduction of telemedicine, however, offers a new pathway to care, although the impacts of it are even less well understood. To this end, simulation is put forward as a method of modeling pathways that capture the timeline of care from the start to the journey's end.
Discrete-Event Simulation Technique
Discrete-event simulation (DES) offers many features to cope with understanding the complex nature of health care systems, of which the patient pathway is a clear example. There are three critical steps to the technique that are advantageous for representing pathways, which can be described as follows: (i) Understanding the system or process to be modeled—this is in terms of its main entities (e.g., patients), events (e.g., clinic visits), and decisions (e.g., referral of patient for consultation) and must be achieved for the subsequent model building to have a good representative basis; (ii) Changing the parameters of the model (e.g., time and cost)—this step can suggest (based on mathematical distributions) the optimum capacity of the system in the present and for the future given different scenarios; and (iii) Understanding the inter-relationships between different entities, events, and decisions in the system—this step can identify the interdependencies of variables and the effect of changing one has upon another.
The simulation's significant feature is its capability of performing “what if…” type analyses through the manipulation of variables to understand the inter-relationships within the model and, hence, the real system. This iterative nature to the modeling process brings about the identification of the optimum system setup. Although simulation is not intended to replace current designs of evaluation studies, acceptance of any new approach to studying health care problems needs to be justified on the professed benefits of the proposed solution option.
Benefits of Simulating the Patient Pathway
Whereas simulation has the potential to overcome many of the problematic issues in telemedicine evaluation (as shown in Table 1), the true value of the approach rarely has been realized, given the narrow focus to which it has been applied previously, typically hospital scheduling problems (11). This quantitative view of simulation modeling—as a way of calculating outcomes—has often failed to produce results that can be readily implemented in real-life applications. Moreover, the telemedicine literature, in particular, reveals a paucity of studies that have undertaken any simulation, except for perhaps two notable examples (6;18). The impetus, therefore, clearly exists for a debate on the conceptual issues of research in telemedicine, given the problems with current evaluation techniques (4). Robinson (24) has called for a debate on simulation study as a mode of practice in various domains. We contend that this debate is necessary within health care and propose that a starting point for this discussion is on evaluating telemedicine from a simulation of the patient pathway.
Patient pathways do not physically exist; therefore, methods of computerizing pathways have demonstrated some degree of success (7). The understanding that is gained through simulation is of a greater value than the pure numerical values produced. We propose, therefore, that simulation be viewed as a tool not to calculate outcomes but to appreciate them. This difference is subtle yet powerful. In this manner, the use of simulation will crucially serve to elicit the intangibles, such as insight into the way the system actually operates, understanding the variables that can affect the system, and informing decisions concerning the system and their possible consequences.
The benefits of simulating patient pathways are increased when qualitative investigations (e.g., interviews, observations, and so on) are directed at critical points along the pathway so as to supplement the models and understand more holistically the relationship between the interpersonal (e.g., patient satisfaction) and technical aspects of telemedicine (9). For example, to return to the three key measures discussed earlier, in diagnostic accuracy, a control patient group can be simulated to compare the outcomes of consultations with a clinical trial group. This approach has been shown previously to provide an educational benefit for informing clinical decision making (14). Furthermore, cost-effectiveness measures can also be extended to produce cost per quality measures of outcome in terms of the quality adjusted life years for specific health care interventions (10). This can have an important personal benefit, particularly for the patient in terms of establishing the relationship between their illness and the likelihood of health care saving their life. Moreover, incorporating patient satisfaction measures is possible by converting patient responses into an appropriate numerical scale and applying these figures to the model. This strategy can have a behavioral benefit in being able to determine fluctuations in patient satisfaction and pinpointing problem areas.
Telemedicine Patient Pathway: An Illustration
To illustrate the potential of simulating patient pathways, leg ulcer sufferers were selected as case examples, given the access to patients who have followed a traditional and telemedicine patient pathway. Figure 1 is a graphic representation of part of a leg ulcer patient pathway. Information to structure the pathway and its critical variables is collected from patient records and interviews with doctors and nurses. Figure 1 serves to illustrate all the health care events along this section of the pathway, for instance a tele-link consultation and the relationships between them. Figure 2 illustrates how identical patients can be treated through two systems, both traditional and telemedicine (the timescale for which is provided for illustrative purposes only).
Leg ulcer patient pathway.
Comparison of a traditional and telemedicine leg ulcer patient pathway.
Variables at strategic decision points along the pathway control the flow of patients. These variables alter based on data parameters (patient volume), patient attributes (age, severity of condition, and general health), and time, which were deemed important for the treatment of leg ulcers from interviews with clinical staff. These variables can be further manipulated to determine different levels of effect when changing sections of the process (i.e., introducing telemedicine).
DISCUSSION AND CONCLUSIONS
This study has discussed simulation modeling—solely as a mode of practice—for the evaluation of telemedicine in the context of the patient pathway, as illustrated by Figures 1 and 2. Future work will need to computerize the pathway into a dynamic running simulation model by comparing the predicted flow of patients with the actual flow along the traditional and telemedicine pathway. This approach will allow the capture of the long-term and organizational impacts of telemedicine implementation. An integral part of the discussion on the potential use of simulation has been the focus on the patient pathway. This focus has provided a compelling example of the importance of evaluating telemedicine in the context of the health care processes into which it is placed. A spotlight on pathways vivifies the collective benefits of simulation that have been put forward by highlighting how telemedicine can be evaluated in relation to the entities, events, and decisions involved in the delivery of care that it impacts upon. On the surface, telemedicine might appear to offer a fast-track system for patients, which can reduce the costs of dressing for leg ulcer patients, for example (as shown in Figure 2). However, Phipps (22) provides a caveat to this in stating that it is important that we understand how optimizing one section of a process (e.g., introducing a tele-link) can affect another section further along in the system and potentially induce a bottleneck (e.g., at the surgery stage) given that this section of the process will not have changed. It is suggested that a new challenge for telemedicine evaluation studies is to compile a more comprehensive view of the technology, in looking across multiple aspects of health-care processes to provide a much-needed commentary on the outcomes of health care and its delivery. To this end, patient pathway simulation has been introduced as a potential evaluation tool for telemedicine to continually monitor clinical practice, the effects of telemedicine, and changing health outcomes.
CONTACT INFORMATION
Jane Coughlan, PhD (Jane-Lisa.Coughlan@brunel.ac.uk), Research Fellow, Julie Eatock, PhD (Julie.Eatock@brunel.ac.uk), Research Fellow; Tillal Eldabi, PhD (Tillal.Eldabi@brunel.ac.uk), Lecturer, Information Systems and Computing, School of Information Systems, Computing and Mathematics, Brunel University, Kingston Lane, Uxbridge, Middlesex UB8 3PH, UK
