Hostname: page-component-7b9c58cd5d-hxdxx Total loading time: 0 Render date: 2025-03-14T03:04:02.755Z Has data issue: false hasContentIssue false
  • English
  • Français

INTERRUPTED TIME SERIES DESIGNS IN HEALTH TECHNOLOGY ASSESSMENT: LESSONS FROM TWO SYSTEMATIC REVIEWS OF BEHAVIOR CHANGE STRATEGIES

Published online by Cambridge University Press:  21 April 2004

Craig R. Ramsay ,
Lloyd Matowe ,
Roberto Grilli ,
Jeremy M. Grimshaw  and
Ruth E. Thomas
Craig R. Ramsay
Affiliation:
University of Aberdeen
Lloyd Matowe
Affiliation:
Kuwait University
Roberto Grilli
Affiliation:
Agenzia Sanitaria Regionale
Jeremy M. Grimshaw
Affiliation:
University of Ottawa
Ruth E. Thomas
Affiliation:
University of Aberdeen
Rights & Permissions [Opens in a new window]

Abstract

Objectives: In an interrupted time series (ITS) design, data are collected at multiple instances over time before and after an intervention to detect whether the intervention has an effect significantly greater than the underlying secular trend. We critically reviewed the methodological quality of ITS designs using studies included in two systematic reviews (a review of mass media interventions and a review of guideline dissemination and implementation strategies).

Methods: Quality criteria were developed, and data were abstracted from each study. If the primary study analyzed the ITS design inappropriately, we reanalyzed the results by using time series regression.

Results: Twenty mass media studies and thirty-eight guideline studies were included. A total of 66% of ITS studies did not rule out the threat that another event could have occurred at the point of intervention. Thirty-three studies were reanalyzed, of which eight had significant preintervention trends. All of the studies were considered “effective” in the original report, but approximately half of the reanalyzed studies showed no statistically significant differences.

Conclusions: We demonstrated that ITS designs are often analyzed inappropriately, underpowered, and poorly reported in implementation research. We have illustrated a framework for appraising ITS designs, and more widespread adoption of this framework would strengthen reviews that use ITS designs.

Keywords

Professional practiceCross-sectional studiesRegression analysisTime factors
Type
GENERAL ESSAYS
Information
International Journal of Technology Assessment in Health Care , Volume 19 , Issue 4 , December 2003 , pp. 613 - 623
Copyright
© 2004 Cambridge University Press

Reliable and valid information on the effectiveness and cost-effectiveness of different interventions is required if policy makers are to make decisions based on these interventions. Rigorous research can help provide such information for policy makers; however, many existing studies are of poor quality (6).

The randomized controlled trial (RCT) is seen as the criterion standard methodology for the evaluation of health care interventions (17). However, there are many interventions for which it is impossible or impractical to use RCTs. For national interventions such as mass media campaigns (35) or within local contexts, the policy makers may wish to assess the impact of specific policies such as the dissemination of clinical guidelines (49). Thus, there is increasing interest in the use of quasi-experimental designs (18).

A common quasi-experimental design used to evaluate such interventions is a before-and-after study design. That is, some measure of compliance or outcome is taken before the intervention and then the same measure is taken after the intervention has occurred. Any changes are then inferred to have occurred because of the intervention. For example, the Working Party of the Royal College of Radiologists undertook a before-and-after study evaluating the impact on general practice of disseminating the Royal College of Radiologists booklet Making the Best Use of a Department of Radiology to general practitioners (58). The booklet provided guidelines for radiological investigations and was introduced on January 1, 1990. In the study, the number and type of referrals were measured for the year before and the year after the introduction of the guidelines. There was a reduction in radiological requests after the guidelines were introduced, and it was concluded that the guidelines had changed practice. Several questions remained unanswered. Was the number of referrals already decreasing before the intervention? Did the major National Health Service reforms introduced during 1989–90 impact upon the number of referrals (23;60)? Did the referrals stay at this lower level throughout 1990, or did they begin to return to the original level? These questions demonstrate the potential weaknesses of the before-and-after design (49).

Interrupted time series (ITS) designs can be used to strengthen before-and-after designs (15). In an ITS design, data are collected at multiple instances over time before and after an intervention (interruption) is introduced to detect whether the intervention has an effect significantly greater than the underlying secular trend. There are many examples of the use of this design for the evaluation of health care interventions (16;22;45;67).

An advantage of an ITS design is that it allows for the statistical investigation of potential biases in the estimate of the effect of the intervention. These potential biases include:

  • Secular trend – the outcome may be increasing or decreasing with time. For example, the observations might be increasing before the intervention; hence, one could have wrongly attributed the observed effect to the intervention if a before-and-after study was performed.
  • Cyclical or seasonal effects – there may be cyclical patterns in the outcome that occur over time.
  • Duration of the intervention – the intervention might have an effect for the first three months only after it was introduced; data collected yearly would not have identified this effect.
  • Random fluctuations – these are short fluctuations with no discernible pattern that can bias intervention effect estimates.
  • Autocorrelation – this is the extent to which data collected close together in time are correlated with each other. The autocorrelation ranges between −1 and 1. A negative autocorrelation suggests that outcomes taken close together in time are likely to be dissimilar. For example, a high outcome is followed by a low outcome that is then followed by a high outcome and so on. In contrast, a positive autocorrelation suggests that outcomes measured close together in time are similar to each other. For example, a high outcome is followed by another high outcome. Ignoring autocorrelation can lead to spuriously significant effects (18).

To investigate such biases, researchers need to use appropriate statistical methods. For example, time series regression techniques (25) or autoregressive integrated moving average models (ARIMA) (10).

Studies using ITS designs are increasingly being considered for inclusion in systematic reviews (7); however, there has been no published research to assess the methodological quality of ITS studies. We, therefore, undertook a critical review of the methodological quality of ITS studies included in two systematic reviews. The first review evaluated the effectiveness of mass media interventions targeted at improving the utilization of health services (35) and the second review evaluated the effectiveness of different clinical guideline dissemination and implementation strategies (36).

METHODS

Inclusion Criteria

All ITS studies in the mass media review were included. One study from the guideline review was excluded, because it was also in the mass media review (48). Both reviews used the Effective Practice and Organisation of Care (EPOC) Cochrane Group definition of an ITS design (7): (i) there were at least three time points before and after the intervention, irrespective of the statistical analysis used; (ii) the intervention occurred at a clearly defined point in time; (iii) the study measured provider performance or patient outcome objectively.

Data Abstraction

Quality criteria were developed from two sources; from the EPOC data collection checklist (5), and from the classification of threats to validity identified by Campbell and Stanley (15). These quality criteria were agreed by consensus among all the authors in the study and are displayed in Box 1. At least two reviewers independently abstracted the data from each study.

Data Analysis

Descriptive statistics for each review were tabulated. If studies had not performed any statistical analysis or had used inappropriate statistical methods, we reanalyzed the results (where possible). For the purpose of reanalysis, data on individual observations over time were derived from tables of results or graphs presented in the original study. Data from graphs was obtained by scanning the graph as a digital image and reading the corresponding values from the images (35).

Statistical Method for Reanalysis of Primary Studies

Time series regression was used to reanalyze the results from each study. The best fit preintervention and postintervention lines were estimated by using linear regression, and autocorrelation was adjusted for by using the maximum likelihood methods where appropriate (25). First-order autocorrelation was tested for statistically by using the Durbin-Watson statistic, and higher-order autocorrelations were investigated by using the autocorrelation and partial autocorrelation function.

Two effect sizes were estimated (see Figure 1). First, a change in the level of outcome at the first point after the introduction of the intervention was estimated. This strategy was performed by extrapolating the preintervention regression line to the first point postintervention. The difference between this extrapolated point and the postintervention regression estimate for the same point estimated the change in level. Further mathematical details are available from the authors on request. Second, a change in the slopes of the regression lines was estimated (calculated as postintervention minus preintervention slope). We classified each study intervention as “significant” if either of these effect sizes were statistically significant.

The effect sizes estimated by time series regression analysis of an ITS design.

RESULTS

Twenty ITS studies were included from the mass media review (8;9;11;18;24;27;39–42;46–48;51;53;55;61;64;69;71), and thirty-eight ITS studies were included from the guidelines review (1–4;12–14;16;22;26;28–33;37;38;43–45;50;52;54;56;57;59;62;63;65–68;70; 72–75). Eighteen studies were published before 1990 and forty studies after 1990. The characteristics of the ITS studies are displayed in Table 1. Both reviews had similar numbers of preintervention data points; however, the average ratio of postintervention points to preintervention points indicated that the guideline studies tended to collect more postintervention points than preintervention points within each study. The time interval between data points varied, with “monthly” the most common in both reviews.

The quality criteria for the ITS studies are shown in Table 2. Thirty-eight (66%) studies did not rule out the threat that another event could have occurred at the same time as the intervention. Reporting of factors related to data collection, the primary outcome, and completeness of the data set were generally done in both reviews. No study provided a justification for the number of data points used or a rationale for the shape of the intervention effect.

Thirty-seven of the ITS studies in the reviews were analyzed inappropriately, of which thirty-three could be reanalyzed. Four “inappropriately analyzed” studies could not be reanalyzed because of the poor quality of the graphs in the primary papers. The results of the reanalyzed studies are shown in Table 3. Eight of the studies had statistically significant preintervention trends (3;11;21;30;52;57;61;75). The mean autocorrelations in the reanalyzed ITS studies were small. All of the interventions in each of the reanalyzed studies were termed “effective” in their original reports, but approximately half of these studies showed no statistically significant differences in slope and level when reanalyzed.

DISCUSSION

We have demonstrated that ITS designs are being increasingly used to evaluate health care behavior change interventions. Many of these studies had apparent threats to their internal validity that made interpretation of the results unreliable. The design of the studies was often poorly reported and analyzed inappropriately leading to misleading conclusions in the published reports of these studies.

There were no substantial differences between the characteristics of the studies in the two reviews. The mass media studies tended to have smaller postintervention phases. Most of the studies in both reviews had short time series. Short time series suggest that model fitting is performed with less confidence, standard errors are increased, type I error is increased, power is reduced, and hence, failure to detect autocorrelation or secular trends.

Over 65% of the studies were analyzed inappropriately. Most of these studies performed a t-test of the preintervention points versus postintervention points. The t-test would give incorrect effect sizes if preintervention trends were present and would decrease the standard error if positive autocorrelation were present (19;49). The t-test, therefore, should not be used to analyze results from an ITS design.

Many of the studies were underpowered. It is difficult to derive a formula for the sample size (34). As a rule of thumb, if one collects ten pre- and ten post-data points, then the study would have at least 80% power to detect a change in level of five standard deviations (of the pre-data) only if the autocorrelation is greater than 0.4 (20). This score is a large effect, and the results from the two reviews suggested that the autocorrelation was around the order of 0.1; therefore, a study with ten pre- and post-data points was likely to be underpowered. In addition, it is beneficial to have a long preintervention phase thereby increasing power to detect secular trends.

This study showed that many ITS studies had analytical shortfalls. In such instances, reviewers may need to consider reanalyzing. The data may be obtained directly from the authors, but if this is not possible, the reviewers can often scan the graphs from the original paper onto a computer as described in the methods section. We recommend reviewers use time series regression techniques for reanalyzing ITS studies. The main advantages of this technique is, first, that it can be performed using most standard statistical packages and, second, that it can estimate the effect sizes more precisely than other time series techniques when the series are short (20;34).

The lack of a randomized control group in an ITS design requires the investigators to be critical of the internal validity of the study. With this in mind, we developed a checklist for reviewers of such designs. This checklist encompassed the major threats to validity of ITS designs as proposed by Campbell and Stanley (15). Generally, the greatest threat to validity is that an event other than the control of the researchers occurred at the same time as the intervention, thereby making causal inferences impossible. It was disappointing, therefore, to note that this threat was not ruled out explicitly in 66% of the studies.

Implications for Researchers

ITS designs may appear superficially simple; however, we have demonstrated that they are often analyzed inappropriately, underpowered, and poorly reported in the implementation research field. To improve the quality of analysis and reporting, we suggest that researchers consider the list of quality criteria described in Box 1.

Implications for Systematic Reviewers

ITS designs are a useful and pragmatic evaluative tool when RCTs are not feasible. It is possible to consider this often neglected study design in systematic reviews and in so doing draw both qualitative and quantitative results on intervention effects. In this study, we have illustrated the usefulness of a framework for appraising ITS designs, and more widespread adoption of such a framework would strengthen reviews that use ITS designs.

Implications for Policy Makers

Policy makers may find ITS designs a useful way to assess the impact of specific policies that could remain unassessable otherwise. In addition, these designs can often be performed inexpensively from routine data collection.

The Health Services Research Unit is funded by the Chief Scientist Office of the Scottish Executive Health Department. The guidelines dissemination and implementation review was funded by the UK NHS R&D Health Technology Assessment Programme. However, the views expressed are those of the authors and not necessarily those of the funding body.

References

Avorn J, Soumerai SB, Taylor W et al. 1988 Reduction of incorrect antibiotic dosing through a structured educational order form. Arch Intern Med. 148: 1720- 1724.Google Scholar
Bareford D, Hayling A. 1990 Inappropriate use of laboratory services: Long term combined approach to modify request patterns. BMJ. 301: 1305- 1307.Google Scholar
Barnett GO, Winickoff R, Dorsey JL, Morgan MM, Lurie RS. 1978 Quality assurance through automated monitoring and concurrent feedback usinga computer-based medical information system. Med Care. 16: 962- 970.Google Scholar
Berbatis CG, Maher MJ, Plumridge RJ, Stoelwinder JU, Zubrick SR. 1982 Impact of a drug bulletin on prescribing oral analgesics in a teaching hospital. Am J Hosp Pharm. 39: 98- 100.Google Scholar
Bero L, Freemantle N, Grilli R et al. 1997 Checklist for data abstraction in EPOC reviews. Aberdeen: EPOC(mimeo)
Bero L, Grilli R, Grimshaw JM et al. 1998 Closing the gap between research and practice: An overview of systematic reviews of interventions to promote implementation findings by health care professionals. BMJ. 317: 465- 468.Google Scholar
Bero L, Grilli R, Grimshaw JM et al. 2002 The Cochrane Effective Practice and Organisation of care Group (EPOC) Module. In: The Cochrane Library. Oxford: Update Software Issue 1.
Blohm M, Herlitz J, Hartford M et al. 1992 Consequences of a media campaign focusing on delay in acute myocardial infarction. Am J Cardiol. 69: 411- 413.Google Scholar
Bonerandi JJ, Grob JJ, Cnudde N, Enel P, Gouvernet J. 1992 Campaign of early detection of melanoma in the Provence-Alpes-Cote-d'Azur area 1989. Lessons of an experience. Ann Dermatol Venereol. 119: 105- 109.Google Scholar
Box EP, Jenkins GM. 1976 Time series analysis. London: Holden-Day
Brasca A, Pigliacampo J, Pollastri E et al. 1987 An awareness campaign on the diagnosis and prevention of colon and rectal cancer. Planning and evaluation of its results. Acta Gastroenterol Latinoam. 17: 325- 329.Google Scholar
Brook RH, Williams KN. 1976 Effect of medical care review on the use of injections: A study of the New Mexico Experimental Medical Care Review Organization. Ann Intern Med. 85: 509- 515.Google Scholar
Brooks RJ. 1996 Implementing guidelines for eye care of diabetic patients: Results from an HMO intervention study. Am J Managed Care. 2: 365- 369.Google Scholar
Brufsky JW, Ross-Degnan D, Calabrese D, Gao X, Soumerai SB. 1998 Shifting physician prescribing to a preferred histamine-2-receptor antagonist. Effects of a multifactorial intervention in a mixed-model health maintenance organization. Med. Care 36: 321- 332.Google Scholar
Campbell DT, Stanley JC. 1966 Experimental and quasi-experimental designs for research. Chicago: Rand McNally
Clarke JA, Adams JE. 1990 The application of clinical guidelines for skull radiography in the accident and emergency department: Theory and practice. Clin Radiol. 41: 152-5.Google Scholar
Cochrane AL. 1989 Effectiveness and efficiency: Random reflections on health services. London: British Medical Journal for Nuffield Provincial Health Trust
Cook TD, Campbell DT. 1979 Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin Company
Crosbie 1993 J. Interrupted time-series analysis with brief single-subject data. J Consult Clin Psychol. 61: 966- 974.
Crosbie J. 1995 Interrupted time-series analysis with short series; why it is problematic; how it can be improved. In: Gottman JM, ed. The analysis of change, New Jersey: Lawrence Erlbaum Associates
Del Mar CB, Green AC, Battistutta D. 1997 Do public media campaigns designed to increase skin cancer awareness result in increased skin excision rates? Aust N Z J Public Health. 21: 751- 754.Google Scholar
Dempsey CL. 1995 Nursing home-acquired pneumonia: Outcomes from a clinical process improvement program. Pharmacotherapy. 15: 33S- 38S.Google Scholar
1989 Department of Health and the Welsh Office. General practice in the National Health Service: A new contract. London: HMSO
Domenighetti G, Luraschi P, Casabianca A et al. 1988 Effect of information campaign by the mass media on hysterectomy rates. Lancet. 2: 1470- 1473.Google Scholar
Draper N, Smith H. 1981 Applied regression analysis. New York: Wiley
Elam K, Taylor V, Ciol MA, Franklin GM, Deyo RA. 1997 Impact of a worker's compensation practice guideline on lumbar spine fusion in Washington State. Med Care. 35: 417- 424.Google Scholar
Eppler E, Eisenberg MS, Schaeffer S, Meischke H, Larson MP. 1994 911 and emergency department use for chest pain: Results of a media campaign. Ann Emerg Med. 24: 202- 208.Google Scholar
Everitt DE, Soumerai SB, Avorn J, Klapholz H, Wessels M. 1990 Changing surgical antimicrobial prophylaxis practices through education targeted at senior department leaders. Infect Control Hosp Epidemiol. 11: 578- 583.Google Scholar
Fowkes FG, Evans RC, Williams LA et al. 1984 Implementation of guidelines for the use of skull radiographs in patients with head injuries. Lancet. 2: 795- 796.Google Scholar
Fowkes FG, Hall R, Jones JH et al. 1986 Trial of strategy for reducing the use of laboratory tests. BMJ Clin Res Ed. 292: 883- 885.Google Scholar
Fraser GL, Wennberg DE, Dickens JD Jr, Lambrew CT. 1996 Changing physician behavior in ordering digoxin assays. Ann Pharmacother 30: 449- 454.Google Scholar
Gama R, Nightingale PG, Ratcliffe JG. 1992 Effect of educational feedback on clinicians' requesting of cardiac enzymes. Ann Clin Biochem. 29: 224- 225.Google Scholar
Gortmaker SL, Bickford AF, Mathewson HO, Dumbaugh K Tirrell PC. 1988 A successful experiment to reduce unnecessary laboratory use in a community hospital. Med Care. 26: 631- 642.Google Scholar
Gottman JM. 1981 Time series analysis: A comprehensive introduction for social scientists. Cambridge, England: Cambridge University Press
Grilli R, Ramsay CR, Minozzi S. 2002 Mass media interventions: Effects on health services utilisation. In: The Cochrane Library. Oxford: Update Software Issue 1.
Grimshaw JM, Thomas RE, MacLennan G et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess (in press).
Gurwitz JH, Noonan JP, Soumerai SB. 1992 Reducing the use of H2-receptor antagonists in the long-term-care setting. J Am Geriatr Soc. 40: 359- 364.Google Scholar
Hammond KW, Snowden M, Risse SC, Adkins TG, O'Brien JJ. 1995 An effective computer-based tardive dyskinesia monitoring system. Am J Med Qual. 10: 133- 137.Google Scholar
Harris DW, Karandikar DS, Spencer MG et al. 1979 Returned-medicines campaign in Birmingham, 1977. Lancet. 1: 599- 601.Google Scholar
Healsmith MF, Graham-Brown RA, Osborne JE, London SP, Fletcher A. 1993 Further experience of public education for the early diagnosis of malignant melanoma in Leicestershire. Clin Exp Dermatol. 18: 396- 400.Google Scholar
Herd RM, Cooper EJ, Hunter JA et al. 1995 Cutaneous malignant melanoma. Publicity screening clinics and survival – the Edinburgh experience 1982-90. Br J Dermatol. 132: 563- 570.Google Scholar
Joshi UY, Cameron SO, Sommerville JM, Sommerville RG. 1988 HIV testing in Glasgow Genito-Urinary Medicine Clinics 1985-1987. Scott Med J. 33: 294- 295.Google Scholar
Kong KM, Johnson WF, Anderson PO, Sato LA, Simonian RI. 1987 Effect of a pharmacy-based cost-containment program on cimetidine costs and use. Am J Hosp Pharm. 44: 2068- 2073.Google Scholar
Legorreta AP, Hasan MM, Peters AL, Pelletier KR, Leung KM. 1997 An intervention for enhancing compliance with screening recommendations for diabetic retinopathy. A bicoastal experience. Diabetes Care 20: 520- 523.Google Scholar
Lomas J, Anderson GM, Domnick-Pierre K et al. 1989 Do practice guidelines guide practice? The effect of a consensus statement on the practice of physicians. N Engl J Med. 321: 1306- 1311.Google Scholar
Lowe JB, Balanda KP, Del Mar CB, Purdie D, Hilsdon AM. 1994 General practitioner and patient response during a public education program to encourage skin examinations. Med J Aust. 161: 195- 198.Google Scholar
Macdonald H, Roder D. 1985 The planning, implementation and evaluation of an immunization promotion campaign in South Australia. Hygie. 4: 13- 17.Google Scholar
Maclure M, Dormuth C, Naumann T et al. 1998 Influences of educational interventions and adverse news about calcium-channel blockers on first-line prescribing of antihypertensive drugs to elderly people in British Columbia. Lancet. 352: 943- 948.Google Scholar
Matowe L, Ramsay CR, Grimshaw JM, Gilbert FI, Macleod MJ, Needham G. 2002 Effects of mailed dissemination of the Royal College of Radiologists' guidelines on general practitioner referrals for radiography: A time series analysis. Clin Radiol. 57: 575- 578.Google Scholar
Morrison JC, Sumrall DD, Chevalier SP et al. 1993 The effect of provider education on blood utilization practices. Am J Obstet Gynecol 169: 1240- 1245.Google Scholar
Nattinger AB, Hoffmann RG, Howell-Pelz A, Goodwin JS. 1998 Effect of Nancy Reagan's mastectomy on choice of surgery for breast cancer by US women. JAMA. 279: 762- 766.Google Scholar
Novich M, Gillis L, Tauber AI. 1985 The laboratory test justified. An effective means to reduce routine laboratory testing. Am J Clin Pathol. 84: 756- 759.Google Scholar
Paunio M, Virtanen M, Peltola H et al. 1991 Increase of vaccination coverage by mass media and individual approach: Intensified measles, mumps, and rubella prevention program in Finland. Am J Epidemiol. 133: 1152- 1160.Google Scholar
Pearce MJ, Begg EJ. 1997 Encouraging consensus cost effective drug therapy: Five years experience with a hospital drug utilisation review programme. N Z Med J. 110: 92- 95.Google Scholar
Pehamberger H, Binder M, Knollmayer S, Wolff K. 1993 Immediate effects of a public education campaign on prognostic features of melanoma. J Am Acad Dermatol. 29: 106- 109.Google Scholar
Poma PA. 1998 Effect of departmental policies on cesarean delivery rates: A community hospital experience. Obstet Gynecol 91: 1013- 1018.Google Scholar
Ratnaike S, Hunt D, Eilermann LJ, Hazen R, Deam D. 1993 The investigation of chest pain: Audit and intervention. Med J Aust. 159: 666- 671.Google Scholar
1993 Royal College of Radiologists Working Party. Influence of Royal College of Radiologists' guidelines on referral from general practice. BMJ. 306: 110- 111.
Santerre RE. 1996 The effect of the ACOG guideline on vaginal births after cesarean. Med Care Res Rev. 53: 315- 329.Google Scholar
1989 Secretaries of State for Social Services WNIaS. Working for patients. London: HMSO
Shelley JM, Irwig LM, Simpson JM, Makaskill P. 1991 Evaluation of a mass-media-led campaign to increase Pap smear screening. Health Educ Res. 6: 267- 277.Google Scholar
Sherman CR, Potosky AL, Weis KA, Ferguson JH. 1992 The Consensus Development Program. Detecting changes in medical practice following a consensus conference on the treatment of prostate cancer. Int J Technol Assess Health Care. 8: 683- 693.Google Scholar
Shorr RI, Fought RL, Ray WA. 1994 Changes in antipsychotic drug use in nursing homes during implementation of the OBRA-87 regulations. JAMA. 271: 358- 362.Google Scholar
Soumerai SB, Ross-Degnan D, Kahn JS. 1992 Effects of professional and media warnings about the association between aspirin use in children and Reye's syndrome. Milbank Q. 70: 155- 182.Google Scholar
Soumerai SB, Avorn J, Gortmaker S, Hawley S. 1987 Effect of government and commercial warnings on reducing prescription misuse: The case of propoxyphene. Am J Public Health. 77: 1518- 1523.Google Scholar
Stuart ME, Macuiba J, Heidrich F et al. 1997 Successful implementation of an evidence-based clinical practice guideline: Acute dysuria/urgency in adult women. HMO Pract. 11: 150- 157.Google Scholar
Studnicki J, Remmel R, Campbell R, Werner DC. 1997 The impact of legislatively imposed practice guidelines on cesarean section rates: The Florida experience. Am J Med Qual. 12: 62- 68.Google Scholar
Suwangool P, Moola-Or P, Waiwatana A et al. 1991 Effect of a selective restriction policy on antibiotic expenditure and use: An institutional model. J Med Assoc Thai. 74: 272- 275.Google Scholar
Tesoriero JM, Sorin MD. 1992 The effect of ‘Magic’ Johnson's HIV disclosure on anonymous HIV counselling and testing services in New York State. AIDS Public Policy J. 7: 216- 224.Google Scholar
Thamer M, Ray NF, Henderson SC et al. 1998 Influence of the NIH Consensus Conference on Helicobacter pylori on physician prescribing among a Medicaid population. Med Care. 36: 646- 660.Google Scholar
Turner GC, Mutton KJ. 1987 HIV testing: Changing trends. BMJ. 295: 502.Google Scholar
van Walraven C, Goel V, Chan B. 1998 Effect of population-based interventions on laboratory utilization: A time-series analysis. JAMA. 280: 2028- 2033.Google Scholar
Vincent EC, Hardin PA, Norman LA, Lester EA, Stinton SH. 1995 The effects of a computer-assisted reminder system on patient compliance with recommended health maintenance procedures. Proc Ann Symp Comput Appl Med Care. 656- 660.Google Scholar
Wong ET, McCarron MM. Shaw ST Jr. 1983 Ordering of laboratory tests in a teaching hospital. Can it be improved? JAMA. 249: 3076- 3080.Google Scholar
Zehr KJ, Dawson PB, Yang SC, Heitmiller RF. 1998 Standardized clinical care pathways for major thoracic cases reduce hospital costs. Ann Thorac Surg. 66: 914- 919.Google Scholar
Figure 0

Box 1. Quality Criteria for ITS Designs

Figure 1

The effect sizes estimated by time series regression analysis of an ITS design.

Figure 2

Characteristics of Included Interrupted Time Series Studies

Figure 3

Quality Criteria for ITS Studies

Figure 4

Characteristics of Reanalyzed Studies