Hostname: page-component-745bb68f8f-l4dxg Total loading time: 0 Render date: 2025-02-06T11:26:24.076Z Has data issue: false hasContentIssue false
  • English
  • Français

Use of systematic reviews in the development of new provincial public health policies in Ontario

Published online by Cambridge University Press:  01 November 2004

Maureen Dobbins ,
Helen Thomas ,
Mary Ann O'Brien  and
Melissa Duggan
Maureen Dobbins
Affiliation:
McMaster University
Helen Thomas
Affiliation:
McMaster University
Mary Ann O'Brien
Affiliation:
McMaster University
Melissa Duggan
Affiliation:
McMaster University
Rights & Permissions [Opens in a new window]

Abstract

Objectives: The study determined whether the results of recently completed systematic reviews evaluating the effectiveness of public health interventions were used in the development of new provincial policies for public health practice.

Methods: This telephone survey included all members from five review groups who updated the Ontario Mandatory Health Programs and Services guidelines for Public Health in 2000. Independent variables included characteristics of the systematic reviews, organization, and the individual. Outcomes included the use of the reviews in developing new policies and the extent to which the reviews led to new recommendations for practice. Descriptive summaries as well as multiple linear regression were conducted.

Results: Eighty-five percent of decision-makers agreed to participate in the study. Ninety-six percent of respondents reported that the systematic reviews played a part in developing the new guidelines, while 47 percent indicated that the reviews contributed a great deal to the development of new recommendations for practice. The multiple linear regression model explained 42 percent of the variation in use of the reviews for developing new recommendations for practice. Significant predictor variables included the importance of the reviews in comparison to other sources of information and relevance of the reviews to the policy decisions.

Conclusion: Public health decision-makers in Ontario have very positive perceptions of the usefulness of systematic reviews in policy development. Therefore, ongoing efforts to promote the usefulness and relevance of systematic reviews to public health decision-makers should remain a priority for health services researchers.

Keywords

Evidence-based decision makingHealth policyPractice guidelinesPublic health
Type
GENERAL ESSAYS
Information
International Journal of Technology Assessment in Health Care , Volume 20 , Issue 4 , November 2004 , pp. 399 - 404
Copyright
© 2004 Cambridge University Press

The National Forum on Health in Canada in 1997 (16) identified the promotion of research transfer and uptake and evidence-based decision making, as two important priorities for the Canadian health-care system for the new millennium. However, currently, there is little consensus on how to promote the transfer and uptake of research evidence from its producers (health services researchers) to potential users (policy-makers, program decision-makers), so as to facilitate evidence-based health policies. Even the conduct of several systematic reviews summarizing the effectiveness of dissemination strategies (1;5;13;18;21) has not led to definitive conclusions of how to ensure the integration of research evidence into health policy decisions.

However, factors such as key stakeholders, organizational culture and values, individual decision-making style, the research evidence itself, and the importance of the decision to the goals and mission of the organization have been shown to influence the use of research evidence in decision making. The purpose of this study was to determine whether the results of systematic reviews evaluating the effectiveness of public health interventions were incorporated into the recommendations for new provincial public health practice guidelines in Ontario.

The provision of public health services in Ontario is governed by the Ministry of Health and Long Term Care, Public Health Branch, through the Mandatory Programs and Services guidelines (17). These guidelines, which were last reviewed in 1989, set a minimum standard of service provision for which each public health unit is responsible. Ongoing yearly resource allocation from the Ministry of Health to individual public health units is dependent on health units achieving these minimum standards.

In 1999, the Public Health Branch of the Ministry of Health and Long Term Care for Ontario, initiated a process whereby the current Mandatory Programs and Services guidelines (1989) would be reviewed and updated to correspond with the changing demographic situation and subsequent health-care needs of residents of Ontario. The process included the creation of five technical review groups composed of public health decision-makers from various levels across the province. After numerous meetings over the course of several months, all five technical review groups came to consensus on recommendations for the new public health practice guidelines in Ontario.

The systematic reviews examined in this study were developed by the Effective Public Health Practice Project (EPHPP), a provincial project initiated in 1998 with the mission of evaluating the effectiveness of public health practice (9). The topics of completed reviews were chosen in collaboration with a provincial advisory group to ensure review relevance to policy and program decisions. The methods used to conduct these systematic reviews followed those outlined by the Cochrane Collaboration (20).

METHODS

Study Sample

This telephone-administered, cross-sectional survey included all members from the five technical review groups to update the Mandatory Programs and Services guidelines. The content areas of the five groups included Chronic Disease Prevention; Child Health; Reproductive Health; Injury Prevention; and the Early Detection of Cancer. Members of the technical review groups included program managers, directors, epidemiologists, medical officers of health, provincial consultants, and local board of health members.

Independent and Dependent Variables

The results of research from several bodies of literature, including the diffusion of innovations and organizational management and decision making, have been instrumental in the identification of characteristics that impact on research transfer and uptake in general (4), as well as among public health decision-makers specifically (6). All of the independent variables were measured as individual items using Likert scales or as continuous or dichotomous variables. The independent variables included characteristics of the innovation (systematic reviews), working group, and the individual. A list of the variables and their operational definitions has been published previously (6).

There were two dependent variables measured in this study. Both measures were chosen based on their previous use in other knowledge transfer studies (11). The first outcome, measured as a dichotomous variable, assessed generally, the use of the results of the reviews in the new guidelines. The remaining outcome, measured using a five-point Likert scale, assessed the extent to which the results of the reviews led to new recommendations for public health practice.

Instruments

The data were collected using a 15-minute telephone administered questionnaire which is available from the author. The questionnaire was modified from previous research transfer and uptake studies (12). The instrument was pretested for content and face validity, as well as test-retest reliability among public health decision-makers at one public health unit. The Cronbach alpha score for reliability was .65, which compares favorably with the reliability coefficients obtained for other research utilization questionnaires (8;10;11).

Statistical Analysis

The analysis included simple descriptive summaries as well as analyses of variance, bivariate correlation analysis, and multiple linear regression analysis using a backward stepwise procedure. A backward stepwise regression approach was used because of the exploratory nature of this study (19).

RESULTS

Sample Description

A summary of results is presented in Table 1. Eighty-five percent of decision-makers agreed to participate in the study (51 of 60). Participants included program managers (23 percent), program directors (16 percent), Medical Officers of Health (8 percent), epidemiologists (8 percent), provincial consultants (18 percent), and elected members of the local boards of health as well as other health professionals (28 percent). Ninety-four percent of respondents indicated that their technical review group strongly or moderately valued the use of the reviews in developing the guidelines and sixty percent indicated that the systematic reviews were very important compared with other forms of evidence (i.e., past experience, preferences, opinions).

Characteristics of the Systematic Reviews

Sixty-one percent of respondents indicated that the executive summary was the most important component of the systematic review, followed by the conclusion (42 percent), abstract (39 percent), and text (36 percent). Seventy-five percent of the sample indicated that the reviews were very good or excellent in terms of being easy to use, whereas only 6 percent reported that they were not easy to use. Finally, 65 percent of respondents reported that the systematic reviews were very good or excellent in terms of being relevant to the guideline decisions being discussed.

Impact of the Reviews on the New Guidelines

Ninety-six percent of respondents reported that the results of the systematic reviews played a part in the development of the new public health guidelines. Just under half (47 percent) of the participants reported that the results of the systematic reviews completely or very extensively led to new recommendations for practice, and no participants reported that the reviews had no impact. There were no significant differences observed between the five technical review groups or between the different decision-making groups (i.e., program managers, directors, consultants) on either outcome measure.

Factors That Significantly Explained Variation in Use

There were no significant characteristics identified in the analysis of variance when the outcome was overall use of the reviews in developing the new recommendations. However, four variables were significant in explaining variation in the extent to which systematic reviews led to new recommendations for practice. These four variables were included in a multiple linear regression analysis, two of which remained in the final model: the importance of the reviews in comparison to other forms of evidence, and the relevance of the reviews to the new guidelines. The adjusted R-square of the model was .42, and the standardized beta coefficients were .536 for the importance of reviews in comparison to other types of evidence and .270 for the relevance of the reviews to the new guidelines. The tolerance values, which measure the extent to which the variables in the model are correlated, were above 0.9, demonstrating no significant multi-collinearity between the two independent variables.

DISCUSSION

Implications for Health Policy

The results reported in this study are encouraging not only for the field of research transfer and uptake but also for public health. These results suggest that provincial health policies can be research-based, given the right environment, relevant systematic reviews, and a culture that values research evidence to a greater extent than other forms of evidence. Similar results were reported by Lavis et al. (14) and Marriott et al. (15), who both found that health services research had influenced health policies. Compared with other health sectors (2), the results of this study identify public health decision-makers in Ontario as being leaders with respect to evidence-based health policy decision making.

Commonly sited reasons for not using research evidence in policy decision making include (i) different goals among policy-makers and researchers; (ii) research evidence being dismissed as irrelevant; (iii) lack of consensus about interpretation of research evidence; (iv) other types of evidence perceived as carrying more weight; and (v) a social environment not conducive to policy change (2;22). In this study, at least three of these issues were not present. For example, the systematic reviews available for use were all highly relevant to the public health policies and practices under revision, decision-makers valued the use of the systematic reviews to a greater extent than they did other types of information, and the social environment was conducive to the use of reviews in decision making. It was surprising that some of the independent variables that previously have been found to predict research use were not significant predictors in this study. This finding is likely explained by the limited variability in the range of observations across participants for these characteristics. This explanation suggests that these characteristics might not be appropriate measures for predicting research utilization among this population or that we need an alternative way of measuring these variables.

Implications for Knowledge Transfer Activities

These results are encouraging for researchers engaged in conducting public health and health promotion-related research. A 1995 Ontario survey composed mostly of public health decision-makers found that almost 60 percent of decision-makers did not perceive the available research evidence to be relevant to the decisions they were faced with (22), and that they rarely consulted research evidence in the decision-making process. Only six years later, this study demonstrated a significant change in this perception with 90 percent of decision-makers reporting the reviews to be good to excellent in terms of their relevance to current decisions. This finding is in part the result of a concerted effort on the part of public health and health promotion researchers to engage in dialogue with public health decision-makers in Ontario to determine current and future information needs. Two examples of such activities come from the Effective Public Health Practice Project, and the McMaster System-Linked Research Unit on Health and Social Service Utilization (3;9). In both instances, efforts were made to work closely with policy-makers, program decision-makers and practitioners, not only in the planning of research projects but also in interpreting research results and developing implications for practice.

Another interesting result from these data is the level of importance decision-makers placed on the various components of systematic reviews. The majority of decision-makers rated the executive summary as being the most important component of the systematic reviews with regard to policy decision making. This finding suggests that a great deal of effort should be placed on writing an executive summary for every systematic review. The executive summary should include the key messages of the research evidence as well as the implications for policy and practice but should also be brief (one to two pages in length) and easy to read and understand.

Policy Implications

The results of this study demonstrated that decisions regarding the development of provincial public health policies incorporated the results of systematic reviews, at least at the preliminary draft stages. It remains to be seen, however, if the final draft of these policies will remain true to the research evidence, when these policies are considered along with all the other health-care needs that are competing for scarce financial resources. However, the integration of research evidence at such a high provincial policy level is very encouraging.

Limitations of the Study

The results of this study are only directly generalizable to the members of these technical review groups. The results may not necessarily be applicable for public health decision-makers not involved in the technical review process within Ontario or other public health professionals outside of Ontario who may have different roles or decision-making responsibilities. However, it is likely that these results would provide a “starting place” for understanding the potential uses and influence of systematic reviews in public health policy and practice. It is also likely that the behaviors and perceptions expressed by the participants of this study are similar to those of other public health decision-makers in Ontario, since in a previous study, Dobbins et al. (7) reported considerable homogeneity across provincial public health decision-makers with respect to their perceptions of the usefulness of systematic reviews in program planning.

There was a relatively small sample in this study (N=51), which was problematic given the multivariate analysis and several independent variables. However, almost all of the technical review group members agreed to participate in the study, making this sample highly representative of this target population. Even if a similar sample of health policy-makers engaged in a similar process from other provinces were used in this study to increase the sample size, the significant variation in the organization of public health services across provinces would have resulted in greater threats to internal and external validity than currently exists.

The number of independent variables analyzed on a relatively modest sample size was the most disturbing limitation of this study. This may have resulted in some variables being significant due to chance alone. However, if a correction for multiple comparisons (i.e., Bonferroni) had been used, the combination of a small sample size and several independent variables would have resulted in such a small alpha level that none of the variables could have been statistically significant in the model. Because this was the first time these characteristics were examined in relation to the use of systematic reviews for provincial public health policy making, it was important to gain an understanding of the impact of each independent variable on the outcomes.

CONCLUSIONS

The results of this study demonstrated a marked progression in the public health sector with respect to the integration of research evidence into health policies over the past decade. In general, a culture that is very supportive of the use of systematic reviews in policy development has emerged, as well as a perception that systematic reviews can be a more important source of information in policy development compared with other sources of information. It will be important over this next decade for researchers conducting public health and health promotion research to maintain existing relationships with decision-makers across the province, as well as to develop new relationships where necessary. This strategy will ensure the ongoing conduct of relevant systematic reviews, as well as contribute to maintaining a culture that supports the use of systematic reviews in policy decision making.

This research study was part of a larger study funded by the Canadian Health Services Research Foundation.

References

Bero LA, Grilli R, Grimshaw JM, et al. 1998; Closing the gap between research and practice: An overview of systematic reviews of interventions to promote the implementation of research findings. BMJ. 317: 465– 468.Google Scholar
Black N. 2001; Evidence based policy: Proceed with care. BMJ. 323: 275– 279.Google Scholar
Browne GB. 1999; Evidence that informs practice and policy: The role of strategic alliances at the municipal, provincial and federal levels. Can J Nurs Res. 31: 79– 94.Google Scholar
Dobbins M. 1999. Characteristics of the innovation, organization, environment and individual that facilitate the utilization of five systematic review among public health decision-makers in Ontario. Toronto: Graduate Department of Health Administration, University of Toronto;
Dobbins M, Ciliska D, DiCenso A. 1998. Dissemination and use of research evidence for policy and practice: A framework for developing, implementing and evaluating strategies. Ottawa: The Canadian Nurses Association;
Dobbins M, Cockerill R, Barnsley J. 2001; Factors affecting the utilization systematic reviews: A study of public health decision-makers. Int J Technol Assess Health Care. 17: 203– 214.Google Scholar
Dobbins M, Cockerill R, Barnsley J, Ciliska D. 2001; Factors of the innovation, organization, environment, and individual that predict the influence five systematic reviews had on public health decisions. Int J Technol Assess Health Care. 17: 467– 478.Google Scholar
Edwards N, Lockett D, Gurd G, Leonard L. 2000. Effectiveness of internet-based dissemination of best practices for public health professionals. Ottawa: Office of Learning Technology;
1999. Effective Public Health Practice Project. Priority setting survey: PHRED systematic reviews. Hamilton: Effective Public Health Practice Project;
Funk SG, Champagne MT, Wiese RA, 1991; Tornquist EM. Barriers: The barriers to research utilization scale. Appl Nurs Res. 4: 39– 45.Google Scholar
Landry R, Amara N. 2001. The uptake of health research evidence by Canadian physicians.Prepared for the 4th International Conference on the Scientific Basis of Health Services. Quebec: University of Laval, Department of Political Science;
Landry R, Amara N, Laamary M. 2001; Utilization of social research science research knowledge in Canada. Res Policy. 30: 333– 349.Google Scholar
Lavis JN. 1999. Towards a new research transfer strategy for the Institute for Work and Health. Toronto, ON: Institute for Work and Health;
Lavis JN, Ross SE, Hohenadel J, et al. 2000. The role of health services research in Canadian provincial policy-making. 1997– 021;Google Scholar
Marriott S, Palmer C, Lelliott P. 2000; Disseminating healthcare information: Getting the message across. Qual Health Care. 9: 58– 62.Google Scholar
1997. National Forum on Health. Canada health action: Building on the legacy. Ottawa: National Forum on Health;
1989. Ontario Ministry of Health. Mandatory health programs and services guidelines. Toronto: Queen's Printer for Ontario;
Oxman AD, Thomson MA, Davis DA, Hayes JE. 1995; No magic bullets: A systematic review of 102 trials of interventions to improve professional practice. Can Med Assoc J. 153: 1423– 1431.Google Scholar
Polit DR. 1996. Data analysis and statistics for nursing research. Stamford, CT: Appleton & Lange;
Sackett D, Rosenberg WM, Gray JAM, Haynes RB, 1996; Richardson WS. Evidence-based medicine: What it is and what it isn't. BMJ. 312: 71– 72.Google Scholar
Thomson MA, Oxman AD, Davis DA, Freemantle N, Harvey EL. 1997; Outreach visits to improve health professional practice and health care outcomes. The Cochrane Library.: 1– 15.Google Scholar
Woodward CA, Feldman W, Snider A. 1997; Health services researchers and decision-makers: Are there really two solitudes. Ann R Coll Phys Surg Canada. 30: 417– 423.Google Scholar
Figure 0

Summary of Independent and Dependent Variables