THE NATURE OF HEALTH SERVICES RESEARCH

Health services research is a typical example of applied research, which aims at the solution of health care policy problems by using theoretical insights and scientific methods from multiple disciplines. In this type of research, the first step to be taken is the transformation of a policy problem into a set of research questions (Figure 1). As opposed to research from basic academic disciplines in which research questions are formulated within scientific disciplines (“science push”), health services research is characterized by “science pull”, which often asks for an interdisciplinary approach. Whether policy makers benefit from the study results is in the first instance determined by the adequacy of this translation of a practical and complex problem into a precise and researchable question and by developing a proper research design using suitable and viable methods. Once data are collected and analyzed, the mission of health care research is to facilitate the utilization of its results by health care policy makers. This concept again requires a transformation step: now from research conclusions to policy implications and solutions to the original problems in the health care sector. Subsequently, research findings and their implications have to be made accessible and communicated to the target groups. Dissemination and implementation strategies have to be elaborated, taking the complexity of the practical situation into account (Figure 1).

Interaction between the societal sector and research.

DOING RESEARCH - MAKING POLICY

Health care policy might benefit from health services research in developing evidence-based policy. However, research will always be only one of the many inputs that determine policy. Weiss (17) uses the metaphor of an arena, where researchers have to deal with the four i's: interests (of civil servants, politicians, or other stakeholders), ideologies (political agendas, prevalent norms and values), information (research is but one source of information that has to compete with many other sources), and institutional forum (the idiosyncrasy of governmental organizations). Moreover, research is not always used in a unidimensional way. Utilization can be instrumental (i.e., the direct application of results), conceptual (i.e., generating new policy ideas), or strategic (i.e., having a role in the political process) (10). Research can also be used for purposes of legitimization or to warn of pending problems. And perhaps the most important function of policy research is its “enlightenment function”: challenging old ideas, providing new perspectives, and helping to re-order the policy agenda (17).

This strategy takes time and a dynamic process. When different stakeholders have conflicting interests, time is needed to negotiate the exact formulation of the research question to avoid discussions on the usefulness of the project afterward. Furthermore, the problem situation, the need for policy interventions and circumstances change over time. In developing this type of research, Lilford and others promote the so-called iterative method. To provide a productive dialogue between researchers and potential users, the scope, form, and content of the research are not specified before the start of the study, but are developed iteratively, in interaction between researchers and policy makers (9).

Most policy makers are well-disposed to health services research, because they need to be informed to make evidence-based decisions. On the other hand, there is a difference of view between researchers and policy makers on the appropriate topics for health services research and on its approaches and methods (13). Whereas researchers opt for simplification of the problem, so that it can be put through scientific methods derived from experimental domains, policy makers need evidence for tackling complex issues and making decisions (see Table 1). Whereas most researchers are primarily driven by a (“science-push”) need to “find the truth,” policy makers are particularly driven by their (“science-pull”) need to reduce uncertainties in the decision-making process. For the same reason, policy-makers favor research that can give a short and clear-cut answer to the policy-problems at hand, whereas researchers may be triggered by their intellectual curiosity to focus on related but separate problems that may creep into their agendas. Another factor that may divide policy makers and researchers is the issue of timing: Policy makers often look to relatively short-term horizons and are not attracted to research that takes a long time (15). However, discrepancies in timing can also work the other way around, when for political reasons policy makers have vested interests in control and delay, whereas researchers want to share their results promptly with the scientific community. Finally, policy makers need practical advice and assistance in finding and using the best methods to implement the research findings, whereas researchers prefer to describe the range of nuance demonstrated by the study in scientific publications, sometimes even omitting the transformation of their conclusions into implications for policy. This complexity in the policy arena with its mutually dependent players has two important consequences: (i) health services researchers must seriously invest in their relationship with the other participants in the policy arena to understand the complexity and context of the policy problems, and (ii) the usefulness of research is at least partly dependent on its fit with the type and timing of information that is needed (“relevance is in the eye of the beholder”).

This finding means that it is extremely important that policy research be closely tuned to the policy arena, that is, to ask the right questions, at the right time, and communicate clearly in the policy arena. If researchers want their research to be useful to policy makers they have to cooperate in defining the research questions and developing a research design. Furthermore, researchers could give practical advice on and assistance with the most effective methods of implementing policy decisions (13).

EX ANTE VERSUS EX POST ASSESSMENT

In assessing the policy relevance of research, it is necessary to distinguish between the potential (ex ante) relevance and the realized (ex post) impact (1;4). An ex ante evaluation of societal relevance focuses on the transformation of societal problems into research questions (see Figure 1). Ex post societal impact refers to the degree to which a research project, program, or institute has been able to answer these research questions and to translate the scientific conclusions into practical solutions or policy implications. The distinction between ex ante and ex post assessment can but does not necessarily coincide with the moment of assessment (research proposal versus research output). In fact, assessment of societal relevance should always keep both in mind. The following review is probably not complete but gives directions for the development of explicit indicators.

Ex Ante Indicators

The first and most important area is the relevance of the policy problem (see Table 2). Utilization research has shown that the priority of the policy problem addressed is one of the main determinants of policy makers' judgment on the usefulness of social science research (along with the degree to which research is action oriented, offers new insights and perspectives that are plausible given prior knowledge, has a high scientific value, and provides challenging results) (14). The priority of a policy problem is, of course, difficult to determine. In the priority setting for medical technology assessment research, the criteria included burden of disease, potential effects of the technology, and potential costs of the technology (11;12;19). Burden of disease is a criterion that often cannot be used outside medical technology assessment in a less broad definition. However, in health services research, relevance can be defined in terms of the general goals of health care systems, such as responsiveness to patients' needs, equity, accessibility, and quality of care in relation to costs (4;5;18). From this point of view, sectors of the health care system, the work of certain professional groups in health care, or health care reforms can be prioritized, based on their contribution to these goals.

Closely related to relevance of the policy problem is the translation of policy problems into research questions. This area has two sides: the extent to which the policy problem has been explored by studying policy documents and contacts with experts in the field on the one side, and the assessment of the problem in terms of whether or not research can be part of the solution on the other. This is not as self-evident as it may seem at first glance; sometimes both researchers and policy makers look to research as a solution, behaving like the proverbial drunken man who lost his keys somewhere, but looks for them under the lamp-post because that is where the light is.

The feasibility of conducting research in complex field situations is also an area of policy relevance. Policy relevant research is usually related to interests of the stakeholders in the phenomenon being studied. This statement means that the research situation has to be structured in such a way that the different interest groups will accept the project (including its methodology) as a potentially relevant contribution to the problem at hand. An indicator of societal quality is that an acceptable research design is developed in a difficult field situation and is not attacked afterward for methodological flaws. This concept is an important criterion, because health services research often does not allow for standard designs, such as randomized controlled trials.

The expected contribution of research to the solution of policy problems and the proposed implementation of research results is another area for indicator development. It refers to the proactive attitude of the researchers in preparing the ground for implementation of the results. A similar indicator is used in priority setting of medical technology assessment research, where the uncertainty of applying a technology is assessed in advance.

In ex ante evaluations of scientific, quality past performance is often used as an indicator area. In assessing societal quality of research, past performance has to be used in a way different from that in scientific quality. In health services research, it is more important that a researcher or a group of researchers be familiar with the set of theories, research methods, and instruments of health services research than with a particular subject. For example, it is more important that a researcher be qualified in manpower planning than that he or she has conducted previous research in the field of orthopedics for a prospective evaluation of the future demand for orthopedic surgeons. Similarly, notwithstanding the need for specific experts as part of the research team, in research about care for the chronically ill (as opposed to cure), disease transgressing theory formation on coping processes is sometimes more important than disease-specific knowledge.

Ex Post Indicators

An important area for assessing policy relevance is knowledge production that is oriented to the policy domain. Ex post evaluation in this area may use quantitative data, such as counts of policy documents and publications in professional journals.

In developing ex post indicators of policy relevance, it is critical that the results of research reach those who have to use it. Assessment of policy relevance, therefore, should be focused on dissemination, that is, the transfer of research findings to the target groups. Indicators at the level of an individual study or a research program are publications in professional journals or policy documents, presentations for a nonscientific audience, and teaching and press publicity. Given that research findings reach the target groups, the question is whether they had any use for the results. Feedback from users is important in assessing the societal quality of research. Looking at the level of an institute or research group, important indicators of societal output include membership of advisory committees, or individual advice, and the organization of invitational conferences (4). In the area of cooperation between research and policy, indicators of relevance can be found in the interaction between researchers and the target groups. Another area by which societal relevance at the level of the institute can be assessed is the mobility of researchers (16). Membership in organs of government can be distinguished as well as the movement of researchers to the policy domain. Finally, perhaps the most important indicator of policy relevance is the degree of (semi) governmental funding of studies, programs, or institutes, as a continuous flow of project grants can probably be considered as the best indicator of researchers' tuning to relevant stakeholders.

It has to be admitted that not all indicators, mentioned in Table 3, will be equally important at the same time. The criteria and indicators used for an evaluation will always be dependent on the mission of a specific research group or institution (8).

SUMMARY AND CONCLUSION

Health services research is a typical instance of applied research, meaning that research findings should contribute to more evidence-based health policy. Nevertheless, from a methodological point of view, this kind of research is not distinguishable from traditional, discipline-oriented scientific research, because studies are performed in a logical sequence of predetermined steps, using objective and systematic methods. A basic assumption for the usefulness of policy research is that it is of good scientific quality, and in that field – the scientific quality of research – there is much more consensus than in the assessment of societal quality. So, good quality research has to meet the standards of science, but evaluation of health services research would be incomplete without an assessment of its societal relevance.

However, there is no generally accepted framework in which criteria for the assessment of societal relevance are defined. In this study, we made an effort to specify criteria and indicators for policy relevance. In general, we concluded that it is important to define what kind of answers are being sought to the questions at hand, who the intended users of these answers will be, and how they can be reached from the start of a research project. Furthermore, to become closely attuned to the policy scene, a productive dialogue between researchers and policy makers is indispensable. In addition, we specified domains and a list of potential criteria for assessing policy relevance. For the purpose of quality assurance, these criteria have to be integrated in a quality improvement cycle. This means that the societal output of health services research should be related to the aims and missions at the level of projects, programs, and institutes.

In defining and measuring indicators of policy relevance, one is confronted with (at least) two problems: how to collect the information needed, and how to weight different criteria and indicators. Collecting information needs to be organized, because most data referring to societal quality are not routinely recorded. Publications are fairly easy to track, but membership of an advisory committee for instance, might only be registered in a researcher's agenda. However, this finding is not a fundamental problem but dependent on the culture and prevalent incentives. People are inclined to find important and consequently keep track of those things that give them status and rewards within their community. Basic to a good system of keeping track of relevant indicators of societal quality of research is that proper incentives are designed, setting certain indicators higher on the agenda.

Weighting the different criteria and indicators of policy relevance is another challenge. The weighting can be arbitrary and can lead to controversy. For publications in professional journals or policy documents, there is no “societal” counterpart to the (Social) Science Citation Index comprising impact scores. In fact, the various indicators of policy relevance should be registered and weighted in relation to each other, resulting in a total profile of a research project; the embedding program and the research institute have solved this problem by means of a transformation of scales and a graphical representation, which provides a profile for each research group (16). But there might be other solutions. As interpretation of the data can differ, these profiles should be used as a tool rather than as an assessment in its own right (7). Furthermore, the central role of stakeholders in the evaluation of the societal quality of health services research demands an assessment system in which facts and figures should be complemented by site visits from a well-balanced committee, that questions the researchers in depth about the relevant elements of societal quality.

POLICY IMPLICATIONS

Current academic assessment systems steer researchers – voluntarily or involuntarily – toward presenting research results predominantly in peer-reviewed international journals. Although this approach may be the perfect medium for dissemination within the academic community, it is not the optimal way to get research results widely known among policy makers who are the principal target group for much health services research. This finding has led to a widely recognized problem with regard to the implementation of results from health services research, rightly or wrongly leading to statements such as, “There is enough knowledge; the main problem is that existing knowledge is not used efficiently.” To tackle this problem, efforts should be made at several levels:

• At the national and international levels: development of measures for societal relevance within existing academic assessment systems;
• At the level of research institutions: creating incentives for researchers who excel in bridging the gap between research and policy;
• At the level of research training: developing courses for PhD students and researchers to enhance awareness of the complicated but fascinating dynamics of applied health services research and to strengthen their competencies to take active part in it.