Karl Marx noted that philosophers have only interpreted the world; “the point, however, is to change it.” While revolution may not be on the agenda, impact is nonetheless the new buzzword in British academia. This “impact agenda” has permeated the British higher education sector since it was decided to expand the criteria for assessing universities to include the “social and economic impact” of the research produced. Universities are under increasing pressure from government to demonstrate that they make a difference. Due to changes in the rules governing the allocation of funding to British higher education, university departments are no longer exclusively assessed on the basis of research excellence, but they are also judged on the broader societal impact stemming from their research. This may include either direct impact on policy or a more indirect contribution to an evidence-based policy debate. These developments in the assessment of British universities raise two interesting questions: first, how is such an impact defined and measured; and, second, what is the likely influence of these new assessment criteria on the public engagement and research of political science departments?
The exact definition of impact can be debated, but it can be described broadly as the influence of academic research on actors outside of higher education (e.g., in business, government, and civil society). All British universities have recently undergone a rigorous assessment of their research under the auspices of the Research Excellence Framework (REF). As part of the REF, the funding bodies aim to (1) identify and reward the impact that excellent research has had on society and the economy, and (2) encourage the sector to build on this impact to achieve its full potential across a broad range of research activities in the future. This article approaches the issue of public engagement of political science from the perspective of the attempt to assess and reward social and economic impact in the British university sector. It reviews how impact is defined and operationalized in the British REF. The discussion then considers the effect that this impact agenda has had on social sciences in the UK and whether there are lessons to be learned beyond the British borders.
THE IMPACT AGENDA
It is common for governments to demand that universities justify public funding of science and research efforts. In Britain, this call for greater public accountability of university funding was crystallized most clearly in the recent REF (completed in 2014) by incorporating “impact” into the overall quality profile of the research activity of a department (or research unit). This new model of university assessment also entails changes in budget allocations: 20% is determined by the societal-influence (i.e., impact) dimension, 65% by research outputs (i.e. publications), and 15% by the research environment. The importance assigned to impact is expected to rise in future assessment rounds.
Impact is defined by the REF as “an effect on, change, or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.” Footnote 1 In the REF panel for the social sciences, it further states that: “The main panel acknowledges that impact within its remit may take many forms and occur in a wide range of spheres. These may include (but are not restricted to): creativity, culture and society; the economy, commerce or organizations; the environment; health and welfare; practitioners and professional services; public policy, law and services.” Examples of impact in political science thus include the influence of research on public policy, law, and services—that is, where the beneficiaries usually are government, public-sector, and charity organizations and societies, through the implementation or nonimplementation of policies, systems, and reforms.
Although these definitions are inclusive, they also clarify what impact is not. First, it is not purely academic impact: the focus is on external impact beyond academia. Second, it is not an exercise in identifying high-profile academics with connections in the public-policy world: the impact must be based on high-quality, published research outputs for it to count as such.
Third, impact according to the REF is not the same as dissemination or even engagement: prominence in the mainstream and social media will not count as impact unless it can be shown that the research has had an effect on, changed, or benefited the economy, society, or culture of public policy. In other words, many activities that political scientists may deem as public engagement—such as speaking to journalists, writing blogs and policy papers about politics, and lecturing at public events—would not be classified as impact unless they are clearly grounded in a peer-reviewed published paper. Moreover, they must be perceived to have more broadly influenced public policy, governments, or society, although such activities clearly may be part of a strategy to achieve impact.
Fourth, impact is not limited to tangible positive changes in social outcome and even less so to claims for a clear-cut social-welfare gain. In other words, it is not necessarily causally linked to a social outcome that has been evaluated as beneficial to society. Stated succinctly, impact therefore is an external influence that is grounded in academic research.
HOW IMPACT IS ASSESSED: CASE-STUDY APPROACH
It is notoriously difficult to measure impact. How do we assess how much influence a particular article or an idea grounded in research has had on other spheres of social life? The REF panel adopted a “mixed-method” approach to the assessment of impact. Not only has impact been measured quantifiably in the REF; expert panels have also reviewed narrative evidence in case studies supported by appropriate indicators, compiled in the so-called impact case studies. Case studies generally are considered superior to a purely metric-based approach, especially in the social sciences, in which it is difficult to identify simple metrics that would capture impact. Each department was required to submit one impact case study for every 10 full-time academics. Hence, it is important to note that the expectation is not that every research project results in measurable external impact but rather that for every 10 academics working in a research unit, one research project should have resulted in substantial impact. The time frame for the attribution of impact to a piece of research is specified by the REF as “15 years between the publication of at least some research output(s) that made a distinctive contribution to the impact and the start of the assessment period.”
Stated succinctly, impact therefore is an external influence that is grounded in academic research.
The REF lists examples for different types of societal impact. For example, “policy debate on climate change or the environment has been influenced by research” and “quality of life in a developing country has improved.” Evidence of external impacts in these case studies can be in the form of references to, citations of, or discussions about a published piece of research (or the core idea, concept, or finding) in media or specialist media outlets; in the records of meetings; in the speeches or statements of authoritative actors; or through the direct involvement of academics in decision making in government agencies, government or professional advisory committees, business corporations or interest groups, and trade unions, charities, or other civil-society organizations. In my own university department, impact case studies in this assessment period ranged from the impact on the design of new macroeconomic institutions in the European Union to the improvement in transparency of legislative activities in the European Parliament and Council. Evidence of impact in these case studies also varied and included letters from ministers and prime ministers, references to published research in policy white papers, and mentions of scholars and key research ideas in mainstream media reporting and social media.
This broad definition of impact, however, also raises a concern that there is such a variety of ways to interpret and measure impact that, as a result, a comparative assessment of different research institutes by reviewers will be difficult and highly subjective. Moreover, the case-study approach may be superior to other metric-based approaches in capturing external influence; however, it also is a costly form of assessment to rely on the peer review of thousands of studies. Peer review has long been the “gold standard” in the British system of evaluating universities, and the assessment of individual research outputs (i.e., books and journal articles) is similarly based on a peer-review system. If we take into account all of the time devoted to preparatory work leading up to a REF submission in universities, as well as the time spent by the REF panels in assessing detailed submissions from each university unit in their field, then the total cost of the previous Research Assessment Exercise was already estimated at $150 million USD, and including the impact component only adds to it (Martin Reference Martin2011; Sastry and Bekhradnia Reference Sastry and Bekhradnia2006). In other words, there is potential difficulty in accurately capturing and measuring impact consistently and completely across university departments without developing an assessment system that is overly complex and costly.
LESSONS LEARNED
Given that the new REF process, with its novel impact assessment, was completed only at the end of 2014, it is premature to appraise the effect of the impact agenda on the nature and influence of social sciences in the United Kingdom. However, initial indications suggest—as might be expected—that institutions matter and that the impact agenda already has had a substantial effect in terms of incentivizing behavior that could strengthen the broader relevance of research, thereby facilitating a stronger science–society relationship. This surely is a positive outcome from the perspective of the public accountability of social sciences—as well as for the ability of our discipline to generate, directly or indirectly, social-welfare gains.
Institutionally, the impact agenda has manifested in various ways. Higher education institutions are making a concerted effort to facilitate impact by incentivizing academics to engage in activities with external actors and to establish relationships beyond narrow academic circles. Higher education institutions also have begun to systematically collect, collate, and record their impact-related work with external actors. The London School of Economics and Political Science (LSE), for example, established a knowledge-exchange and impact group to examine and facilitate these activities at the LSE, and have developed a strategy to strengthen its public-engagement activities.
In terms of public engagement, UK universities increasingly emphasize dissemination of research with public events programs that promote their own research strengths as well as external speakers. Dissemination strategies generally are integrated multimedia and multistage programs that include blogs, social media, media appearances, and podcasts, as well as more traditional town hall events. Another aspect of such dissemination strategies is that academics are increasingly encouraged to publish a form of their research on the open Web or to store it in a university’s online depository to ensure that readers outside of academia have access to it. Academics are also becoming more aware of working with intermediaries and networks (e.g., think tanks and government bodies) to broaden access to potential beneficiaries of research.
These institutional initiatives facilitate greater impact. However, within institutions, certain individuals are more likely to achieve impact than others. Initial studies suggest that more experienced academics—those with greater academic credibility, an external record of successful research, and a larger network of contacts—are more likely to achieve impact. A smaller study of 120 academics shows that academics who are cited more often in the academic literature in social sciences also are cited more often in nonacademic Google references from external actors (LSE Public Policy Group 2011). In other words, it appears that being a well-published scholar in traditional academic outlets also enhances a scholar’s chances of achieving external impact. However, research on impact in the social sciences recommends that academics should move beyond traditional outlets for their research to improve impact (e.g., by starting multiauthor blogs), which enables them to more broadly disseminate their research (LSE Public Policy Group 2011).
Overall, early indications suggest that the inclusion of “impact” in the assessment of British universities and funding allocations encourages institutions and individuals to strengthen the broader relevance of their research and engage more frequently with potential stakeholders. Universities have begun to be more attentive to how they can facilitate the external impact of the research and how they can reward individual academics whose research generates impact.
This can be viewed as a positive outcome; however, we also know that such changes to institutional rules may introduce perverse incentives and generate unintended consequences. A serious risk of the impact agenda is that the focus on external impact could be at the expense of academic independence (e.g., if academics sought to produce research that was more palatable to governments). This agenda also might encourage a shift toward policy-relevant social science at the expense of path-breaking, “blue-sky” research that offers little immediate external impact. A more prosaic concern is the cost associated with impact assessment. As the weight assigned to impact in public-funding allocation increases, so will the time devoted to preparatory work and “game-playing” at universities to ensure that research units perform well in this complex system of assessment. This often will come at the expense of time that could be devoted to carrying out actual research.
However, it is of paramount importance to ensure that this emphasis on impact does not mean sacrificing academic independence and integrity or abandoning research that does not produce immediate policy effects.
The impact agenda in Britain has had a positive effect by bringing attention to how we can ensure the broader relevance and dissemination of social science research. However, it is of paramount importance to ensure that this emphasis on impact does not mean sacrificing academic independence and integrity or abandoning research that does not produce immediate policy effects.
