The challenges and approaches of measuring research impact and influence on public policy making

Abstract

Purpose

Measuring research’s policy influence is challenging, given the complexity of the policy process, the gradual nature of policy influence, and the time lag between research investment and impact. This paper assesses measurement approaches and discusses their merits and applications to overcome various hurdles.

Design/methodology/approach

Relevant articles and studies were selected and analyzed. First, the research-policy interface was revisited to understand their link and how research influences policy making. Second, the most common approaches for measuring policy influence were reviewed based on their features, strengths, and limitations.

Findings

The three approaches reviewed — pyramid, influencing, and results chain — have their respective strengths. Thus, research organizations planning to design a program for monitoring and evaluation (M&E) of policy influence have to adopt the best possible features of each approach and develop a customized method depending on their objectives and overall M&E framework.

Originality/value

This paper fosters a deeper understanding of leveraging the three approaches.

Citation

Siar, S. (2023), "The challenges and approaches of measuring research impact and influence on public policy making", Public Administration and Policy: An Asia-Pacific Journal, Vol. 26 No. 2, pp. 169-183. https://doi.org/10.1108/PAP-05-2022-0046

Introduction

Research plays a vital role in the pursuit of science. Through research, knowledge is created, verified, tested, and refined. As a method for solving problems and acquiring knowledge, research helps science fulfill its purpose to explain, predict (Purtill, 1970), and generate solutions addressing societal needs and global challenges (UNESCO, n.d.).

As the branch of science that deals with society and how people behave, social science helps explain how societies work, what makes an economy grow or fail, or what makes people productive. It supports the development of societies by providing policymakers with “theories, good or bad, about man and about society which underpins his decisions” and “technical solutions to problems” (Cherns, 1968, p. 53). Social science provides vital information for governments and policymakers, business people, non-government organizations, and other entities — public or private — involved in making or influencing public policies, such as health (Topp et al., 2018) and education (Davies, 2000). Through research, social science produces data and evidence that can shape public policies or the “decisions by government and other political actors to influence, change, or frame a problem or issue that has been recognized as in the political realm by policymakers and/or the wider public” (Hassel, 2015, p. 1).

Social scientists, particularly those engaged in applied research and research organizations are compelled to make a dent in policy development and evaluation (Moore, 1983). Influencing the policy process and outcomes is particularly important for think tanks and research organizations established mainly to provide policy analysis and advice (Stone, 2007). They may influence policy — contribute significantly to decision-making or shape policy decisions — in several ways: as a clearinghouse of free information and expertise; as advocates of policy ideas; as “policy entrepreneurs” or “educators, advocates, and networkers” bridging research and policy and “applying (social) science to national problems of economy and society”; and as trusted consultants of government bodies (Stone, 2007, p. 152). These may occur at any stage of the policy cycle (i.e., agenda setting, policy formulation, policy decision-making, policy implementation, and policy evaluation, following the model of Howlett and Ramesh, 2003).

The uptake of research in policy, however, is not a straightforward process for three reasons. First, the policy cycle is neither always orderly nor linear (Howlett and Giest, 2015) like a set of railroad tracks that converge and diverge as they head toward their destinations. One can picture the activities along these tracks as individuals and groups interacting with one another to set agendas and formulate policies.

Second, various actors and factors shape the policy process, making it complex (Jones, 2011), including institutions, or the rules, norms, and practices that influence political behavior; policy networks, or various networks like interest groups with which policymakers engage; policymakers’ ideas and beliefs shaped by their upbringing and education; and the policymakers’ environment, comprising not only the physical environment but also the political, economic, and social systems and the level of technological change (Myeong and Choi, 2010; Matheson, 2016; Gourrier, 2015; Shearer et al., 2016; Swinkels, 2020).

Third, because many actors and factors shape the policy process, it is difficult to establish causality, leading to the attribution problem (Ryan and Garett, 2003). While there are methodologies, such as experimental methods, for determining attribution, these do not suit policy influence studies, given the difficulty of establishing an acceptable counterfactual (Jones, 2011).

Hence, influencing policy is challenging, and so is monitoring and evaluating policy influence. Moreover, the meaning of “policy influence” may vary from one organization to another. Having an organizational consensus on what constitutes influence is thus important in “framing expectations regarding the process and results of efforts”, which can “mitigate conflicts when evaluating or prioritizing actions” (Weyrauch and Echt, 2012, p. 3).

This paper sheds light on the dilemma of assessing the policy influence of research and research organizations. First, it revisits the research-policy interface to fully understand the link between research and policy and how research influences the policymaking process. Second, having established their connection, the article reviews the literature on measuring policy influence to determine available approaches for monitoring and evaluation (M&E) of policy influence. Through this exercise, it shows that assessing the policy influence of research is still doable.

The research-policy interface

There has been a growing recognition of the connection between research and policy and how research helps make better policies. This is articulated in the concept of “research-policy” or “science-policy” nexus, connection, or interface defined by van den Hove (2007, p, 807) as “social processes encompassing the relations between scientists and other actors in the policy process, and which allows for exchanges, co-evolution, and joint construction of knowledge with the aim of enriching decision-making”. Creating a unified and functioning science-policy interface helps align scientific interests and policy priorities toward shared goals that satisfy policymakers’ and scientists’ motivations and benefit society at the same time.

The research-policy interface has various notions. One such notion by van den Hove (2007) emphasized scientific knowledge’s role as a common ingredient in policymaking, with science being “often called upon to provide solutions to societal problems” (p. 809). It is aligned with that of Burton et al. (2019), who noted the opportunities for science and policy to intersect, particularly “issue-driven science”, whereby science is used to address an issue. This is evident in the case of climate change or the COVID-19 pandemic — two global challenges that have seen the confluence of science and policy in searching for solutions.

Issue-driven science, also called “science for action”, contrasts with “science for science” or “curiosity-driven science”, which focuses on science’s primary goal of satisfying one’s curiosity to understand and explain events. It subscribes to the traditional role of science “to discover, communicate, apply knowledge, and train the next generation of scientists” (Lubchenco, 1998).

Boswell and Smith (2017) offer a richer perspective by theorizing the research-policy nexus into four types of connections discussed below.

Research → Policy

The first type of connection is the notion that knowledge and ideas shape policy. This echoes the science for action perspective, but a more nuanced explanation is offered by the concept of knowledge utilization extensively researched by Weiss (1979), who provided four meanings (or ‘models’) of the processes and purposes of knowledge utilization:

1. The knowledge-driven model assumes that basic research opens opportunities for applied research, which identifies areas and tests ideas for practical application; appropriate technologies are then developed and application occurs.

2. The interactive model perceives the process as an interactive search for knowledge, wherein policymakers seek information and advice from social scientists and other sources, such as administrators, practitioners, planners, journalists, and interest groups.

3. The tactical model sees the use of research as “a tactic in bureaucratic politics”, wherein policymakers use research to delay action, evade criticism, or “avoid responsibility for unpopular policy outcomes” by reasoning out that their decisions were based on research or recommended by well-known scientists (p. 429).

4. The enlightenment model proposes that research induces policymakers to recognize policy problems and consider available policy alternatives and solutions. However, research may also influence policymakers to view an issue differently and to prioritize it less.

The Research→Policy interface may work if research findings are disseminated to policymakers in a relevant and accessible form and if there is a high level of trust between researchers and policymakers (Boswell and Smith, 2017). Communication and trust gaps restrict knowledge flow from research to policy and diminish research’s influence on policymaking.

Another important conceptualization of the Research→Policy interface is “knowledge creep”, which suggests that the influence of research on policymaking is not immediate and direct but builds over time through gradual alterations in the policymaker’s ways of thinking (Boswell and Smith, 2017). It assumes that policy influence cannot be attributed to one research alone or its authors but to the collective impact of related, diverse studies. This last point is essential in assessing policy influence to avoid attribution errors.

Research ← Policy/Politics

The second type of connection following Boswell and Smith’s (2017) typology is the opposite of the first, wherein policy/politics shapes research and is focused on the political context of the research-policy interface and how power relations shape research and its use. It assumes that politics constrain and dictate knowledge production and utilization, which is often monopolized by the ruling elites’ dominant ideas and interests. The tactical model of Weiss (1979) is one way of looking at this connection.

Examples of activities demonstrating a politics-driven type of research and research use are policymaker-commissioned studies with specific political agenda and research reviews meant to advance the commercial interests of, for instance, pharmaceutical and tobacco companies (Boswell and Smith, 2017).

Coproduction

The third type of research-policy interface is coproduction, which posits that knowledge (research) and governance (policy) are “mutually constitutive” or interdependent and that research and policy do not only coexist but also reinforce each other. Coproduction presents a more positive way of demonstrating politics’ influence on research in that knowledge production and use are considered inherent parts of governance. Governance may be influenced by scientific knowledge, with the latter contributing to “the construction of political reality” (Boswell and Smith, 2017, p. 5), the production of theories defining social problems, and the creation of tools and technologies used in governance as political responses (Pickering, 1995). The relationship between the two is one of mutual influence and reinforcement.

Autonomous spheres

The fourth and last type of connection assumes research/science and policy as separate domains (Boswell and Smith, 2017). The conventional perspective is that of Caplan (1979) with his “Two Communities Gap”, which underscores the inherent divergence between researchers and policymakers and assumes that they have no formal or informal connection. This is shared by Luhmann (1996, as cited by Boswell and Smith), who described the two as separate, functional systems guided by their own language (‘communicative codes’) and reason, operating without causality and connection. The challenge is to break this gap to enable one domain to pick up the signal of the other and facilitate information flows and diffusion (Boswell and Smith, 2017).

Based on the review of the four types of research-policy interface, the first (Research→Policy) and third (Coproduction) types best represent how research can influence or impact policy. Within the Research→Policy category, the tactical model may be the least ideal, but its honest and realistic take on how the negative side of politics could spoil the integrity of research makes such a model worth considering.

Moreover, research utilization depends on certain assumptions, including the effective and efficient dissemination of research findings, a trusting relationship between researchers and policymakers, and the policymakers’ openness to data and research evidence. These must be considered when analyzing the results of research’s influence on policy.

Also, policy influence builds over time — as the concept of knowledge creep explained. There is a time lag between research investment and research impact. Slote Morris et al. (2011, as cited by Collado et al., 2017) said it could take an average of 17 years to translate research findings into policy and practice. Thus, an approach that tracks the uptake of research across time and investigates the factors that may have either hastened the adoption or contributed to the delay would be a more accurate method.

Lastly, related to knowledge creep, the collaborative effect of individual research projects on policymaking should be acknowledged in measuring policy impact, as knowledge on a particular issue increases over time. Attributing an outcome solely to a research study would be inappropriate, as a policy is often informed by a body of research rather than by a single study.

Approaches to measuring policy influence

This paper discusses the three most common approaches for measuring the influence of research on policymaking. However, it does not discount other methods existing in the literature. To illustrate their advantages and limitations, it examined some examples of how research organizations have adopted such approaches.

Pyramid approach

This approach was an outcome of an 18-month pilot project of AcademyHealth, a nonprofit health services and research organization based in the United States, to assist the Robert Wood Johnson Foundation in understanding the impact of the foundation’s research grants (Collado et al., 2017).

As the name suggests, it is portrayed as a pyramid with several parts. The widest part, the base, is the Awareness level. The middle is the Influence level, and the topmost and highest is the Impact level (Figure 1). This approach depicts research impact as a series of stages, starting with awareness, advancing to influence, and ultimately achieving impact. Influence, thus, is considered only as an intermediate effect, although in this paper, the term “influence” also means impact.

On the right-hand side of the pyramid are the suggested metrics for each level signifying a research’s impact on policy and decision-making. It assumes that the research users are not only the policymakers, a precise depiction of what is happening. Although a study is primarily intended for policymakers, it is also used by researchers, the media, educators, and students. Therefore, the research’s influence/impact is considered a combined result of the research’s use by different audiences. For policymakers, the research may have been useful in crafting legislation. For researchers, it may have helped frame their studies. Considering all the possible users of a research possibly impacted by it makes a lot of sense, as the policy environment has indirect actors that influence policymakers in prioritizing policy problems and policy options. These actors constitute the interest groups with which policymakers continually engage.

At the awareness level, the suggested metrics include the number of page views and downloads of a research and social media mentions. Both can determine the attention or awareness a study can generate, which may lead to a potential policy impact of using the research in policy formulation or evaluation. The effect may be considered superficial and not a policy impact itself but can lead to it. Knowing who downloaded the research and mentioned it on social media — whether policymakers, the academe, civil society, or the public — can reveal the audience(s) aware of the research and whose attention it has caught.

At the influence level, the recommended measures are media mentions, citations in other publications, and the research’s use in academic presentations. This includes metrics that determine the interaction between the researcher and the end-users like collaborative activities, and the research/researcher’s awards or recognitions. The influence being determined is not the research’s influence on the policymakers but on the indirect actors in the policy environment that may influence the policymakers. For example, reporters’ citation of the research in their news articles (media citations) and other researchers’ use of the study in peer-reviewed publications and grey literature (citations in other publications) demonstrate that the effect is beyond awareness and curiosity of the research but more of a deliberate effort by the users to learn more about the study. Such behavior, in turn, may create a ripple effect on policymakers as they all interact — directly or indirectly — in the same policy environment.

The third and highest level of effect is impact. This is achieved when the research is already used in policy- and decision-making. Among the recommended metrics to measure impact are citation of a research article in a policy document or speech, which reflects trust and confidence in the research and/or the organization; briefings for policymakers and other forms of direct interaction between the researcher and the policymaker; and requests received for expert advice on certain policy proposals or programs.

Data for certain metrics to measure the three levels of effect can be obtained by recording observations and counting the number of citations. In some cases, online tracking tools, such as Google Alerts, can capture data more systematically. For some metrics, web and social media analytics are helpful. Other metrics, particularly for the impact level, require augmenting quantitative data with qualitative data, such as the policymaker’s testimony, to determine not only the concrete effect of the research and its utilization but also the impact pathway or how intermediate outcomes have led to the research impact.

The pyramid approach’s merit lies in its simplicity. It is straightforward, and the levels of effect — from awareness to influence to impact — show the progression from one level to the next. Research organizations and think tanks looking for a practical method to monitor their policy influence will find this approach useful. For example, the Philippine Institute for Development Studies (PIDS), a state-funded policy research organization and considered the main government think tank, has started collecting data on the utilization of its policy research studies using indicators like publication downloads and citations in the media, other publications (via Google Scholar and Research Papers in Economics database), and documents of the Philippine Congress. According to its 2021 Annual Report, 4,340 individuals downloaded publications from its website (www.pids.gov.ph), with total downloads of 16,389 (PIDS, 2022). In addition, by requiring downloaders to answer a short survey before they could download a publication, the organization could determine how they would use it. For the past three years, the top reasons that downloaders chose were to use the publication in preparing school reports/papers/theses; writing research studies/projects; developing programs, projects, and services; and formulating policies, laws, and ordinances. These reasons reflect the publications’ use by students, researchers, and policymakers, consistent with the profile of the downloaders — information useful to the organization in determining whether it is reaching its target audience. Using this tool (publication download survey) to augment existing data sources is worth replicating by other organizations.

However, users may find the pyramid approach’s indicators lacking depth compared to those in the succeeding approaches. Its indicators might vaguely show how research has influenced the policy process, such as how it has changed the attitude of policymakers on certain policy issues or how it has shaped public opinion that consequently has compelled policymakers to view an issue differently. This suggests considering additional or other approaches to see the whole picture of the research-policy interface. The second and third approaches fill this gap.

Influencing approach

Another approach in the literature pertains to the measurement of an organization’s influencing tactic. It was developed by researchers of the Research and Policy in Development (RAPID) team of the Overseas Development Institute (ODI) in collaboration with the UK Department of International Development.

Drawing from the works of Keck and Sikkink (1998) and Jones and Villar (2008), this approach organizes policy impact into five categories: attitudinal change, behavior change, procedural change, influencing the policy content, and encouraging discursive commitment from the government.

Moreover, this approach recommends crafting a theory of change (TOC) at the outset as the overall framework for M&E of policy influence. There are different types of TOC, but the most common is the causal chain, which shows a series or chain of elements, namely, inputs, activities, outputs, outcomes, and impacts, and how each element leads to the next (Jones, 2011).

To produce policy impact, it proposes applying different tactics, including Evidence and Advice, Public Campaigns and Advocacy, and Lobbying and Negotiation (Jones, 2011).

The first and last tactics are considered “inside-track approaches” or directly influencing the policymakers. The second tactic is an “outside-track approach” seeking to influence and create change through indirect channels like the media and public meetings. Table 1 shows how to measure the outcome of each tactic.

• (a)

  Evidence and advice

There are three outcomes under evidence and advice, an inside-track approach. The first is Outputs, whose evaluation may be carried out by looking at research reports and other tangible products and assessing them against a set of criteria. Hovland (2007) recommended using the “quality of science” criteria to assess research outputs. She gave the following examples of the criteria used in evaluating the DFID Renewable Natural Resources Research Strategy 1995–2005: program’s contribution to new knowledge, creative use of existing knowledge in new contexts, innovation, use of all current knowledge (journals, books, web-based information), the extent of meeting the program’s expected scientific achievements, contribution to science capacity building, development of long-term institutional partnerships, and the overall result of knowledge dissemination to the science community, policymakers, and end-users (farmers, foresters, fisherfolk) (LTS International et al., 2005 as cited by Hovland, 2007).

The second outcome is Uptake and Use, which is ascertaining the extent to which the research is picked up and used. This may be measured by (a) checking logs or records of informal and anecdotal evidence (e.g., emails, minutes of meetings) about the research’s utilization; (b) tracking citations in academic journals, websites, newspapers, and policy documents; and (c) conducting surveys or focus group discussions with the users to determine how the research is used (Jones, 2011).

The third and last outcome is Influence, whose measurement involves at least three tools. The first is called RAPID Outcome Assessment, a method developed by ODI to examine a project or a research’s contribution to a change in policy or the policy environment (ODI, 2012). It adopts outcome mapping to understand research influence on policy. The RAPID Outcome Assessment has three stages: (1) preparation stage, consisting of a document review and informal discussions with the project staff and stakeholders to illustrate the project’s history, target objectives, and intended changes; (2) workshop with the project team, key stakeholders, and external experts to identify the key policy change processes; and (3) triangulation, wherein the assessment team refines stories of change and does follow-up and in-depth interviews to validate the linkages and effects from the workshop and deepen the analysis of the change processes.

The second tool to measure Influence is episode studies, which are like case studies focusing on a clear policy change and tracking back the impact of research and other factors that led to that change (Tsui et al., 2014). These studies can be a single episode or comparative episodes, with each episode study requiring a historical narrative of the observed policy change, which should present a timeline of the key policy decisions and actors and related documents and events.

Most Significant Change (MSC) is the third recommended tool. It involves collecting significant change stories from various stakeholders and systematically selecting stories that depict the most significant change by a designated panel of stakeholders (Jones, 2011). MSC works best when combined with other options for collecting and analyzing data, as it does not comprehensively show the impacts of an intervention (Better Evaluation, n.d.). The values that the stakeholders held may also influence the selection of the most significant change stories.

• (b)

  Public campaigns and advocacy

An outside-track tool, this refers to indirect channels used to influence policymakers, such as the media and public meetings and campaigns. Three outcomes must be measured under this tactic. Since it does not directly target the policymakers but the secondary audiences or indirect actors of the policy process, measuring the outcomes usually exclude the policymakers.

In assessing the first outcome, which is the target audience’s attitude and behavior, the relevant areas of interest include their awareness of the project or campaign and their attitude and behavior toward it (Jones, 2011). It aims to assess the change over time and the project’s influence on the target audience. Among the tools to measure the outcome are surveys, focus group discussions, and key informant interviews.

Monitoring the extent of the project or study’s media coverage is important. This is the second outcome suggested for measurement under this tactic. The rationale is that the media’s increased reporting of the project can get the message across to the target audience more frequently and consistently and trigger action. To monitor media attention, the simplest tool is using media tracking logs or recording the citations of the project, study, or media campaign, as recommended in the Pyramid approach. Media citation monitoring can be done by manually tracking media mentions and keeping newspaper clippings or its digitized/electronic copies or using online/electronic tools such as Google Alerts or media monitoring programs.

The other tool for measuring media coverage goes beyond monitoring media citations. “Media assessment” entails ascertaining the importance given by the media on the issue or campaign. This may be done by determining the airtime given on television or radio or the space allotted on print media. This can be combined with information on the episode’s audience reach where the campaign is featured and the level of its popularity on social media and the direct feedback from the public, which can help understand the issue’s influence. Useful social media metrics include the number of likes and shares on Facebook and Twitter and trending topics on Twitter.

The last outcome recommended by Jones (2011) to assess the audience’s attitude and behavior is determining how the media framed the message and how this influenced public opinion. It is based on the framing theory, which suggests that how something is presented to an audience affects how they perceive it and use the information (Davie, 2011).

• (c)

  Lobbying

As an inside-track approach, lobbying seeks to influence policymakers directly, such as involvement in negotiations or meetings and direct formal or informal interactions with government officials. Formal spaces may include evidence-based dialogue, while informal ones include informal discussions and debates.

According to Jones (2011, p. 8), measuring the outcome of this tactic entails knowing the actors, “the relationship between them, and the institutions within which they work”. The actors refer to the lobbyist on one side and the target audience, the policymakers, on the other. The success of lobbying depends on the lobbyist’s credibility and his relationship with his target audience (Jones, 2011). The target audience’s interests and ideologies and institutional affiliation may also shape how they perceive and respond to the issue. However, the relationship between the lobbyist and his target audience is not fixed and may change over time. The lobbyist may strongly influence his target audience at some point, but this may wane off and pick up again.

Given the need to understand the actors, their relationship, and how this affects policy influence, the recommended tools to measure the outcome are mostly qualitative. One way is to record how the actors behaved in meetings and negotiations and the issues covered. Another way is to track people — not just the key actors — and the quality of their relationship and how this influences the policymaker’s decision-making process. Interviewing key informants is useful for understanding a project’s policy influence. An in-depth analysis is highly recommended for deeper information on the influence of lobbying efforts. Suggested tools include social network analysis to assess actors’ relationship with each other and how they share information and political economy analysis to investigate the dynamics of the different actors and institutions in decision-making.

Compared to the pyramid approach, the influencing approach provides a deeper analysis of research’s influence on policy. The suggested tools for measuring each tactic of this approach are more comprehensive. It proposes a range of quantitative and qualitative methods, thus, can provide richer data to clearly show how a research or an intervention influences policy and to illustrate the interactions between the different actors of the policy process and how these interactions influence policymaking. The influencing approach delves into the changes attributable to a program and goes beyond citations and mentions as indicators of policy influence. It is useful for conducting a comprehensive assessment of a program, strategy roadmap, sectoral plan, or information/advocacy campaign.

Results chain approach

This last approach tracks policy influence using the results chain, which links inputs, activities, outputs, outcomes, and impacts. It considers these elements as steps to achieve long-term desired objectives. It begins with inputs consisting of the resources (financial, human, and material) to develop the intervention. Next are activities or actions carried out using the inputs to produce specific outputs. The latter consists of the products resulting from the intervention and may include changes relevant to achieving the outcomes, which refer to the outputs’ short- and medium-term effects. Finally, impacts are the interventions’ long-term primary and secondary effects — either directly or indirectly, intended or unintended (Simister, 2017). The results chain is considered a diagram or graphical representation — a logic model — of the theory of change (USAID, n.d.).

Under this approach, indicators are identified for three elements in the results chain: activities and outputs for short- and medium-term results and impacts for long-term results. Tables 2 and 3 show what can be evaluated for activities and outputs, the aspects to evaluate, and examples of indicators. Most indicators are similar to those proposed under the Pyramid approach for measuring awareness and influence.

In terms of measuring impact, the evaluation is more complex. Weyrauch (2012) attributes this to two reasons: (1) the difficulty of detecting and measuring changes as they normally exceed the life of the project and the period of any M&E exercise, and (2) the difficulty of attributing the changes to only one organization or particular research or project, as the changes are “multicausal” and produced or instigated by several actors.

As such, evaluating impact requires determining the changes. The five types of change from the works of Keck and Sikkink (1998) and Jones and Villar (2008) mentioned in the second approach (Influencing) offer a useful way to measure impact under the Results Chain approach. The five types are attitudinal change, behavioral change, procedural change, policy content, and discursive commitments.

Another way to measure impact is to identify various desired long-term outcomes and consider the different types of change mentioned earlier (Weyrauch, 2012). For example, in measuring the increase in policymakers’ knowledge of an issue, the indicators may include the number of meetings and interactions with policymakers, their demand for information or related services, and the number of new joint initiatives on the issue.

The Results Chain approach coherently measures policy influence, as it follows a logical framework that tracks inputs, activities, outputs, outcomes, and impacts. Like the Pyramid approach, it is linear by design. Nevertheless, it is used by many international development organizations, including the Asian Development Bank (ADB). The Independent Evaluation Department (IED), an independent entity of the ADB, uses a results-based framework to systematically evaluate the Bank’s policies, strategies, and operations to provide feedback on its performance and generate and disseminate evaluation lessons for achieving development impact. The framework considers the whole results chain — from inputs to impacts and how these lead to better decision-making for ADB and its stakeholders (ADB 2012).

Conclusion and recommendation

The increasing recognition of the value of evidence-informed policies has intensified the need for greater research-policy interface. Policymakers realize the importance of engaging with researchers more to craft more effective policies, while research organizations are compelled to make a dent in policymaking and ensure that their work has a wider public value. Thus, monitoring and evaluating the influence of research on policy has become more crucial than ever.

Measuring policy influence, however, is not a straightforward exercise. The policy cycle is complex and shaped by various interacting factors and actors. Policy influence is not immediate and direct, nor can it be attributed to a single research or organization. There is often a time lag between research investment and research impact.

Three approaches available in the literature can overcome the hurdles of measuring policy influence: Pyramid, Influencing, and Results Chain. The Pyramid and Results Chain — both linear by design — offer a straightforward model for measuring policy influence. The Pyramid deals with indicators of awareness, influence, and impact, usually through citations and mentions of the research by its intended users. The Results Chain tracks policy influence by analyzing the entire results chain from inputs and activities to outputs, outcomes, and impacts. The Influencing approach focuses on the interaction among the different actors of the policy process and the changes in their attitude, behavior, and commitment and in the policy content and procedure attributed to the research, program, or intervention.

These approaches are not entirely different. In more ways than one, they have similar indicators. Some are missing in one approach but could be found in others. Research organizations and think tanks can build on each approach’s strengths and adopt the indicators they deem relevant in customizing a method to measure their work’s policy influence. In selecting indicators, it is important to consider not only those pertaining to the policymakers but also those that measure the influence of research on the indirect actors. As the three approaches demonstrate, the policy environment encompasses not only the policymakers but also the indirect actors that may influence policymakers in prioritizing policy problems and policy options.

Figures

Figure 1

Pyramid approach to measuring policy impact (Source: Collado et al., 2017)

Table 1

Influencing approach: How to measure policy impact

Influencing approach	Outcomes: what to measure	How: tools
Evidence and advice	Outputs	Evaluating research reports, policy briefs, and websites
	Uptake and use	Logs; new areas for citation analysis; user surveys
	Influence	RAPID outcome assessment; Episode studies; Most significant change
Public campaigns and advocacy	Target audience attitudes, behaviour, etc.	Surveys, focus groups, direct responses
	Media attention	Media tracking logs, media assessment
	Media framing and influence	Framing analysis; coverage
Lobbying and negotiation	Actors; relationships; policy process and institutions	Recording meetings; tracking people; interviewing key informants; probing influence

Source: Jones (2011)

Table 2

Measuring policy impact using the Results Chain approach: Activities

Evaluation focus	What can be evaluated	Aspects to evaluate	Indicators (examples)	Example of tools
Activities	Promotion of new public polices	- Growth levels or interest generated - New opportunities - Efficacy	- Number of meetings granted by relevant policymakers, number of presentations in external events, profile of participants of those events, etc.	- Impact of comments
	Training of public officials and other relevant actors	- Relevance - Quality - Usefulness	- Quantity and level of public officials, degree of application of disseminated knowledge	- Self-evaluation of participants - In-depth interviews
	Technical assistance to implement public policies	- Quality - Usefulness - Efficiency - Efficacy	- Degree and reach of policy implementation - Sustainability - Degree of acknowledgement from those affected by the policy - Quality and level of participation from bureaucrats	- Participant observation - Analysis of official documents
	Monitoring and evaluation of public policies	- Quality - Usefulness - Efficacy	- Inquiries from public officials - Consultation and/or contracts to assist in the reform of the policy under evaluation - Degree of public dissemination of the M&E results	- Peer assistance - Focus groups with relevant public officials

Source: Weyrauch (2012)

Table 3

Measuring policy impact using the Results Chain approach: Outputs

Evaluation focus	What can be evaluated	Aspects to evaluate	Indicators (examples)	Example of tools
Outputs	Papers or research reports	- Quality - Clarity - Relevance - Usefulness	- Quotes in legislative sessions - Feedback from external evaluators - Inquiries from public official	- External Committee of Evaluation
	Policy briefs or public policy documents	- Clarity of identified problem - Suitability of proposed solution - Relevance and opportunity for public policy	- Quote or use in a program or law - Public official inquiries - Organized or called meetings to discuss the problem in depth	- Interviews with targeted public officials
	Blogs/websites	- Website browsability - Quality of content - Feedback from relevant actors	- Number and profile of visitors - Number of downloaded documents	- User interviews
	Publications	- Quality - Clarity - Relevance - Usefulness	- Invitation to present publication - Quotes in public documents - Inquiries from public officials	- Analysis of quotes in academic or specialized publications - Reader surveys
	Seminars/events	- Level of assistance - Quality of the debate - Profile of external presenters	- Number and profile of presenters	- Post-event participant and presenter surveys - After action review

Source: Weyrauch (2012)