Research impact – How to evaluate it? Editorial impact series Part 2

After all, the ultimate goal of all research is not objectivity, but truth.

Helene Deutsch (Polish-born American psychoanalyst 1884–1982).

In our previous editorial (Sandes-Guimarães & Hourneaux Junior, 2020), we presented and discussed the ways research impact is defined and understood. The readers may remember that the various impact conceptualisations may lead to some constraints and difficulties in search of impact itself, especially when it comes to management research.

If understanding what impact means can cause enough trouble for researchers and their stakeholders (universities, research agencies, funding agencies and so on), measuring and evaluating impact can be even a more complicated task. Measuring research impact – in its broader sense – is a much more challenging chore to accomplish than just measuring the traditional scientific impact (Bornmann, 2013; Bozeman & Youtie, 2017; Gunn & Mintrom, 2017). Moreover, given the different expectations towards impact from various stakeholders – related to public policies, business’ activities, social and environmental effects, cultural changes, etc. – the way we understand research impact assessment has turned into a critical path not only for the researchers but also to their institutions (Kellard & Śliwa, 2016; Peter, Kothari & Masood, 2017).

In the past years, we have witnessed an expansion of research impact evaluation models and frameworks, defined either in the literature or by institutions and later adapted to specific contexts (Raftery, Hanney, Greenhalgh, Glover & Blatch-Jones, 2016; Greenhalgh, Jackson, Shaw & Janamian, 2016; Pedersen, Grønvad & Hvidtfeldt, 2020). Regardless of their characteristics, these different initiatives seek to demonstrate in some way how research has contributed to society in its several facets (economy, environment and sustainability, culture, health, public policies, etc.). Assessing the societal impact of research has turned into a great challenge, mainly because of the following:

• the general complexity of the contexts involved in any research; and

• the difficulty to isolate exogenous factors that can affect the research outcomes, that can easily be mixed up with the actual effects of the research (Bornmann, 2013; Bozeman & Youtie, 2017).

In our previous editorial, we presented the main, and sometimes very different, concepts of research impact. This time, we aim to discuss some of the most important models and frameworks for assessing research impact presented in the literature, as follows.

Research impact assessment models and frameworks

Literature shows that there are dozens of different models for assessing research impact (Raftery et al., 2016; Greenhalgh et al., 2016). Nevertheless, these models show wide-ranging impact definitions and different assumptions about the role of scientific knowledge in society. It is possible to split these models into two broad groups. Firstly, those focussed on the performance and how the research results directly or indirectly contribute to society in its various spheres. Secondly, those focussed on the process, targeting the relationships and interactions established during the research process, and how these relationships and interactions are central to contributing or impacting in some sphere of society.

The most known models focussed on performance are the payback framework (Buxton & Hanney, 1996), the research impact framework (RIF) (Kuruvilla, Mays, Pleasant, & Walt, 2006) and, more recently, the research excellence framework (REF), widely used in the UK (REF, 2014, 2019). The most known models focussed on the process are the Social Impact Assessment Methods of Productive Interactions (SIAMPI) (Spaapen & Van Drooge, 2011) and the contribution mapping (Kok & Schuit, 2012). In the next pages, we briefly describe them.

Payback framework

The payback framework is the most used and adapted model for research impact assessment, developed in the mid-1990s at Brunel University (UK) by Buxton and Hanney (1996). This model incorporates academic impacts and broader societal benefits to assess results from research. Initially, the model was focussed on health science research. Later, it was adapted and modified to evaluate research also in the social sciences and humanities (SSH). However, there are still limitations in capturing impact for SSH fields of knowledge given the interactive and complex nature of research in these fields (Pedersen et al., 2020).

The framework comprises two parts. Firstly, it consists of a logic model that contributes to the analysis of the whole history of a research project, from the topic identification, the project specification and the research process, to the various stages of dissemination and also the final results (Buxton & Hanney, 1996; Penfield, Baker, Scoble, & Wykes, 2014). Secondly, there are five categories of “paybacks” (or research results/impacts), reflecting the range of benefits generated by the research. These categories are as follows:

• knowledge (e.g. scientific publications);

• benefits for future research (e.g. training and development of new researchers);

• benefits for both policy and practice (e.g. improving the informational foundation’s basis for political and executive decisions);

• benefits for health and the health system (e.g. cost savings and reducing the inequality of access); and

• broader economic benefits.

The payback framework also considers feedback loops throughout the research stages. It proposes two points of interaction among researchers and potential users:

1. project specification, selection and commissioning; and

2. research dissemination (Hanney, Greenhalgh, Blatch-Jones, Glover, & Raftery, 2017).

Research impact framework

RIF was developed in the UK by Kuruvilla et al. (2006). RIF is directed to the evaluation of research projects in the health area seeking to facilitate the development of research impact narratives and to enable systematic and comparative analysis among projects. RIF is based both on the research impact literature and on the evaluation criteria of publicly funded research in the UK, and it comprises four impact categories:

1. Research-related impacts: type of problem/knowledge; research methods; publications and papers; products, patents and translatability potential; research networks; leadership and awards; research management; and communication.

2. Policy impacts: level of policy-making; type of policy; nature of policy impact; policy networks; political capital.

3. Service impacts: type of services: health/intersectoral; evidence-based practice; quality of care; information systems; services management; cost-containment; and cost-effectiveness.

4. Societal impacts: knowledge, attitudes and behaviour; health; human rights equity; economy; social capital and empowerment; art and culture; and sustainable development (Kuruvilla et al., 2006).

The creators of the framework also highlight that it is not mandatory to use all subitems in the four impact categories in one specific project. These categories are potential research impact areas, and researchers can select them and describe those considered more appropriate for each project (Kuruvilla et al., 2006). The original framework was validated through empirical analysis of research projects from the London School of Hygiene & Tropical Medicine, using semi-structured interviews and documentary analysis to produce a narrative of the research impact (Kuruvilla, Mays & Walt, 2007). Other authors have also used this framework in the literature for addressing research impact assessment (Escribano-Ferrer, Webster & Gyapong, 2017).

Research excellence framework

REF is a system to assess the quality of research produced in higher education institutions (HEI) in the UK. REF was carried out for the first time in 2014 and, since then, it has been the main guideline for evaluating British HEIs, being the primary basis for the decision to allocate research funding resources in these institutions. The assessment is done based on three main fields: quality of the research results (weight of 65%), the vitality of the research environment (15%) and the impact of research on society (20%). This impact on society category will rise to 25% in the next assessment to be carried out in 2021 (with research results decreasing to 60%) (REF, 2019).

For the assessment of the impacts on society category, each HEI presents an impact model (describing its strategy and infrastructure to achieve the impact), along with several four-page impact case studies, each depicting a research program, the claimed impacts and evidence of those impacts. These narratives should follow a linear and temporal structure, and they are supposed to be reviewed by a panel of experts with representatives from academia and research users (industry and policymakers) (Raftery et al., 2016).

In its first evaluation round, 6,975 impact cases were submitted and evaluated. Those cases were assessed in terms of significance (intensity of influence or effect) and reach (the spread or extent of influence or effect on the relevant groups). A scale ranging from 1* (recognised but modest impacts in terms of their reach and significance) to 4* (outstanding impacts in terms of their reach and importance) was used to assess each case (Hughes, Webber & O’Regan, 2019; Morgan Jones, Manville & Chataway, 2017).

Social impact assessment methods of productive interactions of productive interactions

The SIAMPI model was developed from the Dutch project Evaluating Research in Context. It has as a central theme the identification of “productive interactions” among researchers and users or stakeholders, by analysing the networks that evolve during the research programs. Understanding those productive interactions is considered essential for the assessment of societal impact. An interaction is deemed to be productive when it induces efforts from the stakeholder side to apply the research results for achieving social goals, generating behavioural change. The framework distinguishes three types of interaction:

1. direct personal interaction;

2. indirect interaction through a medium (media, for example); and

3. financial or material exchanges (Spaapen & Van Drooge, 2011).

SIAMPI is a model that seeks to understand the impact on society in a non-linear way, focussing on learning instead of judging, trying to unveil the black-box process between research activities and impacts achieved (Quiedeville, Barjolle, Mouret, & Stolze, 2017). It is based on the assumption that societal impact results from dynamic interactions and joint efforts among researchers and actors outside the academic community. SIAMPI grounds the impact discourse in the present, thus bringing it much closer to the research process and its stakeholders, and away from future impact expectations (Boshoff & Sefatsa, 2019). The SIAMPI model is particularly suitable for research projects in the SSH, where research is typically only one component in the middle of several complex political and social processes, where it can be inappropriate to seek a direct effect of research (Pedersen et al., 2020; Penfield et al., 2014).

Contribution mapping

The contribution mapping (CM) framework was created by Kok and Schuit (2012) for assessing the impact of health research, representing research as a complex network ecosystem of people and technologies. The focus of CM is to analyse the research activities, actors involved and alignment efforts among different actors that occur during the research process (problem formulation, research production and its dissemination) (Pedersen et al., 2020; Kok & Schuit, 2012). CM focusses mostly on actors involved or directly interacting with a research project and attempts to assess contributions to action rather than impacts. From the perspective of CM creators, “attributing the ultimate ‘impact’ of research is often unrealistic and practically impossible” (Kok, Gyapong, Wolffers, Ofori-Adjei, & Ruitenberg, 2016). Instead, the focus should be on the active role of users, “who combine research outcomes with existing knowledge and use it for their own purposes in an evolving world full of ongoing processes” (Kok et al., 2016).

In its first step, CM identifies actors that directly shape the research process (investigators and linked actors). Then, it determines the most influential users, the potential key users, i.e. those who seem most capable of using research knowledge to contribute to action (non-linked actors). After that, CM combines this information in a three-phase process model: formulation phase; production phase; knowledge extension phase. Based on these phases and the identified actors, the framework distinguishes four contribution categories:

1. change in the ability and actions of the investigators and linked actors;

2. contributed knowledge products;

3. contributions to action through the utilisation of the knowledge produced by the investigators or linked actors; and

4. contributions realised through utilisation by non-linked actors (Kok & Schuit, 2012).

As a result, a research contribution map is produced, covering the three phases of the research, investigators, linked actors, key users and displaying the research’s contribution and its connections and to the involved actors (Raftery et al., 2016). CM is an appropriate model for research that involves and supports the production of knowledge directly with non-academic actors in the process of knowledge co-production (Pedersen et al., 2020).

Summary and analysis of the models

Table 1 summarises the models presented here and their main features.

Table 1.
Main models for research impact assessment

Type	Model	Main features	Advantages
Performance	Payback framework (Buxton & Hanney, 1996)	Logical model + Five categories of paybacks	Explicit research planning Defined standard impact categories
	Research impact framework (Kuruvilla et al., 2006)	Four impact categories	A valuable guide on societal impact categories
	Research excellence framework (REF, 2014, 2019)	Research impact plan + Defined criteria and scale for assessment	Definition of a research impact plan Specific products as research deliverables (case studies)
Process	SIAMPI (Spaapen & Van Drooge, 2011)	Identification of interactions among researchers and users	More suitable for social sciences and humanities
	Contribution mapping (Kok & Schuit, 2012)	Map with the contribution of research and the interaction with key users	Combines the phases of research with associated contributions and interactions with users

These research impact assessment models, as well as other impact studies in the literature, mostly use interviews and case studies as a method to capture the impact. They also use, to a lesser extent, surveys, peer review or expert panels, commercialisation statistics, bibliometrics, among others (Pedersen et al., 2020). Regarding the evidence of the claimed impacts, the most used are testimonials from research users (policymakers, practitioners, among others), project reports, media attention and intellectual property (Digital Science, 2016; Hughes et al., 2019).

Based on this overview, it is possible to notice there are diverse ways of evaluating the research impact on society. The five models described in this editorial are only some of the more than 20 different models and frameworks identified in a comprehensive review by Raftery et al. (2016). Each one of them has specific characteristics and was developed in a particular context or field of knowledge (three out of the five models were developed for health research). Therefore, one should be careful while using these models for different purposes and in diverse contexts.

Even though we attempted to divide these frameworks into two types, performance-oriented and process-oriented, those are not distinct categories and they can, or should, be used in a complementary or hybrid way. An evaluation can focus both on process and performance at the same time, assessing the impact and contribution of research to numerous societal areas and also taking into account the mutual influences among researchers and users through the diverse interactions among them along the research process.

Thus, we can reach some conclusions. Firstly, these frameworks represent specific and different ways of evaluating societal research impact. Secondly, up to now, there is no universal method or a particular set of indicators that can be applied for this assessment. At last, the paths for scientific discoveries to reach and be appropriated by society are multiple and complex, involving several actors along the process.

In the next editorial, the last of this three-part series, we will go back to this discussion on research impact, focussing on the general problems and criticisms faced in research impact management and possible improvements and contributions that can be provided to the topic in the academic context.

References

• Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233, doi: 10.1002/asi.22803.
  » https://doi.org/10.1002/asi.22803
• Bozeman, B., & Youtie, J. (2017). Socio-economic impacts and public value of government-funded research: Lessons from four US national science foundation initiatives. Research Policy, 46(8), 1387–1398, doi: 10.1016/j.respol.2017.06.003.
  » https://doi.org/10.1016/j.respol.2017.06.003
• Boshoff, N., & Sefatsa, M. (2019). Creating research impact through the productive interactions of an individual: an example from South African research on Maritime piracy. Research Evaluation, 28(2), 145–157, doi: 10.1093/reseval/rvz001.
  » https://doi.org/10.1093/reseval/rvz001
• Buxton, M., & Hanney, S. (1996). How can payback from health services research be assessed? Journal of Health Services Research & Policy, 1(1), 35–43, doi: 10.1177/135581969600100107.
  » https://doi.org/10.1177/135581969600100107
• Digital Science. (2016). The Societal and Economic Impacts of Academic Research: International Perspectives on Good Practice and Managing Evidence, London, Digital Science.
• Escribano-Ferrer, B., Webster, J., & Gyapong, M. (2017). Assessing the impact of health research on health policies: a study of the dodowa health research Centre. Bmc Health Services Research, 17(1), 435, doi: 10.1186/s12913-017-2383-0.
  » https://doi.org/10.1186/s12913-017-2383-0
• Greenhalgh, T., Jackson, C., Shaw, S., & Janamian, T. (2016). Achieving research impact through co-creation in community-based health services: Literature review and case study. The Milbank Quarterly, 94(2), 392–429, doi: 10.1111/1468-0009.12197.
  » https://doi.org/10.1111/1468-0009.12197
• Gunn, A., & Mintrom, M. (2017). Evaluating the non-academic impact of academic research: design considerations. Journal of Higher Education Policy and Management, 39(1), 20–30, doi: 10.1080/1360080X.2016.1254429.
  » https://doi.org/10.1080/1360080X.2016.1254429
• Hanney, S., Greenhalgh, T., Blatch-Jones, A., Glover, M., & Raftery, J. (2017). The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews. Health Research Policy and Systems, 15(1), 26, doi: 10.1186/s12961-017-0191-y.
  » https://doi.org/10.1186/s12961-017-0191-y
• Hughes, T., Webber, D., & O’Regan, N. (2019). Achieving wider impact in business and management: analysing the case studies from REF 2014. Studies in Higher Education, 44(4), 628–642, doi: 10.1080/03075079.2017.1393059.
  » https://doi.org/10.1080/03075079.2017.1393059
• Kellard, N. M., & Śliwa, M. (2016). Business and management impact assessment in research excellence framework 2014: analysis and reflection. British Journal of Management, 27(4), 693–711, doi: 10.1111/1467-8551.12186.
  » https://doi.org/10.1111/1467-8551.12186
• Kok, M., & Schuit, A. J. (2012). Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Research Policy and Systems, 10(1), 21, doi: 10.1186/1478-4505-10-21.
  » https://doi.org/10.1186/1478-4505-10-21
• Kok, M., Gyapong, J., Wolffers, I., Ofori-Adjei, D., & Ruitenberg, J. (2016). Which health research gets used and why? An empirical analysis of 30 cases. Health Research Policy and Systems, 14(1), 36, doi: 10.1186/s1296101601072.
  » https://doi.org/10.1186/s1296101601072
• Kuruvilla, S., Mays, N., & Walt, G. (2007). Describing the impact of health services and policy research. Journal of Health Services Research & Policy, 12(1), 23–31, doi: 10.1258%2F135581907780318374.
  » https://doi.org/10.1258/2F135581907780318374
• Kuruvilla, S., Mays, N., Pleasant, A., & Walt, G. (2006). Describing the impact of health research: a research impact framework. BMC Health Services Research, 6(1), 134, doi: 10.1186/1472-6963-6-134.
  » https://doi.org/10.1186/1472-6963-6-134
• Morgan Jones, M., Manville, C., & Chataway, J. (2017). Learning from the UK’s research impact assessment exercise: a case study of a retrospective impact assessment exercise and questions for the future. The Journal of Technology Transfer, doi: 10.1007/s10961-017-9608-6.
  » https://doi.org/10.1007/s10961-017-9608-6
• Pedersen, D. B., Grønvad, J. F., & Hvidtfeldt, R. (2020). Methods for mapping the impact of social sciences and humanities – a literature review. Research Evaluation, 29(1), 4–21, doi: 10.1093/reseval/rvz033.
  » https://doi.org/10.1093/reseval/rvz033
• Penfield, T., Baker, M. J., Scoble, R., & Wykes, M. C. (2014). Assessment, evaluations, and definitions of research impact: a review. Research Evaluation, 23(1), 21–32, doi: 10.1093/reseval/rvt021.
  » https://doi.org/10.1093/reseval/rvt021
• Peter, N., Kothari, A., & Masood, S. (2017). Identifying and understanding research impact: a review for occupational scientists. Journal of Occupational Science, 24(3), doi: 10.1080/14427591.2016.1277547.
  » https://doi.org/10.1080/14427591.2016.1277547
• Quiedeville, S., Barjolle, D., Mouret, J.-C., & Stolze, M. (2017). Ex-post evaluation of the impacts of the science-based research and innovation program: a new method applied in the case of farmers’ transition to organic production in the camargue. Journal of Innovation Economics, 22(1), 145, doi: 10.3917/jie.022.0145.
  » https://doi.org/10.3917/jie.022.0145
• Raftery, J., Hanney, S., Greenhalgh, T., Glover, M., & Blatch-Jones, A. (2016). Models and applications for measuring the impact of health research: update of a systematic review for the health technology assessment programme. Health Technology Assessment, 20(76), doi: 10.3310/hta20760.
  » https://doi.org/10.3310/hta20760
• Research Excellence Framework [REF]. (2014). About the REF. Retrieved from www.ref.ac.uk/2014/about/
  » www.ref.ac.uk/2014/about/
• Research Excellence Framework [REF]. (2019). REF 2021: Panel criteria and working methods. Retrieved from www.ref.ac.uk/publications/panel-criteria-and-working-methods-201902/
  » www.ref.ac.uk/publications/panel-criteria-and-working-methods-201902/
• Sandes-Guimarães, L., & Hourneaux Junior, F. (2020). Research impact – what is it, after all? Editorial impact series part 1. RAUSP Management Journal, 55(3), 283–288, doi: 10.1108/RAUSP-07-2020-202.
  » https://doi.org/10.1108/RAUSP-07-2020-202
• Spaapen, J., & Van Drooge, L. (2011). SIAMPI final report. Retrieved from www.siampi.eu/Content/SIAMPI/SIAMPI_Final%20report.pdf
  » www.siampi.eu/Content/SIAMPI/SIAMPI_Final%20report.pdf

Publication Dates

• Publication in this collection
  26 Feb 2021
• Date of issue
  Oct-Dec 2020