1 |
Pawson, Tilley (1997) United Kingdom Sociology and Social Politics(66 Pawson R, Tilley N. Realistic Evaluation. London: SAGE; 1997.)
|
New evaluation paradigm, based on realism, which requires the use of different methods and data analysis that favor the explanation of the regularity and theory that underlies the logic of operation of the iteration of the contexts, mechanisms, and outcomes of the studied reality. |
2 |
Henry, Rog (1998) Did not describe other data(1313 Henry GT, Rog DJ. A realist theory and analysis of utilization. In: Henry GT, Julnes G, Mark MM, editors. Realist evaluation: an emerging theory in support of practice: new directions for evaluation. San Francisco: New Directions for Evaluation; 1998. p. 3-32.)
|
It develops a contextual understanding which reveals the mechanisms that generate different outcomes. It seeks to understand what types of evaluations are useful to which audiences, under which political conditions. The realist evaluation can be applied using the outcomes directly in making decisions about programs; using the outcomes to influence the way a program or its effects are seen; or using the outcomes to justify the decision about the programs. |
3 |
Mark, Henry, Julnes (1998) Did not describe other data(1414 Mark MM, Henry GT, Julnes G. A realist theory of evaluation practice. In: Henry GT, Julnes G, Mark MM, editors. Realist evaluation: an emerging theory in support of practice. San Francisco: New Directions for Program Evaluation; 1998. p. 3-32.)
|
The emerging realist evaluation estimates the effects of programs and assesses activities, such as which social values are served by certain programs. Thus, it seeks to identify the mechanisms underlying the effects of the program, the conditions under which these mechanisms operate, and the types of individuals for whom they operate. |
4 |
Tilley (2000) Did not describe other data(1515 Tilley N. Realistic evaluation: an overview. In: Founding Conference of the Danish Society; 2000 Sep. [Internet]. Nottingham: Notthingham Trent University; 2000. Available from: http://healthimpactassessment.pbworks.com/f/Realistic+evaluation+an+overview+-+UoNT+England+-+2000.pdf http://healthimpactassessment.pbworks.co...
)
|
It provides information to support public policy decision making and implementation. The realist evaluation question comprises the following: What works, for whom, and in what circumstances? Based on an understanding of how the measures will produce varying impacts on different circumstances, it is believed that the policy maker will be better prepared to decide which policies to implement under which conditions. |
5 |
Kazi, Rostila (2002) England Evaluation studies(1616 Kazi MAF, Rostila I. The practice of realist evaluation in two countries. [nternet]. Huddersfield: University of Huddersfield; 2002 [cited Sep 25, 2019]. Available from: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.510.423&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/dow...
)
|
It incorporates the main mechanisms, contexts, and components of the programs in the evaluation process. One of the main contributions of the realist evaluation paradigm is the concept of inserting the evaluator in the organizational process/structure. The realist evaluation considers the influence of the social relationships and organizational structures that make up the open system on the program's outcomes. As it is an open system, it is necessary to identify some regularities, that is, characteristics, factors, and mechanisms that lead to better or worse outcomes, and to identify the conditions under which the causal mechanisms would be activated to produce the outcomes. |
6 |
Pawson, Tilley (2004) Did not describe other data(1717 Pawson R, Tilley N. Realist evaluation. [Internet]. 2004 [cited Dec 3, 2018]. Available from: http://www.communitymatters.com.au/RE_chapter.pdf http://www.communitymatters.com.au/RE_ch...
)
|
An evaluation guided by theory, with an explanatory search. In the evaluation process, the theories are tested with the purpose of refining them. The authors highlighted the need to understand the nature of the programs and how they work, considering the following: 1) The nature of the programs and how they work; 2) Basic concepts for understanding programs that involve mechanism, context, and results. 3) Strategies and methods of the realist evaluation; 4) Presentation and use of the outcomes of the realist evaluation by the policy maker, in order to understand the issues related to the policy, practice, and organizational limits involved in the implementation of a program. |
7 |
Wilson, McCormack (2006) Australia Nursing(1818 Wilson V, McCormack B. Critical realism as emancipatory action: the case for realistic evaluation in practice development. Nurs Philos. 2006 Jan 12;7(1):45-57. doi: 10.1111/j.1466-769X.2006.00248.x https://doi.org/10.1111/j.1466-769X.2006...
)
|
It is strongly linked to the Emancipatory Practice Development programs, supporting effective research questions that will test research outcomes and inform the possibility of transferring mechanisms in different contexts. The result depends on the context as it interferes with the mechanisms. It is based on the principles of realism and seeks to apprehend what is true (mechanisms that may or may not trigger), real (events that may or may not be observable but that exist), and empirical (evidence of experiences and observations made). The explanations mainly require interpretations of qualitative data to discover the reasoning and circumstances of the actors in specific contexts, not in their abstraction, which necessarily involves the participation of stakeholders and the identification of the local history. |
8 |
Westhorp (2008) United Kingdom Philosophy(1919 Westhorp G. Development of realist evaluation models and methods for use in small-scale community based settings. [thesis]. Nottingham: Nottingham Trent University; 2008.)
|
It requires the theories and/or assumptions implicit in a program to be made explicit in order to determine what and how to assess. It implies identifying theoretical assumptions, resources, and activities of the program, related to the mechanisms that lead to short, medium, and long-term outcomes. |
9 |
Keller, et al. (2009) Sweden Business(2020 Keller C, Gäre K, Edenius M, Lindblad S. Designing for complex innovations in health care: design theory and realist evaluation combined. In: Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology; 2009 May 7-8; Philadelphia, Pennsylvania. New York: Association for Computing Machinery; 2009. doi: 10.1145/1555619.1555623 https://doi.org/10.1145/1555619.1555623...
)
|
It must be applied by those who wish to plan and implement innovations considering the receptive context for change, the readiness of the system for innovation, power relations, and the external socio-political context. After the innovation has been implemented, the realist evaluation can be applied to provide an explanation of the outcome patterns, depending on the mechanisms and on the contextual constraints. |
10 |
Kontos, Poland (2009) Canada Rehabilitation(2121 Kontos PC, Poland BD. Mapping new theoretical and methodological terrain for knowledge translation: contributions from critical realism and the arts. Implement Sci. [Internet]. 2009 Jan 5 [cited Dec 3, 2018];4:1. Available from: http://www.implementationscience.com/content/4/1/1 http://www.implementationscience.com/con...
)
|
The critical realist evaluation derives from critical realism, an approach that is advocated for implementing evidence-based innovation in health. The evaluation is inherent to the implementation process and not something out of place to measure outcomes. The context and the process are considered, that is, the conditions that promoted or hindered the changes. These can be assessed by combining quantitative and qualitative data, which will promote an understanding of why the intervention worked, for whom, and under which circumstances. |
11 |
Coryn, Noakes, Westine (2011) USA Evaluation(2222 Coryn CLS, Noakes LA, Westine CD. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval. 2010 Nov 12;32(2):199-226. doi: 10.1177/1098214010389321 https://doi.org/10.1177/1098214010389321...
)
|
Five principles for a theory-oriented evaluation: 1) Theory-oriented evaluations/evaluators should create a plausible program theory; 2) Theory-oriented evaluations/evaluators should create and prioritize evaluation questions around a program theory; 3) The theory of the program should be used to guide the planning, design, and execution of the evaluation considering relevant contingencies; 4) Theory-oriented evaluations/evaluators must measure the constructs postulated in the program theory; 5) Theory-oriented evaluations/evaluators must identify breaks, side effects, determine the effectiveness (or ineffectiveness) of the program, and explain cause and effect associations between the theoretical constructs. |
12 |
Westhorp, et al. (2011) Australia Did not describe(2323 Westhorp G, Prins E, Kusters CSL, Hultink M, Guijt IM, Brouwers JHAM. Realist Evaluation: an overview. [Internet]. Wageningen: Wageningen UR Centre for Development Innovation; 2011 [cited Dec 3, 2018]. Available from: http://library.wur.nl/WebQuery/wurpubs/fulltext/173918 http://library.wur.nl/WebQuery/wurpubs/f...
)
|
It starts from the precept that social organization occurs in the form of systems. Social systems are open: the elements can enter and leave the system. As a result, any event has many causes and, at the same time, can have many consequences. It also means that every result of a program is the result of multiple causes. The findings are likely to focus on a subset of mechanism-contexts-outcomes. In general, they can indicate that: • a specific intervention works separately; • it is implemented in different ways; • it is more effective with some groups than with others; • it will have a greater use in one location than in another; • it has intentional and unintentional consequences; • its effects are likely to last. |
13 |
Pawson, Manzano-Santaella (2012) United Kingdom Sociology and Social Politics(2424 Pawson R, Manzano-Santaella A. A realist diagnostic workshop. Evaluation. 2012 Apr 22;18(2):176-91. doi: 10.1177/1356389012440912 https://doi.org/10.1177/1356389012440912...
)
|
Guided by theory and capturing the outcomes of all the interventions, aiming to identify what works, for whom, in which circumstances, under what aspects, for how long, and why. The complex range of outcomes must be explained to verify the program's effectiveness. Another objective of the realist evaluation is to improve the programs, distinguishing the effectiveness of the implementation (effective or ineffective). |
14 |
Luskin, Ho (2013) USA Did not describe(2525 Luskin RJ, Ho T. Comparing the intended consequences of three theories of evaluation. Eval Program Plann. 2013 Jun;38:61-6. doi:10.1016/j.evalprogplan.2012.03.015 https://doi.org/10.1016/j.evalprogplan.2...
)
|
Social improvement must be considered as the objective of the collective evaluations so that program developers, participants, policy makers, and the general public can make decisions about programs and policies. |
15 |
Souza (2013) Singapore Education(2626 ouza DE. Elaborating the Context-Mechanism-Outcome configuration (CMOc) in realist evaluation: a critical realist perspective. Evaluation. 2013 Apr 12;19(2):141-54. doi: 10.1177/1356389013485194 https://doi.org/10.1177/1356389013485194...
)
|
It is possible to consider a social program as the input that will reconfigure or differently activate the underlying causal mechanisms within pre-existing social structures to generate change or a different potential within the action context. An action context comprises aspects of structure, culture, agency, and relationships. |
16 |
Manzano (2016) United Kingdom Did not describe(2727 Manzano A. The craft of interviewing in realist evaluation. Evaluation. 2016 Apr 5;22(3):342-60. doi: 10.1177/1356389016638615 https://doi.org/10.1177/1356389016638615...
)
|
The research process will begin by creating theories, which will be tested, refined and re-tested and, in this iterative process, the understanding of the real world is also refined. |
17 |
Wong, et al. (2016) United Kingdom Primary Care(2828 Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Med. [Internet]. 2016 Jun 24 [cited Dec 03, 2018];14(1):96. Available from: https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-016-0643-1 https://bmcmedicine.biomedcentral.com/ar...
)
|
It must explain the underlying theories of a program, developing clear hypotheses about how and for whom the programs can work. It implies collecting data, not only about the impacts of the program or about the program implementation processes, but also about specific aspects of the context that can affect the intended and unintended outcomes of the program and about the specific mechanisms that may be creating changes. The evaluation must be properly described: what it consists of, who the target is, who provides it, what the geographic reach is, what is expected to be achieved, and so on. The data collected must include information about program impacts and implementation processes, specific aspects of the program context that can affect the program outcomes and how those contexts shape the specific mechanisms that may be creating changes. When seeking information from the participants, it is assumed that different participants have different perspectives, information, and understandings about how programs should work and whether they actually work. The realistic methodology is well suited to the study of Community-Based Participatory Research. General and specific limitations of the realist evaluations must be explained so that the readers can interpret the findings based on them. Points or limitations imposed by any changes made to the evaluation processes must also be reported and described. |
18 |
Wong, et al. (2017) United Kingdom Primary Care(2929 Wong G, Westhorp G, Greenhalgh J, Manzano A, Jagosh J, Greenhalgh T. Quality and reporting standards, resources, training materials and information for realist evaluation: the RAMESES II project. [Internet]. Southampton: NIHR Journals Library; 2017 [cited Dec 03, 2018]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK459059/pdf/Bookshelf_NBK459059.pdf)
|
It collects specific aspects of the context that may impact on the intended and unintended outcomes of the program, and the specific mechanisms that may be changing the outcomes. |
19 |
Wong (2018) United Kingdom Primary Care(3030 Wong G. Getting to grips with context and complexity - the case for realist approaches. Gac Sanit. 2018 Mar/Apr;32(2):109-10. doi: 10.1016/j.gaceta.2017.05.010 https://doi.org/10.1016/j.gaceta.2017.05...
)
|
It is initiated by the construction of a theory, that is, an explanation of how, why, for whom, in which contexts, and on which basis an intervention is designed to "work". The result of any phenomenon is derived from the context and from the (C+M=O) mechanism. Realist evaluations are research methodologies that explicitly and consistently link the context to the outcomes and set out to address complexity issues. |