Conditions for outcomes evaluation in public policies: a debate on the role of institutionality

Th e goals of this analysis are (i) to off er elements of refl ection to the actors who design and implement public policies; (ii) to keep the debate on outcomes evaluation alight; (iii) to contribute toward integrating the debate on evaluation to the complexity of actions in the public sector. Th is theoretical essay researched works published on the subject from 1979 to 2019. Th e investigation of the approaches off ered by the selected authors allowed identifying variables that are relevant to the analysis of public policies. Th e variables are expressed in the infl uences of institutional trajectories, actors, and organizational context. Th e analysis brings the following concluions: (i) the link between institutionality and the capacity to conduct results evaluations in public policies deserves attention in future studies; (ii) the debate on the topic can be strengthened by analyses that consider not only the decisions adopted, but also the rules, norms and strategies that defi ne the political-institutional scene in which public policies are implemented.


INTRODUCTION
This essay sought to examine how literature has dealt with the relations between the institutionality of public policy and the adoption of processes aimed at their outcomes evaluation. In the long run, the objectives of the work are to keep the debate on the subject of outcomes evaluation in public policies alight and to contribute towards integrating the analytical aspects presented here with the complexity of the actions in the public sector.
There are several reasons given by the authors consulted to explain the importance of evaluation actions in the cycle of public policies and their influence on the definition of the agenda, formulation, and implementation of policies. Mark, Henry, and Julnes (2000) have identified that assessments should be made with the aim of providing the means for recognition of merit and value; for organizational and programmatic improvement in the governmental environment; and for the supervision and development of public policy knowledge. Anderson (2008), Mark and Henry (2004), Meny and Thoenig (1992) and Weiss (1988a) consider that the rationale for evaluation lies in the link between the evaluation processes and the search for social improvement, given that they offer government agents the opportunity to adjust policies in order to improve them for the benefit of society (social betterment).
However, in spite of the presence and relevance of the topic in the literature, Weiss (1999) stimulated the debate when analyzing contexts of implementation of public policies and identified contradictions between the recognition in the literature about the contribution of evaluative processes and their depreciation by public policy managers. The analysis conducted by Weiss (1999) linked the discredit and fragility of evaluation processes to the most prevalent models of governmental decision-making processes, and to the attributes of the public agents that operate in the governmental arena in which public policies are formulated and conducted. Weiss (1999) identified important variables for the understanding of the relationships that are established in scenarios of public policy formulation: contexts with a high level of informality of the policy and with low level of debate would be associated to the low valuation of evaluation actions. In informal contexts, Weiss (1999) observed the tendency for the valuation of knowledge inherent to policymakers to the detriment of advice drawn from evaluation processes; while the decentralized, democratic and participatory contexts were associated with the valorization of the evaluation actions. Weiss (1999) compared the ease of identifying the values associated with evaluation processes to the barriers faced in their entry into the public policy arena, noting that although evaluation actions have much to offer to public policy managers, they rarely base new policies on evaluation results. As stated by Weiss (1999) this contradiction in the context of governmental management can be attributed to the pressure exerted by competitiveness, as well as to the ideological factors and the fragmentation in the scenarios of implementation of public policies.
In the light of the analytical elements raised by Weiss's work (1999Weiss's work ( , 1998Weiss's work ( , 1993Weiss's work ( , 1979, the following questions were asked: Are more institutionalized public policies more likely to have their results evaluated? What variables can be used to evidence the institutionality of a public policy? The method used to conduct the study consisted of a systematic review on the subject of the evaluation of results in public policies, both on Brazilian and international bases, using search terms in Portuguese (avaliação de resultados; avaliação de impacto; políticas públicas; institucionalidade) and English (evaluation of results, impact evaluation, public policy, institutionality) for the period ranging from 1979 to 2019.
Papers which adopted the neo-institutionalist theoretical and methodological framework to address the factors involved in the evaluation of results in public policies were selected for the accomplishment of this analysis.

THE CONCEPTUAL BASIS OF DEBATE
In the field of public administration, the action of evaluating involves aspects that place it between science and art. It is, therefore, a reflection organized both on the content of public policies, and on their context of implementation. In this sense, evaluation implies providing analyses on the conditions that exist for the operation of a policy, the objectives that were achieved and the expected and unexpected effects that were produced on the reality that it sought to transform (Anderson, 2008;Bovaird, 2014;Hall, 2004;Hansen, Klejsntrup, & Andersen, 2013;Hill & Hupe, 2005;Mark et al., 2000;Meny & Thoenig, 1992;Oliver et al., 2019;Scott, 1998;Trosa, 2001).
Among the studies considered classic for the study of public policies, Pressman and Wildavsky (1984) related the existing interfaces between the evaluation of public policies, organizational learning and implementation. According to those authors (Pressman & Wildavsky, 1984), the evaluation broadens organizational learning and learning guides implementation. Thus, organizational learning would contain the essence for the understanding of the variables involved in the implementation and evaluation processes of public policies. Pressman and Wildavsky (1984) emphasized the role of the evaluation of public policies for the conduct of processes of adaptation and of political-institutional learning. In this sense, they highlighted the potential of evaluative actions to reconcile the knowledge obtained about the formulation and implementation of a given policy with the political-institutional power, generating possibilities for changing directions in the governmental arena.
In these terms, besides representing opportunities for learning and improving the organizational capacities involved in the implementation of public policies (Cousins et al., 2014;Mark & Henry 2004;Patton, 1988;Cousins & Leithwood, 1986), evaluation processes guide public managers on the quality and importance of the public policies implemented under their responsibility, providing them with information and elements of analyses (feedbacks) capable of contributing to improve, as a whole, the organizational functioning of administrative structures. As maintained by Anderson, (2008), Trosa, (2001, Mark and Henry (2004) and Mark et al. (2000) evaluation actions are fundamental tools to increase the transparency and visibility desired by democratic states, especially in the contexts that value the systematic institution of accountability to society as an exercise of citizenship. Weiss (1999) placed citizens as the primary beneficiaries of evaluation actions, defining that the central objective of evaluation processes is to help public organizations improve their policies, plans and practices, always in the interest of society.
For Mark and Henry (2004) evaluation actions are linked with the way people attribute meaning to public policies, which are implemented with the aim of meeting the needs of society, as well as with the efforts made by public agents to correct and expand public policies implemented in democratic environments.
In this sense, Hanberger (2011) linked the conceptualization of evaluation to the fulfillment of six functions (or variables): the improvement of policies and programs; internal accountability, reporting on process and performance; legitimation; the promotion of organizational learning; democratic and external accountability; and the fulfillment of symbolic / ritual functions.
Although the definitions for evaluation actions are many and broad, there is in common among them the idea that evaluations contribute decisively to generate the necessary knowledge to guide the decision-making process in public policies. Evaluations are therefore unique and valuable opportunities for organizational learning and policy improvement, based on four pillars -knowledge, learning, decision making, and improvement of the governmental environment. Hence, for the purposes of this analysis, the outcomes evaluation in public policies comprises the actions aimed at generating knowledge about the products and the results obtained by public policies, signaling if they are in line with its intentions and initial objectives. The evaluation actions are those that seek to provide information about the expected or unexpected consequences of the implementation of public policies, in order to maintain or correct the decisions adopted in their design or implementation phase, aiming to improve them based on the learning outcome (Anderson, 2008;Hanberger, 2011;Mark et al., 2000;Mark & Henry, 2004;Weiss, 1979Weiss, , 1988bWeiss, , 1999. Douglass North's (1991) contributions to this approach stem from his emphasis on the role played by institutions in societies. North (1991) expanded the concept of institutions, drawing attention to the possibilities for reducing uncertainties and to the limitations involved in institutionally mediated interactions, considering them as tools capable of connecting past, present and future. As stated by North (1991), institutions should be defined as formal and informal procedures, routines, norms and conventions immersed in the organizational structure of politics or economics, resulting from the competition between interest groups that interact with each other and with the State. Furthermore, North (1991) distinguished institutions from organizations, conceptualizing institutions as rules, and organizations as both the teams and the main agents of transformation in societies.
The role of national states in the development of innovative institutional changes and the repercussions on the political-institutional dynamics of governments was studied by North (1991), for whom such changes must be elaborated both on the basis of the relationships established between the structures that give them institutional support, as well as in the structural organizations that result from them. Skocpol (1985) and Evans (1993Evans ( , 2008 analyses have added that organizational rules and factors influence political culture, bureaucracy, and state capabilities, identifying, among them, state capacity for policy evaluation. Ostrom (2007) conceptualized rules as prescriptions, norms as internalized prescriptions, and strategies as plans, which tend to produce rules, norms, and expectations. In doing so, Ostrom (2007) associated rules, norms, and strategies to the contexts of implementation of public policies. Thus, Ostrom (2007) considered levels of operational analysis and collective, constitutional, and metaconstitutional choices, but her main focus was on the effect of rules on policy implementation.
Hofstede (2003), based on his studies on the cultural influences exerted by countries in organizational contexts, suggested that the term culture be considered, in organizational analyses, as a form of mental programming, noting that the collective programming of the mind distinguishes members from one group or category against another (Hofstede, 2003). Culture, in the broad sense, would be acquired and shared, coming from the social environment of individuals and groups. While recognizing the relevance of cultural factors to organizational structuring and decision making, McSweeney (2009) and Hall (2004) cautioned against the difficulties inherent in conceptualizing what are the cultural factors that surround an organization (Hall, 2004).
The term culture, in the broad sense of the word and adopted in the studies of social anthropology, has been considered as an attribute or internal quality to individuals or groups (Scott, 2004). Culture is also a collective phenomenon, which encompasses ways of thinking, feeling and behaving, while involving the sharing of these forms of expression among people living in the same social environment in which they are acquired (Hofstede, 2003).
To Weiss (1999) the degree of institutionality of a policy is associated with the cultural, historical, operational, budgetary and regulatory circumstances of public organizations. In this sense, the institutional path and the resolutions that precede the implementation of a policy impose strong determination on the conduct of the evaluation processes that will be established (path-dependence). Likewise, Weiss (1999) argues that evaluation actions participate in the institutional process by contributing to the incorporation of knowledge into new rules, new habits, and new organizational practices, in addition to concurring to changes in institutional culture and ideology.
In this perspective, the construct of institutionality adopted in this present analysis considers the term as the organizational assimilation of rules, both formal and informal, that derive from decisions and choices made during the period of formulation and implementation of public policies. The presence of these organizational regulation processes is relevant because it makes it difficult to reverse policies in the face of the expectations they generate, especially in situations where the political (institutional) cost of altering or suppressing policies is greater than the cost of maintaining them. In this sense, the clarity of rules, norms, and strategies is considered a factor associated with the stability and the possibility of obtaining results by public policies (Lane, 2014;North, 1991;Ostrom, 2007;Pierson, 2000;Streeck & Thelen, 2005;Weiss, 1999).
Therefore, the institutionality of a public policy can be evidenced by attributes such as (i) the insertion of politics in the organizations that participated in the process of institutional change in the area (Streeck & Thelen, 2005); (ii) the degree of formality of the policy (Anderson, 2008;Trosa, 2001); (iii) the level of debate between the organizations that participated in its formulation and implementation (Hall, 2004;Meny & Thoenig, 1992); and (iv) the degree of decentralization it presents, with regard to decision-making aspects and the execution of actions (Weiss, 1999).

THE EMPIRICAL GROUNDS OF DEBATE
In the literature, on the subject of the evaluation of public policies, there are several studies that point to the existence of associations between the institutionality of public policies and the chances that their results will be evaluated (Farrell & Héritier, 2004;Hall, 2004;Hanberger, 2011;Hofstede, 2003;Ingraham, 1987;Ingram, Schneider, & Deleon 2007;Kuhlmann & Shapira, 2009;Lane, 2014;Mahoney & Thelen, 2010;March, 2009;North, 1991;Ostrom, 2007;Pierson, 2000;Sabatier & Weible, 2007;Skocpol, 1985;Streeck & Thelen, 2005;Wandersman, 2014;Weiss, 1999Weiss, , 1993Weiss, , 1988aWeiss, , 1988bWeiss, , 1979. The exploration of the studies that linked the outcome evaluation and the institutionality of public policies highlighted the set of analyses conducted by Weiss (1999Weiss ( , 1993Weiss ( , 1988aWeiss ( , 1988bWeiss ( , 1979, which deepened the relations established between the evaluation of public policies and the decisions made by policymakers or decision-makers. For Weiss (1999), the evaluation actions have the power to instruct the decisions of public policy managers, provided they are willing to consider the information derived from the evaluation process in policy making. As stated by Weiss (1999) the involvement of public policy managers in the conduct of evaluative actions is associated to the degree of comprehension they have about the power of instruction of evaluations for the formulation and implementation of public policies. Provided policy managers are committed to the so-called "good policies" and understand the power that evaluation actions have in order to maintain the adequacy of policies to the needs of the population, they tend to value them by using knowledge for the redesign of policies and the readjustment of its objectives and goals. Thus, for these public agents, the motivation to conduct evaluation processes would be based on the knowledge obtained in these processes, to obtain better and more adequate results, resulting from the implementation of public policies.
Based on the studies he carried out in the 1980s, Ingraham (1987) emphasized that the understanding of the processes of public agents' choices regarding the implementation of evaluative processes requires conducting analyses that consider the variables present in the context of policy formulation.
In this sense, Ingraham (1987) focused on the complexity that involves the decisions implied in the design process of a public policy, which includes evaluation actions, and emphasized the relevance of the choices made by public agents in the formulation of specific indicators for the organization of evaluation actions. It is worth mentioning that there are situations in which difficulties to show the results of public policies may not be due to the lack of results in regard to the objectives proposed by the policies. In some cases, the evaluation of results is hampered by gaps arising from inconsistent policy designs (Ingraham, 1987;Hanberger, 2011;Howlett, 2014).
Among the reasons identified by Weiss (1999) to explain the low frequency for conducting public policy evaluation actions, the instances of institutional disconnection between the formulation process and the policy evaluation process deserve special mention. In contrast, the decentralized deployment of policies, competitive political systems, and democratic systems were identified as facilitators of the institutionalization of evaluation actions. Moreover, Weiss (1999) pointed out that, despite the relevance of evaluation processes to pressure groups, members of the legislature and legislators, party leaders, taxpayers, groups potentially benefited by policies, researchers, among others, the public agents or policymakers constitute the group that is most directly impacted, while also impacting on the results of the implementation of the public policies in which they are involved. Pierson (2000) has associated the occurrence of specific events (particularly those that are rulemakers) to institutionalized organizational practices, in which process and outcomes evaluations can be included. Pierson's analyses (2000) called attention to conjunctures in which great political changes occur, which are characterized by interfering in the positions assumed by the actors in the political game, changing the rules and instruments of institutional power.
On the one hand, the theoretical matrix of Pierson (2000) has contributed to explain how the chain of previous decisions explains later trajectories assumed in the implementation and results of public policies, configuring them as path dependent. On the other hand, when analyzing how government actions complement the decisions adopted in periods of great changes, his analyses addressed other theoretical models focused on the institutionality of public policies, also present in the studies carried out by Mahoney and Thelen (2010), Streeck and Thelen (2005) and North (1991). Notwithstanding the specificities of each of these models, they all affirmed the importance assumed by the changes in the rules as a form of construction of viability, with repercussions on the implementation and results of the public policies.
Some studies have clarified the situations in which variables of a cultural nature would be related to the absence of evaluative actions in processes of implementation of public policies (McSweeney, 2002(McSweeney, , 2009Ostrom, 2007;Scott, 2004;Trompenaars, 1994). Based on the link between cultural resistance, learning, and organizational changes, Hofstede (1997) found that the behavioral, feeling and thinking patterns of certain policymakers are likely to provoke interference in decisions regarding the formulation of public policies. For Hofstede (2003) these standards interfere in the design of public policies and in the way in which their results will be evaluated. Kuhlmann and Shapira (2009) and Farrell and Héritier (2004) found associations between the institutionality of public policies and the results obtained by them in industrialized countries. Kuhlmann and Shapira (2009) analyzed the relationships between governance styles, policy, and results on innovation and found evidence that state regulation influences both the form of evolution and the results produced by innovation systems. Hanberger (2011) associated the assembly of evaluation systems with good governance, and with the political, institutional and procedural arrangements related to the internal and external environment of public policies. He emphasized that policy design and managerial support are critical factors for the implementation and operation of evaluation systems.
In the same analytical trend, Mahoney and Thelen (2010) and Streeck and Thelen (2005) emphasized the importance of institutionality for the recognition and outcome evaluation in public policy implementation processes, highlighting both the weight of institutional characteristics of policies and their interactions with the public agents, in processes of institutional change. Streeck and Thelen (2005) highlighted the relevance of the rules and standards established for the recognition of the degree of institutionality reached in public policymaking processes, and emphasized the importance of political contexts, of institutional characteristics and their interaction with public agents in processes of institutional change.
The emphasis of the institutionality of the policy for the conduction of outcomes evaluations was developed in other studies carried out by authors who are affiliated to the institutionalist side, among which the analyses carried out by March (2009) and Sabatier and Weible (2007).
The model of analysis conjectured by Sabatier and Weible (2007), the Advocacy Coalitions Framework (ACF), predicted that the institutionality of a policy leads to situations of stability, this being the case of public policies that have been maintained for more than a decade. Sabatier and Weible (2007) associated stable public policies with normatively constructed results evaluation models, whose objectives included: (i) the link between public policy objectives and the values they sought to build; and (ii) the focus on the possibilities for reformulation and revision as a result of appraisal-based learning.
In the Advocacy Coalitions Framework (Sabatier & Weible, 2007), the policy-making process is viewed from a belief system and understood as the mode of interpretation of problems, which is both capable of generating cohesion and mistrust among process participants. Sabatier and Weible (2007) distinguished policies external factors as stable (those which remain for more than a decade) and dynamic ones (those which vary in shorter periods). For Sabatier and Weible (2007), the temporal factor is considered as a relevant variable for the identification of the stability of public policies and should be considered both for the proposal of their evaluation indicators and in their evaluation design. Sabatier and Weible (2007) proposed a policy analysis' model that contemplated the decisions adopted in the public policy formulation processes and, therefore, also contributed to the construction of approaches aimed at outcome evaluation. Those authors strongly related the evaluation of results in public policies to four factors: (i) the set of decisions that were manifested during the formulation of the policy; (ii) the formulators' ability to clearly state the objectives of the policy; (iii) the factors related to the products and the results of the policy; and, (iv) the possibilities of readjustments contemplated in the policy design, provoked by the knowledge obtained in the course of the implementation process.
Within the approach proposed by Sabatier and Weible (2007), results evaluation processes should be defined through the elaboration of proposals built with strong normative content. According to those authors, evaluation actions should contemplate and express the rules' design, whose comprehensiveness conveys both the link between the values and the objectives of the policy, as well as possibilities of adjustments and course changes based on knowledge and learning obtained from the evaluation process.
In a subsequent work than that of Sabatier and Weible (2007), March (2009) presented an institutional learning model for the evaluation of public policies whose starting point consists of inquiries about what happened, why it happened and whether what happened was satisfactory or unsatisfactory (March, 2009). March's work has greatly valued the historical course of organizations and societies, recommending that the answers to the questions posed at the initial moment of evaluation processes should incorporate historical events, which are sometimes not considered as objects of observation. In addition to these historical events, March suggests that be considered, among others, both the concrete events and the events that have been observed, as well as the interpretations of policymakers about them. As stated by March (2009) the institutional learning processes that guide the evaluation of results use institutional analyses, as well as selections, bargains, and imitations. Thus, these institutional factors began to influence and, therefore, to assume relevance for analyses that consider the relation between learning and the evaluation of results.
Additionally, March (2009) drew attention to the fact that institutional analyses involving the participation of different individuals should consider that the assimilation of learning in these contexts occurs differently. Thereby, even if the participants in the implementation of public policy have experienced the same historical processes, their experiences are affected by factors such as recalls, interpretations, and evaluations made by themselves (March, 2009).
Situations in which public policy evaluation processes produce contradictory results, indicating success or failure according to the evaluator's situation, were also analyzed by March (2009). The author identified a tendency to self-congratulation by the evaluators, which would lead to adjustments both in the aspiration levels and in their definitions of what should and should not be valued in evaluation processes. In this sense, variables such as experience and the elaboration of value judgments by the evaluators are important variables to explain the results of evaluations that consider the gap between the initial expectations and the results obtained.
In these terms, Anderson (2008) pointed out that the lack of information about the results obtained from a policy would act as an obstacle to the identification of factors that contribute to its success or failure. These obstacles would be expressed as difficulties in making decisions about the necessary adjustments in regard to the definition and reformulation of the policies, in order to continue, modify, strengthen or even end their implementation. Wandersman (2014) addresses perspectives developed by Anderson (2008) and Weiss (1999). Wandersman (2014) proposed that evaluations should be empowered and that the evaluative capacity be considered organizational constructs, involving such factors as: (i) the assessment of needs and resources; (ii) the definition of the desired objectives and results; (iii) the selection of evidence-based practices for policies; (iv) the necessary adjustments; (v) the implementation and process evaluations; and, (vi) sustainability.
For Wandersman (2014) evaluation processes should be viewed from a perspective of continuous improvement of public organizations, in which information is organized and applied both for the selfassessment of policymakers and for the implementation of quality public policies. He suggests that the ability to plan in a systematic way is a function of the ability to use the knowledge and evaluation tools, which are, ultimately associated with the assessment capabilities of public policy managers.
An effort to synthesize the main contributions of the target authors, the relationships observed between the selected variables and the deriving conclusions are seen in Box 1.
The institutionality of a policy is associated with political formality, the high level of debate between the organizations participating in the formulation and implementation of the policy, and the decentralization of the decision making and execution aspects (Weiss, 1999).
Policy design that adopts clear rules and information systems, norms and strategies capable of influencing the political culture, bureaucracy and state capabilities, increases the likelihood of evaluating the policies (Anderson, 2008;March, 2009;Ostrom, 2007;Streeck & Thelen, 2005;Trosa, 2001).
The concern for social betterment or the opportunity to redesign politics assumed by policymakers, in order to adapt it to the proposed objectives for obtaining better results, increases the chances of having public policies' results evaluated (Hanberger, 2011;Mark et al., 2000;Mark & Henry, 2004;Meny & Thoenig, 1992;Trosa, 2001;Weiss, 1988a).
The high degree of informality and the low level of debate in the process of formulating and implementing policies tend not to make way to the knowledge offered by the outcomes of evaluation actions. In these contexts each actor tends to act within their own level of personal knowledge, not valuing actions and evaluation systems (Weiss, 1999).
Cultural resistance and frequent changes in the organizational environment that alter thinking, feeling and behavior patterns of policymakers have been negatively associated with the processes of evaluation, due to the instability that they generate in the decision making processes and in the trajectory of policy implementation (Hofstede, 2003;McSweeney, 2009;Meny & Thoenig, 1992;Ostrom, 2007;Scott, 2004;Trompenaars, 1994).

Trajectories matter:
The institutional path and the resolutions that precede the implementation of a policy impose strong determination on the execution of the evaluation processes that will be established ( p a t h -d e p e n d e n c e ) (Weiss, 1999).
Actors matter: Policymakers committed to the adequacy of public policies to cater for the needs of the population (social betterment) value the evaluation actions and its contributions to the redesign, adequacy of policies and the readjustment of its objectives and goals (Weiss, 1999).

Contexts
matter: Good governance and political-institutional arrangements that value the internal and external contexts of public policy are relevant to the implementation and functioning of evaluation systems (Hanberger, 2011).
Source: Elaborated by the authors.

CONCLUSION
As a conclusion, it is worth recalling the two questions that guided the study, namely: (i) which variables can be used to evidence the institutionality of public policies? (ii) are more institutionalized public policies more likely to have their results evaluated?
Regarding the first question, several of the studied authors identified variables capable of indicating the degree of institutionality of public policies, pointing to associations that, although remaining fluid, can be used in future studies on the subject. Viewed in this way, the studies conducted by Howlett (2014), Cousins et al. (2014), Hanberger (2011), Anderson (2008, Mark et al. (2000), Meny and Thoenig (1992), Ingraham (1987) and Weiss (1999Weiss ( , 1988bWeiss ( , 1979 revealed that public policies whose designs contains processes aimed at knowledge management, learning, improvement of structures and organizational processes have greater institutionality. For authors like Mahoney and Thelen (2010), Streeck and Thelen (2005), Evans (1993Evans ( , 2008, Pierson (2000), Weiss (1999), Meny and Thoenig (1992) and Skocpol (1985) public policies whose organizational rules and factors influence political culture, bureaucracy, and state capacity for action, among which the ability to evaluate results, can be considered more institutionalized than those policies in which these attributes are not identified. It is worth mentioning that Weiss (1999) adds to these associations the political formality, the high level of debate between the organizations participating in the formulation and implementation of the policy and the decentralization in the decision making and execution aspects, emphasizing their relevance to the definition of the institutionalized aspect of public policies.
The second aspect that guided this study concerns the associations between the institutionality of public policies and outcomes evaluations, that is, whether and how the institutionality of a public policy favors or not the evaluation of its results. In this sense, the analyzed studies identified variables that, when present, indicate that the institutionality of public policy can be positively or negatively associated with the processes of outcomes evaluation. Hence, the positive associations between public policies with a high degree of institutionality and the conduction of results evaluations were grouped, for didactic purposes, into three sets of variables: (i) variables that reflect the characteristics of the decision-making process and the policy implementation context; (ii) identifiable variables in the policy design; and (iii) analyzable variables based on the behavior of policymakers.
The first set of variables reflects the characteristics of the decision-making process and the policy implementation context, evoking the way in which the policy objectives are linked to the values they seek to construct. In this set, are relevant the variables that express the link between the objectives of the policy, the clarity of its purposes and the possibility of making adjustments in the policy, based on the lessons learned, (Hanberger, 2011;Sabatier & Weible, 2007), as well as the variables related to the decentralized implementation of policy and the adoption of democratic and participatory decision-making processes (Trosa, 2001;Weiss, 1999). The political-institutional contexts that include values such as visibility, transparency, accountability to citizens and the possibility of correcting the evaluated policy were also identified as favorable to outcome evaluations (Faria, 2012;Mark et al., 2000;Meny & Thoenig, 1992;Trosa, 2001). These contexts value the support and the way people, either individually or collectively, give meaning to the policies implemented to meet human needs (Meny & Thoenig, 1992;Trosa, 2001).
The second set of variables values not only the policy design but also the organizational commitment towards learning, political-institutional improvement, adaptation and the possibility of reformulation and revision of the policy, which increases the chances of policies to have its results evaluated (Cousins & Leithwood, 1986;Cousins et al., 2014;Hanberger, 2011;March, 2009;Mark & Henry, 2004;Meny & Thoenig, 1992;Pressman & Wildavsky, 1984). According to studies by March (2009), Anderson (2008, Ostrom (2007), Streeck and Thelen (2005) and Trosa (2001), when policy design covers the adoption of clear rules and information systems, as well as of norms and strategies capable of influencing the political culture, bureaucracy, and state capabilities, the likelihood of evaluating the results of the policies are increased.
The third set consists of the variables whose analysis is based mainly on the behavior of policymakers. Those analyses are based on the recognition of the managers of the policy on the merit and the value of the evaluation of results for the organizational and programmatic improvement, for the supervision and the development of the knowledge about the policy. In other words, when the concern for social betterment or the opportunity to redesign politics is manifested and assumed by policymakers, in order to adapt it to the proposed objectives, with the aim of obtaining better results, the chances of evaluating results of public policies increases (Hanberger, 2011;Mark et al., 2000;Mark & Henry, 2004;Meny & Thoenig, 1992;Trosa, 2001;Weiss, 1988a).
The negatively established associations between the institutionality of public policies and the conduction of outcomes evaluations were explained by factors such as the degree of informality and cultural resistance. The high degree of informality and the low level of debate in the process of formulating and implementing a public policy tend not to make way to the knowledge offered by the outcomes of evaluation actions, given that in these contexts each actor involved in the process tends to act with its own level of personal knowledge, not valuing the delineation offered by the actions and evaluation systems (Weiss, 1999). Cultural resistance and frequent changes in the organizational environment that alter thinking, feeling and behavior patterns of policymakers have been negatively associated with the processes of evaluation, due to the instability that they generate in the decision making processes and in the trajectory of policy implementation (Hofstede, 2003;McSweeney, 2009;Meny & Thoenig, 1992;Ostrom, 2007;Scott, 2004;Trompenaars, 1994).
In summary, the survey carried out on the evaluation of public policy results identified that the understanding of the path taken by public policies may benefit from approaches based on explanatory models that are guided by the neoinstitutionalist perspective. In this regard, it should be pointed out that, for the purposes of this essay, the term institutionality derives from an expanded concept of institutions, such as proposed by North (1991) and Ostrom (2007), which encompasses many types of entities, and includes the organizations and rules used to structure interaction models with and within organizations.
In these terms, it is recommended that analyses on the subject of outcomes evaluation consider the influences exerted by variables related to institutional trajectories, actors and organizational contexts both at the moment of the formulation of public policies and during the phase of their implementation, since all these influences act in the situations in which decisions are made regarding the evaluation process.
The work carried out on the subject of outcomes evaluation revealed that the institutionality of policies deserves to remain an object of attention in studies conducted on evaluation in public policies. In addition, the study showed that the debate on the evaluation of public policies can be strengthened by analytical strategies and the conduction of empirical studies that consider the decisions adopted by the public policy makers, as well as the rules, norms and strategies that define the political-institutional scene in which public policies are implemented.