The Governance of Public Policy Evaluation Systems: Policy Effectiveness and Accountability

Abstract

Do different institutional arrangements of public policy evaluation systems produce different results in promoting accountability and government effectiveness? Our study aims at discussing the governance of public policy evaluation systems. For such purpose, the evaluation systems of Canada, Chile, France, Mexico, Spain, Sweden, the United Kingdom, and the United States were analyzed using the QCA methodology. Our analysis seeks to evaluate the institutional arrangements for the governance of evaluation systems and the conditions necessary and sufficient for them to promote greater governmental effectiveness and accountability. In conclusion, the process of delegating power to evaluation systems should include the involvement of the Legislative Branch to ensure that they help increase governmental effectiveness and promote accountability.

governance; policy evaluation; governmental effectiveness; accountability

Introduction

Studies on the policy evaluation focuses on evaluation methods and techniques and their role in the concept of result-oriented public management (Anderson, 2008; Hill & Hupe, 2005; Weiss, 1998). Governments and political systems have adopted various evaluation models, which have focused on the results or effects of policy on society. On the other hand, the literature has paid little attention to governance systems that organize institutions for evaluation systems. The processes used in selecting which policies to evaluate, how they will be evaluated, and which evaluation results are relevant in measuring the effectiveness of public policy and accountability. This article contemplates a broad concept of evaluation, aiming at contributing to the improvement of governmental policy and practices to the benefit of its citizens (Anderson, 2008; Hill & Hupe, 2005; Laurian et al., 2010; Mark & Henry, 2004; Robichau & Lynn, 2009; Weiss, 1979, 1988a, 1988b, , 1988a, 1988b, 1993, 1998, 1999).

We assume that the role of evaluating public policy is not just a managerial mechanism. The decisions concerning what to evaluate, how to evaluate, which methodologies to use, and what results will be achieved involve conflicts, world views, available technologies, and capacities that are not always available or organized for the evaluators. We start from the premise that the evaluation system institutional arrangements explain the results achieved by promoting governmental effectiveness and accountability of public policies (Anderson, 2008; Hanberger, 2011; Mark, Henry, & Julnes, 2000).

The first objective of our article is to compare the institutional arrangement of eight public policy evaluation systems. For each of the cases, aspects related to institutional autonomy and its capacity to evaluate public policies were identified. The first objective, therefore, is descriptive. The concept of governance is used here as an analytical resource to understand public policy evaluation systems, relating political questions, institutional arrangements, and the delegation process. In this sense, the use of the term governance has broadened to encompass an innovative concept evolved from traditional forms of coordination to negotiated interactions among actors involved in public policy implementation and institutions (Hanberger, 2011)

The second objective is explanatory and consists of responding why some institutional arrangements of public policy evaluation systems perform better. What are the institutional conditions that explain the different performances of the public policy evaluation system?

Our article seeks to understand how the evaluation role occurs by its practices, guidelines, and normative objectives. The hypothesis that defines the analytic strategy used is that the Legislative Branch performance in the evaluation process and the evaluation system institutional autonomy creates conditions for accountability and governmental effectiveness.

We use a comparative methodological approach. We examined cases in Canada, Chile, France, Mexico, Spain, Sweden, the United Kingdom, and the United States to understand how the different institutional arrangements related to evaluation influence the policy effectiveness and accountability. These cases were chosen because they are institutionalized systems that exist in diverse political contexts and institutional arrangements, allowing the analysis of the results achieved by the different types of public policy evaluation systems. Cases were chosen by convenience to approach countries with accumulated learning and public policy evaluation professionalization.

Our study adopts a qualitative approach to the institutional arrangements of the public policy evaluation systems in the countries analyzed. From an analytical point of view, the public policy evaluation systems are evaluated considering: (a) the political systems within which they operate; (b) the type of institutional arrangements adopted; (c) the system autonomy and integration; (d) capacity for evaluation; (e) the methodologies applied; (f) the communication strategies adopted by the agencies. By applying the Qualitative Comparative Analysis (QCA) technique, we showed which relationships are necessary and sufficient to explain the results of evaluation systems to result in the greater effectiveness and accountability of public policy.

Theoretical background

Evaluation is associated with the final moment of the public policy cycle and seeks to indicate if a policy has contributed to change the reality it has set out to address (Anderson, 2008; Hill & Hupe, 2005; Weiss, 1998). Thus, an evaluation incorporates value judgments of public policy, seeking to register its objectives, instruments, costs, and benefits, and the actors and roles of policymakers and implementers, which requires care and expertise in the analysis equivalent to the complexity of the policy involved (Faria, 2005; Sabatier & Weible, 2007).

Studies on policy evaluation usually concentrates on specific topics related to the evaluation itself or the evaluation process results. One topic is the distinction between products and results in the evaluation process (Anderson, 2008). Second deals with the imbalance in the literature in terms of the scientific analysis of evaluation, since there are fewer studies of the evaluation of results than there are about the evaluation of processes and products (Anderson, 2008; Hill & Hupe, 2005; Laurian et al., 2010; Mark & Henry, 2004; Robichau & Lynn, 2009). A third is the matter of institutional learning. Organizational learning is an actual result of policy evaluation implementation (Presman & Wildavsky, 1984). Decisions about implementation and design define what results are measured (Hofstede, 1997). Finally, if public policy managers can use evaluation knowledge and tools to plan and implement with greater quality policy improvement (Wandersman, 2014). Policy improvement occurs when the knowledge generated by policy evaluation are incorporated into rules, habits, and organizational culture (Weiss, 1999).

Another critical perspective for evaluation studies is the organizational use in public policy processes to revise or discontinue policies. Public policy evaluation involves deciding what to assess, how to evaluate, under what conditions, and in a context of interests. Thus, evaluation occurs in a political process, depending on the evaluation decision (Hojlund, 2014a, 2014b; Pattyn, 2014; Raimondo, 2018; Rutkowski & Sparks, 2014).

Besides offering opportunities for learning and improving the organizational structures and processes instituted in the policy cycle, evaluation offers elements of analysis that public managers can use to obtain feedback about the relevance and quality of public policy (Cousins & Leithwood, 1986; Cousins, Goh, Elliott, & Bourgeois, 2014; Mark & Henry, 2004; Patton, 1988, 1999), 1999). In this sense, evaluation is a potential instrument that can confer desired levels of visibility and transparency in democratic states, essential elements in providing public accountability (Mark & Henry, 2004; Ramos & Schabbach, 2012; Trosa, 2001). Evaluation offers a crucial opportunity for public policy social betterment (Mark & Henry, 2004; Mény & Thoenig, 1992; Weiss, 1988a).

However, there is a gap in the literature regarding public policy evaluation and the institutional arrangements that organize this role in the public policy cycle (Pattyn & Brans, 2015). The relevance of institutional factors and rules as elements of institutional learning, which emphasizes the influence of learning on the evaluation of results, has also been understood as part of a process based on analyses, bargains, imitations, selections, and learning (March, 2009). Theories on evaluation cover their internal dilemmas and policymakers’ roles and expectations in the evaluation process. There are few analyses of what is involved in the evaluation process and the institutional arrangements associated with them. Alternatively, studies excessively focused the use of evaluation as an instrument, ignoring the institutional arrangements that make it possible for evaluators to perform their duties (Hanberger, 2011; Johnson, 1998; Mark et al., 2000; Patton, 1988, 19, 1999).

The learning effects of evaluation on public organizations depend on decision-making and control structures that make evaluations result in more transparent processes of creating public value and the outcomes and impacts of a policy (Mark & Henry, 2004; Weiss, 1988b). A governance structure is necessary for public organizations to decide what to evaluate, with what focus, and for what purpose within a structure of their own (Hanberger, 2011). The governance of public policy evaluation systems is an institutional arrangement that allows the evaluation process efficiency and legitimacy in a political system and society. The governance of the evaluation system improves the State capacity to learn from its own mistakes and promotes a more significant influence and efficiency of policies for society (Patton, 1988, 19, 1999; Presman & Wildavsky, 1984).

In this context, two fundamental aspects help to understand public policy evaluation. First, public policy evaluation involves a systemic vision. This systemic vision focuses not only on one phase of the cycle but also on an organized group of complex activities and procedures that define the objective limits. The systemic vision establishes a panorama of results, accountability, and organizational learning. Public policy evaluation provides valuable wisdom in the context of technical knowledge within a political and bureaucratic environment. Thus, evaluation systems institutional arrangements matter in creating evaluation results and if agents can perform accurate evaluations that can fulfill their normative functions (Vedung, 2009).

Secondly, implementers and policymakers dispute the results, impacts, and concepts of policy with evaluators, and they tend to block evaluations of policies they have designed and implemented. In addition to all the technical aspects, a public policy evaluation system needs to be performed with political and institutional autonomy to conduct its functions successfully (Vedung, 2009).

Policy evaluation means constructing a governance, in which this delegation creates organizational learning, correct errors, provide accountability and encourage society’s participation in public policy discussion (Weiss, 1999). The implementation of an evaluation system means, therefore, creating governance. The evaluation role is a field of institutionalized knowledge of practices that may or may not be reflective, which expresses the internal power structure and courses of action shaped by the interaction of agents. Therefore, the evaluation system institutional arrangements are essential in explaining its results and the exercising of its role within the government. Thus, the evaluation system institutional arrangements require a governance structure that may contribute to or complicate the policy evaluation.

The evaluation systems need to be developed in a distinctive epistemological perspective and in organizational structures and institutions, or institutional models that characterize its governance (Vedung, 2009). In this sense, an important criterion to label a system of evaluation is analyzing if its activities are performed by organizational entities such as national governments (Leeuw & Furubo, 2008). In public policy evaluation literature, governance of evaluation systems is associated with institutional models that value merit, organizational improvement, and development of policy knowledge (Mark & Henry, 2004; Mény & Thoenig, 1992).

Thus, the governance of evaluation systems depends heavily on public managers' attention and sensitivity to capture the information offered and use it in proposing or maintaining policies. Governance of evaluation systems should be understood as the elements that define the evaluation policy that guides the evaluation practice (Trochim, 2009). Conceptually, an evaluation policy is “any rule or principle that a group or organization uses to guide its decisions and actions when doing an evaluation” (p. 16). The rules and principles that guide the evaluation process may be formal or informal. Observing the evaluation process within the governance context means observing how the evaluation role is delegated, implemented, and its results.

The definition of governance does not just consist of management and managerial mechanisms. It represents a process that connects administrative mechanisms to the political dimensions and infrastructure of power. Governance recognizes the qualitative nature of government by the policies and services. It recognizes equality as a fundamental quality of a political regime and a fundamental public value of its administration (Rothstein & Teorell, 2008). Governance means steering policy, considering the legitimacy of public action and its efficiency and effectiveness. In this concept of governance, public services and policies should promote equality and add public value by the State actions (Bevir, 2010; Rhodes & Bevir, 2016). Governance is a new way to govern (Stocker, 1997). In the problem outlined here, public policy evaluation needs to be governed. Understanding how evaluation is governed is the main objective of our study.

Given that the concept of governance is central to our analysis, our discussion of public policy evaluation systems should include the political aspects of delegating power to evaluation systems, the institutional arrangements, and their role in public management performance. Governance of evaluation systems means observing how their role is institutionalized in the State and how they describe the nature of power relations within the public policy cycle and their effectiveness in fulfilling their mission. In this sense, the evaluation must be understood considering specific, historical, and institutional embeddedness (Dahler-Larsen, 2015).

Institutionalization and practices adopted in the evaluation process will be analyzed to understand how these factors affect public policy effectiveness and accountability.

Associating political context with institutional arrangements to explain how evaluation works imply how autonomous the system is in performing its role. Such autonomy depends on the delegation process. Delegation to bureaucratic agencies is advantageous to politicians because it leads to specialization due to labor division, and it also offers political control over the agency (Lupia, 2001; McCubbins, Noll, & Weingast, 1987). Regarding governance of public policy evaluation systems, the process and delegation are vital to establishing or not establishing institutional autonomy. The autonomy of evaluation systems, in turn, makes greater specialization possible, meaning exclusive dedication to evaluation, therefore, giving the agency a greater ability to perform its functions. Suppose the agency is more autonomous and capable of performing the evaluation. In that case, it will be more able to use its knowledge to produce greater accountability and improve public policy effectiveness.

The theory of delegation can be an interesting analytical tool to understand governance models for evaluation systems. Under democratic regimes, governing means delegating power. Since evaluation has an essential role in the governmental function in the policy cycle, agencies depend on the delegation of power that authorizes them to create knowledge about policy. Delegation can benefit those that govern; however, it can also lead to problems, given that the agents to whom this work is delegated can act in opposition to the interests of the principal. In this case, politicians can lose control over policy formulation and implementation, making delegation expensive in political resources. The strategy usually adopted by politicians, thus, is abdicating delegation, making it ineffective. The expertise and the ability of the agency, in this sense, determine if delegation indeed occurs. Delegation reduces the delegator’s cost of information and presents measurable results (McCubbins, Noll, & Weingast, 1987).

The political context in which the evaluation system works is important for the construction of its institutional autonomy and its capacity to provide organizational knowledge and strengthen the effectiveness and accountability of public policies. The evaluation system autonomy delineates the capacity to produce knowledge about public policies and provide intervention and feedback that allow for greater governmental effectiveness and accountability.

Methodology

The methodology adopted is a comparative study of the institutional arrangements of the evaluation system of public policies of eight countries. The comparative analysis will be qualitative, relating the institutional autonomy of the evaluation system, the delegation process, and the type of political system. Cases from Canada, Chile, Spain, France, the United States, Mexico, the United Kingdom, and Sweden will be examined. The following cases were chosen because these countries own institutionalized systems for public policy evaluation. The sample is for convenience, choosing cases with different institutional inputs. That is, we want to understand how different political systems and different organizational arrangements of evaluation systems produce results regarding policy effectiveness and accountability¹.

Our study uses an exploratory approach for each case, describing their characteristics, identifying the delegation process to the agencies responsible for evaluating public policy, internal politics, and evaluation capacity. Next, it identifies the associations between the public policy evaluation governance models and the autonomy and political context in which the delegation to the agencies responsible for the evaluation in each case occurs, along with the results related to governmental management. Based on the governmental management, our study will analyze accountability and government effectiveness indicators by looking at the governance indicators used by the World Bank.

Qualitative Comparative Analysis (QCA) will be used to analyze the relationships among variables. This technique uses Boolean algebra to compare different case configurations and identify causality and sufficient and necessary conditions between independent variables and the results (Rihoux & Ragin, 2009). By using the QCA technique, we can make a systematic comparative analysis of complex cases based on a low number of observations. It permits identifying systematic differences and similarities between cases and the conditions necessary for a given phenomenon to occur in a small number of observations, thus making allowing the identification of causal processes in which given conditions produce given results.

Data collection was based on evaluating the legal and organizational structure of the evaluation system, identifying the roles and the relationship between the political system, the Legislative Power's role, and the institutional autonomy with the evaluation systems. The investigation is qualitative, identifying and analyzing the relationships between the independent and dependent variables pointed out to understand how the evaluation systems increase accountability and government effectiveness. The QCA technique helps to promote different classifications and analysis, allowing the identification of the relationships established between the Legislative Power's role in the evaluation system, the type of political system, and institutional autonomy with the promotion of accountability and governmental effectiveness by the evaluation systems. In this case, indicators of accountability and effectiveness were adopted as proxies of the evaluation systems performance.

Governance models for public policy evaluation systems will be analyzed to identify differences and similarities among the cases. Our study will examine the institutional arrangements concerning autonomy and their ability to produce evaluations based on the political system context, the system functional autonomy and capacity for integration, and its evaluation practices. The second step is an analysis on they way these arrangements can produce effective government and accountability. These will be the dependent variables, based on the assumption that evaluation systems, which theoretically produce learning and improve public policy, increase government effectiveness, and broaden accountability.

Regarding our study hypotheses, the assumption is that the Legislative Branch involvement and the autonomy of evaluation systems promote better results of accountability and governmental effectiveness. Legislators are stakeholders of the evaluation system and play the critical role of being evaluation advisors and users. It is the role of legislators to establish the continuity or discontinuity of a policy and make governments more or less accountable (Speer, Pattyn, & De Peuter, 2015).

H1: The Legislative Branch performance in the evaluation process and the institutional autonomy of the evaluation system creates conditions for accountability

The Legislature is part of the evaluation system. Parliamentarians use evaluation results to inform legal construction and control the Executive Branch (Varone, Bundi, & Gava, 2020). Parliaments may either exercise evaluation functions or delegate these activities to specific agencies. In this case, the Legislature performance in the evaluation process means how much it controls delegated agents, exerting influence and building policy learning.

H2: The Legislative Branch performance in the evaluation process and the institutional autonomy of the evaluation system creates conditions for governmental effectiveness

This second hypothesis depends on how autonomous the evaluation systems are due to the Legislative Power delegation process. Government effectiveness depends on how well-designed policies are, depending on a permanent evaluation process that can generate institutional learning (Peters et al., 2019). Evaluation systems play a vital role in organizational learning, collecting information, and making judgments about public policy performance. Institutional autonomy depends on the process of delegating principals (Legislative) to agents performing tasks on their behalf (Bendor, Glazer & Hammond, 2001). In the case of evaluation systems, the hypothesis is that their autonomy is based on the Legislative Power delegation process.

With these defined hypotheses, the qualitative data about each case is collected, analyzing its institutional arrangements based on the delegation process, institutional autonomy, and the type of political system that contextualizes public policy evaluation system. The data collected are qualitative, considering the documentary and legal analysis of each country’s evaluation system. The delegation process metric is the involvement of the Legislative Branch with the evaluation of public policies. Each case is treated according to the Legislative Power involvement in the delegation process to the evaluation system.

Results: governance of public policy evaluation systems

Evaluation systems are characterized by the way of delegation of power to the evaluation agencies. Thus, the political system defines different strategies delegate evaluation. These strategies define institutional arrangements. In this sense, the delegation and institutional autonomy of public policy evaluation systems result from different processes, which may or may not affect their ability to perform accurate evaluations.

Delegation can occur in two forms. First, the decision process implies that delegation can be a technical process. Public policy evaluation can be delegated to bodies that already perform similar activities, given that the accumulation of expertise reduces the learning curve. In the case of the United Kingdom and Sweden, this delegation begins with Parliament. It is related to characteristics of their political systems and the role of auditing institutions associated with Parliament. In Canada, Parliament delegates this role to the Executive Branch agencies; however, the Parliament retains decision-making power and control of the evaluation system bureaucracy.

Second, evaluation is delegated to two different agencies, creating a competitive system, in which two agencies perform the same function. This competitive system is a safety criterion to ensure delegation occurrence, since two agencies competing discourage each other from doing a poor job (Bendor et al., 2001). In the United States, the separation of powers implies two delegating powers that compete as evaluators. There are two principals in presidential political systems, namely the Executive Branch and the Legislative Branch, which delegate power to a single agency or competing agencies. In the case of Chile and France, power is delegated to bodies of the Executive Branch. In Spain, the Prime Minister Cabinet delegates power to AEVAL, without the Legislative Branch direct involvement. In Mexico, the President of the Republic delegates this role, without the Legislative Branch involvement. However, the institutional design of public policy evaluation systems maintains a direct relationship with the Legislative Branch in the delegation process.

The Legislative Branch can have either a large or a small role in evaluation, which directly influences an agency capacity and autonomy concerning policy-implementing bodies. The interaction between autonomy and capacity, essential to governance arrangements, occurs according to the degree to which the Legislative Branch is involved in the evaluation.

The Legislative Branch involvement in the evaluation of public policy ensures greater institutional autonomy for the evaluation system concerning the government. Autonomy reduces the government’s control – as the implementer of policy – regarding policy results or outcomes. Governments seek to maintain control over the evaluation system in several different ways. The government can increase or diminish the budget of the agencies responsible for evaluating public policy, making this a priority or not in terms of its policy. The government can also control an evaluation agency by nominating its leaders, creating a link between evaluation content and the presented results. By this practice, the government can assume control of the political agenda of the evaluation and control its implementation process, distressing from or hiding results that will harm its public image. The government can also control the responsible agency by exaggerating results that favor the government’s agenda.

The expectation is that the Legislative Branch involvement is essential to the context of the governance of public policy evaluation systems because it facilitates institutional autonomy in the performance of the evaluation. It also offers greater capacity for evaluation, since the delegation process ensures a greater capacity for institutional action and the power to promote institutional learning and improvement. Finally, the Legislative Branch involvement in the public policy evaluation system closes the cycle by involving the representative system in the analysis of the results.

Table 1 shows a synthesis of the cases analyzed. An examination of the cases of Canada, Chile, France, Mexico, Spain, Sweden, the United Kingdom, and the United States shows different institutional formats concerning their public policy evaluation systems. This diversity is due more to institutional arrangements than inherent practices. These cases reveal that the delegation made to the responsible agencies can be done in several ways. There are systems in which the Legislative Branch has a large or small amount of involvement. There are other forms, in which the Legislative Branch delegates power to an evaluation system or competing systems.

Table 1
Comparison of the governance of evaluation systems

Case	Political system	Involvement of the Legislative Branch in evaluation	Institutional organization of the evaluation system	Autonomy and degree of system integration	Evaluation capacity	Applied methodologies	Communication strategies
Canada	Constitutional monarchy	High	Inter-ministerial commission under the control of Parliament	High, ensured by the Law of Finance Management.	High, featuring interdisciplinary teams and external consultants.	Quantitative & qualitative	Reports to the Treasury Board
Chile	Presidential	High	The organization is linked to the government.	High, ensured by the consensus between Congress and the government.	High, performed by external evaluators.	Quantitative & impact assessment	Public reports protected by the Law on Information Access
Spain	Constitutional monarchy	Low	Regulatory agency	Low, ensured by Regulatory Agency Law, with system integration.	High, featuring teams of managers and hired consultants.	Quantitative, qualitative & public opinion research	Reports to Ministry Boards and society
United States	Presidential	High	Competing organizations, with one linked to the Cabinet and the other to Congress	High autonomy, but a low degree of system integration	High, featuring interdisciplinary teams within competing structures.	Quantitative & qualitative	Uses ICTs and participative mechanisms
France	Semi-presidential	Low	The organization linked to the Cabinet.	Low, but with a large degree of system integration	High, featuring interdisciplinary teams.	Quantitative, qualitative & impact research	Report to Prime Minister and society
United Kingdom	Constitutional monarchy	High	The organization linked to Parliament	High, ensured by a mandate from Parliament	High, featuring interdisciplinary teams.	Quantitative & qualitative research and impact analysis	Reports to Parliament and service and policy users
Mexico	Presidential	Low	The organization is linked to the government.	High autonomy	High, featuring technical teams.	Quantitative & qualitative research	Reports to the President
Sweden	Constitutional monarchy	High	The organization linked to Parliament	High, ensured by a mandate from Parliament and the Law governing State auditing activities.	High, featuring teams of auditors.	Quantitative & qualitative	Reports sent to Parliament and published on digital portals.

Source: Elaborated by the authors based on the documentary analysis.

Data were collected in the evaluation system legislation, how the legislation defines the evaluation system organization, and the delegation process involved. The degree of the involvement of the Legislative in the evaluation process depends on the existence of a legal provision for legislative involvement. Likewise, the evaluation system organization was examined in the documents of the evaluation system, making it possible to describe the degree of autonomy and integration of the system. These variables will define the capacity of the evaluation system and the description of the applied methodologies and communication strategies.

Based on the exploratory study of these cases, the Legislative Branch involvement in the evaluation of public policy is more significant in Canada, Chile, Sweden, the United Kingdom, and the United States. In Canada, the evaluation system is delegated by the Parliament to the government in the parliamentary political system dynamic. In the Canadian model, even though this power is delegated to governmental bodies within the Executive Branch, Parliament maintains control over the evaluation process. The evaluation system is more complex due to the cost control system that the Parliament uses. The Parliament delegates the evaluation to the ministers within the Canadian government. However, it has a higher cost of control due to the Inspector General’s institution, in addition to the costs of the entire system and the cost of information. The Canadian system is onerous in public policy evaluation due to the more significant architectural complexity of the institutions of this system.

In Chile, the delegation is conducted according to a memorandum of understanding between the Congress and the Department of Treasury. The Congress delegates public policy evaluation to the Department of Treasury, configuring its principles and scope in the budgeting process. This characteristic ensures autonomy since the Department of Treasury evaluations must be external. However, they have a more limited scope in terms of the quality of public expenditure. The Chilean system is dynamic and efficient in its institutional arrangements, concentrating the evaluation activities within a single autonomous agency. The Chilean system reduces the learning curve and the costs of evaluation and diminishes the evaluation focus on the quality of public expenditure.

In the British and Swedish systems, the role of evaluation is delegated by the Parliament to the government’s auditing bodies in a technical manner. The boundaries between evaluation and auditing are somewhat vague, and, as a result, there are gray areas. However, this system has a lower cost of learning and offers a greater capacity to intervene in the government’s work in striving to improve its institutions. The institutional arrangements in both are dynamic in producing evaluations focused on institutional learning and accountability. The ties to the institutional model in public policy evaluation systems occur by political representation, reducing the cost of information and control for the Parliament.

In the United States, the Legislative Branch involvement strongly competes with that of the Executive Branch due to the country’s presidential system dynamics. The same principle applies to the evaluation of public policy. The Office of Information and Regulatory Affairs (OIRA), linked to the Cabinet, and the Government Accountability Office (GAO), linked to the Congress, compete in their roles as evaluators and perform the same functions. The cost of delegation is low, since both agencies have the great technical capacity. However, the system costs are higher, which also raises the costs of this information for society. This system was created for security reasons. Each principal (the President and the Congress) delegates to a specific agency that competes with its rival roles as evaluators. This competition between these evaluation systems results in a high level of internal knowledge in both agencies. It also gives both an elevated level of institutional knowledge in terms of public policy.

In Spain, unlike the other cases examined, the government specifies that evaluation be performed by a regulatory agency that operates separately from auditing and budgeting. Regarding the Parliament, the Spanish arrangement is also different. The Parliament plays a small role in this process. The Council of Ministers oversees all the work of the evaluation system. The Spanish system results in less accountability and a diminished role of the Parliament in the public policy evaluation process. The Parliament delegates this role to the government, which, in turn, delegates the evaluation role to AEVAL. Thus, it abdicates its role of controlling the delegation of the evaluation process. The costs of the delegation are low in this instance, creating a centralized system, highly coordinated.

The French case represents a model, in which its Parliament delegates the entire public policy evaluation process to the Executive Branch. The Parliament’s role is small in the dynamic of a semi-presidential system. The evaluation system directed by the government has a broader role in the reform of the State. The French government has created a more centralized system under the auspices of the SGMAP, lowering the delegation cost. The French system is dynamic in its evaluation of public policy, which is motivated by the normative goal of reforming the State and is focused on the institutional improvement of policy. However, this institutional arrangement does not ensure the public policy evaluation system autonomy and offers little accountability.

Congress plays almost no role in the evaluation process within the dynamics of the presidential system of Mexico. In this system, the Congress is removed from the public policy evaluation process, which features an institutional arrangement full of deviations and does not seek to improve institutions or promote accountability. The entire evaluation role is tied directly to the government, without institutional means to ensure its autonomy.

The Legislative Branch central role is fundamental to the construction of the institutional arrangements of a public policy evaluation system. The evaluation system governance should consist of a delegating process that links evaluation to political representation to create institutional learning and policy improvement. Therefore, the evaluation system will not be beholden to special interest groups and government interests, which often have little interest in an independent evaluation.

In this sense, to what extent do the institutional arrangements of public policy evaluation systems matter in delivering greater accountability and more effective government? Accurate policy evaluations can contribute to the formation of governments that are more accountable to their citizens and can offer essential information about government performance and its ability to promote social intervention and change. That is, public policy evaluation systems can contribute to the broadening of public debate about public policy, allowing the involvement a larger portion of society. Therefore, public policy evaluation systems can contribute to creating more effective governments, capable of absorbing the information collected in the evaluation process and using it to promote the improvement of policy institutions, leading to a better use and a better quality of public spending.

Considering these questions, QCA has been used to test which public policy evaluation systems create a more effective and accountable government. By using this technique, one can associate necessary conditions or relationships – independent variables – that produce results – dependent variables. With the use of these dependent variables, it is questioned if governance arrangements can produce a more effective government – in terms of institutional improvement and learning – with better accountability. Governmental effectiveness and accountability indicators, available in Worldwide Governance Indicators, published by the World Bank, were selected. Governmental effectiveness and accountability are aggregate indicators that combine people's perceptions in business, citizens, and experts collected by surveys. Since 1996, these indicators have been available for 215 countries. The use of these indicators allows the comparison of results, starting from the premise that the evaluation systems allow greater effectiveness of policies and accountability strengthening.

For our study, indicators of accountability quality and governmental effectiveness were selected for each of the cases analyzed. Table 2 shows the values of the indicators for each case. Observing the aggregate values, those with values above 85 were classified as high and those with values below 85 as low.

Table 2
Indices of governmental effectiveness and the quality of accountability

Case	Quality of accountability indicator	Value	Governmental effectiveness indicator	Value
Canada	95.6	High	95.2	High
Chile	80.3	Low	84.1	Low
Spain	77.3	Low	84.6	Low
United States	79.8	Low	89.9	High
France	89.2	High	88.9	High
United Kingdom	92.1	High	92.8	High
Mexico	47.8	Low	61.1	Low
Sweden	99.5	High	95.7	High

World Bank (2014)

Our comparative/qualitative analysis of the eight countries above generated impressive results in exploring the relationship between different institutional arrangements for evaluation systems. Truth tables below present these configurations. Considering the accountability indicator as a dependent variable, the political system, the institutional autonomy of the evaluation system, and the Legislative Branch involvement in the evaluation process are conditions for producing results. In the case of the political system variable, 0 is the value of the presidential system, and 1, the parliamentary system value. The case of France, which is semi-presidential, is considered to have a value of 0.

Regarding the institutional autonomy variable, 0 is the value of low autonomy level, and 1, a high level of autonomy. In terms of the Legislative Branch involvement in the evaluation process variable, 0 is considered to a low level of involvement and 1, a high level of involvement. The truth table produced when we consider the accountability indicator as a result, as follows:

In Table 3, result 1 – high accountability – occurs only when the three conditions are satisfied. Therefore, an autonomous evaluation system, a parliamentary political system, and a high degree of involvement on the part of the Legislative Branch in the evaluation process are conditions that promote greater accountability. We may understand that the political system itself does not matter in terms of accountability, as long as there are institutional autonomy and a high level of involvement on the Legislative Branch in the public policy evaluation system. Therefore, the established finding is that institutional autonomy and the involvement of the Legislative Branch, mobilizing the representation system, are fundamental to ensure that evaluation systems have greater capacity and, therefore, result in greater accountability. This result is consistent with the idea that evaluation systems governance should ensure autonomy and democratic representation in its institutional arrangements.

Table 3
Truth Table for the dependent variable accountability

Institutional autonomy	Political system	Involvement of the Legislative Branch	Accountability indicator	Cases
1	1	1	1	Canada, Sweden, and the United Kingdom
1	0	1	0	Chile and the United States
0	1	0	0	Spain
0	0	0	1	France
1	0	0	0	México

World Bank (2014)

The Venn diagram – Figure 1 – represents of the configuration of these conditions and results. Canada, Sweden, and the United Kingdom, which are constitutional monarchies with significant autonomy and a high degree of involvement on the part of the Legislative Branch in the public policy evaluation process, present conditions for the governance of evaluation systems are more likely to produce accountability. Chile and the United States, which have presidential systems, present the best conditions for governance. Spain is an exception, because it is a parliamentary system with a low degree of involvement in the Legislative Branch. Finally, France and Mexico present the lowest results in terms of governance of public policy evaluation system. Compared to Chile and the United States cases, which have presidential systems, the case of Mexico illustrates the fundamental role played by the Legislative Branch in the construction of accountability.

Figure 1
Venn Diagram for the dependent variable accountability

In France’s case, the semi-presidential system with an imperial president may be a determinant factor for the construction of an evaluation system with low autonomy. The case of France is an outlier. A semi-presidential system with low involvement of Parliament in the evaluation system produces high accountability. There is no immediate answer to this finding, but Parliament’s low involvement in public policy evaluation is offset by a bureaucracy with a long professionalization path. The evaluation function is based on the context of a professional bureaucracy and an autonomous state structure (Bourdieu, Wacquant, & Farage, 1994). This professional bureaucracy makes it possible that, even with the low involvement of Parliament, evaluation results in accountability.

In parliamentary systems, the evaluation role is delegated directly by Parliament, which allows evaluations to promote contexts that are more favorable to accountability. In presidential systems, the best results depend on two possible configurations of governance. In the United States, the presidential system establishes a delegation process based on competitive institutional arrangements, in which the Legislative Branch plays an essential role in evaluation. Chile’s good results are due to a consensual form of government, in which the Congress and the government, together, establish the evaluation process, its principles, and agency by agreement. The logical corollary is that in presidential political systems, the Legislative Branch involvement in public policy evaluation system governance is an essential variable in terms of its ability to produce accountability.

Public policy evaluation systems can promote institutional learning and improvement, resulting in a more effective government.

By observing the truth table – Table 4 – for the dependent variable governmental effectiveness, we see the same configuration for the cases in Table 3. We can see that an autonomous evaluation system and a high degree of involvement on the Legislative Branch in the evaluation system are conditions for producing governmental effectiveness.

Table 4
Truth table for the dependent variable governmental effectiveness

Institutional autonomy	Political system	Involvement of the Legislative Branch	Governmental effectiveness indicator	Cases
1	1	1	1	Canada, Sweden, and the United Kingdom
1	0	1	1	Chile and the United States
0	1	0	0	Spain
0	0	0	1	France
1	0	0	0	Mexico

World Bank (2014)

Independent of the political system, the necessary and sufficient conditions for evaluating system governance to promote greater governmental effectiveness are greater evaluation system autonomy and a high degree of involvement on the part of the Legislative Branch in the public policy evaluation process. This result is consistent with the idea that governance of public policy evaluation systems should be autonomous and representative to promote a more effective government. We assume that evaluation systems can help improve public policy institutions and, therefore, help make them more robust. The Venn diagram – Figure 2 – represents these configurations.

Figure 2
Venn Diagram for the dependent variable governmental effectiveness

According to this configuration of conditions, Canada, Sweden, and the United Kingdom present all the conditions necessary to ensure that the governance of evaluation systems will result in greater governmental effectiveness. The autonomy of public policy evaluation systems and a high degree of involvement of the Legislative Branch in the evaluation process are favorable conditions for producing a more effective government. The cases with presidential systems present two distinct groups with logical contradictions. In Chile and the United States, we have high and low degrees of effectiveness, respectively, which indicates that the type of political system alone is not sufficient to explain governmental effectiveness. Likewise, France and Mexico present high and low levels of governmental effectiveness but a low level of involvement on the Legislative Branch in public policy evaluation. We can then that presidential systems need to be combined with other conditions so that public policy evaluation systems can produce institutional learning and improvement. Therefore, presidential political systems require more complex conditions, so as evaluation systems may result in public policy institutional learning and improvement and, consequently, a greater governmental effectiveness.

These cases of presidential systems underline the fact that the delegation process for evaluation systems is more complex and depends on other variables to create the autonomy and capacity needed to produce institutional improvement. The governance of public policy evaluation systems, based on governmental effectiveness, is more dependent on presidents’ ability to articulate policy, which can determine the viability of evaluation systems. An examination of the institutional role of presidents was not possible in our study. However, we have assumed that presidents tend to maintain control over the agenda of policy evaluation, which may present problems concerning the autonomy of the agents in charge.

In examining the delegation process for public policy evaluation systems, the links between institutional autonomy and the involvement of the Legislative Branch in the policy evaluation process were clearer. The governance of public policy evaluation systems is varied, but it depends on explicit delegation from the representative system and institutional autonomy. Both in parliamentary and in presidential political systems, legislative branches play a fundamental role in the governance of public policy evaluation systems, since they stipulate the conditions for the responsibility of the agencies, establishing their autonomy and control.

In the United Kingdom, the Parliament delegates control, with a low cost of information, to the National Audit Office’s activities – NAO. The delegation allows a centralized system with autonomy and the capacity to produce evaluations consistent with the public interest and capacity to provide institutional improvement. In Mexico, the evaluation system is subject to significant institutional instability and lacks autonomy to produce consistent evaluations and improve public policy. Autonomous evaluation systems are essential in creating more accountable and effective governments, in which the permanent scrutiny of policy by agencies with competence in evaluation contributes to the improvement of the institutional arrangements of public policy. In short, an in-depth analysis of these cases shows the mechanisms that explain the links between different governance models for public policy evaluation systems and their specific results about accountability and government effectiveness.

Conclusions

The institutional arrangements by which the evaluation of public policy are conducted in the various experiences examined in our study vary in their ability to promote public policy management and accountability improvements. The cases of Canada, Chile, France, Mexico, Spain, Sweden, the United Kingdom, and the United States have shown the importance of the role of Legislative Branch in delegating power to institutions responsible for public policy evaluation. Since the Legislative Branch is essential to accountability, its role as principal is essential to ensure that the institutional arrangements of evaluation systems provide institutional improvement to operate with autonomy and evaluation capacity.

The political system does not necessarily affect institutional arrangements for public policy evaluation. Parliamentary systems have a slight advantage due to the delegation process nature to their rules and policy institutions. However, being parliamentary does not determine the reason why evaluation systems produce effective results; presidential systems can promote institutional arrangements that may be competitive or consensual, according to the relationship between the Legislative and Executive Branches. In this case, ensuring the Legislative Branch performance in the evaluation process is essential, so that evaluation systems contribute to a greater accountability. In the context of democracies, if the Legislative Branch controls governmental actions, the evaluation system has greater autonomy and greater capacity, as long as the Legislative Branch assumes the role of principal and controls the agencies responsible for evaluation.

Therefore, evaluation system autonomy depends on the degree of involvement of the Legislative Branch is in the process. Evaluation agency autonomy can be high or low depending on this power delegation, since the capacity of a system is based on its autonomy. The Legislative Branch performance in the governance of public policy evaluation systems depends on its role in controlling the government. Evaluation systems are more useful when they are autonomous concerning the government. Since the government is responsible for making and implementing public policy, the Legislative Branch involvement is of paramount in creating autonomous evaluation systems that can provide institutional learning and accountability. Strategies for institutional arrangements of evaluation systems vary and consist of an independent institution, the government, a regulatory agency and an inter-ministerial commission, or competing government bodies. For all strategies to deal with public policy evaluation systems governance, the Legislative Branch involvement is vital to promote accountability and strengthen institutional arrangements of public policy.

Governance of evaluation systems depends on a delegation process that involves the Legislative Branch and promotes institutional autonomy and capacity. The delegation process results in evaluation autonomy, allowing a trustworthy public policy evaluation and an institutional learning. Autonomy also promotes specialization for the responsible agency, with knowledge from different public policy areas. It also allows public communication, facilitating open debate on the government performance and the public policy implementation. More than merely examining and judging government expenditure effectiveness and quality, public policy evaluation plays a central role in democracy. More important, public policy evaluation requires proper governance structure to provide instruments to democratic government.

Our study made it possible to classify different governance strategies for evaluation systems. The study is limited by the number of cases approached, requiring replication with a larger number of cases. The increase in the number of cases enables a more significant densification of the initial findings described in this article. Likewise, it is necessary to understand better the organizational learning process and its relation to government effectiveness, deepening the link between evaluation and effectiveness. Future studies should address this problem and improve the initial findings of our study.

In general, we can say that the institutional arrangements of public policy evaluation produce different results. Increasing the number of variables and hypothesis testing conditions, however, might enable more assertive conclusions, thus refining our study findings.

References

• Anderson, J. (2008). Public Policymaking: an Introduction (7th ed.). Boston: Houghton-Mifflin.
• Bendor, J., Glazer, A., Hammond, T. (2001). Theories of Delegation. Annual Review of Political Science, 4, 235-269. doi:10.1146/annurev.polisci.4.1.235
  » https://doi.org/10.1146/annurev.polisci.4.1.235
• Bevir, M. (2010). Democratic Governance. Princeton: Princeton University Press.
• Bourdieu, P., Wacquant, L. J. D., Farage, S. (1994). Rethinking the State: Genesis and Structure of the Bureaucratic Field. Sociological Theory, 12(1), 1-18. doi: 10.2307/202032
  » https://doi.org/10.2307/202032
• Cousins, J. B., Leithwood, K. A. (1986) Current Empirical Research on Evaluation Utilization. Review of Educational Research, 56(3), 331-364. doi:10.3102/00346543056003331
  » https://doi.org/10.3102/00346543056003331
• Cousins, J. B., Goh, S. C., Elliott, C. J., Bourgeois, I. (2014). Framing the Capacity to do and Use Evaluation. In J. B. Cousins & I. Bourgeois (Eds.). New Directions for Evaluation: Organizational Capacity to Do and Use evaluation, number 141 (pp. 7-23). Hoboken: John Wiley & Sons.
• Dahler-Larsen, P. (2015). The Evaluation Society: Critique, Contestability and Skepticism. Spazio Filosofico, 1(13), 21-36. Retrieved from https://bit.ly/3aW7q2Y
  » https://bit.ly/3aW7q2Y
• Faria, C. A. P. (2005). A Política da Avaliação de Políticas Públicas. Revista Brasileira de Ciências Sociais, 20(59), 97-109. doi:10.1590/S0102-69092005000300007
  » https://doi.org/10.1590/S0102-69092005000300007
• Hanberger, A. (2011). The Real Functions of Evaluation and Response Systems. Evaluation, 17(4), 327-349. doi:10.1177/1356389011421697
  » https://doi.org/10.1177/1356389011421697
• Hill, M., Hupe, P. (2005). Implementing Public Policy (2nd ed.). Thousand Oaks: Sage.
• Hofstede, G. (1997). Os níveis de cultura. In G. Hofstede. Culturas e Organizações, (pp. 17-33). Lisboa: Sílabo.
• Hojlund, S. (2014a). Evaluation Use in the Organizational Context – Changing Focus to Improve Theory. Evaluation, 20(1), 26-43. doi:10.1177/1356389013516053
  » https://doi.org/10.1177/1356389013516053
• Hojlund, S. (2014b) Evaluation Use in Evaluation Systems – the Case of the European Commision. Evaluation, 20(4), 428-46. doi:10.1177/1356389014550562
  » https://doi.org/10.1177/1356389014550562
• Johnson, R. B. (1998). Toward a Theoretical Model of Evaluation Utilization. Evaluation and Program Planning, 21(1), 93-110. doi:10.1016/S0149-7189(97)00048-7
  » https://doi.org/10.1016/S0149-7189(97)00048-7
• Laurian, L., Crawford, J., Maxine, D., Kouwenhoven, P., Mason, G., Ericksen, N., Beattie, L. (2010). Evaluating the Outcomes of Plans: Theory, Practice and Methodology. Environment and Planning B: Planning and Design, 37, 740-757. doi:10.1068/b35051
  » https://doi.org/10.1068/b35051
• Leeuw, F. L., Furubo, J. E. (2008) Evaluations Systems. What are They and Why Study Them? Evaluation, 14(2), 157-169. doi:10.1177/1356389007087537
  » https://doi.org/10.1177/1356389007087537
• Lupia, A. (2001). Delegation Power: Agency Theory, In N. Smelser, P. Baltes (Eds.) International Encyclopedia of the Social and Behavioral Sciences (pp. 3375-3377). Oxford: Elsevier.
• March, J. G. (2009). Como as Decisões Realmente Acontecem: Princípios de Tomada de Decisões. São Paulo, SP: Leopardo.
• Mark, M. M., Henry, G. T. (2004). The Mechanisms and Outcomes of Evaluation Influence. Evaluation, 10(1), 35-57. doi:10.1177/1356389004042326
  » https://doi.org/10.1177/1356389004042326
• Mark, M. M., Henry, G. T., Julnes, G. (2000). Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs. San Francisco: Jossey-Bass.
• McCubbins, M., Noll, R., Weingast, B. (1987). Structure and Process, Politics and Policy: Administrative Arrangements and Political Control of Agencies. Virginia Law Review, 75, 431-482. doi:10.2307/1073179
  » https://doi.org/10.2307/1073179
• Mény, I., Thoenig, J. C. (1992). Las Políticas Públicas. Barcelona: Ariel.
• Patton, M. Q. (1988). Reports on Topic Areas. The Evaluator’s Responsibility for Utilization. Evaluation Practice, 9(2), 5-24. doi:10.1177/109821408800900201
  » https://doi.org/10.1177/109821408800900201
• Patton, M. Q. (1999). Organizational Development and Evaluation. The Canadian Journal of Program Evaluation, 14 (Special Issue), 93-113. Retrieved from: https://bit.ly/3aWqBtn
  » https://bit.ly/3aWqBtn
• Pattyn, V. (2014). Why Organizations (do not) Evaluate? Explaining Evaluation Activity through the Lens of Configurational Comparative Methods. Evaluation, 20(3), 348-367. doi:10.1177/1356389014540564
  » https://doi.org/10.1177/1356389014540564
• Pattyn, V., Brans, M. (2015). Organisational Analytical Capacity: Policy Evaluation in Belgium, Policy and Society, 34(3-4), 183-196. doi:10.1016/j.polsoc.2015.09.009
  » https://doi.org/10.1016/j.polsoc.2015.09.009
• Peters, G., Capano, G., Howlett, M., Mukherjee, I., Chou, M.H., & Ravinet, P. (2019). Design for Policy Effectiveness. Defining and Understand a Concept. Cambridge: Cambridge University Press.
• Presman, J. L., Wildavsky, A. (1984). Implementation (3rd ed.). Berkeley: University of California Press.
• Raimondo, E. (2018). The Power and Dysfunctions of Evaluation Systems in International Organizations. Evaluation, 24(1), 26-41. doi:10.1177/1356389017749068
  » https://doi.org/10.1177/1356389017749068
• Ramos, M. P., Schabbach, L. M. (2012). O Estado da Arte da Avaliação de Políticas Públicas: Conceituação e Exemplos de Avaliação no Brasil. Revista de Administração Pública, 46(5), 1271-1294. doi:10.1590/S0034-76122012000500005
  » https://doi.org/10.1590/S0034-76122012000500005
• Rihoux, B., Ragin, C. (2009). Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques. Thousand Oaks: Sage.
• Rhodes, R. A. W., Bevir, M. (2016). The “3Rs” in Rethinking Governance: Ruling, Rationalities, and Resistance. In R. A. W. Rhodes & M. Bevir (Eds.). Rethinking Governance: Ruling Rationalities and Resistance. Abingdon: Routledge.
• Robichau, R. W., & Lynn, L. E., Jr. (2009). The Implementation of Public Policy: Still the Missing Link. The Policy Studies Journal, 37(1), 21-36. doi:10.1111/j.1541-0072.2008.00293.x
  » https://doi.org/10.1111/j.1541-0072.2008.00293.x
• Rothstein, B., Teorell, J. (2008). What is Quality of Government? A Theory of Impartial Institutions. Governance: an International Journal of Policy, Administration, and Institutions, 21(2), 165-190. doi:10.1111/j.1468-0491.2008.00391.x
  » https://doi.org/10.1111/j.1468-0491.2008.00391.x
• Rutkowski, D., Sparks, J. (2014). The New Scalar Politics of Evaluation: An Emerging Governance Role for Evaluation. Evaluation, 20(4), 492-508. doi:10.1177/1356389014550561
  » https://doi.org/10.1177/1356389014550561
• Sabatier, P. A., Weible, C. (2007) The Advocacy Coalition Framework: Innovations and Clarifications. In P. A. Sabatier & C. Weible (Eds.) Theories of the Policy Process (pp. 189-220). London: Routledge.
• Speer, S., Pattyn, V., & De Peuter, B. (2015). The Growing Role of Evaluation in Parliaments: Holding Governments Accountable? International Review of Administrative Sciences, 81(1), 37-57. doi:10.1177/0020852314546583
  » https://doi.org/10.1177/0020852314546583
• Stoker, G. (1997). Governance as Theory: Five Propositions. International Social Science Journal, 50(155), 17-28. doi: 10.1111/1468-2451.00106
  » https://doi.org/10.1111/1468-2451.00106
• Trochim, W. M. K. (2009). Evaluation Policy and Evaluation Practice. In W. M. K. Trochim, M. M. Mark, L. J. Cooksy (Eds.), Evaluation Policy and Evaluation Pratice: New Directions for Evaluation. Hoboken: Jossey Bass.
• Trosa, S. (2001). Gestão Pública por Resultados. Quando o Estado se compromete. Rio de Janeiro, RJ: Revan; Brasília, DF: Enap.
• Varone, F., Bundi, P., Gava, R. (2020). Policy Evaluation in Parliament: Interest Groups as Catalysts. International Review of Administrative Sciences, 86(1), 98-114. doi:10.1177/0020852317750461
  » https://doi.org/10.1177/0020852317750461
• Vedung, E. (2009). Public Policy and Program Evaluation. New Brunswick: Transaction.
• Wandersman, A. (2014). Getting to Outcomes an Evaluation Capacity Building Example of Rationale, Science, and Practice. American Journal of Evaluation, 35(1), 100-106. doi:10.1177/1098214013500705
  » https://doi.org/10.1177/1098214013500705
• Weiss, C. H. (1979). The Many Meanings of Research Utilization. Public Administration Review, 39(5), 426-431. doi:10.2307/3109916
  » https://doi.org/10.2307/3109916
• Weiss, C. H. (1988a). Reports on Topic Areas. Evaluations for Decisions: Is Anybody There? Does Anybody care? American Journal of Evaluation, 9, 5-19. doi:10.1177/109821408800900101
  » https://doi.org/10.1177/109821408800900101
• Weiss, C. H. (1988b). If Program Decisions Hinged Only on Information: A Response to Patton. American Journal of Evaluation, 9, 15-28. doi:10.1177/109821408800900302
  » https://doi.org/10.1177/109821408800900302
• Weiss, C. H. (1993). Where Politics and Evaluation Research Meets. Evaluation Practice, 14(1), 93-106. doi:10.1016/0886-1633(93)90046-R
  » https://doi.org/10.1016/0886-1633(93)90046-R
• Weiss, C. H. (1998). Have We Learned Anything New About the Use of Evaluation? American Journal of Evaluation, 19(1), 21-33. doi:10.1016/S1098-2140(99)80178-7
  » https://doi.org/10.1016/S1098-2140(99)80178-7
• Weiss, C. H. (1999). Speeches and Addresses. The interface between Evaluation and Public Policy. Evaluation, 5(4), 468-486. doi:10.1177/1356389005059385
  » https://doi.org/10.1177/1356389005059385
• World Bank. (2014). Worldwide Governance Indicators. Washington: World Bank. Retrieved from http://bit.ly/2Mqaknn
  » http://bit.ly/2Mqaknn

Notes

• 1
  . Data were collected in documents, which are broken down at the end of this article as secondary sources.
• Secondary sources
  Canada
  Government of Canada. (2002, April 1st). Rescinded [2016-07-01] - Policy on Evaluation. Retrieved from http://bit.ly/3pVDMjG
  Government of Canada. (1985). Loi sur la gestion des finances publiques. Retrieved from http://bit.ly/3oST1Z8.
  Government of Canada. (2016, December 14th). Secrétariat du Conseil du Trésor du Canada. Retrieved from http://bit.ly/3pPTaOm
  Secrétariat du Conseil du Trésor du Canada. (2004, July 6th). La function d’évaluation au gouvernement du Canada. Retrieved from http://bit.ly/3jjfsWr
  Treasury Board of Canada Secretariat. (2004, July 6th). Evaluation Function in the Government of Canada. Retrieved from http://bit.ly/3rjjqRB
  Chile
  Gobierno de Chile, Ministerio de Hacienda, Dirección de Presupuestos. (2015, June). Evaluación Ex-post: conceptos y metodologias. Retrieved from https://bit.ly/39Lxbm7
  Gobierno de Chile, Ministerio de Hacienda, Dirección de Presupuestos. (2020, December 18th). DIPRES – Institucional. Retrieved from http://bit.ly/3jh3axu
  Gobierno de Chile, Ministerio de Hacienda, Dirección de Presupuestos. (2020, December 18th). Sistema de Evaluación y Control de Gestión. Retrieved from http://bit.ly/2O6sbQG
  Gobierno de Chile, Ministerio de Hacienda, Dirección de Presupuestos. (2020). Antecedentes generales. Retrieved from http://bit.ly/3trjb8W
  S pain
  Gobierno de España, Ministerio de Hacienda y Administraciones Públicas, Agencia de Evaluación y Calidad. (2008, January 11th). Normativa. Retrieved from http://bit.ly/3cFfUgj
  The United States
  The White House, Office of Management and Budget. (n. d.). Evidence and Evaluation. Retrieved from http://bit.ly/3rmHvae
  Office of Information and Regulatory Affairs, Office of Management and Budget, Executive Office of the President. (2016, October 11th). Reginfo.gov. Retrieved from http://bit.ly/3oNlaAP
  France
  Gouvernement de France. (2015, April 24th). Le portail de la modernisation de l’action publique. Retrieved from https://bit.ly/3rhM7hy
  Évaluation des politiques publiques. (2011, February 12th). In Wikipedia. Retrieved from http://bit.ly/3azkudh
  Gouvernement de France. (2015, April 24th). Mémo évaluation des politiques publiques: principes, processus et méthode. Retrieved from http://bit.ly/2MZKYN6
  United Kingdom
  National Audit Office. (2013, December 20th). Evaluation in Government. Retrieved from http://bit.ly/3jhEiG1
  Mexico
  Gobierno de México. (2021, January 28th). Secretaría de la Función Pública. Retrieved from http://bit.ly/2MsXNzA
  Sweden
  Swedish National Audit Office. (2002). Act containing Instructions for the Swedish National Audit Office. Retrieved from https://bit.ly/3rmfr6H
  Swedish National Audit Office. (2002). Act on Audit of State Activities etc. Retrieved from https://bit.ly/3oIgoog
  Swedish National Audit Office. (2020, April 1 st). Governance. Retrieved from http://bit.ly/3as88U4
• Plagiarism check
  O&S submits all documents approved for publication to a plagiarism check, using specific tools.
• Data availability
  O&S encourages data sharing. However, in compliance with ethical principles, it does not demand the disclosure of any means of identifying research participants, fully preserving their privacy. The practice of open data seeks to ensure the transparency of the research results, without requiring research participants to disclose their.

Associate Editor: Wescley Xavier

Publication Dates

• Publication in this collection
  07 May 2021
• Date of issue
  Jan-Mar 2021

History

• Received
  06 Mar 2019
• Accepted
  18 May 2020