PERFORMANCE AUDIT AND PUBLIC POLICY EVALUATION: THE CASE OF SUPREME AUDIT INSTITUTIONS

ABSTRACT

This paper presents two public governance instruments at the disposal of Supreme Audit Institutions (SAIs), performance audit and policy evaluation, aimed at promoting accountability and learning in public policy development. Respective definitions and types are provided, and in relation to performance audit the paper elaborates on the pervasive tension between independence and relevance of audit practices, associated with the stylized functions of watchdog and sheepdog. The regulatory environment for the development of policy evaluation by SAIs is briefly described. The contribution of the paper is the design of a set of parameters and associated content for each of the two instruments that allows their comparison and evolution, contrasting with the trend in the literature to identify similarities and differences between them.

Keywords:
accountability; performance audit; public policy evaluation; public management; supreme audit institutions

RESUMO

Este artigo apresenta dois instrumentos de governança pública à disposição das Instituições Superiores de Controlo (ISC), a auditoria de resultados e a avaliação de políticas, destinados a promover a accountability e a aprendizagem no desenvolvimento de políticas públicas. São fornecidas as respetivas definições e tipos e, em relação à auditoria de resultados, o documento discorre sobre a tensão entre independência e relevância das práticas de auditoria, associada às funções estilizadas de “cão de guarda” e de “cão pastor”. O ambiente regulatório para o desenvolvimento da avaliação de políticas pelas ISC é brevemente descrito. A contribuicao do artigo e o desenho de um conjunto de parâmetros e conteúdos associados a cada um dos dois instrumentos que permite a sua comparação e evolução, contrastando com a tendência da literatura que insiste na identificação de semelhanças e diferenças entre eles.

Palavras-chave:
accountability ; auditoria de resultados; avaliação de políticas públicas; gestão pública; Instituições Superiores de Controlo

RESUMEN

Este artículo presenta dos instrumentos de gobernanza pública disponibles para las entidades fiscalizadoras superiores (EFS): la auditoría operativa o de gestión y la evaluación de políticas, destinadas a promover la responsabilidad y el aprendizaje en el desarrollo de políticas públicas. Se proporcionan sus definiciones y tipologías y, en relación con la auditoría operativa, el documento analiza la tensión generalizada entre la independencia y la relevancia de las prácticas de auditoría, asociada a los estilizados papeles de “perro guardián” y “perro pastor”. Se describe brevemente el entorno normativo para el desarrollo de la evaluación de políticas por parte de las ISC. La aportación del artículo es el diseño de un conjunto de parámetros y contenidos asociados para cada uno de los dos instrumentos que permite su comparación y evolución, contrastando con la tendencia en la literatura a identificar similitudes y diferencias entre ellos.

Palabras Clave:
responsabilidad; auditoría operativa; evaluación de políticas públicas; gestión pública; entidades fiscalizadoras superiores

INTRODUCTION

Performance audit (PA) and public policy evaluation (PPE) are two key instruments that support public governance, understood as the set of mechanisms through which public objectives are formulated and achieved, ensuring the equitable delivery of public services. These instruments contribute to fostering institutional trust, upholding public integrity, and mitigating the risk of corruption (Institute of Internal Auditors [IIA], 2012; Organization for Economic Co-operation and Development [OECD], 2016, 2020).

Traditionally, PA has been the responsibility of SAIs, while PPE is typically conducted by agencies within the structure of public administration, usually subordinated to the executive branch. However, some SAIs have expanded their mandates to include PPE, such as the Government Accountability Office (GAO) in the United States and the Cour des Comptes (CdC) in France, both pioneers in this area (Benzerafa-Alilat et al., 2021; Moscovici, 2021; OECD, 2020). Together with compliance and financial audits, PA form a major category of public audit (Cordery & Hay, 2021; INTOSAI, 2019f), with the term “auditing” encompassing an increasingly diverse and expanding range of activities in the public sector (Grossi et al., 2023). INTOSAI, established in 1953, is the international organization responsible for regulating public auditing. It develops and maintains a global framework of principles, standards, and guidance that reflect the evolving roles of SAIs.

In the second half of the 20th century, following recovery from the world wars—which had expanded the role of the state and its social dimension—parliaments in countries with political and administrative systems grounded in representative democracy, the rule of law, checks and balances entrusted their SAIs not only with monitoring legality but also with assessing how governments use public funds to implement public policies. In this context, instruments such as PA and PPE began to emerge in SAIs inspired by the Westminster model (or auditor-general model), particularly in the United States, Australia, Canada, and the United Kingdom. The rise of the New Public Management paradigm in the 1980s and 1990s further increased the relevance of the PA within SAIs, making them the subject of extensive academic study (e.g., Barzelay, 1997; Funnell, 2015).

SAIs have developed audit and evaluation tools focused on the performance of public entities, administrative systems, programs, or policies, reflecting the broader functions these institutions may perform. The adoption of such instruments may also result from institutional isomorphism – coercive, mimetic, or normative in nature (Cordery & Hay, 2021)—even among SAIs with different administrative traditions. This includes both Westminster-model SAIs, such as controllers and auditors general, and court-type SAIs, like the French Cour des Comptes, a paradigm of the continental or Napoleonic model, as adopted by the Portuguese Tribunal de Contas (Court of Accounts).

Although the distinction between the PA and the PPE remains unresolved and is subject to ongoing debate, developing a clearer understanding of the parameters for comparing these instruments may enhance our interpretation of the roles SAIs play. As independent institutions, SAIs seek to reinforce their legitimacy by demonstrating their contributions to the political-administrative system and to whole society.

Accordingly, this article aims to: (1) present the definitions, objectives, and typologies of the PA; (2) highlight the tensions and balance the PA must maintain between independence and relevance; (3) outline the concept, objectives, and types of the PPE, while briefly illustrating how two paradigmatic SAIs with distinct organizational models have incorporated this governance instrument into their mandates; and (4) based on the literature, illustrate the relationship between PA and PPE by proposing a set of parameters for substantive comparison between these two public governance instruments.

PERFORMANCE AUDIT

This section presents the PA concept and objectives; a brief overview of the different types identified in the literature and discusses the necessary balance between independence and relevance associated with this public governance instrument.

Since PA are developed through the selection of themes, topics, and the formulation of audit questions, akin to the development of research questions in academic studies, they frequently examine the implementation of public policies and the administrative structures that support their execution. Consequently, this may lead to perceptions that SAIs are interfering in political and executive functions (Hazgui et al., 2022). This creates a tension that continually challenges the independence of SAIs from the Executive Branch, as the ultimate assessment concerns whether a government is making effective use of public resources to achieve its political objectives.

Definition and objectives

Since the Lima Declaration, produced at the 1977 INTOSAI Congress, PA has reached increasingly advanced conceptual and normative levels at the international level (INTOSAI, 2019b). Subsequently, at the INTOSAI Congress held in Sydney in 1986, PA was not only recognized as an extension of the mandate of SAIs, but its objective was also defined as the improvement of public management. The concept approved at that time described it as “[an] audit which is concerned with the evaluation of economy, efficiency, and effectiveness of public sector management” (International Congress of Supreme Audit Institutions [INCOSAI], 1986, cited in Kells, 2011, p. 383).

PA, or value-for-money (VfM) audit, is an independent, reliable, and objective examination of whether public expenditure complies with the principles of economy, efficiency, and effectiveness (Shand, 2016).

INTOSAI (2019e, p. 8) provides a more detailed definition, stating that PA assesses whether interventions, programs, or institutions perform in line with the principles of economy, efficiency, and effectiveness (the three “Es”), and whether there is room for improvement in achieving these principles. The performance achieved must be compared against appropriate criteria, and the causes of non-compliance with these criteria are analyzed. The goal is to answer audit questions in reference to these three principles and issue recommendations that promote compliance with the stated criteria.

Regarding its objectives, PA aims to enhance VfM in the public sector and strengthen governance, accountability, and transparency by placing new information in the public domain and, where appropriate, issuing recommendations for improvement (INTOSAI, 2019d).

As a preliminary point, it is important to clarify the concept of accountability adopted here, as it is multifaceted, ambiguous, and can be viewed either as a virtue (a guiding value for the behavior of public agents) or as a mechanism (Bovens, 2010). Taking accountability as a mechanism that requires transparency (i.e., information) to be effective, we adopt the following definition: it is a social and institutional relationship established between an entity that renders accounts (accountor) and another that reviews those accounts (accountee), taking place in a public forum where the first entity is obliged to explain and justify its conduct, and the second may pose questions and issue a judgment, which may include sanctions for the accountor (Bovens, 2007, 2010).

These objectives are achieved because PA incorporates a range of benefits, particularly through “identifying (INTOSAI IDI, 2021, p. 13):

• waste and inefficiency in delivering public services;

• opportunities to maximise return on investment in public services;

• risks to the achievement of policy goals; and

• matters of social and economic concern to citizens.”

By providing a publicly accessible information base and producing analytical knowledge, PA can foster learning and improvement without questioning political intentions or decisions.

Types of Performance Audit

The PA encompasses a broad conceptual spectrum, which is why the literature seeks to identify and characterize its various types (e.g., Barzelay, 1997; Bowerman, 1996; Grönlund et al., 2011; Pollitt et al., 1999).

A key distinction is between substantive PA and systems PA (governance and management systems), depending on the focus of the audit. Substantive PA examines specific dimensions of the activities undertaken, such as economy, efficiency, and effectiveness (i.e., control of activities). Systems PA, on the other hand, focuses on the conditions under which activities are performed rather than their outcomes (i.e., control of governance and management systems). The audit criteria in systems PA are typically based on administrative rules, regulations, literature review, or good management practices. It is important to note that elements of both substance and systems may coexist within a single PA.

Based on the literature, Table 1 summarizes the different types of PA.

Tabela 1
Types of Performance Audit

TYPE	DESCRIPTION
Economy audit	Focuses on examining inputs in order to assess whether costs are being minimized, ensuring the level of quality
Efficiency audit	Analyzes the relationship between outputs and resources used in the production of public goods and services
Program effectiveness audit	Focuses on assessing whether outcomes have been achieved
Performance management capacity audit	Assesses whether the auditee has the conditions and skills to manage processes and programs efficiently and effectively
Performance information audit	The information disclosed by the auditees on performance is audited by reference to a set of attributes (e.g. reliability, relevance, timeliness)
Risk assessment audit	Aims to alert those responsible to risks that may jeopardize the success of a public policy
Best practice review	The aim is, through research methodologies, identify the factors that explain the best practices found, contributing to define criteria for other types of audits
General management review	An assessment is made of an organization rather than a policy or program, seeking to identify the causes of any performance problems within the organization

It is important to note that these types of audits are associated with the selected criteria, and different criteria can coexist within a single PA, including some related to compliance auditing (Grönlund et al., 2011; Pollitt et al., 1999, pp. 89–92).

Balancing Independence and Relevance

The independence of SAIs is fundamental to their social legitimacy and impact (INTOSAI, 2019c). Posner and Shahan (2014) identify four types of independence: (i) political, (ii) organizational, (iii) functional, and (iv) financial.

Political independence means that SAIs are not directly influenced by the executive (government) or legislative (parliament) branches, allowing it to plan and carry out audits based on professional judgment and standards. This does not mean that the institution disregards the expectation of the parliament. In fact, SAIs can be seen as a mechanism available to parliaments to oversee government actions. Streim (1994) emphasizes that SAIs are justified by agency problems in the political system. Moral hazard is more likely when the actions of the agent (government) are not fully observable by the principal (parliament), which may result in the agent not exerting the effort expected to implement government programs. To serve as a transparency-enhancing mechanism and reduce agency problems, SAIs must maintain their independence.

As for the other three types of independence, organizational independence concerns the appointment, tenure, and dismissal of the SAI’s president and members. These processes must be clearly defined in the legal framework (e.g., through an organic law), along with the SAI’s autonomy in managing human resources; functional independence involves unrestricted access to relevant information, autonomy in selecting audit topics and methodologies, the authority to issue recommendations, and the freedom to publish reports; and Financial independence means the SAIs are empowered to determine the level of financial resources required to fulfill their mandate and implement their plan of activities.

However, the way SAIs interprets the relationship between independence and relevance can influence the effectiveness of their audits. Tension may arise when audits fail to produce meaningful improvements in public governance or management (Lonsdale, 2008; Triantafillou, 2020).

For example, Triantafillou (2020) argues that if independence is interpreted as consistently issuing reports dominated by criticism – overemphasizing negative findings while ignoring positive outcomes – this may lead to counterproductive effects. These include diminished innovation, reluctance to assume responsibility, and demotivation within the public administration (Kells, 2011).

Conversely, an excessive emphasis on relevance may compel the SAI to audit the outcomes of its own recommendations, potentially compromising its perceived independence. Johnsen (2019) posits that the equilibrium between independence and relevance can be swayed by the partisan composition of parliament. For example, if the government holds only a parliamentary minority, SAIs might refrain from issuing improvement advice to maintain their appearance of neutrality, thereby prioritising independence over relevance.

This tension between independence and relevance relates to two general functions of SAIs identified in the literature (Bourne, 2007; English et al., 2010; Ling, 2003). The first is the “watchdog” function, which reflects auditors’ traditional comfort zone and emphasizes public accountability. It typically involves ex post analysis of compliance with rules, standards, and good governance expectations, resulting in information disclosure to parliament and the public. The second is the “sheepdog” function, where audits are geared toward enhancing public governance and management, reflecting an ex-ante approach and more proactive engagement with auditees. The limitation of the “sheepdog” function is that it may jeopardize the independence necessary for the “watchdog” role, blurring the distinction between audit and public policy advice (Pierre et al., 2018). Power (2003) associates the “watchdog” function with a logic of accountability and the “sheepdog” function with a logic of learning. Ideally, audits should seek to balance both logics, although these functions are often seen as incompatible due to the practical challenges involved in reconciling them.

PUBLIC POLICIES EVALUATION

Policy evaluation is the final stage of the public policy cycle, which also includes: (i) issue emergence; (ii) agenda setting; (iii) alternative selection; and (iv) implementation (Birkland, 2020, pp. 33–34).

Conceptualization and goals

There are different definitions of PPE, or the evaluation of a public intervention aimed at producing change in each context. In the United Kingdom, HM Treasury (2020, p. 9) defines evaluation as the systematic assessment of an intervention’s design, implementation, and outcomes. It involves understanding how an intervention is being or has been implemented, its effects, for whom, and why. It also includes identifying possible improvements to the process and how to implement them, as well as estimating impacts and cost-effectiveness.

Within the framework of SAIs, PPE consists of an examination intended to assess the usefulness of a policy, based on its performance in terms of objectives, implementation, outputs, outcomes, and final impacts—with the latter two (outcomes and impacts) representing its effects (INTOSAI, 2019a).

A central step in evaluation is the development of a theory of how change will occur through a given public intervention, commonly referred to in the literature as a Theory of Change. This involves constructing a model that identifies the goals and explains how the intervention will contribute to achieving them, thereby enhancing the policy’s effectiveness. This conceptualization allows for the identification of additional influencing factors that need to be measured and monitored to determine whether the observed effects align with expectations, understand the causes of the policy’s success or failure, identify who is benefiting, and clarify under what conditions (Birckmayer & Weiss, 2000; HM Treasury, 2020). It serves as a roadmap comprising multiple stages that require a detailed understanding of the steps leading to a long-term goal, as well as the connections between program activities and the outcomes that emerge along the way (Weiss, 1995). In this sense, a Theory of Change is a specific explanation, through projection or prediction, of how a given policy is expected to achieve its intended objectives.

Like PA, the primary objectives of PPE are to enhance public accountability and to encourage the learning necessary for improving public governance and management (HM Treasury, 2020; INTOSAI, 2019a).

From a learning perspective, PPE contributes by (i) managing the risks associated with a given intervention and its implementation; (ii) improving ongoing interventions by providing evidence-based support for decision-making, thereby helping management to achieve results; and (iii) identifying what works (e.g., at the governance and management level), for whom, and under what conditions, thereby generating knowledge for future policy design.

The promotion of accountability through PPE derives primarily from the evidence generated on the effects of public interventions. This evidence can give rise to various forms of political accountability, such as interpellations and questions posed by members of parliament to the government, the appearance of government officials before specialized parliamentary committees, or even the establishment of parliamentary inquiries triggered by findings from a public policy evaluation. The same argument applies to PA.

Types of Public Policy Evaluation

Table 2 briefly describes the types of assessments, based on the most recent professional literature (GAO, 2021; HM Treasury, 2020; National Audit Office [NAO], 2022; OECD, 2020).

Tabela 2
Types of Public Policy Evaluation

TYPE	DESCRIPTION
Process evaluation (or implementation)	The evaluation has a formative purpose; that is, it seeks to examine the extent to which a policy (or program) is being implemented as intended, produces the expected results, and follows the requirements, design, standards, or expectations of stakeholders. Aims to answer the question: What can be learned from the implementation of the public policy?
Evaluation of outcomes	The evaluation has a summative purpose, insofar as it seeks to assess whether a given policy (or program) achieved its outcomes and how they were (or were not) achieved Aims to answer the question: Did the public policy work?
Impact evaluation	The evaluation serves a summative purpose aimed at measuring the effects of a policy (or program) by estimating what would have occurred in the absence of that policy (or program). Aims to answer the question: What difference did the public policy make?
Cost-benefit and cost-effectiveness analysis	Cost-benefit analysis seeks to identify all the qualitative and quantitative costs and benefits associated with a policy (or program), typically expressed in monetary units. Cost-effectiveness analysis aims to identify all costs to achieve a goal or objective, and can be used to identify the lowest cost alternative to achieve that goal or objective Aims to answer the question: Does the public policy justify the resources consumed?

Public Policy Evaluation by SAIs

As noted in this section, the U.S. Supreme Audit Institution (SAI) was a pioneer in incorporating the evaluation of public policies and programs into its mandate.

Within the framework of INTOSAI, PPE received particular attention in 1992 when a working group on policy and program evaluation was established. This group currently comprises 26 members and was coordinated by the French SAI from its inception until 2022; it is now coordinated by the Swiss SAI. (More information on the Working Group on Evaluation of Public Policies and Programs [WGEPPP] is available at: https://wgeppp.intosai-community.net/.)

One of the main outcomes of this working group was the approval, in 2016, by the INTOSAI Congress of INTOSAI GOV 9400 (currently GUID 9020), a guide intended to support SAIs in conducting PPE.

GUID 9020 (INTOSAI, 2019a) defines the main characteristics of a PPE, including its relationship with the administrative and political actors responsible for implementing policies, the planning and execution of the evaluation, and the presentation of results, among other aspects.

Figure 1 illustrates the process, highlighting the points of contact with the performance audit (PA) and the specific elements of PPE based on a logical model (McLaughlin & Jordan, 2010).

Figura 1
Public Policy Evaluation Process

Note: Adapted from INTOSAI (2019a).

It is crucial to clarify the concepts specific to PPE, as defined in GUID 9020. While the relevance of a public policy refers to the adequacy of its objectives concerning social, economic, and environmental needs, its utility concerns the policy’s merit – specifically, how the outputs (i.e., public goods and services produced) generate direct effects on outcomes and indirect effects on the level of impacts, whether expected or unexpected. These effects are assessed based on the needs the policy aims to address.

The following section briefly describes how the PPE was incorporated into the mandates of two SAIs from different traditions and models: (i) the U.S. Government Accountability Office (GAO), which follows the Westminster model, operates as an Auditor General-type institution and was a pioneer in integrating the PPE into its work; and (ii) the French Cour des Comptes (CdC), which is based on the Napoleonic model.

Government Accountability Office

The U.S. Government Accountability Office (GAO), led by a Comptroller General, was established in 1921 through the Budget and Accounting Act as an agent of Congress, from which it derives its powers, responsibilities, and resources. Its primary purpose was to provide Congress with better information and oversight of public spending, particularly regarding probity and legality. At the time, the GAO also performed prior inspection activities, such as approving warrants, and played a role in the authorization processes for expenditures, debt, and revenue. Additionally, it was granted authority in accounting standardization (Mosher, 1979).

The development of the Planning-Programming-Budgeting System in the 1960s, followed by the introduction of management by objectives and zero-based budgeting in the 1970s, created the conditions for expanding the GAO’s mandate. The Congressional Budget and Impoundment Control Act of 1975 (Chelimsky, 1985, p. 483) formally extended the GAO’s functions to include the PPE. This encompassed effectiveness assessments, cost-benefit analyses of policy alternatives, and special studies, with some reports offering recommendations that effectively proposed changes in public policy. The GAO was the first SAI to incorporate the evaluation of programs and policies into its official responsibilities (Chelimsky, 1985; Mosher, 1979).

Today, GAO evaluates the use of public resources, reviews federal programs and activities, conducts in-depth analyses, offers policy options, and issues recommendations and other forms of support to Congress to facilitate effective political oversight, policymaking, and funding decisions. Its instruments include financial audits, performance audits, program evaluations, attestation engagements, legal opinions, investigations, and other services (Shaw, 2016, p. 58).

Cour des Comptes

The Cour des Comptes (CdC) is France’s Supreme Audit Institution (SAI), founded in 1807 by Emperor Napoleon. It operates independently of both Parliament and the government, with its mandate structured around four main missions: i) Judging public accounts by verifying compliance with public accounting standards by public managers, which may lead to financial sanctions; ii) Monitoring through analysis of the regularity, efficiency, and effectiveness in the use of public funds; iii) Certifying accounts, a function established by the 2001 revision of the French Budget Framework Law (Loi organique relative aux lois de finances – LOLF), whereby the CdC issues an opinion on the faithful, transparent, and appropriate presentation of the State and Social Security accounts; and iv) Evaluating public policies as part of the CdC’s broader role in supporting both Parliament and the government (Benzerafa-Alilat et al., 2021; Moscovici, 2021; NAO, 2021).

The fourth mission – PPE – was formally established through the constitutional revision of 23 July 2008, which empowered the French SAI to assess public interventions beyond traditional economic, efficiency, and effectiveness criteria. These evaluations emphasize the relevance and impact of public policies while also considering how well the policies align with the needs they are intended to address (NAO, 2021).

PERFORMANCE AUDIT AND PUBLIC POLICY EVALUATION: COMPARATIVE PERSPECTIVE

There are four main ways to conceptualize the relationship between PA and PPE. Without implying any order of priority, the first view holds that they are distinct practices. The second considers PA a component of PPE. Conversely, the third sees PPE as a component of PA. The fourth, which offers a more balanced perspective, suggests that similarities and differences exist between the two, depending on the criteria used for comparison (Barzelay, 1997; Peters & Pierre, 2020).

However, we argue that rather than focusing solely on their relationship, it is more useful to attempt a characterization of both practices, albeit in a stylized form, based on a set of parameters. Some elements may be common to both practices, while others may be unique to one. With this in mind, and following the brief descriptions provided in the previous sections, it is now important to systematize, based on the literature, the parameters that enable a comparative understanding of these two disciplines. Accordingly, Table 3 summarizes the elements present in PA and PPE, drawing on a set of parameters identified in the literature (Barrados, 2020; Chelimsky, 1985, 1996; Divorski, 1996; INTOSAI, 2019a; Leeuw, 1992; Moscovici, 2021; Pierre et al., 2018).

Tabela 3
Parameters to Compare Performance Audit and Public Policy Evaluation

PARAMETER	PERFORMANCE AUDIT	PUBLIC POLICY EVALUATION
Focus	Policy administration	Policy conceptualization
Criteria	Economy, efficiency, effectiveness	Economy, efficiency, effectiveness, relevance, utility Identify the unforeseen consequences of the policy as well
Object	Entities and programs	Programs
Actors independency	SAIs are largely independent of the audited entities or programs and of political power. The PA topics are chosen by the SAI and may take into account requests from political power.	The evaluation entities are usually dependent on political power Third parties usually indicate the evaluation topics
Methodology	Follows standards and guidelines associated with audit objectives. More descriptive in nature, establishing facts rather than their causes (what)	Originating from the social sciences, it aims to answer questions related to the efficiency and effectiveness of policy programs. More explanatory in nature, oriented towards the causes of a given effect (the why)
Approach	Emphasizes the identification of issues rather than providing advice for improving governance and management of public policies	Emphasizes the improvement of governance and management of public policies, with a learning objective. It can also serve reward and punishment purposes (accountability)
Institutional positioning	Developed by SAI, distant from the political process	Developed by the entity implementing the public policy, by public administration agencies and/or consultants (e.g., companies, academia) It can be part of the mandate of the SAIs. It is part of the political process, being better positioned to advise on the design of policies.
Timeframe	More short-term, being able to detect problems earlier, but drawing premature conclusions about the usefulness of a public intervention	Due to the need to assess impacts (see Figure 1), it requires a long time for, in addition to the outcomes, produce impacts in society. Often incorporates the perspectives of those who were affected by the public intervention.
Recipients of reports	The general public, the legislative branch (parliament), and the executive branch (government)	Typically, the executive branch (government)

Part of the literature suggests that PA is increasingly integrating elements traditionally associated with PPE, which also reflects the broadening scope of PA (Peters & Pierre, 2020; Pierre et al., 2018). Table 3 indicates that certain elements of PPE linked to specific parameters that define its content are now being incorporated into the parameters of PA. For example, methodological aspects of PPE are being adopted within the methodology of a given PA, which begins to pose different and more complex questions. The answers to these questions require analytical skills capable of producing scientifically valid knowledge about the outcomes and impacts of a given public policy.

It is important to emphasize that developing public policies without an appropriate knowledge base can lead to failure in analyzing socioeconomic problems and selecting the right policy instruments to address them. For this reason, institutions must take responsibility for generating this knowledge through practices such as PA and PPE. While each relies on a different professional foundation, both practices (or disciplines) produce knowledge essential for achieving public accountability and enhancing public governance and management.

Rather than competing for space within a specific institutional field, we view PA and PPE as complementary. Both are crucial for producing information that supports the quality of the policy cycle (see the section on Public Policy Evaluation), thereby contributing to more effective public interventions, those that are better aligned with citizens’ needs, more accountable, and better governed and managed (Chelimsky, 1996).

CONCLUSIONS

This article aims to provide a brief description of PA and PPE as two public governance instruments that share the objectives of promoting the transparency required for public accountability and improving public governance and management.

The literature review suggests the emergence of a path characterized by a transferability of elements between PA and PPE, specifically the elements of the parameters identified in Table 3 presented above. This cross-fertilization is responsible for the ongoing relevance of the debate surrounding the parameters that distinguish these instruments and the elements that constitute them.

We argue that the main challenge lies not in differentiating between PA and PPE, but in understanding their relevance regarding their usefulness for public decision-makers, legislators, and other stakeholders. These instruments contribute to promoting accountability, facilitating change, and enhancing both administrative activity and the formulation and management of public policies.

Unlike evaluation, PA does not assess the merit, utility, or relevance of public policy objectives. The dual purpose of PA – promoting accountability and fostering improvement – leads to a focus on the success of implemented solutions. When PA influences political agendas or contributes to decision-making and policy reformulation or termination – either by attracting the attention of political decision-makers or through direct requests from legislators or governments – it may overlap in function with PPE. However, these two instruments should not be confused. The PAs are rooted in the external control function of SAIs, not in ministerial technical units or other public or non-governmental agencies. Nevertheless, this distinction does not preclude SAIs from conducting both PA and PPE – as is the case with institutions such as the Government Accountability Office (GAO) in the United States and the Cour des Comptes (CdC) in France – wherein SAIs may adopt a more explicitly advisory role in the political process.

In this context, it is possible to say that the more relevant an SAI aims to be to its parliament, the greater the inclusion of elements traditionally associated with PPE within PA, or alternatively, the more likely it is that PPE becomes part of its mandate, as has occurred with the CdC (Benzerafa-Alilat et al., 2021; Moscovici, 2021). This development may help better delineate the “watchdog” and “sheepdog” functions.

Thus, PPEs conducted by SAIs may shift these institutions away from the traditional “watchdog” function, focused on identifying, correcting, and sanctioning wrongdoing, and toward a more pedagogical profile associated with the “sheepdog” function. This function involves scenario analysis, risk alerts, and the provision of advice, with the overarching goal of enhancing public governance and management (Ling, 2003; Viana et al., 2022).

It is broadly acknowledged that public management and governance at various levels, especially in increasingly complex societies, must rely on high-quality evidence (or information) generated through scientific methods, both quantitative and qualitative. PA and PPE contribute to the production of this evidence. However, many political choices and associated implementation tools have historically been based more on ideology than on scientifically derived evidence (Peters & Nagel, 2024).

Finally, considering the attention PPE has received from paradigmatic SAIs such as GAO and CdC, as well as from INTOSAI itself, a phenomenon of institutional isomorphism – either mimetic or normative – may be underway, particularly among SAIs from different political-administrative traditions (Cordery & Hay, 2021). The evolution of this phenomenon is likely to shape the dynamics of future debates concerning the relationship between PA and PPE.

• Evaluated through a double-anonymized peer review.

References

• Barrados, M. (2020). Introduction. In M. Barrados & J. Lonsdale (Eds.), Crossover of audit and evaluation practices (pp. 1-12). Routledge.
• Barzelay, M. (1997). Central audit institutions and performance auditing: A comparative analysis of organizational strategies in the OECD. Governance, 10(3), 235-260. https://doi.org/10.1111/0952-1895.411997041
  » https://doi.org/10.1111/0952-1895.411997041
• Benzerafa-Alilat, M., Tandilashvili, N., & Friscia, M. (2021). New development: Diversification of the Cour des comptes’ activities – A necessary modernization or an identity evolution? Public Money & Management, 41(6), 491-493. https://doi.org/10.1080/09540962.2021.1934230
  » https://doi.org/10.1080/09540962.2021.1934230
• Birckmayer, J. D., & Weiss, C. H. (2000). Theory-Based evaluation in practice: What do we learn? Evaluation Review, 24(4), 407-431. https://doi.org/10.1177/0193841x0002400404
  » https://doi.org/10.1177/0193841x0002400404
• Birkland, T. (2020). An introduction to the policy process: Theories, concepts, and models of public policy making (5fh ed.). Routledge.
• Bourne, J. (2007). Public sector auditing: Is it value for money? John Wiley and Sons.
• Bovens, M. (2007). Analysing and assessing accountability: A conceptual framework. European Law Journal, 13(4), 447-468. https://doi.org/10.1111/j.1468-0386.2007.00378.x
  » https://doi.org/10.1111/j.1468-0386.2007.00378.x
• Bovens, M. (2010). Two concepts of accountability: Accountability as a virtue and as a mechanism. West European Politics, 33(5), 946-967. https://doi.org/10.1080/01402382.2010.486119
  » https://doi.org/10.1080/01402382.2010.486119
• Bowerman, M. (1996). The Rise and fall of value for money auditing. In I. Lapsley & F. Mitchell (Eds.), Accounting and performance measurement: Issues in the private and public sectors (pp. 193-212). London: Paul Chapman Publishing
• Chelimsky, E. (1985). Comparing and contrasting auditing and evaluation: Some notes on their relationship. Evaluation Review, 9(4), 483-503. https://doi.org/10.1177/0193841X8500900406
  » https://doi.org/10.1177/0193841X8500900406
• Chelimsky, E. (1996). Auditing and evaluation: Whither the relationship? New Directions for Evaluation, 1996(71), 61-67. https://doi.org/https://doi.org/10.1002/ev.1043
  » https://doi.org/https://doi.org/10.1002/ev.1043
• Cordery, C., & Hay, D. (2021). Public sector audit Routledge.
• Divorski, S. (1996). Differences in the approaches of auditors and evaluators to the examination of government policies and programs. New Directions for Evaluation, 1996(71), 7-14. https://doi.org/https://doi.org/10.1002/ev.1039
  » https://doi.org/https://doi.org/10.1002/ev.1039
• English, L. M., Guthrie, J., Broadbent, J., & Laughlin, R. (2010). Performance audit of the operational stage of long‐term partnerships for the private sector provision of public services. Australian Accounting Review, 20(1), 64-75. https://doi.org/10.1111/j.1835-2561.2010.00075.x
  » https://doi.org/10.1111/j.1835-2561.2010.00075.x
• Funnell, W. (2015). Performance auditing and adjudicating political disputes. Financial Accountability & Management, 31(1), 92-111. https://doi.org/10.1111/faam.12046
  » https://doi.org/10.1111/faam.12046
• Government Accountability Office. (2021). Program evaluation: Key terms and concepts https://www.gao.gov/products/gao-21-404sp
  » https://www.gao.gov/products/gao-21-404sp
• Grönlund, A., Svärdsten, F., & Öhman, P. (2011). Value for money and the rule of law: The (new) performance audit in Sweden. International Journal of Public Sector Management, 24(2), 107-121. https://doi.org/10.1108/09513551111109026
  » https://doi.org/10.1108/09513551111109026
• Grossi, G., Hay, D. C., Kuruppu, C., & Neely, D. (2023). Changing the boundaries of public sector auditing. Journal of Public Budgeting, Accounting & Financial Management, 35(4), 417-430. https://doi.org/10.1108/JPBAFM-05-2023-0079
  » https://doi.org/10.1108/JPBAFM-05-2023-0079
• Hazgui, M., Triantafillou, P., & Christensen, S. E. (2022). On the legitimacy and apoliticality of public sector performance audit: Exploratory evidence from Canada and Denmark. Accounting, Auditing & Accountability Journal, 35(6), 1375-1401. https://doi.org/10.1108/AAAJ-04-2020-4508
  » https://doi.org/10.1108/AAAJ-04-2020-4508
• HM Treasury. (2020). The magenta book https://assets.publishing.service.gov.uk/media/5e96cab9d3bf7f412b2264b1/HMT_Magenta_Book.pdf
  » https://assets.publishing.service.gov.uk/media/5e96cab9d3bf7f412b2264b1/HMT_Magenta_Book.pdf
• Institute of Internal Auditors. (2012). The role of auditing in public sector governance https://na.theiia.org/standards-guidance/Public%20Documents/Public_Sector_Governance1_1_.pdf
  » https://na.theiia.org/standards-guidance/Public%20Documents/Public_Sector_Governance1_1_.pdf
• International Organization of Supreme Audit Institutions. (2019a). GUID 9020 – Evaluation of public policies. https://www.issai.org/pronouncements/guid-9020-evaluation-of-public-policies/
  » https://www.issai.org/pronouncements/guid-9020-evaluation-of-public-policies/
• International Organization of Supreme Audit Institutions. (2019b). INTOSAI-P 1 – The Lima Declaration https://www.issai.org/pronouncements/intosai-p-1-the-lima-declaration/
  » https://www.issai.org/pronouncements/intosai-p-1-the-lima-declaration/
• International Organization of Supreme Audit Institutions. (2019c). INTOSAI-P 10 – The Mexico Declaration on SAI Independence
• International Organization of Supreme Audit Institutions. (2019d). ISSAI 300 – Performance audit principles https://www.issai.org/pronouncements/issai-300-performance-audit-principles/
  » https://www.issai.org/pronouncements/issai-300-performance-audit-principles/
• International Organization of Supreme Audit Institutions. (2019e). ISSAI 3000 - Performance audit standard. https://www.issai.org/professional-pronouncements/?n=3000-3899
  » https://www.issai.org/professional-pronouncements/?n=3000-3899
• International Organization of Supreme Audit Institutions. (2019f). ISSAI - 100 – Fundamental principles of public-sector auditing https://www.issai.org/pronouncements/issai-100-fundamental-principles-of-public-sector-auditing/
  » https://www.issai.org/pronouncements/issai-100-fundamental-principles-of-public-sector-auditing/
• International Organization of Supreme Audit Institutions IDI. (2021). Performance audit: ISSAI implementation handbook. https://www.idi.no/work-streams/professional-sais/work-stream-library/performance-audit-issai-implementation-handbook
  » https://www.idi.no/work-streams/professional-sais/work-stream-library/performance-audit-issai-implementation-handbook
• Johnsen, Å. (2019). Public sector audit in contemporary society: A short review and introduction. Financial Accountability & Management, 35(2), 121-127. https://doi.org/10.1111/faam.12191
  » https://doi.org/10.1111/faam.12191
• Kells, S. (2011). The seven deadly sins of performance auditing: Implications for monitoring public audit institutions. Australian Accounting Review, 21(4), 383-396. https://doi.org/10.1111/j.1835-2561.2011.00150.x
  » https://doi.org/10.1111/j.1835-2561.2011.00150.x
• Leeuw, F. L. (1992). Performance auditing and policy evaluation: Discussing similarities and dissimilarities. Canadian Journal of Program Evaluation, 7(1), 53-68. https://doi.org/10.3138/cjpe.07.004
  » https://doi.org/10.3138/cjpe.07.004
• Ling, T. (2003). Ex ante evaluation and the changing public audit function: The scenario planning approach. Evaluation, 9(4), 437-452. https://doi.org/10.1177/135638900300900405
  » https://doi.org/10.1177/135638900300900405
• Lonsdale, J. (2008). Balancing independence and responsiveness: A practitioner perspective on the relationships shaping performance audit. Evaluation, 14(2), 227-248. https://doi.org/10.1177/1356389007087541
  » https://doi.org/10.1177/1356389007087541
• MacLaughlin, J., & Jordan, G. (2010). Using logic models. In J. Wholey, H. Hatry, & K. Newcomer (Eds.), Handbook of practical program evaluation (3rd ed., pp. 55-80). John Wiley & Sons.
• Moscovici, P. (2021). La experiencia de la Cour des comptes de Francia en la evaluación de las políticas públicas. Revista Española de Controlo Externo, XXIII(68), 10-25.
• Mosher, F. C. (1979). The GAO: The quest for accountability in American government Westview.
• National Audit Office. (2021). Peer review of the Cour des Comptes https://www.ccomptes.fr/system/files/2021-03/202102-UK-NAO-Peer-review-of-the-French-Cour-des-comptes-English.pdf
  » https://www.ccomptes.fr/system/files/2021-03/202102-UK-NAO-Peer-review-of-the-French-Cour-des-comptes-English.pdf
• National Audit Office. (2022). Evaluating government spending: an audit framework https://web.archive.org/web/20220412080353id_/https:/www.nao.org.uk/wp-content/uploads/2022/04/Evaluating-government-spending-an-audit-framework.pdf
  » https://web.archive.org/web/20220412080353id_/https:/www.nao.org.uk/wp-content/uploads/2022/04/Evaluating-government-spending-an-audit-framework.pdf
• Organization for Economic Co-operation and Development. (2016). Supreme audit institutions and good governance: Oversight, insight and foresight OECD Publishing.
• Organization for Economic Co-operation and Development. (2020). Improving Governance with policy evaluation OECD Publishing.
• Peters, B. G., & Nagel, M. L. (2024). Zombie Ideas: Policy pendulum and the challenge of effective policymaking. Policy Quarterly, 20(1), 10-15. https://doi.org/10.26686/pq.v20i1.9047
  » https://doi.org/10.26686/pq.v20i1.9047
• Peters, B. G., & Pierre, J. (2020). From evaluation to auditing and from programs to institutions: Causes and consequences of the decline of the program approach. Governance, 33(3), 585-597. https://doi.org/10.1111/gove.12455
  » https://doi.org/10.1111/gove.12455
• Pierre, J., Peters, B. G., & Licht, J. de F. (2018). Is auditing the new evaluation? Can it be? Should it be? International Journal of Public Sector Management, 31(6), 726-739. https://doi.org/10.1108/IJPSM-08-2017-0219
  » https://doi.org/10.1108/IJPSM-08-2017-0219
• Pollitt, C., Girre, X., Lonsdale, J., Mul, R., Summa, H., & Waerness, M. (1999). Performance or compliance? Performance audit and public management in five countries Oxford University Press.
• Posner, P., & Shahan, A. (2014). Audit institutions. In M. Bovens, R. Goodin, & T. Schillemans (Eds.), The Oxford handbook of public accountability (pp. 488-506). Oxford University Press.
• Power, M. (2003). Evaluating the audit explosion. Law & Policy, 25(3), 185-202. https://doi.org/10.1111/j.1467-9930.2003.00147.x
  » https://doi.org/10.1111/j.1467-9930.2003.00147.x
• Shand, M. (2016). External audit. In R. Allen, R. Hemming, & B. Potter (Eds.), The international handbook of public financial management (pp. 817-836). Palgrave Macmillan.
• Shaw, T. (2016). Performance budgeting practices and procedures. OECD Journal on Budgeting, 15(3), 65-136. https://doi.org/10.1787/budget-15-5jlz6rhqdvhh
  » https://doi.org/10.1787/budget-15-5jlz6rhqdvhh
• Streim, H. (1994). Agency problems in the legal political system and supreme auditing institutions. European Journal of Law and Economics, 1, 177-191. https://doi.org/10.1007/BF01552469
  » https://doi.org/10.1007/BF01552469
• Triantafillou, P. (2020). Playing a zero‐sum game? The pursuit of independence and relevance in performance auditing. Public Administration, 98(1), 109-123. https://doi.org/10.1111/padm.12377
  » https://doi.org/10.1111/padm.12377
• Viana, L., Moreira, J. A., & Alves, P. (2022). State audit of public–private partnerships: Effects on transparency, auditor’s roles, and impact on auditee’s reactions. Financial Accountability & Management, 38(4), 633-660. https://doi.org/10.1111/faam.12342
  » https://doi.org/10.1111/faam.12342
• Weiss, C. (1995). Nothing as practical as good theory: Exploring Theory-based evaluation for comprehensive community initiatives for children and families. In J. Connell, A. Kubisch, L. Schorr, & C. Weiss (Eds.), New approaches to evaluating community initiatives: Concepts, methods, and contexts (pp. 65-92). The Aspen Institute.

Edited by

• Guest Editors:
  Albino Alves Simione, Ana Lúcia Romão, Daniel Matos Caldeira, Elisa Rangel Nunes, Luís Ortet da Veiga, Marco Antônio Carvalho Teixeira, Maria das Neves Ceita Batista de Sousa

Publication Dates

• Publication in this collection
  25 Aug 2025
• Date of issue
  2025

History

• Received
  06 June 2023
• Accepted
  23 Aug 2024