A new classification for the results of the performance audit follow up

Abstract

For several decades, performance audits (AOPs) have significantly helped the improvement of public management in Brazil and the world. However, few studies have investigated this audits follow-up process and how they have adapted to Brazilian culture. This research proposes a new classification for the results of the performance audits follow-up observing whether the determinations and recommendations resulting from AOPs carried out by the Court of Accounts of the State of Santa Catarina (TCE/SC) are implemented. The classification provided in the AOP Manual prepared by the Federal Court of Accounts, was adapted by the TCE/SC, with changes that were described in the follow-up reports of 14 AOPs with information from the fiscal years 2004 to 2015. A first draft of a new classification was submitted to the auditors for analysis and suggestions. After that, it was sent to the analysis and contributions of the chief auditors and audit coordinators. The study, therefore, registered the changes that had occurred in the AOPs and proposed a new classification for the results of the AOPs follow-up that considers the degree of implementation of determinations and recommendations in the performance audit.

Keywords:
audit; performance audits; follow-up; results classification

Resumo

Há décadas, as auditorias operacionais (AOPs) constituem importante ferramenta para o aprimoramento da gestão pública no Brasil e no mundo. Contudo, poucos estudos tratam do processo de monitoramento dessas auditorias e de sua adaptação à cultura brasileira. O objetivo desta pesquisa foi propor uma nova classificação para o acompanhamento das determinações e das recomendações resultantes das AOPs realizadas pelo Tribunal de Contas do Estado de Santa Catarina (TCE/SC). A partir da classificação prevista no manual de AOP do Tribunal de Contas da União (TCU), o TCE/SC adotou alterações na classificação descrita nos relatórios de monitoramento das 14 AOPs encerradas até 2015. A atual classificação foi submetida aos auditores e, posteriormente, aos chefes e aos coordenadores de auditoria, para sugestões. Por essa razão, sugere-se uma nova classificação para os resultados das AOPs.

Palavras-chave:
auditoria; auditoria operacional; monitoramento; classificação de resultados

Resumen

Durante décadas, las auditorías de desempeño (AOP) han sido una herramienta importante para la mejora de la gestión pública en Brasil y en el mundo. No obstante, pocos estudios abordan el proceso de seguimiento de esas auditorías y su adaptación a la cultura brasileña. El objetivo de esta investigación es proponer una nueva clasificación para el seguimiento de las determinaciones y recomendaciones resultantes de las AOP, llevadas a cabo por el Tribunal de Cuentas del Estado de Santa Catarina (TCE/SC). A partir de la clasificación establecida en el Manual de AOP del Tribunal de Cuentas de la Unión, el TCE/SC adoptó cambios en la clasificación descrita en los informes de seguimiento de las 14 AOP concluidas hasta 2015. La clasificación actual se sometió a los auditores y posteriormente a los jefes y coordinadores de auditoría para posibles sugerencias. Por esta razón, se sugiere una nueva calificación para los resultados de las AOP.

Palabras clave:
auditoría; auditoría de desempeño; seguimiento, clasificación de resultados

1. INTRODUCTION

The necessity of more rigorous evaluation of the activities of policies, programmes and projects in Brazil emerged from the economic transformations in the 1980s, especially from the debate on the crisis of the State,which strengthened over the subsequent decade (Antero, 2008).

The Federal Court of Accounts of Brazil (TCU) was a pioneer institution, in the 1980s, in the function of the external public control, in our country, in carrying out performance audits that were in line with the international trend in assessment the results achieved in programmes, projects, activities, bodies and legal entities of the Public Management. After a while, with the training project of evaluating public programmes, the TCU inaugurated, in 1995, new methods of evaluating governmental programmes and projects (Tribunal de Contas da União (TCU, 1998).

This activity has begun, in Santa Catarina, with the course of Management Auditing in the Public Sector, which happened from 2002 to 2004 and trained 92 fiscal auditors of external control. The achievement of performance audits gained prominence and intensity in 2007, from the creation of a control office with the attribute of conducting performance audits, among other responsabilities, to training courses provided in the extent of Program for Modernization of the External Control System of Brazilian States, Federal District and Cities Programa de Modernização do Controle Externo dos Estados, Distrito Federal e Municípios Brasileiros (PROMOEX) (Fleischmann, 2015).

The Santa Catarina State Court of Accounts (TCE/SC) has been utilising performance auditing manuals of the Federal Court of Accounts (TCU) as the cornerstone document of its activities since 2007. Nevertheless, the use of the classification that is included in the Performance Audit Manual, Manual de Auditoria Operacional (TCU, 2010) for positioning itself in relation to the degree of implementing recommendations, as well as for observing determinations, has undergone some changes over the past years, which are the outcome of the contribution of court´s peers and the experience gained from the follow-up to performance audits that were conducted by TCE/SC. Such changes still were not sufficient for expressing amplitude of situations that happen during these follow-ups.

For this reason, what is being emphasised is the necessity of describing and analysing the evolution of the classification of the observance of determinations and the implementation of recommendations that were counterposed by the TCE/SC in its performance audits to, from de facto situation, have the power to propose a new classification in the case of the observance of determinations and the implementation of recommendations.

The research method applied in this study consists of bibliographic and documentary sources as well as in a quantitative and qualitative comparative analysis of final considerations of the last follow-up reports on performance audits that ended up by financial year 2015. It represents 14 processes involving 101 determinations and 204 recommendations. In that case, the Santa Catarina Court of Accounts (TCE/SC) did not regularize in a specific manner the classification of the result of the performance auditing, but even so the institution regurlaly has used the Performance Auditing Manual (TCU, 2010) as a directive.

The TCE/SC encouraged changes in the classification, which are registered in this article, by virtue of accumulating experience of conducting its follow-up audits. The research result was presented to the fiscal auditors of external control who instruct performance auditing processes and their follow-ups in order to validate informations to the construction of a new proposal for classifying the evaluation of compliance with determinations and the implementation of recommendations.

Limitations of the study were originated from the lack of register in the TCE/SC of decisions and bases, which have been used for changing classifications in the course of fiscal years (FYs), as well as from the deficiency of the evidences description that led to conclude the compliance or not with some determinations, without presenting elements that can evaluate the degree of improvement on the performance of the task. In that case, the previous classification was maintained. Despite the existence of several publications on evaluation and follow-up of Brazilian public policies, such as Ceneviva and Farah (2012), Pinto and Gerhardt (2012) and Santos and Raupp (2015), there are few texts that deal with the follow-up of performance audits, which is an important modality of public policies evaluation.

The article is organised from this introduction, followed by a brief overview of performance auditing, which conceptualises the activity. Performance auditing in Brazil and Santa Catarina is commented in the third section; and the follow-up process of performance audits and its classification are in the fourth. Classification changes carried out by the TCE/SC are approached in the fifth section; and it is discussed the focus group technique and the new proposal of classification in the sixth section. Finally, the conclusion is presented.

2. BRIEF OVERVIEW ON PERFORMANCE AUDITING

The performance audits as they are understood today began in Europe, Australia and North America in the 1970s and 1980s; and they have created roots in many democracies and audit institutions when observing the evaluation of the spending efficiency and the promotion of good results (Lonsdale, 2011). Shand and Anand (1996) consider that the movements towards evaluation of programmes or effiency audits began at the end of the 1960s and beginning of the 1970s in the United States of America (USA), followed by Australia, in 1979; Noruega, in 1980; and the United Kingdom, in 1983.

According to Pollitt, Girre, Lonsdale and Summa (2008), the motivation for carrying out performance audits came from extensive programmes of reforming the public management, which had the objective of modernising, making more efficient and, in some cases, decresing the general size of the state apparatus.

The development of this type of audit in different continents has generated different nomenclatures. In addition to the term performance audit (performance auditing), especially adopted in Anglo-Saxon countries, it is used the term audit of management (Auditoría de Gestión) mainly in Latin countries (Rocha & Quintiere, 2013). The European Court of Auditors (ECA), for its part, has adopted in portuguese the term auditoria de resultados¹(União Europeia UE, 2015). In Brazil, the term auditoria operacional² has been chosen by federal auditors of external control of the TCU to describe audit work, which specific objective is to improve examining operations (Oliveira, 1996).

Regarding this matter, to this day Supreme Audit Institutions (SAI) of Canada, United States (US), United Kingdom (UK), European Union (EU) and Australia have adopted different nomenclatures and objectives for performance audits. However, respecting the cultural and legal differences of each country, the International Organization of Supreme Audit Institutions (INTOSAI) has contributed to the standardization of definitions and objectives of performance audits by publishing International Standars of Supreme Audit Institutions (ISSAI). In this respect the latest Intosai´s definition of performance auditing is published on ISSAI 300 (International Organization of Supreme Audit Institutions (INTOSAI, 2015, p. 2), which description refers to “an independent, objective and reliable examination of whether government undertakings, systems, operations, programmes, activities or organisations are operating in accordance with the principles of economy, efficiency and effectiveness³, and whether there is room for improvement”.

The ISSAI 300 (INTOSAI, 2015) establishes a series of principles and procedures to be observed for conducting a performance auditing. Among the general principles, it is observed the establishment of the objective, approach, criteria, risk, communication, required skills, judgement and professional scepticism, quality control, materiality and necessary documents.

The ISSAI 300 also highlights the observance of the principles related to the performance auditing process, which involves the planning, the execution of the audit, the report elaboration and the follow-up.

About the follow-up, Yetano (2014) discusses the importance of this process after the efforts to accomplish the audit. The writer cites that, in Germany and the United Kingdom, it is common to exist a follow-up process for verifing whether the recommendations originated in performance audits were implemented; thus, a certain amount of pressure is created in order to be adopted by the audited organisation the proposed reforms.

3. PERFORMANCE AUDITING IN BRAZIL AND SANTA CATARINA

According to Cunha (1997), the first legal instrument that predicted the performance auditing achievement in the Brazilian federal public administration was a decree, Decreto n. 93.874 (1986), which assigned to the internal control system of the executive branch the role of examining the acts of management, with a view to certifying as accurate and regular the accounts and ascertaining the efficiency and effectiveness of the public resources application.

Corroborating this trend, the Federal Constitution of 1988 (CF/1988) has assigned the responsibility for the performance audits achievement to the bodies of control in order to verify if the public resources were applied in an efficient, economic and effective way.

The TCU pioneered, in the 1980s, in the achievement of performance audits in line with the international trend. In the 1990s, it was launched Project for Training in the Evaluation of Public Programs, Projeto de Capacitação em Avaliação de Programas Públicos, in partnership with Fundação Getulio Vargas (Brazil), the Virginia Polytechnic Institute and State University (United States), or simply Virginia Tech, and the National Academy of Public Administration (United States), subsequently new methods of evaluating governmental programmes and projects were created .The consolidated work turned into the Performance Auditing Manual (TCU, 1998), which has served as a reference for several bodies of government in the country (Mendonça, 2005).

Thereafter the first revision of the Performance Auditing Manual (TCU, 1998) approved, it was renamed Operational Audit Manual, Manual de Auditoria de Natureza Operacional (TCU, 2000); and when approved its second revision in 2010, the denomination changed for Performance Audit Manual, Manual de Auditoria Operacional (TCU, 2010).

An important factor in Brazil has contributed to the diffusion of performance audits in state and municipal courts of accounts (TCs), it was the restructuring process promoted by the Programme for the Modernisation of External Control of Brazilian States and Municipalities (PROMOEX), which had support of the Inter-American Development Bank (IADB).

This process of TCs’ modernisation originated in the financial crisis of 1997 and 1998, forcing Brazil to seek financial assistance from the International Monetary Fund (IMF) and to devalue the exchange rate. According to Loureiro and Abrucio (2004), the government had to make a strict fiscal adjustment and proposed to the National Congress a law that would assure the balance of public accounts, thus being the Fiscal Responsibility Law, Lei de Responsabilidade Fiscal (LRF), Complementary Law No 101 (2000).

Hence the Brazilian TCs were incumbents to supervise the LRF´s assumptions, emerging the need for its technological modernisation and internal restructuring, with a view to appreciating its supervisory function (Loureiro, Teixeira, & Moraes, 2009). That way, PROMOEX was born with the objective of institutionally strengthening and modernizing TCs by increasing the effectiveness, efficiency and effectiveness of control actions and mechanisms of social articulation (Brasil, 2006).

Following along, TCU’s Project for the Development of Operational Auditing Techniques, Projeto de Desenvolvimento de Técnicas de Auditoria de Natureza Operacional, started in April 1998 and carried out 47 audits up to 2004 (Albuquerque, 2006); the first ones delt with administrative aspects, and the subsequent ones aimed for evaluating government programmes, which emphasis was on examining the quality of service delivery and its capacity of contributing to the solution of social problems. Afterwards, with the Project for Improving the External Control with Focus on Reducing Social Inequalities, Projeto de Aperfeiçoamento do Controle Externo com Foco na Redução da Desigualdade Social (Cerds), the TCU adopted as a priority the examination of fostering equity and poverty reduction.

In 2002, the TCE/SC started to conduct performance audits in Santa Catarina, promoting the course in Management Audit in the Public Sector, which trained 92 fiscal auditors in external control from 2002 to 2004; consequently, they conducted 20 audits as final assignment. However, from 2004 to 2006, only two performance audits were accomplished by the technical offices of the Santa Catarina State Court of Accounts (TCE/SC).

In this regard, with the objective to increase participation, in 2007, the Santa Catarina State Court of Accounts created an office that has, among others, the responsability to conduct performance audits.

Back to the fiscal year 2007, the TCU commanded, through PROMOEX, a performance auditing training that involved almost all Brazilian TCs. That event was attended by four representatives of the TCE / SC, who replicated the training to other civil servants that were making the same sort of supervision at that time. Shortly after, as a result of the training, the State Courts of Accounts (TCs) carried out a joint performance audit, under the coordination of TCU. The practice of achieving joint audits occurred in the following years, increasing the professionalisation of fiscal auditors involved and the interest of the court of accounts in this type of auditing (Fleischmann, 2015).

In the period from 2004 to 2015, the TCE / SC carried out 48 performance audits, which could be separated into areas of the scope presented in graph 1.

Graph 1
Coverage area of performance audits achieved from 2004 to 2015

Fourteen out of the 48 audits carried out were filed until end 2015, after the achievement of 2 or 3, follow-ups that were duly judged by the TCE / SC plenary.

4. THE FOLLOW-UP PROCESS OF PERFORMANCE AUDITS

ISSAI 300 (INTOSAI, 2015) establishes that auditors should follow up the findings and recommendations of previous audits, whenever appropriate. Follow-up should be conducted suitably in order to provide feedback to the Legislative branch, if possible, with the conclusions and impacts of all relevant corrective actions.

INTOSAI (2015) adds that the follow-up encourages audited entities and other users of the reports to take seriously the procedure, besides providing auditors with useful lessons and performance indicators.

Follow-up audit procedures should take the form of “limited reviews” (UE, 2015), that is, be limited to the initial audit findings, and assess to what extent the audited entity has responded to the findings of initial audit and recommendations of the supervisory body.

Ribeiro, Lopes, Pederneiras, Silva and Mulatinho (2010) describe that, at the end of the audit, the manager should prepare an action plan that will serve as a basis for the follow-up achievement. They also add that two or three follow-ups should be carried out for each audit, and the process will be closed in approximately 24 months after the publication of the TC decision, when the final audit impact report will be prepared.

Regarding the recommended periodicity for the follow-up achievement, the TCU argues that it should not take too long for this to happen, so that it will not lose the expectation for control, and that doesn’t worth following up the activity for an extended time, because, according to the experience of the US Government Accountability Office (US GAO), most determinations are implemented up to three years after the audit achievement (TCU, 2010). On the other hand, the European Court of Auditors (ECA) (EU, 2015) establishes that it is normal a follow-up taking place 2 or 3 years after the publication of the audit report.

The ISSAI 3000 (INTOSAI, 2004) prescribes documenting the follow-up report and presenting the timely information in an objective and independent manner. The TCU (2010) establishes that it must be an autonomous document, having sufficient information to understand the subject without needing to read the audit report or previous follow-up.

However, INTOSAI (2004) makes a caveat at this point: follow-up should not be restricted to implement the recommendations, and should direct its efforts to verify whether the responsible person has appropriately dealt with the problems, and whether the indicated deficiencies have been corrected.

After the procedures for collecting evidence, according to the Performance Auditing Manual, Manual de auditoria operacional (TCU, 2010), the team should be positioned in to the degree of implementating the recommendations, using the classification presented in Box 1.

Box 1
Form of classification of recommendations according to the degree of implementation

In this systematics, especially describing the classification “partially fulfilled”, it is observed prevalence of the information provided by the manager on the evidences identified in the follow-up. Thus, the audit risk increases, as observed by Santos and Raupp (2015), since the information manually generated by the main areas responsible for the provision of goods or services; due to their fear of being followed, ends up being manipulated in order to be more efficient and effective.

Finally, although the Performance Audit Manual (TCU, 2010) does not describe the difference between determination and recommendation⁴, it makes a distinction for the follow-up classification, replacing the terms, when it comes to determination, “implemented” and “in implementation” with “fulfilled” and “in fulfilment of “.

5. THE CHANGES OCCURRED IN THE CLASSIFICATION OF FOLLOW-UPS CONDUCTED BY TCE/SC

For each performance audit, the TCE/SC carries out at least two follow-ups to verify fulfilment of its determinations. There were cases of processes that underwent the third follow-up, by means of a requisition of the audit team that performed the second follow-up or the process rapporteur. Nevertheless, in both cases, the request occurs by virtue of low compliance of the decision and the importance of audit findings.

For the exercise of this supervision activity, supervision, the operational audit manuals of TCU have always been the basic document of assessments of the Court of Accounts of Santa Catarina. The first follow-ups were classified using its premises in respect.

However, over time, some items were questioned by external auditors and more attentive rapporteurs, such as DAE Instruction Report 9/2012 (Estado de Santa Catarina, 2012a), which deals with the second follow-up of performance audit conducted in the Sewage Treatment System (Efluent Treatment Plant - ETE) of Florianópolis. At the end of the last follow-up, it was assessed the determination 6.2.1.2－ on treating sewage in accordance with environmental standards and the ETE manual－, which was classified as “in fulfilment of”.

That conclusion was in line with the presuppositions established by the TCU, since there was an improvement in the treatment of effluent, considering that, in the audit (in 2004), there were seven parameters of non-standard laboratory analysis established in the environmental standard (sedimentable solids, pH, oils and greases, total phosphorus, total nitrogen, total coliforms and fecal coliforms); while in the second follow-up, only three.

On the other hand, because that was the last follow-up, it was questioned an audit team’s positon on compliance with the determination of the plenary session of the TCE/SC, taking into account that non-compliance with environmental legislation, which served as the basis for the determination, could impose a fine. In that case, the expression “in fulfilment of” was not used in the last follow-up, only in the first one, it can be inferred from which that entailed the observed difference in the fulfillment of the analysis parameters.

Even in the recommendations, the classification “in implementation” was no longer employed in the last follow-up. As an example, it is cited the first follow-up of the same performance audit that is include in the Report No. 020/2010 (Estado de Santa Catarina, 2010a). In that case, fissures and leaks were identified in the aeration tank structure. Considering the gravity and urgency of the situation, the TCE/SC made new recommendations for the solution of the problem. In the second follow-up occurred in 2012, the responsible body for the activity made an evaluation of the sand crack and hired a waterproofing company for its treatment, which solved the leaks. It also launched a bidding procedure for hiring a probing company and pathology of the reinforced concrete structure, without success, because none of the interested party attended the competition.

In the sequence, the Catarinence Water and Sanitaton Company, Companhia Catarinense de Águas e Saneamento (Casan), signed with Federal University of Santa Catarina (UFSC) a scientific and technical cooperation agreement on evaluating states of degradation and diagnosis of pathological manifestations of concrete structure of the insular ETE aeration unit, and, till the end of the inspection procedure, the last service had not been concluded. In this regard the last follow-up of the performance audit classified the item as “in implementation”. However, as already noted, it was not appropriate to use that classification in the last follow-up, and it should have been classified as “partially implemented”, since the manager encouraged improvements, but he has not fully complied with recommendation yet.

The changes in classification are not limited to the last follow-up, as it is observed right after. In the performance audit follow-up that evaluated the environment supervisory activity under the responsability of the Environment Foundation, of Santa Catarina, Fundação do Meio Ambiente (Fatma), held in partneship with Environmental Military Police Battalion of Santa Catarina, Batalhão da Polícia Militar Ambiental (BPMA), in the first follow-up (Estado de Santa Catarina, 2010b), the determination 6.2.1.2 — at least, for being available one camera for the team on duty — has been considered “partially fulfilled”. The classification took into account the information of the manager who had purchased equipments, and the evidence of a number of teams on duty without camera was the same as the one in the audit.

In the same process, the determination 6.2.1.15 － to be included in the administrative process of environmental supervision a copy of the official bill for collecting the fine － was similarly classified in the first monitoring, “partially fulfilled”. However, though the manager informed that he had implemented the determinations, in the audit it was found that in 32.9% of the analysed processes did not contain the bill for the collection of fines, whereas in the first follow-up it was only 14%. Thus, it is observed that determination 6.2.1.2 should have been classified as “not fulfilled” as there was no improvement in the performance in the period; and the determination 6.2.1.15 should have been classified as “in fulfilment of”, considering there was an improvement in the indicator analyzed, although it did not fully comply with the legislation.

The third item of this process has ended up generating another change of understanding that occurred in the classification of the items 6.2.2.1, 6.2.2.2, 6.2.2.8, and 6.2.2.14 as they were considered “implemented” in the first follow-up and “not implemented” in the second (Estado de Santa Catarina, 2013a). This phenomenon arose because of the items that were considered as continuous or periodic fulfilment, such as the item 6.2.2.2 — elaborating campaign and material to disseminate the existing and expanded channels for charges.

In this case, the implementation is not static and arises only once, but cyclical, it should occur year after year. Thus, it was considered that there was compliance with first, and the second, not, since the manager did not repeat the action. However, after the peers’ provocation, it has started considering periodic compliance, when attending the first follow-up, which was considered “in implementation”; this highlighted the need to maintain the good practice in the coming years. In the last follow-up, if the manager repeated the action, it woul be classified as “implemented”, otherwise as “partially implemented”, in consideration of the fulfilment in the first follow-up.

Another stir arised with reference to DAE Instruction Report 21/2013 (Estado de Santa Catarina, 2013b) － second monitoring of the performance audit, auditoria operacional, that evaluated the public school transport service provided by the municipality of Bom Jardim da Serra/SC－, the 6.2.1.6 determination was classified as partially fulfilled because of a supply of seatbelts with conditions of use for the municipality own vehicles could conduct the school transport.

In the first follow-up (Estado de Santa Catarina, 2012c), it was observed that five out of a six own vehicles rendering the schoolchildren transport did not have a seatbelt in conditions of use in all benches. Subsequently, in the second follow-up, only one vehicle was still in an irregular situation. Considering the significant improvement in this item, the team classified as partially fulfiled.

However, there was a query again about the suitability of the classification, since it would not be up to the court to attest partial compliance with legal provisions, considering the legislation should be completely fulfilled.

Thereafter a more rigid posture has been adopted in reference to the determinations; classifying them as “in fulfilment of” in the first follow-up only when there is a significant improvement in the performance; and in the last follow up, it is required the full accomplishment to classify the determination as “Fulfilled”, otherwise it is classified as “not fulfilled”.

On the other had this more rigid posture was directly reflected in the fulfilment indicator of determinations of each follow-up. Even so, it serves a greater good, because some determinations that aim to regularise an administrative procedure, for instance, the control of the workday frequency of civil servants daily manner and on electronic media, which apparently may be a more bureaucratic artifice, end up concealing the practice of the unfulfilled working day.

As an example, since it has not yet been closed and is not part of the sample, the performance audit that evaluated the services of the Joana de Gusmão Children´s Hospital (Estado de Santa Catarina, 2012b) is here mentioned. This supervision has combined the frequency register of the surgeons with records of productivity in the surgical center and the fulfillment of the on duty. The audit procedure has identified professionals who received for 216 hours worked: 80 hours of services contract, 80 hours on duty, and 56 hours on call. One professional did not register his/her frequency in the control system and had productivity of 12 hours in the ambulatory and 6 hours and 20 minutes in the surgical center; therefore, he/she did not do justice to the remuneration paid of more than R$15,000.00 (fifteen thousand reais) .

The first follow-up of this performance audit was the subject of the DAE Instruction Report 18/2014 (State of Santa Catarina, 2014), which adopted this new, more rigid position. Even though the controller had reported that all determinations and recommendations were resolved, it was concluded, after the follow-up procedure, that 9% of the determinations were “in fulfilment of” and 91% “not fulfilled”. This low compliance motivated the audit team to suggest the application of fines to those who were responsible, in particular, for the seriousness of the facts and the possibility of generating damage to public funds. However, the suggestion was not accepted by the rapporteur of the case, who considered hasty in applying a fine in the first follow-up of a performance audit.

In fact, the proposed and consolidated changes in the classification of performance audits conducted by the TCE/SC are clearer and more objective than the considerations in the Performance Audit Manual, Manual de Auditoria de Natureza Operacional (TCU, 2010). Moreover, they assign greater importance to evidence of improved performance of the activity, to the detriment of information provided by the public manager, which does not always represent the situation found in the follow-up.

In summary, considering the practice of performance auditing follow-ups carried out by TCE/SC, it was elaborated the classification of the assessment of determinations and recommendations given by the public body and presented on Box 2.

Box 2
Classification of the fulfilment evaluation of the determinations and the implementation of recommendations observed in the tce / sc

Considering the classification of Box 2 was implemented gradually, there was a need to reclassify some items, especially the older follow-us. Thus, it was evaluated that the 14 performance audits filed by the Santa Catarina Court of Accounts(TCE/SC) that involved 101 determinations, and five were considered damaged at the end of the process. There were also 204 recommendations made, and six of them were considered damaged. Thus, the new classification of the valid 96 determinations and 198 valid recommendations have presented the results illustrated in graphs 2 and 3 regarding their fulfilment / implementation.

Graph 2
Fulfilment of determinations after the last follow-up

Graph 3
Implementation of audit recommendations after the last follow-up

Taking into account what was evaluated, rounding up the values and joining the determinations and recommendations, it is observed that the ceased performance audit processes present approximately 60% of fulfilled / implemented, 15% of partially fulfilled / implemented and 25% of not fulfilled / implemented. Probably, by adopting a more rigid stance on complying with determinations, this scenario should change in the next audits to be closed, increasing the non-compliance rate, while decreasing the number of determinations classified as fulfilled.

It is noteworthy, as a good practice, that almost all follow-up reports have Table synthesis and graphical representation in the determination fulfilment and in the recommendations implementation. In addition, it is emphasized that, the most recent reports, there was a standardisation of the graphic representation, using the three colours universally used in the traffic lights, being the green indicative of implemented / implemented; yellow of partially fulfilled / implemented; and red of not fulfilled / implemented.

It is verified that the adoption of universal colors for graphic representation facilitates the reader’s interpretation, in order to avoid misunderstandings on the part of the target audience.

6. PRESENTATION TO THE FISCAL AUDITORS AND THE NEW PROPOSAL

On July 21, 2016, ten external control fiscal auditors who work directly in the instruction of performance audit processes, and follow-ups, met at the TCE/SC headquarters to para participate in a presentation for discussing in a presentation for discussing the study result.

The group did not reach a consensus, especially because of the variety of situations and combination of results. In addition, a question was raised regarding the classification of the last monitoring, whether it should be positioned in relation to the entire follow-up period, or only the time lapse between the previous and the current ones.

In this regard it was observed that the second follow-up classified the whole period refraining from expressing a opinion on the base period. If it was classified the lapse between the previous and current follow-up, some items might perhaps be considered “fulfilled” or “not fulfilled”, or it might even be “partially fulfilled”. But, on the other hand, the way in which the follow-ups were drawn up (the period between the audit and the last follow-up), the need of a greater evaluation prevails that addresses the impact of supervision, not necessarily the period assessed.

After the first presentation, a new proposal was developed covering most of the circumstances and possible combinations, followed by an individual explanation for division heads and auditing coordinators. Three rounds of contributions later, the validated version by the participants has switched follow-ups denominations from ordinal numbers to intermediate and final; and the last one would have the classification of the period to which it refers (from the previous follow-up to the current one) and additional classification related to the audit impact by adding a column, which would cover the entire follow-up period as systematized in Box 3.

Box 3
New classification proposal of evaluating the fulfilment of determinations

A new classification proposal was applied in the same sample in order to evaluate the final follow-up outcome and the audit impact, presenting the results in graphs 4 e 5.

Graph 4
Fulfilment of determinations related to the final follow-up

Graph 5
Fulfilment of determinations related to the audit impact

In relation to graphs 5 e 6, it has been attested that the classification has provided another parameter for “in fulfilment of ” in the final follow-up, and the same items has begun to be reclassified as “partially fulfilled” in relation to the audit impact.

IMoreover, it is observed that there was a reduction of 9.7% in relation to the determinations considered not fulfilled, particularly the possibility of classifying stagnation or a slight worsening in performance as “in fulfilment of “ in the final monitoring.

There was also a reduction of 4.1% in relation to the determinations considered fulfilled, being reclassified as “in fulfilment of” or “partially fulfilled”. This characteristic stems from the fact that the recommendation was originally considered “fulfilled”, even if it was not fully met.

As for the recommendations, the new classification corresponds to the representation of graphs 6 and 7.

Graph 6
Implementation of audit recommendations after the final follow-up

In the same way as the evaluation of determinations, there was a reduction in the recommendations that were considered “implemented” (4,4% and 4%), and the ones “not implemented” (2,8% and 3%); with a respective increment in the recommendations that were considered “in implementation” or “partially implemented”.

Although the new classification is more accurate in relation to situations that may be encountered, the auditor´s experience is still necessary to deal with the subjectivity in some situations. For example, on the recommendation for recomposing the teams of the Family Health Strategy (ESF) and substituting its employees by reason of suspensions or leaves for prolonged periods. In the auditing report it is cited that 95,83% of the teams were incomplete; they were followed by 83,3% incomplete at intermediate follow-up and 76,7%, in the final one. Thus, it has been concluded that the recommendation was not fulfilled in both follow-ups. But considering the manager efforts, because of the opening of civil service entrance examination and and selection processes for filling vacancies, besides the improvement indicator — despite the high index of teams with an absence of members — , it has otherwise become indispensable, before the new proposal to classify the recommendation as “in fulfilment of “, in the final follow-up, and “partially fulfilled” as to the audit impact.

Graph 7
Implementation of audit recommendations after the final follow-up

It also highlights the stagnation cases, for example, the school transport in which none of the eight buses in a certain municipality had authorization from the transit authority to transport schoolchildren at the timing of audit. Subsequently seven vehicles had the aforementioned authorization as observed at intermediate follow-up, that situation persisted in the final follow-up. Thus, according to the new classification proposal, the status of “in fulfilment” was maintained in the final monitoring, and “partially fulfilled”, in the audit impact. It was taken into account significant improvement of compliance with what was recommended at first follow-up, as well as stagnation in the last evaluation. Moreover, it is considered important to describe if the body has insisted on taking a kind of action which has a prolonged effect or, for example, besides promoting training, it has instituted a lifelong learning policy.

The classification division into final follow-up and audit impact has also made possible identifying situations that were not perceived. As an example, in the first evaluation follow-up of urban solid waste management system in the city of Itajaí, it is cited the classification as fulfilled for the requirement of repairing the isolation (the fence) in order to avoid animals entering into the landfill. Although it had not been the object of assessment in the final follow-up, it was considered as fulfilled the recommendation. Based on the new methodology, the final follow-up was classified as faulty, since it had not been verified; but the audit impact remained as fulfilled.

In the new classification proposal, it is perceived the intention of recognizing, in a simple way, the public manager efforts to solve proposals of the Court of Auditors (TC), considering the evaluation of the situation pinpointed in the auditing or assessed in the final follow-up. It is worth highlighting the need to redo the audit procedures in the follow-up, in order to identify if there was any improvement in the activity, not only basing on information provided by the person subject to the jurisdiction of the Court of Auditors.

The recommendations would follow the same reasoning, requiring only the performance improvement, that is, without the necessity of “significant” improvement for considering them “in implementation”. It would be also maintained the replacement of the term fulfilled and its variations by implemented and its variations.

7. CONCLUSION

More than a decade elapsed from the beginning of the realization of performance audits in the TCE/SC, several relevant programmes and activities were evaluated. However, the effort to assess and follow up its decisions did not have the same stimulus to describe its audit procedures and convert them into positive norms, this is demonstrated by situations occurred in the classification of the evaluation of compliance with the determinations and implementation of the court’s (TCE/SC) recommendations in these performance audits.

According to exposed facts, it is observed that the simple reading and interpretation of the classification form that is suggested in the Manual de auditoria operacional (TCU, 2010) do not cover in particular all situations faced by external control; considering the public manager often informs that a problem has been solved , the deficiency still persists.

But, on the other hand, from the analysis of the evolution that occurred in the classification that is currently adopted by Santa Catarina State Court of Accounts (TCE/SC), it is also highlight its improprieties because certain itens are deemed to be fulfilled, although they did not comply the legislation in its fullness, which raises greatly the audit risk of issuing an inadequate conclusion.

An advantage of the new classification proposal is identified with the necessity of describing in the report evidences taken into consideration in order to estimate the implementation degree of the recommendation such as mild or significant improvement/ worsening of performance, or a stagnant situation. Hence, it is necessary to create indicators that ends up mitigating an identified limitation.

In this regard it is also suggested, in the face of imposed challenges, the inclusion of guidalines on the description in the follow-up report of solutions that were adopted by the manager and had aggravating or mitigating effects on the classification of the results of performance audits.

Therefore, it can be affirmed that the objective of this research was achieved since, besides recording the changes occurred in the classification of the result of performance audits that were conducted by the TCE/SC, a new classification was also proposed taking into account the degree of implementation of the recommendations and of compliance with determinations, which provides better organization and definition of all various situations. Thus, an assessment of the audit impact becomes evident while it is provided an effective contribution to the other courts (TCs) that will encourage the discussion.

Future studies may apply this new classification that was proposed in the follow-ups of performance audits conducted by other courts of accounts, particularly on the part of the TCU, in order to improve the methodology and discuss sensitive points, such as considering that there is “in compliance with” in the last follow-up when there is stagnation or slight worsening in the performance indicator.

REFERÊNCIAS

• Albuquerque, F. F. T. (2006). A auditoria operacional e seus desafios: um estudo a partir da experiência do Tribunal de Contas da União(Dissertação de Mestrado). Universidade Federal da Bahia, Salvador, BA.
• Antero, S. (2008). Monitoramento e avaliação do Programa de Erradicação do Trabalho Escravo. Revista de Administração Pública, 42(5), 791-828.
• Brasil. (2006). Programa de Modernização do Sistema de Controle Externo dos Estados, Distrito Federal e Municípios Brasileiros - PROMOEX Brasília, DF: Ministério do Planejamento, Orçamento e Gestão.
• Ceneviva, R., & Farah, M. F. S. (2012). Avaliação, informação e responsabilização no setor público.Revista de Administração Pública, 46(4), 993-1016.
• Cunha, M. C. M. S. (1997). Auditoria operacional no setor público federal. Revista de Contabilidade do Mestrado em Ciências Contábeis da UERJ, 2(2), 47-52.
• Decreto Decreto n. 93.874, de 23 de dezembro de 198 6. (1986). Dispõe sobre os sistemas de administração financeira, contabilidade e auditoria, e de programação financeira, organiza o sistema de controle interno do poder executivo, e dá outras providências. Brasília, DF.
• Estado de Santa Catarina. (2013). Resolução TC n. 79/2013: dispõe sobre a fiscalização por meio de auditoria operacional realizada pelo Tribunal de Contas do Estado de Santa Catarina Florianópolis, SC: Tribunal de Contas do Estado de Santa Catarina. Recuperado de http://www.tce.sc.gov.br/sites/default/files/leis_normas/resolucao_n_79-2013_consolidada.pdf
  » http://www.tce.sc.gov.br/sites/default/files/leis_normas/resolucao_n_79-2013_consolidada.pdf
• Estado de Santa Catarina. (2010a). Relatório n. 20/2010: primeiro monitoramento da auditoria operacional realizada no Sistema de Tratamento de Esgoto Insular de Florianópolis Florianópolis, SC: Tribunal de Contas do Estado de Santa Catarina. Recuperado de http://www.tce.sc.gov.br/sites/default/files/1o_relatorio_monitoramento_ete_insular_3.pdf
  » http://www.tce.sc.gov.br/sites/default/files/1o_relatorio_monitoramento_ete_insular_3.pdf
• Estado de Santa Catarina. (2010b). Relatório n. XXX/2010: primeiro monitoramento da auditoria operacional na atividade de fiscalização ambiental de responsabilidade da Fundação do Meio Ambiente (Fatma) Florianópolis, SC: Tribunal de Contas do Estado de Santa Catarina. Recuperado dehttp://www.tce.sc.gov.br/sites/default/files/relatorio_Monitoramento_FATMA_BPMA_fiscalizacao_0.pdf
  » http://www.tce.sc.gov.br/sites/default/files/relatorio_Monitoramento_FATMA_BPMA_fiscalizacao_0.pdf
• Estado de Santa Catarina. (2012a). Relatório de instrução DAE 9/2012: segundo monitoramento da auditoria operacional realizada no Sistema de Tratamento de Esgoto Insular de Florianópolis Florianópolis, SC: Tribunal de Contas do Estado de Santa Catarina . Recuperado de http://www.tce.sc.gov.br/sites/default/files/relatorio%20de%20impacto%20ete%20Insular.pdf
• Estado de Santa Catarina. (2012b). Relatório de instrução DAE 10/2012: avaliação dos serviços prestados pelo Hospital Infantil Joana de Gusmão Florianópolis, SC: Tribunal de Contas do Estado de Santa Catarina . Recuperado dehttp://www.tce.sc.gov.br/sites/default/files/Relatorio_Auditoria_Operacional_HIJG_0.pdf
  » http://www.tce.sc.gov.br/sites/default/files/Relatorio_Auditoria_Operacional_HIJG_0.pdf
• Estado de Santa Catarina. (2012c). Relatório de instrução DAE 22/2012: segundo monitoramento da auditoria operacional no serviço de transporte escolar público prestado pelo município de Bom Jardim da Serra Florianópolis, SC: Tribunal de Contas do Estado de Santa Catarina. Recuperado dehttp://www.tce.sc.gov.br/sites/default/files/Relatorio_monitoramento_Transporte_Escolar_Bom_Jardim_da_Serra_0.pdf
  » http://www.tce.sc.gov.br/sites/default/files/Relatorio_monitoramento_Transporte_Escolar_Bom_Jardim_da_Serra_0.pdf
• Estado de Santa Catarina. (2013a). Relatório de instrução DAE 17/2013: segundo monitoramento da auditoria operacional na atividade de fiscalização ambiental de responsabilidade da Fundação do Meio Ambiente (Fatma) Florianópolis, SC: Tribunal de Contas do Estado de Santa Catarina . Recuperado dehttp://www.tce.sc.gov.br/sites/default/files/Segundo%20e%20%C3%BAltimo%20monitoramento%20fiscaliza%C3%A7%C3%A3o%20ambiental.pdf
  » http://www.tce.sc.gov.br/sites/default/files/Segundo%20e%20%C3%BAltimo%20monitoramento%20fiscaliza%C3%A7%C3%A3o%20ambiental.pdf
• Estado de Santa Catarina (2013b). Relatório de instrução DAE 21/2013: segundo monitoramento da auditoria operacional no serviço de transporte escolar público prestado pelo município de Bom Jardim da Serra Florianópolis, SC: Tribunal de Contas do Estado de Santa Catarina. Recuperado dehttp://www.tce.sc.gov.br/sites/default/files/Relatorio_segundo_monitoramento_Bom_Jardim_da_Serra_0.pdf
  » http://www.tce.sc.gov.br/sites/default/files/Relatorio_segundo_monitoramento_Bom_Jardim_da_Serra_0.pdf
• Estado de Santa Catarina. (2014). Relatório de instrução DAE 18/2014: primeiro monitoramento da auditoria operacional para avaliação dos serviços prestados pelo Hospital Infantil Joana de Gusmão Florianópolis, SC: Tribunal de Contas do Estado de Santa Catarina . Recuperado de http://www.tce.sc.gov.br/sites/default/files/Relat%C3%B3rio%20Primeiro%20Monitoramento%20Hospital%20Infantil_HIJG_Dezembro.pdf
  » http://www.tce.sc.gov.br/sites/default/files/Relat%C3%B3rio%20Primeiro%20Monitoramento%20Hospital%20Infantil_HIJG_Dezembro.pdf
• Fleischmann, R. S. (2015). Auditoria operacional: avaliação da eficiência de centros cirúrgicos de quatro hospitais públicos catarinenses. In Anais do 1oCongresso Internacional de Controle e Políticas Públicas. Belo Horizonte, MG.
• International Organization of Supreme Audit Institutions. (2004). ISSAI 3000: las Normas Internacionales de las Entidades Fiscalizadoras Superiores - normas y directrices para la auditoria del rendimiento basadas en las Normas de Auditoría y la experiencia práctica de la INTOSAI Estocolmo, Suecia: Comité de Normas de Auditoría.
• International Organization of Supreme Audit Institutions. (2015). ISSAI 300: princípios fundamentais de auditoria operacional. In Tribunal de Contas da União. Normas Internacionais das Entidades Fiscalizadoras Superiores (ISSAI): princípios fundamentais de auditoria (nível 3) Brasília, DF: Autor. Recuperado de https://portal.tcu.gov.br/lumis/portal/file/fileDownload.jsp?fileId=8A8182A15D3169 CE015D56CC4DE61545
  » https://portal.tcu.gov.br/lumis/portal/file/fileDownload.jsp?fileId=8A8182A15D3169 CE015D56CC4DE61545
• Lei Lei Complementar n. 101, de 4 de maio de 2000. (2000). Estabelece normas de finanças públicas voltadas para a responsabilidade na gestão fiscal e dá outras providências. Brasília, DF.
• Lonsdale, J. (2011). Introduction. In J. Lonsdale, P Wilkins, & T Ling. (Eds.). Performance auditing: contributing to accountability in democratic government (pp. 1-21). Cheltenham, England: Edward Elgar.
• Loureiro, M. R., & Abrucio, F. L. (2004). Política e reformas fiscais no Brasil recente.Revista de Economia Política, 24(1), 50-72.
• Loureiro, M. R., Teixeira, M. A. C., & Moraes, T. C. (2009). Democratização e reforma do Estado: o desenvolvimento institucional dos tribunais de contas no Brasil recente.Revista de Administração Pública, 43(4), 739-772.
• Mendonça, B. C. C. (2005). Auditoria operacional no âmbito do Poder Judiciário do Estado do Rio de Janeiro (Trabalho de Conclusão de Curso). Fundação Getulio Vargas, Rio de Janeiro, RJ.
• Oliveira, A. M. (1996). Controle e auditoria governamental com enfoque em auditoria operacional Brasília, DF: ESAF.
• Pinto, J. M., & Gerhardt, T. E. (2013). Práticas avaliativas na gestão da atenção básica à saúde: estudo de caso em Camaquã e Canguçu (RS).Revista de Administração Pública, 47(2), 305-326.
• Pollitt, C., Girre, X., Lonsdale, J., & Summa, H. (2008). Desempenho ou legalidade? Auditoria operacional e de gestão pública em cinco países (tradução de Pedro Buck). Belo Horizonte, MG: Fórum.
• Ribeiro, J. F., Filho Lopes, J E G., Pederneiras, M. M. M., Silva, A. J. M., & Mulatinho, C. E. S. (2010). Recomendações em auditoria operacional: uma prospecção de fragilidades, com base na inteligência competitiva.Revista de Administração da UFSM, 3(2), 191-204.
• Rocha, A. C., & Quintiere, M. M. R. (2013). Auditoria governamental. Uma abordagem metodológica da auditoria de gestão Curitiba, PR: Juruá.
• Santos, G. K., & Raupp, F. M. (2015). Monitoramento e avaliação de resultados dos programas governamentais delineados no PPA. Revista de Administração Pública, 49(6), 1429-1451.
• Shand, D; & Anand, P. (1996). Performance auditing in the public sector: approaches and issues in OECD member countries. In Organisation for Economic Co-Operation and Development. Performance auditing and the modernisation of government (pp. 57-78). Paris, France: Puma.
• Tribunal de Contas da União. (1998). Manual de auditoria de desempenho Brasília, DF: Secretaria de Auditoria e Inspeções.
• Tribunal de Contas da União. (2000). Manual de auditoria de natureza operacional Brasília, DF: Coordenadoria de Fiscalização e Controle.
• Tribunal de Contas da União. (2010). Manual de auditoria operacional Brasília, DF: Secretaria de Fiscalização e Avaliação de Programas de Governo. Recuperado de https://portal.tcu.gov.br/lumis/portal/file/fileDownload.jsp?fileId=8A8182A14D8D5AA6014D8D8277322DA6
  » https://portal.tcu.gov.br/lumis/portal/file/fileDownload.jsp?fileId=8A8182A14D8D5AA6014D8D8277322DA6
• União Europeia. (2015). Manual da auditoria de resultados Luxemburgo: Tribunal de Contas Europeu. Recuperado dehttp://www.eca.europa.eu/Lists/ECADocuments/PERF_AUDIT_MANUAL/PERF_AUDIT_MANUAL_PT.PDF
  » http://www.eca.europa.eu/Lists/ECADocuments/PERF_AUDIT_MANUAL/PERF_AUDIT_MANUAL_PT.PDF
• Yetano, A. (2014). La difusión de la auditoría operativa en el ámbito regional: un estudio comparado del Reino Unido, Alemania y España.Gestión y Política Pública, 23(2), 421-464.

• 1
  Translation note: A literal translation for auditoria de resultados would be “audit of results”.
• 2
  TN: A translation for auditoria operacional would be “operational auditing”.
• 3
  The TCU has translated “effectiveness” from English into Portuguese as “effectivity”, efetividade. However, the organ recognises in a translation note that the “term in English encompasses two different concepts in the Portuguese language: efectivity (concept related to results) and efficacy (concept related to the extent of achieving the programmed goals)” (INTOSAI, 2015, p. 2). Although INTOSAI uses the term "efficacy" in the Spanishlanguage publication the term “efficacy” in the Spanish language publication, the definition in both languages encompasses two meanings (goals and results).
• 4
  ResolutionTC n. 79/2013: it provides for supervision by means of performance audit conducted by Santa Catarina State Court of Accounts. “Article Five. The Court’s decision in the procedures relating to the performance audit may contain: I — determinations of acts and procedures correction, when it is verified infraction of legal norm, regulatory or contract; II — recommendations towards impromevent of the audit object, as well as towards optimization of the application of the public resources.”

• [Translated version] Note: All quotes in English translated by this article’s translator

Publication Dates

• Publication in this collection
  Jan-Feb 2019

History

• Received
  26 Sept 2016
• Accepted
  18 Oct 2018