What is the quality of the regulatory impact analysis prepared by Brazilian regulatory agencies?

Abstract

The regulatory impact analysis - RIA is a tool that seeks to identify the regulatory policy’s relevant impacts to support decision-making. Its use has become mandatory in Brazil, but no studies have investigated whether the quality of the analysis has reached the desired level in the country. Thus, the objective of this research was to evaluate the quality of RIAs carried out by Brazilian federal regulatory agencies. An instrument was developed and applied with 50 criteria for quality assessment in a set of 21 RIAs. The results indicated the good quality of the RIAs in some aspects, such as the definition of problems and objectives, but they pointed out weaknesses in others, such as the analysis of impacts and the comparison of alternatives. As an agenda for future studies, investigations of hypotheses and factors that may explain the absence of a quality standard for RIA in the country were proposed.

Keywords:
regulatory impact analysis; regulatory impact assessment; RIA; regulation; quality

Resumo

A análise de impacto regulatório (AIR) é uma ferramenta que busca identificar os impactos relevantes que uma política regulatória pode ter, com vistas a subsidiar a tomada de decisão. O seu uso tornou-se obrigatório no Brasil, mas não há estudos que tenham investigado se a qualidade da análise tem alcançado o nível desejado no país. Assim, o objetivo da presente pesquisa foi avaliar a qualidade das AIRs realizadas por agências reguladoras federais brasileiras. Para a condução deste trabalho, foi desenvolvido e aplicado um instrumento com 50 critérios para a verificação da qualidade de um conjunto de 21 AIRs. Os resultados indicaram boa qualidade das AIRs em alguns aspectos, como na definição de problemas e objetivos, mas apontaram fragilidades em outros, como na análise de impactos e na comparação de alternativas. Como agenda para pesquisas futuras, foi proposta investigação de hipóteses e fatores que podem explicar a ausência de um padrão de qualidade da AIR no país.

Palavras-chave:
análise de impacto regulatório; AIR; regulação; qualidade

Resumen

El análisis de impacto regulatorio (AIR) es una herramienta que busca identificar los impactos relevantes que puede tener una política regulatoria, con miras a apoyar la toma de decisiones. Su uso se ha vuelto obligatorio en Brasil, pero no hay estudios que hayan investigado si la calidad del análisis ha alcanzado el nivel deseado en el país. Así, el objetivo de la presente investigación fue evaluar la calidad de los AIR realizados por las agencias reguladoras federales brasileñas. Para realizar la investigación se elaboró y aplicó un instrumento con 50 criterios de evaluación de la calidad en un conjunto de 21 AIR. Los resultados indicaron buena calidad de los AIR en algunos aspectos, como en la definición de problemas y objetivos, pero señalaron debilidades en otros, como en el análisis de impactos y en la comparación de alternativas. Como agenda de estudios futuros, se propusieron investigaciones de hipótesis y factores que puedan explicar la ausencia de un estándar de calidad del AIR en el país.

Palabras clave:
análisis de impacto regulatorio; AIR; regulación; calidad

1. INTRODUCTION

Regulatory Impact Analysis (RIA) has no single concept, and its use has varied depending on the country or context of application (Peci, 2011; Radaelli, 2010). In general, it is an analysis that seeks to identify the relevant impacts of a regulatory policy, in order to support decision making and achieve balanced and effective policies (Ellig, McLaughlin, & Morrall, 2013; Jacobs, 2004).

RIA has been widely spread around the world, with growing acceptance (Adelle et al., 2016). The adoption of RIA in regulatory processes started slowly in the 1970s, when it was adopted in the United States, and advanced a lot in the 1990s and 2000s, when it reached most countries of the European Union and member countries of the Organization for Economic Cooperation and Development (OECD) (Francesco, 2012). Recently, it has also progressed in developing countries as a strategy to improve governance (Carvalho, Costa, Marques, & Cordeiro, 2019).

In Brazil, the government showed a first institutional commitment to RIA in 2007, by creating the Program for Strengthening Institutional Capacity for Regulation Management (Proreg). Its goal was to adopt measures for regulatory improvement, which included encouraging the use of RIA in policy making in the country (Peci, 2011). Despite the efforts, RIA had a modest progress until mid-2018. Its application was incomplete, non-systemic, and not understood as a government strategy (Castro, 2014).

In recent years, the Brazilian government has fostered the topic of RIA, and passed two laws¹ and a presidential decree² that made its use mandatory in regulatory policy formulation. Since April 15, 2021, Brazilian regulatory agencies - at the federal level - started to use RIA as a requirement for policy making

Despite the growing use of RIA around the world, there is still a shortage of studies indicating if the quality of the analysis has achieved the intended analytical depth. The available papers are concentrated in the United States and Europe, and make only partial assessments on the diverse aspects of quality (Achtnicht, Rennings, & Hertin, 2009; Belfield, Bowden, & Rodriguez, 2019; Fritsch, Radaelli, Schrefler, & Renda, 2013; Saab & Silva, 2021b). The results of these studies indicate that, in recent decades, RIA quality was low, and to identify ways to improve it involves carrying out new evaluations in different contexts (Belfield et al., 2019; Ellig & McLaughlin, 2012; Hahn, Burnett, Chan, Mader, & Moyle, 2000; Saab & Silva, 2021b; Souto-Otero, 2013).

Based on the available knowledge on the subject, there is a gap that justifies the present study, whose objective was to evaluate the quality of RIAs done by Brazilian federal regulatory agencies. Hence, this paper addressed a reality not yet explored in previous studies - Brazil’s - and used more comprehensive evaluation criteria, which can contribute to a broader debate on RIA’s quality.

2. ASSESSMENT OF RIA QUALITY

The use of knowledge, evidence, and information to support decision makers is increasingly frequent in developed democracies (Staroňová, 2014). When a decision maker seeks knowledge before choosing a policy, he/she aims to know and measure the expected impacts, in order to make better decisions and track its results (Desmarais & Hird, 2014).

RIA can be considered a tool that allows approaching science and knowledge on regulatory policy making. Its use can mean an important strategy to use qualified information and arguments for policy decisions (Costa, 2016).

RIA defines the problem and intervention’s goals, and compares the different alternatives available to the regulator (Jacobs, 2004; Radaelli, 2004, 2010). Throughout the process, it uses methods and evidence to support each step, as well as encourage wide social participation. In the final part of the analysis, it presents the most favorable regulatory path for society (Radaelli, 2010).

RIA can also play a relevant role against regulatory capture. The use of standardized methods to assess costs and benefits, and the availability of results for public scrutiny can help detect situations where data are manipulated in order to promote regulations that meet the interests of certain groups (Livermore & Revesz, 2013; Saab & Silva, 2021a).

Merely carrying out RIA does not ensure good regulatory policies. It is necessary to overcome misgivings about its adoption, develop technical capabilities, and secure greater political commitment to its use (Carroll, 2010). Impact analyses should have appropriate quality in order to provide good inputs to the decision maker (Daskal, Ayalon, & Shechter, 2019; Ellig et al., 2013; Fritsch et al., 2013).

The next subsections present dimensions, criteria, and approaches that have been used to assess RIA quality, and provide measures for improvement.

2.1. Dimensions and evaluation criteria

In general, RIA quality assessment can be done by investigating four dimensions (Figure 1): i) RIA’s formal aspects; ii) the use of social participation; iii) the use of evidence in the process; and iv) its usefulness for decision-making (Saab & Silva, 2021b).

Source: Adapted from Saab and Silva (2021b).

FIGURE 1 DIMENSIONS FOR ASSESSING RIA QUALITY

In the first dimension, we should assess if the formal aspects of RIA were properly conducted and registered. The aim is to investigate criteria that measure the appropriate definition of the problem and of the regulation goals (Staroňová, Pavel, & Krapež, 2007), in addition to the identification of regulatory alternatives (Dudley et al., 2017), as well as their comparison (Souto-Otero, 2013), in order to recommend the best regulatory option to the decision maker (Ellig & McLaughlin, 2012).

In the social participation dimension, we should assess criteria that indicate if RIA’s development process was transparent and participatory (Ellig & McLaughlin, 2012). It is important to investigate if the government authority involved society, encouraged data collection, captured perceptions from different social groups, and answered their needs (Staroňová, 2010; Staroňová et al., 2007).

As for the use of evidence in the process, we should look for criteria that indicate if the arguments and assumptions used in RIA were based on properly proven and verifiable elements, facts, data, and information (Russel & Turnpenny, 2009; Souto-Otero, 2013).

Finally, the dimension of usefulness for decision-making is peculiar. It is not an intrinsic and verifiable element in the RIA report, but we know that there is a bidirectional correlation between the quality of the analysis and its use in decision-making. When RIA has a better quality, it influences the decision-making process more, just as the decision maker’s intention to use it tends to increase its quality (Ellig & McLaughlin, 2012).

2.2. Approaches for assessing RIA quality

The quality of RIA is measured by applying different approaches (or methods) to the pre-established criteria and dimensions. In the last 40 years, we found four approaches in the literature: i) checklist; ii) point scale; iii) retrospective comparison; and iv) case study.

The checklist consists in verifying the criteria and contents that RIA should have, by assigning “yes” or “no” values. The point scale is similar to the checklist, but sophisticates the assessment by assigning points to each criterion. The case study involves a qualitative investigation on RIA, based on capturing and analyzing perceptions and information. Finally, a retrospective comparison assigns quality to RIA, according to the accuracy in estimating costs and benefits (Saab & Silva, 2021b).

2.3. Related studies

Hahn et al. (2000) were pioneers in proposing studies to assess RIA quality. According to the authors, RIA was created as an instrument to facilitate the development of efficient and effective regulatory policies, but the level of information provided by the analyses was insufficient to achieve that goal. Assessing RIA quality would be an essential initiative for its strengthening and, consequently, for improving regulatory policies.

Some studies mark the trajectory of knowledge building on the subject. To assess the quality of 45 RIAs conducted in the United States in 2008, Ellig and McLaughlin (2012) proposed 12 criteria and used the point scale approach. The criteria generally measured RIA’s ability to define the problem, identify and compare alternatives, and estimate the costs and benefits of regulation. In addition, they considered the ease for a citizen to locate and understand the RIA report, and if RIA affected the decision. RIAs’ examination showed that the average quality of the analyses was low, with many weaknesses related to problem definition and use of evidence for decision-making.

Fritsch et al. (2013) used the checklist approach to evaluate 773 RIAs from the United Kingdom and European Union, between 2005 and 2010. They used 18 assessment criteria, focusing on those related to cost and benefit calculation, impacts, quantification, and profitability - “we left aspects such as problem definition, consultation, monitoring, and evaluation for further work arising from this project” (Fritsch et al., 2013, p. 447). The result showed that cost and benefit analysis, quantification, and monetization were problematic issues in RIAs.

More recently, Belfield et al. (2019) assessed the quality of 28 RIAs that supported regulatory policies in the education area, in the United States. The authors used 11 criteria and the checklist approach. Similar to Fritsch et al. (2013), their examination of RIA quality emphasized criteria linked to the determination of costs and benefits. Other aspects of the RIA process, such as problem analysis, goals, and social participation, were not used for assessing quality. The study concluded that, in the few situations where RIA was used in educational policies, methodological standards were low, costs were underestimated, and there was little transparency regarding methods and assumptions. In addition, benefits were rarely estimated, and the adopted method did not ensure high reliability levels.

Some studies have contributed to evaluating RIA quality in different contexts, helping to understand its strengths and weaknesses. Improving assessment methods, approaches, and criteria, as well as conducting new evaluations, can help identify ways to improve the analysis. The next section presents the methodological procedures adopted to assess the quality of RIA in Brazilian regulatory agencies.

3. METHODOLOGICAL PROCEDURES

According to Yin (1994), an empirical investigation is based on a research plan that relates data and objectives to reach conclusions. Figure 2 presents the steps and procedures used in this study, between its inception and conclusion.

Source: Elaborated by the authors.

FIGURE 2 RESEARCH PLAN

In the first stage, we explored the scientific literature on RIA quality, in publications found on Google Scholar, Web of Science, and Scopus databases. We also examined official documents (laws, decrees, and guides) that regulate the topic in Brazil. With these scientific studies and official documents, we designed a preliminary instrument to assess RIA quality in Brazil. The preliminary version of the instrument was sent to a referee - a PhD in Social Sciences with extensive experience in public management, regulation and RIA - who judged its adequacy. The final version was prepared after adjustments resulting from referee’s observations.

The final version of the assessment instrument had 50 quality criteria, and was prepared for applying the point scale approach. The proposed methodologies cover the dimensions of quality investigation suggested by Saab and Silva (2021b), and enable more complete assessments than those available in the literature. We chose the point scale because this approach differs from others, by allowing a qualitative perception of how well each quality criterion was evaluated (Saab & Silva, 2021b). To apply the instrument, we adapted a point scale from Ellig and McLaughlin (2012), as shown in Box 1.

BOX 1
MEANING OF THE POINT SCALE

In the second stage of the study, we conducted a survey of all RIA reports that followed regulatory policies approved between April 15 and December 31, 2021. We requested information from the 11 federal regulatory agencies, through the Brazilian government platform ‘Fala.BR’, used to handle requests for information access (Bourges, 2020). The choice of period aimed to select only RIAs prepared after its mandatory implementation in Brazil, when concepts and practices were defined. All regulatory agencies answered the request for information through the ‘Fala.BR’ platform.

After the RIA survey, we carried out a floating reading of the reports, in order to check if the documents fit the research scope. Next, we organized them, and applied the quality assessment instrument to each report. The two authors of this paper conducted the evaluation. The first analyzed the content of the reports and applied the evaluation instrument; the second reviewed and discussed the results of the analysis.

In the third and last stage, we analyzed and described the results achieved through the assessment of RIA quality.

4. RESULT ANALYSIS AND DISCUSSION

We carried out the analysis and discussion of results by assessing the quality of a set of 21 RIAs prepared by Brazilian federal regulatory agencies. Among the 11 agencies, only Anvisa, Ancine, and ANM did not have their RIAs assessed, because they did not approve regulatory policies, in the period from April 15 to December 31, 2021, following the new concepts and practices for RIAs defined in the Brazilian legislation, as shown in Box 2.

BOX 2
NUMBER OF RIAS EVALUATED, BY REGULATORY AGENCY^(a)

After applying the assessment instrument, we present the results in seven subsections, explained below: (i) problem analysis and definition; (ii) goal setting; (iii) identification of regulatory alternatives; (iv) analysis of alternatives’ impacts; (v) comparing alternatives; (vi) indication of the best alternative; and (vii) transparency and social participation.

i) Problem analysis and definition

Proper problem analysis and definition is the most important stage in designing a regulatory policy (Nweke, 2011). When analyzing a problem, the undesired situation must be deeply understood for the intervention to be effective, justified, and targeted to real and legitimate problems of society (Souto-Otero, 2013).

The results of the evaluation showed that the analysis and definition of the problem were not neglected by Brazilian regulatory agencies but were stronger in some aspects than in others (Table 1).

TABLE 1
SCORE - PROBLEM ANALYSIS AND DEFINITION

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
1. Problem definition	Dudley et al. (2017); Souto-Otero (2013)	2.57	51.4%	1.12
2. Nature (Market failure, regulatory failure, etc.)	Dudley et al. (2017); Ellig and McLaughlin (2012)	0.48	9.5%	1.08
3. Use of evidence	Dudley et al. (2017); Ellig and McLaughlin (2012)	1.14	22.9%	1.59
4. International experience	Decreto nº 10.411, de 30 de junho de 2020	0.57	11.4%	1.25
5. Problem uncertainties	Ellig and McLaughlin (2012)	0.00	0.0%	0.00
6. Cause mapping	Mota et al. (2020); Snowdon, Schultz, and Swinburn (2008)	0.81	16.2%	1.47
7. Identification of root cause	Mota et al. (2020); Snowdon et al. (2008)	0.38	7.6%	1.20
8. Consequence mapping	Mota et al. (2020); Snowdon et al. (2008)	0.95	19.0%	1.53
9. Problem trends	Dudley et al. (2017)	0.43	8.6%	0.98
10. Mapping of affected players	Staroňová et al. (2007)	2.57	51.4%	1.12
11. Distribution of effects on each affected group	Staroňová et al. (2007)	1.10	21.9%	1.73
12. Legal basis	Decreto nº 10.411, de 30 de junho de 2020	4.43	88.6%	0.98

• Source: Elaborated by the authors.

In all RIA reports evaluated, there was a section dedicated to understanding the undesired situation. In general, there were significant efforts to delimit and define the problems to act upon. However, some regulators did not understand the problem correctly, as “the lack of regulation” or “the need for regulation”. In these cases, they confused the problem with a possible solution - the regulation. Some also presented long descriptions of the problem, without clearly delimiting it.

In general, regulators showed no familiarity with the concept ‘problem nature’. Few RIAs showed some level of discussion on market failures or regulatory failures. Best practices recommend identifying these failures, so that agencies understand the problem, the need to act upon it, and promote effective regulation in economic and welfare terms (Dudley et al., 2017; Ellig & McLaughlin, 2012).

Another issue that requires improvement by Brazilian agencies is the use of evidence on the identified problem. The research indicated that the use of evidence was incipient in the RIAs assessed. In general, regulators mentioned their own understanding of the undesired situation they intended to face (problem), and used little data and evidence to support their points of view. Not even the search for data and information on international experiences was used systematically. Given the low body of evidence in RIAs, it would be appropriate to carry out an uncertainty analysis on the problems (Ellig & McLaughlin, 2012), but none of them addressed uncertainties objectively.

Problem analysis and definition, when properly done, include the identification of its causes and consequences. Often, policies are based on the analysis of the factors that cause the problem and the resulting consequences. The development of a diagram or problem tree can be crucial in this process (Mota et al., 2020; Snowdon et al., 2008).

In Brazilian RIAs, regulators rarely used problem diagrams to illustrate the causes, root causes, and consequences of the problem they intended to address. Most of the time, causes and consequences were little explored. Analyses of trends in problem evolution, essential to guide the urgency and need for regulators’ action, were also rare and fragile.

Despite the weakness of the analysis on problem’s consequences, Brazilian regulators made good mappings of the actors who suffered the effects of the problem. The identification of the affected players was present in all investigated RIAs. In the best analyses, regulators recognized different stakeholders, and discussed how each of them was affected. In lower quality evaluations, regulators just listed the groups of players, using generic names that did not allow their exact identification. They used terms like ‘users’, ‘agency areas involved with the topic’, and ‘investors’.

Finally, the last criterion related to problem analysis and definition refers to the legal basis. This is about defining the legal mechanisms that support government action upon an identified problem. In general, this criterion was very well evaluated in Brazilian RIAs. As a rule, there was citation and explanation of the laws, regulations, and instruments related to the institutional competence for acting on the problems. The good quality of RIA in this aspect prevents regulators from wasting time and effort to solve problems upon which they have no legal competence to act.

ii) Goal setting

After identifying the problem to be faced, the regulator must define the intervention goals. This stage can be the change from the negative statement of the problem into a positive statement of goals (Snowdon et al., 2008), which need to be clear and measurable in order to guide policy formulation (Souto-Otero, 2013).

Goal setting was found in almost all of the 21 Brazilian RIAs evaluated. In just one, the intervention’s objective was not presented. In general, goals were clearly described and connected to the problem. This logic, followed by Brazilian regulators, is important to ensure that goal achievement addresses the confrontation, mitigation, or elimination of the problem.

Although the definition of goals has generally fulfilled its role in the assessed RIAs, the score presented in Table 2 shows that the quality of the information provided could be better. Especially when the problem was not well defined, the regulator used generic and non-measurable verbs to determine the intervention goals. In some RIAs, they were established based on expressions like “improve consistency”, “keep a favorable regulatory environment”, or “optimize efforts”. These are ambiguous objectives that allow several interpretations, which can lead to developing policies that are ineffective, detached from the problems, and difficult to monitor results.

TABLE 2
SCORE - GOAL DEFINITION

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
13. Goal definition	Snowdon et al. (2008); Souto-Otero (2013)	2.67	53.3%	1.15
14. Connection of goals with the problem	Mota et al. (2020); Souto-Otero (2013)	3.24	64.8%	1.55
15. Goal measurement	Souto-Otero (2013)	1.52	30.5%	0.93

• Source: Elaborated by the authors.

iii) Identification of regulatory alternatives

When defining the goals of an intervention, the trend is to consider only the regulations for achieving them (Hepburn, 2006). However, the same objective can be achieved through several paths. In RIA, regulators must identify all feasible alternatives (Souto-Otero, 2013).

In this process, besides thinking of different possibilities for regulation, regulators must think on non-normative alternatives - regulatory means that are more flexible and less prescriptive than traditional regulation (Hepburn, 2006). Regulators should also consider the possibility of “no action”. An intervention should only be taken if it is better than the alternative of doing nothing (Mazzocchi, Ragona, & Zanoli, 2013).

In all RIAs assessed in this study, Brazilian agencies presented regulatory alternatives to achieve the intended goals. As indicated by the scores in Table 3, the use of the “no action” alternative was more consistent than the use of non-normative alternatives. The main limitation to the use of the “no action” option was its early elimination. Some regulators excluded this possibility based on superficial arguments. In the case of the non-normative alternatives, the main weakness was their limited use in the investigated RIAs. On the rare occasions when regulators suggested non-normative alternatives, they were eliminated early or considered only as additional measures to traditional regulation.

TABLE 3
SCORE - IDENTIFICATION OF REGULATORY ALTERNATIVES

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
16. Identification of alternatives	Ellig and McLaughlin (2012); Souto-Otero (2013)	3.10	61.9%	0.94
17. Non-normative alternatives	Hepburn (2006)	1.14	22.9%	1.62
18. ‘No action’ alternative	Mazzocchi et al. (2013)	2.81	56.2%	1.33
19. Risk analysis of alternatives	Hepburn (2006)	0.05	1.0%	0.22

• Source: Elaborated by the authors.

We also noticed that the language used by Brazilian regulators in RIA reports was oriented towards traditional regulation. In the texts, we identified little willingness to adopt more flexible regulatory alternatives. The preference for known and traditional norms was already found in previous studies that evaluated RIA quality in other countries (Posner, 2003; Radchenko & Parshina, 2014; Russel & Turnpenny, 2009). According to Hepburn (2006), this preference may occur because of regulators’ perception of risk in the process. They are not aware of many successful experiences with regulations other than traditional, have little information on the use of these alternatives, and are afraid of the government losing control over the problem, if it does not adopt conventional regulatory measures.

To deal with regulators’ risk aversion, good practice in RIA suggests the use of risk analysis tools for the alternatives. By gathering information on risk, they could overcome their constraints and concerns (Hepburn, 2006). Despite its relevance, the risk analysis of alternatives was seldom done in the assessed RIAs. There was a huge difficulty of regulators to consider regulatory alternatives in the process, which hindered the quality of Brazilian agencies’ analyses.

iv) Analysis of alternatives’ impacts

If implemented, each regulatory alternative will have different impacts on society. RIA should include a strong analysis to provide a complete scenario of the harmful and beneficial effects of each alternative. It should explain how each alternative will contribute to achieve the regulatory goals (benefits) and the costs incurred to meet the established requirements. It is essential to present evidence, in order to support the analysis (Dudley et al., 2017).

In previous studies on RIA quality assessment, authors found that social, environmental, and economic impacts, and those for complying with regulation, were the most important (Fritsch et al., 2013; Russel & Turnpenny, 2009; Staroňová, 2010). In Brazil, government guidelines further suggest that impacts on micro and small firms, competition and competitiveness, international trade, security, and public budget be included in the analysis (Ministério da Economia, 2021).

As shown in Table 4, RIAs prepared by Brazilian regulatory agencies were weak in covering these impacts. There were few analyses that considered the effects on the economy, health, and environment. The effects on competition and competitiveness, security, micro and small firms, international trade and public budget, although explicit in government’s guidelines, were not taken into consideration properly.

In general, RIAs conducted in Brazil have considered two types of impacts. Regulatory costs - those arising from compliance with obligations imposed by regulation - and other impacts. These include more specific effects, such as “customer service”, “participation in concession’s management” and “conciliation of interests”. Another weakness found in RIAs was the rare use of evidence in the analysis of these impacts. Regulators, in general, used just impact expectations, developed through simplistic arguments, with little support from evidence.

TABLE 4
SCORE - ANALYSIS OF ALTERNATIVES’ IMPACTS

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
20. Impact on regulatory costs	Fritsch et al. (2013); Staroňová (2010)	1.14	22.9%	1.15
21. Impact on the economy	Souto-Otero (2013); Staroňová (2010)	0.14	2.9%	0.48
22. Impact on health	Souto-Otero (2013); Staroňová (2010)	0.14	2.9%	0.48
23. Impact on security	Souto-Otero (2013); Staroňová (2010)	0.25	5.0%	0.72
24. Impact on the environment	Fritsch et al. (2013); Staroňová (2010)	0.29	5.7%	0.72
25. Impact on competition and competitiveness e	Ministério da Economia (2021)	0.24	4.8%	0.77
26. Impact on micro and small firms	Ministério da Economia (2021)	0.14	2.9%	0.48
27. Impact on international trade	Ministério da Economia (2021)	0.00	0.0%	0.00
28. Impact on public budget	Ministério da Economia (2021)	0.43	8.6%	1.12
29. Other impacts	Jacobs (2004)	2.33	46.7%	1.06
30. Evidence on the impact of alternatives	Dudley et al. (2017)	0.90	18.1%	1.18

• Source: Elaborated by the authors.

v) Comparing alternatives

The comparison of alternatives, based on calculated impacts, should follow consistent methods that assess all positive and negative effects relevant for decision-making (Jacobs, 2004). It is possible to compare them through the appropriate use of quantitative methods of analysis, such as cost-benefit, multicriteria, and cost-effectiveness (Aquila et al., 2019). Qualitative methods of comparison can also be used, as long as procedures for maximizing the consistency of the analyses are adopted (Jacobs, 2004).

A key element to guide the comparison of alternatives is to present the “no action” alternative as a baseline. This means that the differences between the baseline and the incremental changes imposed by each alternative will indicate the best regulatory alternatives available (Dudley et al., 2017).

As Table 5 shows, the use of the baseline was a neglected practice in the investigated RIAs. Moreover, in only a few cases they used consistent comparative methods. Justification for choosing the methods was also not presented systematically.

TABLE 5
SCORE - COMPARING ALTERNATIVES

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
31. Alternative of “no action” as baseline	Dudley et al. (2017); Ellig and McLaughlin (2012)	0.15	3.0%	0.49
32. Method for comparing alternatives	Ellig and McLaughlin (2012); Mazzocchi et al. (2013)	0.90	18.1%	1.64
33. Justification for choosing the method	Decreto nº 10.411, de 30 de junho de 2020	0.38	7.6%	1.16
34. Costs or disadvantages of alternatives	Mazzocchi et al. (2013); Staroňová (2010)	1.86	37.1%	0.96
35. Quantification of costs or disadvantages of alternatives	Dudley et al. (2017)	0.43	8.6%	1.08
36. Benefits or advantages of alternatives	Mazzocchi et al. (2013); Staroňová (2010)	1.86	37.1%	0.96
37. Quantification of benefits or advantages of alternatives	Dudley et al. (2017)	0.43	8.6%	1.08
38. Update of costs (disadvantages) and benefits (advantages) by means of discount rate	Belfield et al. (2019)	0.00	0.0%	0.00
39. Sensitivity analysis of alternatives	Dudley et al. (2017)	0.14	2.9%	0.65

• Source: Elaborated by the authors.

As a rule, Brazilian regulators carried out the comparison of alternatives based on tables or texts that indicated their own understanding on the advantages and disadvantages of each alternative. These procedures have been classified by many regulators as a ‘qualitative method’ for comparing alternatives. These comparisons were often done without any methodological accuracy, and mistakenly classified as ‘qualitative’. The risk of this methodological flexibility resulting in analyses without method had already been mentioned by Jacobs (2004) and Lussis (2004). If Brazilian agencies wish to use qualitative methods to compare alternatives, they should adopt procedures for data collection and analysis to achieve consistent results.

Regarding the use of quantitative methods for comparing alternatives, the practice proved to be incipient. The most used was multicriteria analysis, but without a methodological standard. Some regulators applied multicriteria analysis with accuracy, while others ranked very simple analyses as multicriteria. Because of the rare use of quantitative methods, Table 5 shows that the quality scores of RIAs related to quantification criteria and the use of discount rate and sensitivity analysis were very low. Reports indicated that the agencies did not have the necessary maturity to use quantitative methods consistently.

vi) Indication of the best alternative

The indication of the best alternative is the direct result of the impact analysis and the comparison between different alternatives. An alternative is recommended to the decision maker when it is identified as better than the others (Ellig & McLaughlin, 2012).

TABLE 6
SCORE - INDICATION OF THE BEST ALTERNATIVE

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
40. Recommendation of the best alternative	Ellig and McLaughlin (2012)	3.10	61.9%	0.70
41. Risks and effects expected by adopting the best alternative	Decreto nº 10.411, de 30 de junho de 2020	0.90	18.1%	1.37
42. Indicators for monitoring and evaluating	Souto-Otero (2013); Staroňová (2010)	1.29	25.7%	1.74
43. Data sources of indicators for monitoring and evaluating	Ellig and McLaughlin (2012)	0.71	14.3%	1.45
44. Implementation strategy	Decreto nº 10.411, de 30 de junho de 2020	1.38	27.6%	1.16

• Source: Elaborated by the authors.

The recommendation of the best regulatory alternative was present in all RIA reports evaluated. In general, Brazilian regulators concluded the rationale for comparing alternatives by highlighting the one that proved to be the best for society. However, in some cases, the arguments supporting the choice of the best alternative lacked the expected consistency and clarity. Incompleteness in the stages of impact analysis and comparison of alternatives led to unsubstantiated recommendations.

In addition, Table 6 shows that Brazilian regulators provided limited information on the expected risks and effects of adopting the best alternative, and showed poor information on its implementation, monitoring, and evaluation.

vii) Transparency and social participation

Throughout all RIA stages, regulators may find gaps in knowledge and information for developing sound and thorough analyses. Therefore, a quality RIA needs to adopt systematic procedures for data collection and analysis, in order to provide realistic and useful results to the decision maker (Kirkpatrick, Parker, & Zhang, 2004).

Moreover, it is up to the regulator to encourage society to critically comment on RIA (Ballantine & Devonald, 2006). When social participation is effective, and different standpoints are considered in the analysis, we achieve greater credibility and legitimacy, and the chances of the regulatory policy attaining its goals increase (Radaelli, 2004). For society to participate, it is essential that the RIA process be transparent. The reports and documents related to the analysis should be public and prepared in a simple and accessible language (Saab & Silva, 2021a).

As shown in Table 7, transparency proved to be a well evaluated element in the investigated RIAs. The scores regarding the access to reports and the ease of reading the information were high. Most reports were found on the regulatory agencies’ websites, and the language adopted was appropriate for society’s understanding. In this aspect, we identified an opportunity for improvement, by better structuring the executive summaries. Sometimes they were used to introduce the report’s content, and not to summarize its main findings.

TABLE 7
SCORE - TRANSPARENCY AND SOCIAL PARTICIPATION

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
45. RIA reports published on agencies’ websites	Ellig and McLaughlin (2012)	4.71	94.3%	0.72
46. RIA understandable to the lay citizen	Ellig and McLaughlin (2012)	3.95	79.0%	0.80
47. Executive summary	Decreto nº 10.411, de 30 de junho de 2020	2.36	47.3%	1.62
48. Social participation during RIA	Ballantine and Devonald (2006); Radaelli (2004)	1.86	37.1%	1.65
49. Participation of affected groups in RIA	Staroňová et al. (2007)	1.43	28.6%	1.69
50. Contributions to RIA received and analyzed	Staroňová (2010); Staroňová et al. (2007)	0.90	18.1%	1.45

• Source: Elaborated by the authors.

As for society participation, we identified some elements that need improvement. In some cases, regulators encouraged the participation of society after completing RIA, through public consultations on draft regulations. However, this was a late moment. When participation occurs when there is already a government positioning, the likelihood that other standpoints will be considered in the regulation is lower (Saab, Bermejo, Garcia, Pereira, & Silva, 2018).

Furthermore, cases where participation was restricted to areas and departments of the agency itself were not uncommon. There was no systematic concern for ensuring that groups affected by the problem were heard. Even when regulators received contributions from society, they seldom showed the analyses and justifications for accepting or rejecting them.

5. FINAL REMARKS

This study advanced previous research and assessed the quality of a set of 21 RIAs prepared by Brazilian federal regulatory agencies, by developing and applying an evaluation tool with 50 quality criteria.

The results of the evaluation showed that RIAs developed in Brazil were appropriate in some aspects, and weak in others. The definition of the regulatory problem and the objectives of the intervention showed good quality, as well as the process transparency, which also received good scores. On the other hand, the results indicated important weaknesses, especially in investigating impacts and comparing regulatory alternatives. The impacts considered in the analysis were limited, and the methods used for comparing alternatives were poorly consistent.

We raised some hypotheses to explain the absence of an analytical standard in RIAs done in Brazil. Insufficient technical capacity to carry out sophisticated impact analyses can explain the methodological limitations and weaknesses found. In addition, interest groups may have influenced RIA quality, through pressure and resistance to its use in certain contexts. Likewise, political commitment to conducting RIA may have varied according to the interests involved in the regulatory proposals under analysis. Finally, the recent institutionalization of RIA in the country may have generated suspicions and doubts regarding its use, which must be overcome in order to achieve the desired standard of quality in the analyses.

Future studies that investigate in depth the hypotheses raised can contribute for building knowledge on the subject. Interviews with actors involved in the implementation and execution of RIAs in Brazil can provide insights and new data for explaining RIA quality in the country.

The results of this research can contribute for managers and civil servants to know the aspects of highest and lowest quality in RIA practice in the country. Resources and efforts can be oriented to train and qualify regulators more effectively. Additionally, the evaluation instrument proposed in this research can be adopted by public administration bodies to assess quality and improve impact analyses.

From a theoretical standpoint, this research added knowledge to previous studies that evaluated the quality of RIA around the world, and confirmed the need for improvements in the analyses. In addition, we designed a new evaluation tool, based on the literature, which can be criticized and improved, so that other assessments can be carried out in different contexts and realities.

REFERÊNCIAS

• Achtnicht, M., Rennings, K., & Hertin, J. (2009). Experiences with integrated impact assessment - empirical evidence from a survey in three European member states. Environmental Policy and Governance, 19(5), 321-335. Recuperado de https://doi.org/10.1002/eet.517
  » https://doi.org/10.1002/eet.517
• Adelle, C., Weiland, S., Dick, J., Olivo, D. G., Marquardt, J., Rots, G., … Zasada, I. (2016). Regulatory impact assessment: a survey of selected developing and emerging economies. Public Money and Management, 36(2), 89-96. Recuperado de https://doi.org/10.1080/09540962.2016.1118930
  » https://doi.org/10.1080/09540962.2016.1118930
• Aquila, G., Pamplona, E. O., Ferreira, J. A. Filho, Silva, A. S., Mataveli, J. V. A., Correa, J. E., ... Garcia, G. C. (2019, agosto). Quantitative regulatory impact analysis: Experience of regulatory agencies in Brazil. Utilities Policy, 59, 100931. Recuperado de https://doi.org/10.1016/j.jup.2019.100931
  » https://doi.org/10.1016/j.jup.2019.100931
• Ballantine, B., & Devonald, B. (2006). Modern regulatory impact analysis: the experience of the European Union. Regulatory Toxicology and Pharmacology, 44(1), 57-68. Recuperado de https://doi.org/10.1016/j.yrtph.2005.06.016
  » https://doi.org/10.1016/j.yrtph.2005.06.016
• Belfield, C. R., Bowden, A. B., & Rodriguez, V. (2019). Evaluating Regulatory Impact Assessments in Education Policy. American Journal of Evaluation, 40(3), 335-353. Recuperado de: https://doi.org/10.1177/1098214018785463
  » https://doi.org/10.1177/1098214018785463
• Bourges, F. (2020). Digital public administration as expansion and redemption of citizen participation - the Brazilian system Fala.Br. Revue Internationale Des Gouvernements Ouverts, 9, 149-164.
• Carroll, P. (2010). Does regulatory impact assessment lead to better policy? Policy and Society, 29(2), 113-122. Recuperado de https://doi.org/10.1016/j.polsoc.2010.03.009
  » https://doi.org/10.1016/j.polsoc.2010.03.009
• Carvalho, B. E., Costa, A. S., Marques, R. C., & Cordeiro O, Netto. (2019, agosto). The presence of governance: a system assessment based on innovative core regulatory principles for Brazilian regulators. Expert Systems, 36(4), e12406. Recuperado de https://doi.org/10.1111/exsy.12406
  » https://doi.org/10.1111/exsy.12406
• Castro, C. M. (2014). Some aspects of implementing Regulatory Impact Analysis in Brazil. Revista de Administração Pública, 48(2), 323-342. Recuperado de https://doi.org/10.1590/0034-76121359
  » https://doi.org/10.1590/0034-76121359
• Daskal, S., Ayalon, O., & Shechter, M. (2019, março). Closing the Loop: the Challenges of Regulation in Municipal Solid Waste Management. Detritus, 5, 3-10. Recuperado de https://doi.org/10.31025/2611-4135/2019.13775
• Decreto nº 10.411, de 30 de junho de 2020 (2020). Regulamenta a análise de impacto regulatório, de que tratam o art. 5º da Lei nº 13.874, de 20 de setembro de 2019, e o art. 6º da Lei nº 13.848, de 25 de junho de 2019. Brasília, DF. Recuperado dehttp://www.planalto.gov.br/ccivil_03/_ato2019-2022/2020/decreto/d10411.htm
  » http://www.planalto.gov.br/ccivil_03/_ato2019-2022/2020/decreto/d10411.htm
• Desmarais, B. A., & Hird, J. A. (2014). Public policy’s bibliography: the use of research in US regulatory impact analyses. Regulation and Governance, 8(4), 497-510. Recuperado de https://doi.org/10.1111/rego.12041
  » https://doi.org/10.1111/rego.12041
• Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11
  » https://doi.org/10.1017/bca.2017.11
• Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x
  » https://doi.org/10.1111/j.1539-6924.2011.01715.x
• Ellig, J., Mclaughlin, P. A., & Morrall, J. F. (2013). Continuity, change, and priorities: the quality and use of regulatory analysis across US administrations. Regulation and Governance, 7(2), 153-173. Recuperado de https://doi.org/10.1111/j.1748-5991.2012.01149.x
  » https://doi.org/10.1111/j.1748-5991.2012.01149.x
• Francesco, F. (2012). Diffusion of regulatory impact analysis among OECD and EU Member States. Comparative Political Studies, 45(10), 1277-1305. Recuperado de https://doi.org/10.1177/0010414011434297
  » https://doi.org/10.1177/0010414011434297
• Fritsch, O., Radaelli, C. M., Schrefler, L., & Renda, A. (2013). Comparing the content of regulatory impact assessments in the UK and the EU. Public Money and Management, 33(6), 445-452. Recuperado dehttps://doi.org/10.1080/09540962.2013.836007
  » https://doi.org/10.1080/09540962.2013.836007
• Hahn, R. W., Burnett, J. K., Chan, Y. H. I., Mader, E. A., & Moyle, P. R. (2000). Assessing regulatory impact analyses: the failure of agencies to comply with executive order 12,866. Harvard Journal of Law and Public Policy, 23(3), 859-885.
• Hepburn, G. (2006). Alternatives do Traditional Regulation Recuperado de https://www.oecd.org/gov/regulatory-policy/42245468.pdf
  » https://www.oecd.org/gov/regulatory-policy/42245468.pdf
• Jacobs, S. (2004). Regulatory impact assessment and the economic transition to markets. Public Money and Management, 24(5), 283-290. Recuperado dehttps://doi.org/10.1111/j.1467-9302.2004.00435.x
  » https://doi.org/10.1111/j.1467-9302.2004.00435.x
• Kirkpatrick, C., Parker, D., & Zhang, Y. F. (2004). Regulatory impact assessment in developing and transition economies: a survey of current practice. Public Money and Management, 24(5), 291-296. Recuperado dehttps://doi.org/10.1111/j.1467-9302.2004.00436.x
  » https://doi.org/10.1111/j.1467-9302.2004.00436.x
• Lei nº 13.848, de 25 de junho de 2019 (2019). Dispõe sobre a gestão, a organização, o processo decisório e o controle social das agências reguladoras, altera a Lei nº 9.427, de 26 de dezembro de 1996, a Lei nº 9.472, de 16 de julho de 1997, a Lei nº 9.478, de 6 de agosto de 1997, a Lei nº 9.782, de 26 de janeiro de 1999, a Lei nº 9.961, de 28 de janeiro de 2000, a Lei nº 9.984, de 17 de julho de 2000, a Lei nº 9.986, de 18 de julho de 2000, a Lei nº 10.233, de 5 de junho de 2001, a Medida Provisória nº 2.228-1, de 6 de setembro de 2001, a Lei nº 11.182, de 27 de setembro de 2005, e a Lei nº 10.180, de 6 de fevereiro de 2001. Brasília, DF. Recuperado de http://www.planalto.gov.br/ccivil_03/_ato2019-2022/2019/lei/l13848.htm
  » http://www.planalto.gov.br/ccivil_03/_ato2019-2022/2019/lei/l13848.htm
• Lei nº 13.874, de 20 de setembro de 2019 (2019). Institui a Declaração de Direitos de Liberdade Econômica; estabelece garantias de livre mercado; altera as Leis nos 10.406, de 10 de janeiro de 2002 (Código Civil), 6.404, de 15 de dezembro de 1976, 11.598, de 3 de dezembro de 2007, 12.682, de 9 de julho de 2012, 6.015, de 31 de dezembro de 1973, 10.522, de 19 de julho de 2002, 8.934, de 18 de novembro 1994, o Decreto-Lei nº 9.760, de 5 de setembro de 1946 e a Consolidação das Leis do Trabalho, aprovada pelo Decreto-Lei nº 5.452, de 1º de maio de 1943; revoga a Lei Delegada nº 4, de 26 de setembro de 1962, a Lei nº 11.887, de 24 de dezembro de 2008, e dispositivos do Decreto-Lei nº 73, de 21 de novembro de 1966; e dá outras providências. Brasília, DF. Recuperado dehttp://www.planalto.gov.br/ccivil_03/_ato2019-2022/2019/lei/L13874.htm
  » http://www.planalto.gov.br/ccivil_03/_ato2019-2022/2019/lei/L13874.htm
• Livermore, M. A., & Revesz, R. L. (2013). Can executive review help prevent capture? In D. Carpenter, & D. A.Moss(Eds.), Preventing Regulatory Capture: special interest influence and how to limit it (pp. 420-450). Campridge, UK: Campridge University Press.
• Lussis, B. (2004, dezembro). EU extended impact assessment review(Working Paper, 9). Sombreffe, Belgique: Institut Pour Un Développement Durable.
• Mazzocchi, M., Ragona, M., & Zanoli, A. (2013). A fuzzy multi-criteria approach for the ex-ante impact assessment of food safety policies. Food Policy, 38(1), 177-189. Recuperado dehttps://doi.org/10.1016/j.foodpol.2012.11.011
  » https://doi.org/10.1016/j.foodpol.2012.11.011
• Ministério da Economia. (2021). Guia para Elaboração de Análise de Impacto Regulatório (AIR) Recuperado de https://www.gov.br/mma/pt-br/acesso-a-informacao/analise-de-impacto-regulatorio-2013-air-1/guia-para-elaboracao-de-air-2021_vdefeso.pdf
  » https://www.gov.br/mma/pt-br/acesso-a-informacao/analise-de-impacto-regulatorio-2013-air-1/guia-para-elaboracao-de-air-2021_vdefeso.pdf
• Mota, D., Saab, F., Vazzoler, R., Schunig, K., Donagema, E., & Troncoso, G. (2020). Regulatory impact assessment in pandemic times: a practical exercise on the COVID-19 context. Revista do Serviço Público, 71, 165-88. Recuperado dehttps://doi.org/10.21874/rsp.v71i0.4824
  » https://doi.org/10.21874/rsp.v71i0.4824
• Nweke, O. C. (2011). A framework for integrating environmental justice in regulatory analysis. International Journal of Environmental Research and Public Health, 8(6), 2366-28385. Recuperado dehttps://doi.org/10.3390/ijerph8062366
  » https://doi.org/10.3390/ijerph8062366
• Peci, A. (2011). Avaliação do impacto regulatório e sua difusão no contexto brasileiro. Revista de Administração de Empresas, 51(4), 336-348. Recuperado dehttps://doi.org/10.1590/S0034-75902011000400003
  » https://doi.org/10.1590/S0034-75902011000400003
• Posner, E. A. (2003). Transfer regulations and cost-effectiveness analysis. Duke Law Journal, 53(3), 1067-1110. Recuperado de https://scholarship.law.duke.edu/dlj/vol53/iss3/3
  » https://scholarship.law.duke.edu/dlj/vol53/iss3/3
• Radaelli, C. M. (2004). The diffusion of regulatory impact analysis - best practice or lesson-drawing? European Journal of Political Research, 43(5), 723-477. Recuperado dehttps://doi.org/10.1111/j.0304-4130.2004.00172.x
  » https://doi.org/10.1111/j.0304-4130.2004.00172.x
• Radaelli, C. M. (2010). Rationality, power, management and symbols: four images of regulatory impact assessment. Scandinavian Political Studies, 33(2), 164-188. Recuperado dehttps://doi.org/10.1111/j.1467-9477.2009.00245.x
  » https://doi.org/10.1111/j.1467-9477.2009.00245.x
• Radchenko, T., & Parshina, E. (2014). Regulatory impact assessment in Russia: Application and lessons from economic theory. Economic Policy, 3, 36-60.
• Russel, D., & Turnpenny, J. (2009). The politics of sustainable development in UK government: what role for integrated policy appraisal? Environment and Planning C: Government and Policy, 27(2), 340-354. Recuperado dehttps://doi.org/10.1068/c0810j
  » https://doi.org/10.1068/c0810j
• Saab, F., Bermejo, P. H. S., Garcia, G. C., Pereira, J. S., & Silva, S. A. M. (2018). Does public consultation encourage social participation? Journal of Enterprise Information Management, 31(5), 796-814. Recuperado de https://doi.org/10.1108/JEIM-11-2017-0169
  » https://doi.org/10.1108/JEIM-11-2017-0169
• Saab, F., & Silva, S. A. M. (2021a). Análise de impacto regulatório e prevenção da corrupção: um estudo exploratório sobre a AIR. Revista da CGU, 13(24), 167-179. Recuperado dehttps://doi.org/10.36428/revistadacgu.v13i24.451
  » https://doi.org/10.36428/revistadacgu.v13i24.451
• Saab, F., & Silva, S. A. M. (2021b). Evaluating the quality of regulatory impact analysis: a literature review. Revista do Serviço Público, 72, 34-57.
• Snowdon, W., Schultz, J., & Swinburn, B. (2008). Problem and solution trees: a practical approach for identifying potential interventions to improve population nutrition. Health Promotion International, 23(4), 345-53. Recuperado dehttps://doi.org/10.1093/heapro/dan027
  » https://doi.org/10.1093/heapro/dan027
• Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725
  » https://doi.org/10.1332/174426413X662725
• Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201
  » https://doi.org/10.1017/S0143814X09990201
• Staroňová, K. (2014). Analyzing scientific knowledge in documents: the case of regulatory impact assessment. Human Affairs, 24(3), 299-306. Recuperado dehttps://doi.org/10.2478/s13374-014-0228-7
  » https://doi.org/10.2478/s13374-014-0228-7
• Staroňová, K., Pavel, J., & Krapež, K. (2007). Piloting regulatory impact assessment: a comparative analysis of the Czech Republic, Slovakia and Slovenia. Impact Assessment and Project Appraisal, 25(4), 271-80. Recuperado dehttps://doi.org/10.3152/146155107X246314
  » https://doi.org/10.3152/146155107X246314
• Yin, R. K. (1994). Pesquisa estudo de caso - desenho e métodos (2a ed.). Porto Alegre, RS: Bookman.

• 1
  Act no. 13,848, of June 25, 2019 (Lei nº 13.848, de 25 de junho de 2019), and Act no. 13,874, of September 20, 2019 (Lei nº 13.874, de 20 de setembro de 2019).
• 2
  Decree no. 10,411, of June 30, 2020 (Decreto nº 10.411, de 30 de junho de 2020).

• [Translated version] Note: All quotes in English translated by this article’s translator.

Edited by

Florence Cavalcanti Heber Pedreira de Freitas (Universidade Federal de Sergipe, São Cristóvão / SE - Brazil). ORCID: https://orcid.org/0000-0002-9498-3268

Elize Massard da Fonseca (Fundação Getulio Vargas, São Paulo / SP - Brazil). ORCID: https://orcid.org/0000-0003-3847-3105

Publication Dates

• Publication in this collection
  09 Sept 2022
• Date of issue
  Jul-Aug 2022

History

• Received
  04 Apr 2022
• Accepted
  07 Aug 2022