What is the quality of the regulatory impact analysis prepared by Brazilian regulatory agencies?

Saab, Flavio; Silva, Suylan de Almeida Midlej e

doi:10.1590/0034-761220220111x

Flavio Saab

Universidade de Brasília / Programa de Pós-Graduação em Administração, Brasília / DF - Brazil

https://orcid.org/0000-0002-3913-4175

Suylan de Almeida Midlej e Silva

Universidade de Brasília / Programa de Pós-Graduação em Administração, Brasília / DF - Brazil

https://orcid.org/0000-0002-9395-3657

Flavio Saab - Undergraduate degree in Economics; Master and Ph.D. student in Administration at University of Brasília (UnB); Collaborating Professor at the Department of Public Policies Management (GPP/UnB); Federal civil servant at the National Sanitary Surveillance Agency (Anvisa). E-mail: flaviosaab@gmail.com

Suylan de Almeida Midlej e Silva - Undergraduate degree in Social Communication from the Pontifical Catholic University of Minas Gerais (PUC/MG); Master in Contemporary Communication and Culture from Federal University of Bahia (UFBA); Ph.D. in Sociology from University of Brasília (UnB); Associate Professor at the Graduate Program in Administration (PPGA/UnB), at the Professional Master Program in Public Administration (MPA/UnB), and of the Department of of Public Policies Management (GPP/UnB). E-mail: suylan@unb.br

Florence Cavalcanti Heber Pedreira de Freitas (Universidade Federal de Sergipe, São Cristóvão / SE - Brazil). ORCID: https://orcid.org/0000-0002-9498-3268

Elize Massard da Fonseca (Fundação Getulio Vargas, São Paulo / SP - Brazil). ORCID: https://orcid.org/0000-0003-3847-3105

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
1. Problem definition	Dudley et al. (2017Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11 https://doi.org/10.1017/bca.2017.11... ); Souto-Otero (2013Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725 https://doi.org/10.1332/174426413X662725... )	2.57	51.4%	1.12
2. Nature (Market failure, regulatory failure, etc.)	Dudley et al. (2017Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11 https://doi.org/10.1017/bca.2017.11... ); Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... )	0.48	9.5%	1.08
3. Use of evidence	Dudley et al. (2017Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11 https://doi.org/10.1017/bca.2017.11... ); Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... )	1.14	22.9%	1.59
4. International experience	Decreto nº 10.411, de 30 de junho de 2020Decreto nº 10.411, de 30 de junho de 2020. (2020). Regulamenta a análise de impacto regulatório, de que tratam o art. 5º da Lei nº 13.874, de 20 de setembro de 2019, e o art. 6º da Lei nº 13.848, de 25 de junho de 2019. Brasília, DF. Recuperado dehttp://www.planalto.gov.br/ccivil_03/_ato2019-2022/2020/decreto/d10411.htm http://www.planalto.gov.br/ccivil_03/_at...	0.57	11.4%	1.25
5. Problem uncertainties	Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... )	0.00	0.0%	0.00
6. Cause mapping	Mota et al. (2020Mota, D., Saab, F., Vazzoler, R., Schunig, K., Donagema, E., & Troncoso, G. (2020). Regulatory impact assessment in pandemic times: a practical exercise on the COVID-19 context. Revista do Serviço Público, 71, 165-88. Recuperado dehttps://doi.org/10.21874/rsp.v71i0.4824 https://doi.org/10.21874/rsp.v71i0.4824... ); Snowdon, Schultz, and Swinburn (2008Snowdon, W., Schultz, J., & Swinburn, B. (2008). Problem and solution trees: a practical approach for identifying potential interventions to improve population nutrition. Health Promotion International, 23(4), 345-53. Recuperado dehttps://doi.org/10.1093/heapro/dan027 https://doi.org/10.1093/heapro/dan027... )	0.81	16.2%	1.47
7. Identification of root cause	Mota et al. (2020Mota, D., Saab, F., Vazzoler, R., Schunig, K., Donagema, E., & Troncoso, G. (2020). Regulatory impact assessment in pandemic times: a practical exercise on the COVID-19 context. Revista do Serviço Público, 71, 165-88. Recuperado dehttps://doi.org/10.21874/rsp.v71i0.4824 https://doi.org/10.21874/rsp.v71i0.4824... ); Snowdon et al. (2008Snowdon, W., Schultz, J., & Swinburn, B. (2008). Problem and solution trees: a practical approach for identifying potential interventions to improve population nutrition. Health Promotion International, 23(4), 345-53. Recuperado dehttps://doi.org/10.1093/heapro/dan027 https://doi.org/10.1093/heapro/dan027... )	0.38	7.6%	1.20
8. Consequence mapping	Mota et al. (2020Mota, D., Saab, F., Vazzoler, R., Schunig, K., Donagema, E., & Troncoso, G. (2020). Regulatory impact assessment in pandemic times: a practical exercise on the COVID-19 context. Revista do Serviço Público, 71, 165-88. Recuperado dehttps://doi.org/10.21874/rsp.v71i0.4824 https://doi.org/10.21874/rsp.v71i0.4824... ); Snowdon et al. (2008Snowdon, W., Schultz, J., & Swinburn, B. (2008). Problem and solution trees: a practical approach for identifying potential interventions to improve population nutrition. Health Promotion International, 23(4), 345-53. Recuperado dehttps://doi.org/10.1093/heapro/dan027 https://doi.org/10.1093/heapro/dan027... )	0.95	19.0%	1.53
9. Problem trends	Dudley et al. (2017Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11 https://doi.org/10.1017/bca.2017.11... )	0.43	8.6%	0.98
10. Mapping of affected players	Staroňová et al. (2007Staroňová, K., Pavel, J., & Krapež, K. (2007). Piloting regulatory impact assessment: a comparative analysis of the Czech Republic, Slovakia and Slovenia. Impact Assessment and Project Appraisal, 25(4), 271-80. Recuperado dehttps://doi.org/10.3152/146155107X246314 https://doi.org/10.3152/146155107X246314... )	2.57	51.4%	1.12
11. Distribution of effects on each affected group	Staroňová et al. (2007Staroňová, K., Pavel, J., & Krapež, K. (2007). Piloting regulatory impact assessment: a comparative analysis of the Czech Republic, Slovakia and Slovenia. Impact Assessment and Project Appraisal, 25(4), 271-80. Recuperado dehttps://doi.org/10.3152/146155107X246314 https://doi.org/10.3152/146155107X246314... )	1.10	21.9%	1.73
12. Legal basis	Decreto nº 10.411, de 30 de junho de 2020Decreto nº 10.411, de 30 de junho de 2020. (2020). Regulamenta a análise de impacto regulatório, de que tratam o art. 5º da Lei nº 13.874, de 20 de setembro de 2019, e o art. 6º da Lei nº 13.848, de 25 de junho de 2019. Brasília, DF. Recuperado dehttp://www.planalto.gov.br/ccivil_03/_ato2019-2022/2020/decreto/d10411.htm http://www.planalto.gov.br/ccivil_03/_at...	4.43	88.6%	0.98

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
13. Goal definition	Snowdon et al. (2008Snowdon, W., Schultz, J., & Swinburn, B. (2008). Problem and solution trees: a practical approach for identifying potential interventions to improve population nutrition. Health Promotion International, 23(4), 345-53. Recuperado dehttps://doi.org/10.1093/heapro/dan027 https://doi.org/10.1093/heapro/dan027... ); Souto-Otero (2013Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725 https://doi.org/10.1332/174426413X662725... )	2.67	53.3%	1.15
14. Connection of goals with the problem	Mota et al. (2020Mota, D., Saab, F., Vazzoler, R., Schunig, K., Donagema, E., & Troncoso, G. (2020). Regulatory impact assessment in pandemic times: a practical exercise on the COVID-19 context. Revista do Serviço Público, 71, 165-88. Recuperado dehttps://doi.org/10.21874/rsp.v71i0.4824 https://doi.org/10.21874/rsp.v71i0.4824... ); Souto-Otero (2013Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725 https://doi.org/10.1332/174426413X662725... )	3.24	64.8%	1.55
15. Goal measurement	Souto-Otero (2013Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725 https://doi.org/10.1332/174426413X662725... )	1.52	30.5%	0.93

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
16. Identification of alternatives	Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... ); Souto-Otero (2013Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725 https://doi.org/10.1332/174426413X662725... )	3.10	61.9%	0.94
17. Non-normative alternatives	Hepburn (2006Hepburn, G. (2006). Alternatives do Traditional Regulation. Recuperado de https://www.oecd.org/gov/regulatory-policy/42245468.pdf https://www.oecd.org/gov/regulatory-poli... )	1.14	22.9%	1.62
18. ‘No action’ alternative	Mazzocchi et al. (2013Mazzocchi, M., Ragona, M., & Zanoli, A. (2013). A fuzzy multi-criteria approach for the ex-ante impact assessment of food safety policies. Food Policy, 38(1), 177-189. Recuperado dehttps://doi.org/10.1016/j.foodpol.2012.11.011 https://doi.org/10.1016/j.foodpol.2012.1... )	2.81	56.2%	1.33
19. Risk analysis of alternatives	Hepburn (2006Hepburn, G. (2006). Alternatives do Traditional Regulation. Recuperado de https://www.oecd.org/gov/regulatory-policy/42245468.pdf https://www.oecd.org/gov/regulatory-poli... )	0.05	1.0%	0.22

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
20. Impact on regulatory costs	Fritsch et al. (2013Fritsch, O., Radaelli, C. M., Schrefler, L., & Renda, A. (2013). Comparing the content of regulatory impact assessments in the UK and the EU. Public Money and Management, 33(6), 445-452. Recuperado dehttps://doi.org/10.1080/09540962.2013.836007 https://doi.org/10.1080/09540962.2013.83... ); Staroňová (2010Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201 https://doi.org/10.1017/S0143814X0999020... )	1.14	22.9%	1.15
21. Impact on the economy	Souto-Otero (2013Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725 https://doi.org/10.1332/174426413X662725... ); Staroňová (2010Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201 https://doi.org/10.1017/S0143814X0999020... )	0.14	2.9%	0.48
22. Impact on health	Souto-Otero (2013Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725 https://doi.org/10.1332/174426413X662725... ); Staroňová (2010Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201 https://doi.org/10.1017/S0143814X0999020... )	0.14	2.9%	0.48
23. Impact on security	Souto-Otero (2013Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725 https://doi.org/10.1332/174426413X662725... ); Staroňová (2010Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201 https://doi.org/10.1017/S0143814X0999020... )	0.25	5.0%	0.72
24. Impact on the environment	Fritsch et al. (2013Fritsch, O., Radaelli, C. M., Schrefler, L., & Renda, A. (2013). Comparing the content of regulatory impact assessments in the UK and the EU. Public Money and Management, 33(6), 445-452. Recuperado dehttps://doi.org/10.1080/09540962.2013.836007 https://doi.org/10.1080/09540962.2013.83... ); Staroňová (2010Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201 https://doi.org/10.1017/S0143814X0999020... )	0.29	5.7%	0.72
25. Impact on competition and competitiveness e	Ministério da Economia (2021)Ministério da Economia. (2021). Guia para Elaboração de Análise de Impacto Regulatório (AIR). Recuperado de https://www.gov.br/mma/pt-br/acesso-a-informacao/analise-de-impacto-regulatorio-2013-air-1/guia-para-elaboracao-de-air-2021_vdefeso.pdf https://www.gov.br/mma/pt-br/acesso-a-in...	0.24	4.8%	0.77
26. Impact on micro and small firms	Ministério da Economia (2021)Ministério da Economia. (2021). Guia para Elaboração de Análise de Impacto Regulatório (AIR). Recuperado de https://www.gov.br/mma/pt-br/acesso-a-informacao/analise-de-impacto-regulatorio-2013-air-1/guia-para-elaboracao-de-air-2021_vdefeso.pdf https://www.gov.br/mma/pt-br/acesso-a-in...	0.14	2.9%	0.48
27. Impact on international trade	Ministério da Economia (2021)Ministério da Economia. (2021). Guia para Elaboração de Análise de Impacto Regulatório (AIR). Recuperado de https://www.gov.br/mma/pt-br/acesso-a-informacao/analise-de-impacto-regulatorio-2013-air-1/guia-para-elaboracao-de-air-2021_vdefeso.pdf https://www.gov.br/mma/pt-br/acesso-a-in...	0.00	0.0%	0.00
28. Impact on public budget	Ministério da Economia (2021)Ministério da Economia. (2021). Guia para Elaboração de Análise de Impacto Regulatório (AIR). Recuperado de https://www.gov.br/mma/pt-br/acesso-a-informacao/analise-de-impacto-regulatorio-2013-air-1/guia-para-elaboracao-de-air-2021_vdefeso.pdf https://www.gov.br/mma/pt-br/acesso-a-in...	0.43	8.6%	1.12
29. Other impacts	Jacobs (2004Jacobs, S. (2004). Regulatory impact assessment and the economic transition to markets. Public Money and Management, 24(5), 283-290. Recuperado dehttps://doi.org/10.1111/j.1467-9302.2004.00435.x https://doi.org/10.1111/j.1467-9302.2004... )	2.33	46.7%	1.06
30. Evidence on the impact of alternatives	Dudley et al. (2017Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11 https://doi.org/10.1017/bca.2017.11... )	0.90	18.1%	1.18

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
31. Alternative of “no action” as baseline	Dudley et al. (2017Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11 https://doi.org/10.1017/bca.2017.11... ); Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... )	0.15	3.0%	0.49
32. Method for comparing alternatives	Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... ); Mazzocchi et al. (2013Mazzocchi, M., Ragona, M., & Zanoli, A. (2013). A fuzzy multi-criteria approach for the ex-ante impact assessment of food safety policies. Food Policy, 38(1), 177-189. Recuperado dehttps://doi.org/10.1016/j.foodpol.2012.11.011 https://doi.org/10.1016/j.foodpol.2012.1... )	0.90	18.1%	1.64
33. Justification for choosing the method	Decreto nº 10.411, de 30 de junho de 2020Decreto nº 10.411, de 30 de junho de 2020. (2020). Regulamenta a análise de impacto regulatório, de que tratam o art. 5º da Lei nº 13.874, de 20 de setembro de 2019, e o art. 6º da Lei nº 13.848, de 25 de junho de 2019. Brasília, DF. Recuperado dehttp://www.planalto.gov.br/ccivil_03/_ato2019-2022/2020/decreto/d10411.htm http://www.planalto.gov.br/ccivil_03/_at...	0.38	7.6%	1.16
34. Costs or disadvantages of alternatives	Mazzocchi et al. (2013Mazzocchi, M., Ragona, M., & Zanoli, A. (2013). A fuzzy multi-criteria approach for the ex-ante impact assessment of food safety policies. Food Policy, 38(1), 177-189. Recuperado dehttps://doi.org/10.1016/j.foodpol.2012.11.011 https://doi.org/10.1016/j.foodpol.2012.1... ); Staroňová (2010Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201 https://doi.org/10.1017/S0143814X0999020... )	1.86	37.1%	0.96
35. Quantification of costs or disadvantages of alternatives	Dudley et al. (2017Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11 https://doi.org/10.1017/bca.2017.11... )	0.43	8.6%	1.08
36. Benefits or advantages of alternatives	Mazzocchi et al. (2013Mazzocchi, M., Ragona, M., & Zanoli, A. (2013). A fuzzy multi-criteria approach for the ex-ante impact assessment of food safety policies. Food Policy, 38(1), 177-189. Recuperado dehttps://doi.org/10.1016/j.foodpol.2012.11.011 https://doi.org/10.1016/j.foodpol.2012.1... ); Staroňová (2010Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201 https://doi.org/10.1017/S0143814X0999020... )	1.86	37.1%	0.96
37. Quantification of benefits or advantages of alternatives	Dudley et al. (2017Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11 https://doi.org/10.1017/bca.2017.11... )	0.43	8.6%	1.08
38. Update of costs (disadvantages) and benefits (advantages) by means of discount rate	Belfield et al. (2019Belfield, C. R., Bowden, A. B., & Rodriguez, V. (2019). Evaluating Regulatory Impact Assessments in Education Policy. American Journal of Evaluation, 40(3), 335-353. Recuperado de: https://doi.org/10.1177/1098214018785463 https://doi.org/10.1177/1098214018785463... )	0.00	0.0%	0.00
39. Sensitivity analysis of alternatives	Dudley et al. (2017Dudley, S., Belzer, R., Blomquist, G., Brennan, T., Carrigan, C., Cordes, J., … Zerbe, R. (2017). Consumer’s guide to regulatory impact analysis: ten tips for being an informed policymaker. Journal of Benefit-Cost Analysis, 8(2), 187-204. Recuperado dehttps://doi.org/10.1017/bca.2017.11 https://doi.org/10.1017/bca.2017.11... )	0.14	2.9%	0.65

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
40. Recommendation of the best alternative	Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... )	3.10	61.9%	0.70
41. Risks and effects expected by adopting the best alternative	Decreto nº 10.411, de 30 de junho de 2020Decreto nº 10.411, de 30 de junho de 2020. (2020). Regulamenta a análise de impacto regulatório, de que tratam o art. 5º da Lei nº 13.874, de 20 de setembro de 2019, e o art. 6º da Lei nº 13.848, de 25 de junho de 2019. Brasília, DF. Recuperado dehttp://www.planalto.gov.br/ccivil_03/_ato2019-2022/2020/decreto/d10411.htm http://www.planalto.gov.br/ccivil_03/_at...	0.90	18.1%	1.37
42. Indicators for monitoring and evaluating	Souto-Otero (2013Souto-Otero, M. (2013). Is “better regulation” possible? Formal and substantive quality in the impact assessments in education and culture of the European Commission. Evidence and Policy, 9(4), 513-529. Recuperado de https://doi.org/10.1332/174426413X662725 https://doi.org/10.1332/174426413X662725... ); Staroňová (2010Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201 https://doi.org/10.1017/S0143814X0999020... )	1.29	25.7%	1.74
43. Data sources of indicators for monitoring and evaluating	Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... )	0.71	14.3%	1.45
44. Implementation strategy	Decreto nº 10.411, de 30 de junho de 2020Decreto nº 10.411, de 30 de junho de 2020. (2020). Regulamenta a análise de impacto regulatório, de que tratam o art. 5º da Lei nº 13.874, de 20 de setembro de 2019, e o art. 6º da Lei nº 13.848, de 25 de junho de 2019. Brasília, DF. Recuperado dehttp://www.planalto.gov.br/ccivil_03/_ato2019-2022/2020/decreto/d10411.htm http://www.planalto.gov.br/ccivil_03/_at...	1.38	27.6%	1.16

Criteria	Supporting Literature	Average Score (n = 21)	(%)	Std. Dev.
45. RIA reports published on agencies’ websites	Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... )	4.71	94.3%	0.72
46. RIA understandable to the lay citizen	Ellig and McLaughlin (2012Ellig, J., & Mclaughlin, P. A. (2012). The Quality and use of regulatory analysis in 2008. Risk Analysis, 32(5), 855-880. Recuperado de https://doi.org/10.1111/j.1539-6924.2011.01715.x https://doi.org/10.1111/j.1539-6924.2011... )	3.95	79.0%	0.80
47. Executive summary	Decreto nº 10.411, de 30 de junho de 2020Decreto nº 10.411, de 30 de junho de 2020. (2020). Regulamenta a análise de impacto regulatório, de que tratam o art. 5º da Lei nº 13.874, de 20 de setembro de 2019, e o art. 6º da Lei nº 13.848, de 25 de junho de 2019. Brasília, DF. Recuperado dehttp://www.planalto.gov.br/ccivil_03/_ato2019-2022/2020/decreto/d10411.htm http://www.planalto.gov.br/ccivil_03/_at...	2.36	47.3%	1.62
48. Social participation during RIA	Ballantine and Devonald (2006Ballantine, B., & Devonald, B. (2006). Modern regulatory impact analysis: the experience of the European Union. Regulatory Toxicology and Pharmacology, 44(1), 57-68. Recuperado de https://doi.org/10.1016/j.yrtph.2005.06.016 https://doi.org/10.1016/j.yrtph.2005.06.... ); Radaelli (2004Radaelli, C. M. (2004). The diffusion of regulatory impact analysis - best practice or lesson-drawing? European Journal of Political Research, 43(5), 723-477. Recuperado dehttps://doi.org/10.1111/j.0304-4130.2004.00172.x https://doi.org/10.1111/j.0304-4130.2004... )	1.86	37.1%	1.65
49. Participation of affected groups in RIA	Staroňová et al. (2007Staroňová, K., Pavel, J., & Krapež, K. (2007). Piloting regulatory impact assessment: a comparative analysis of the Czech Republic, Slovakia and Slovenia. Impact Assessment and Project Appraisal, 25(4), 271-80. Recuperado dehttps://doi.org/10.3152/146155107X246314 https://doi.org/10.3152/146155107X246314... )	1.43	28.6%	1.69
50. Contributions to RIA received and analyzed	Staroňová (2010Staroňová, K. (2010). Regulatory impact assessment: formal institutionalization and practice. Journal of Public Policy, 30(1), 117-36. Recuperado de https://doi.org/10.1017/S0143814X09990201 https://doi.org/10.1017/S0143814X0999020... ); Staroňová et al. (2007)Staroňová, K., Pavel, J., & Krapež, K. (2007). Piloting regulatory impact assessment: a comparative analysis of the Czech Republic, Slovakia and Slovenia. Impact Assessment and Project Appraisal, 25(4), 271-80. Recuperado dehttps://doi.org/10.3152/146155107X246314 https://doi.org/10.3152/146155107X246314...	0.90	18.1%	1.45

[1] Flavio Saab - Undergraduate degree in Economics; Master and Ph.D. student in Administration at University of Brasília (UnB); Collaborating Professor at the Department of Public Policies Management (GPP/UnB); Federal civil servant at the National Sanitary Surveillance Agency (Anvisa). E-mail: flaviosaab@gmail.com

Brazil

Brazil

What is the quality of the regulatory impact analysis prepared by Brazilian regulatory agencies?

Abstract