META-RESEARCH: THE IMPLEMENTATION OF EDUCATIONAL PUBLIC POLICIES IN THE FIELD OF EDUCATION

ABSTRACT:

This paper aims to analyze the theoretical references and analytical concepts (called descriptors) mobilized by studies on implementing public policies in the field of education. Considering that academic production on the theme of implementation in the Education field is scarce, we conducted a meta-research based on the definition of some descriptors and theoretical references from the Political Science field. We tried to verify, surveying the academic production in the Education field — theses, dissertations, and articles — which was the presence and how these references were mobilized. The time frame was limited to the years 2018 and 2019, and, in total, we identified 10 articles and 25 theses and dissertations. The analysis and categorization of this material, identifying the theoretical-conceptual referential used, indicated that among the selected theses and dissertations, there is a higher concentration of studies on implementing agents and, among the articles, a higher concentration of studies on governance structures. From the results, we consider that there is, among researchers in the educational area, an approach “under construction” to the specific literature of the Political Science field on implementation.

Keywords:
implementation of public policies; meta-research; education; political science; knowledge production

RESUMO:

Este trabalho tem como objetivo fazer uma análise dos referenciais teóricos e conceitos analíticos (denominados de descritores) mobilizados pelos estudos sobre implementação de políticas públicas no campo da Educação. Considerando que a produção acadêmica sobre o tema da implementação, no campo da Educação, é escassa, realizamos uma metapesquisa a partir da definição de alguns descritores e teóricos de referência do campo da Ciência Política. Procuramos verificar, no levantamento da produção acadêmica no campo da Educação - teses, dissertações e artigos -, qual era a presença e como eram mobilizados esses referenciais. O recorte temporal se limitou aos anos de 2018 e 2019 e, no total, identificamos 10 artigos e 25 teses e dissertações. A análise e categorização deste material, identificando o referencial teórico-conceitual utilizado, indicou que entre as teses e dissertações selecionadas há uma maior concentração de estudos sobre os agentes implementadores e, entre os artigos, uma maior concentração de estudos sobre estruturas de governança. A partir dos resultados, consideramos que há, entre os pesquisadores da área educacional, uma aproximação “em construção” à literatura específica do campo da Ciência Política sobre implementação.

Palavras-chave:
implementação de políticas públicas, metapesquisa, educação, ciência política; produção de conhecimento

RESUMEN:

Este trabajo tiene como objetivo hacer un análisis de las referencias teóricas y los conceptos analíticos (llamados descriptores) movilizados por los estudios sobre la implementación de políticas públicas en el ámbito de la Educación. Considerando que la producción académica sobre el tema de la implementación en el campo de la Educación es escasa, realizamos una meta-investigación a partir de la definición de algunos descriptores y teóricos de referencia del campo de la Ciencia Política. Intentamos verificar, en el relevamiento de la producción académica en el campo de la Educación - tesis, disertaciones y artículos - cuál era la presencia y cómo se movilizaban esas referencias. El corte temporal se limitó a los años 2018 y 2019 y, en total, identificamos 10 artículos y 25 tesis y disertaciones. El análisis y categorización de este material, identificando el referencial teórico-conceptual utilizado, indicó que entre las tesis y disertaciones seleccionadas hay una mayor concentración de estudios sobre agentes ejecutores y, entre los artículos, una mayor concentración de estudios sobre estructuras de gobernanza. A partir de los resultados, consideramos que existe, entre los investigadores del área educativa, un acercamiento "en construcción" a la literatura específica del campo de la Ciencia Política sobre la implementación.

Palabras clave:
implementación de políticas públicas; metainvestigación; educación; ciencia política; producción de conocimiento

INTRODUCTION

The national academic production in Educational Policy primarily uses the theoretical contribution of the policy cycle approach, developed by Stephen Ball and Richard Bowe (1992). We can say that the authors highlight that a public policy is constituted by a continuous cycle formed by three contexts — influence, text production, and practice — and that, although they are interrelated, they are not necessarily temporal or sequential and therefore are characterized as a cycle. This approach is widely used for investigations into the trajectory of policy, which involves four phases: agenda, formulation, implementation, and results.

In the field of academic research, the phases of the agenda, formulation (analysis of the legal texts of the policy), and policy outcomes (analysis of the effects and impacts that the policy has on the quality of education) are the focus of many studies. In contrast, research on the implementation phase is still incipient in the various areas of knowledge (Lotta, 2015), especially in the educational field (Oliveira, 2019).

Despite the few studies conducted on the implementation of educational policies, some advances can be observed in the field of Political Science, with researchers developing studies on the implementation of High School Reform in Brazilian states, for example (Lotta et al., 2021), and in the field of education, which has, since 2014, a network of researchers, called the Network of Studies on Implementation of Public Education Policies (REIPPE)¹, which develops studies on the implementation of educational policies in various national contexts.

The scientific gap in the implementation of educational policies is the object of this work, whose purpose consisted of a survey of academic production in the years 2018 and 2019 on the topic of the implementation of educational policies. The main objective was to map the number of studies on the subject in the qualified scientific production and analyze the theoretical contributions used. To this end, we analyzed the articles published in journals classified as A1 of Qualis of the Coordination for the Improvement of Higher Education Personnel (CAPES) and the theses and dissertations published in the CAPES portal in the reference period.

Data collection and analysis took place in two stages: the first, of quantitative nature, for the survey and selection of the number of articles, theses, and dissertations published in 2018 and 2019, and the second, of qualitative nature, for the analysis of the theoretical contribution. Following this organization of the empirical work, the text is divided into four sections, including this brief introduction. In the next section, we present the methodology used, including the description of how the survey and the selection of studies on implementing educational policies took place. In section three, we analyze the selected studies to identify the main theoretical contributions adopted in the articles, theses and dissertations analyzed. Moreover, we present some final considerations in the fourth section.

METHODOLOGY

This study is aligned with the methodological approach called meta-research, or research of research (Mainardes, 2018), which, unlike the literature review, the state of knowledge and state of the art, is concerned with identifying and analyzing the theoretical foundations of the studies developed in a given area or field of research. Mainardes (2018, p. 306) explained the difference between meta-research and the other methodological approaches:

Meta-research differs from literature review, systematic review, state of the art, and state of knowledge. Meta-research is discipline (area or field) oriented and is engaged with research advances in the discipline (area or field). Review studies (literature review, systematic review, state of knowledge, state of the art) are oriented towards research projects: previous research is reviewed primarily as a preparatory phase for undertaking new research projects. Through the literature review, the researcher becomes familiar with the knowledge already built on the research topic and identifies possible gaps that need to be filled in new research projects. Generally, literature review papers are more concerned with synthesizing the results of a body of research, paying less attention to the theoretical foundations of the research reviewed. Meta-research, on the other hand, seeks to analyze, especially, the theoretical foundations of research and their significance in the theoretical development of the research field.

In this sense, the meta-research focuses on analyzing theoretical references mobilized by research on a specific theme. Tello and Mainardes (2015, p. 169) point out that meta-research

a) allows the expansion of the knowledge produced in the field; b) helps in the reflection on the possibilities of the criteria of scientificity and epistemological surveillance and c) contributes to intensifying the exchange of information and criticism on the production of knowledge in the field.

Thus, it enables “identify how researchers in the field are developing their ideas and propositions, based on certain theoretical references” (Tello and Mainardes, 2015, p. 169). Therefore, meta-research is not to be confused with a literature review because its purpose is to systematize the work on a specific theme and analyze their theoretical and epistemological perspectives. In contrast, the literature review focuses on the results obtained by research.

Considering that this study aims to map the research conducted in 2018 and 2019 on the implementation of educational policies in the field of education, the work is close to a meta-research, as it seeks to identify the main theoretical approaches of the studies aligned to the theme of policy implementation in the field of education. It is not to synthesize the research results but to understand which concepts mobilized and the main theoretical references used in the studies on implementing educational policies in the field of education.

For this, we first selected studies published in the area from the definition of some descriptors (keywords) and reference authors who investigate the theme of implementation in the field of Political Science. The definition of which descriptors and which authors we would use as reference was based on the reading of various research studies (articles, theses, and dissertations) developed in the field of Political Science on the theme of policy implementation in areas such as Labor, Health and Social Assistance² . This literature was presented to us especially within the scope of REIPPE - which has been using this theoretical and methodological framework to develop investigative work. In recent years, this study network has promoted a fruitful interaction between researchers from other areas (especially Health and Social Assistance), which has contributed to expanding the theoretical and methodological framework in the field of policy implementation based on studies dedicated especially to this phase of the classic policy cycle. Thus, from the contact with this literature, we elected 13 reference authors (Brazilian and foreign) and 18 descriptors to guide our survey, as presented in Box 1.

Box 1
Descriptors and reference authors

Obviously, the academic discussion on policy implementation is not exhausted in this set of descriptors and authors, although they are undoubtedly important concepts and references for studies on public policy implementation. After defining the descriptors and reference authors, we dedicated ourselves to selecting the studies that focused on the implementation theme and were published in the field of education. Next, we present the selection process of articles and then the selection process of theses and dissertations.

Article Selection

The articles were selected in two stages: first, we selected the journals and then the articles. For the journals’ selection, we used the classification in CAPES as a criterion. As the objective was to map qualified productions, we used as a first filter the A1 classification in the Qualis CAPES of the 2017-2018 quadrennium³, in which we identified 107 national and international journals and visited their respective online pages to map the focus and scope of each of them, which should be, explicitly, education and all the sub-areas of Social Sciences (Anthropology, Sociology, and Political Science), since education is a Social Science and, therefore, studies on education are publishable in these journals. This filter reduced our list to 35 journals, followed by the application of a third filter: selecting only national journals, since we intended to verify the national academic production in educational policy implementation. We arrived, then, at a list of 20 journals.

Once the journals were selected, we defined the criteria for article selection. We chose to analyze all articles published in continuous demand or open call, excluding the articles published in thematic dossiers and the publications related to interviews, reviews, and presentations/editorials of the journals. The selection of the articles was based on two subsequent criteria: at least one of the 18 descriptors, as presented in chart 1, should be identified in the abstract and/or in the keywords of the articles and, once one of the descriptors was identified, we looked at the bibliographic references in order to check whether any of the 13 reference authors were cited in the articles. Hence, the articles that presented at least one descriptor (in the abstract and/or keywords) and one of the authors (in the bibliographic reference) were selected for reading. In total, 25 articles were selected. Table 1 shows the distribution of these articles in the analyzed journals, revealing the low representativeness of the theme in the analyzed publications. As observed, of the 1747 articles published in the period in the 20 selected journals, only 25 met the criteria defined in the study, corresponding to only 1.43% of the analyzed production. This data reveals that the qualified journals publish few articles addressing implementing public educational policies from the concepts and theoretical frameworks defined in the study.

Table 1
Number of articles selected per journal

No.	Journal	Scope of the journal	No. of articles - 2019	No. of articles - 2018	Selected articles
1	Caderno CRH	Studies on current affairs in the Social Sciences.	15	14	1
2	Cadernos Cedes	Studies on Education.	25	23	0
3	Cadernos de Pesquisa	Studies on Education, Gender and Ethnicity.	53	38	1
4	Currículo Sem Fronteiras	Studies on critical and emancipatory education, which strengthens the dialogue between Portuguese-speaking countries.	25	23	0
5	Dados - Revista de Ciências Sociais	Studies on substantive issues in society and politics.	28	29	5
6	Educação & Sociedade	Studies on the relationship between education and society.	45	55	0
7	Educação e Pesquisa	Studies in Education.	105	106	0
8	Educação e Realidade	Studies in Education.	51	57	1
9	Educação em Revista	Studies in Education.	35	30	0
10	Educar em Revista	Studies in Education.	48	48	0
11	Ensaio - Avaliação e Políticas Públicas em Educação	Studies on evaluation and public policies in education.	40	59	2
12	Interfaces da Educação	Studies in Education.	50	51	0
13	Revista Brasileira de Educação	Studies in Education.	57	89	3
14	Revista Brasileira de Estudos Pedagógicos (RBEP-INEP)	Studies in Education.	36	36	0
15	Revista Brasileira de Políticas Públicas	Public Policy Studies.	54	105	3
16	Revista de Políticas Públicas da UFMA	Public Policy Studies.	48	50	6
17	Revista de Sociologia e Política	Studies on the political practice of social agents.	26	27	3
18	Revista Sociedade e Estado	Social Science Studies.	34	27	0
19	Sociologia, Problemas e Práticas	Studies on the areas of the Social Sciences.	19	19	0
20	Sociologias	Studies that promote the exchange between social scientists from Brazil and abroad.	32	35	0
Total			826	921	25

• Source: Prepared by the authors.

In the analysis of the articles we were led to make another filter, empirically defined in the dynamic reading of the articles. This is because some articles had extended thematic scope not dealing with implementing educational policies. This is the case of the study of Lotta (2018), which used several descriptors and reference authors (even one of these references), although it dealt with implementing policies in the field of Health. The work of Lavalle, Rodrigues, and Guicheney (2019) is also in this exclusion category because it addresses a policy in the field of Social Assistance. The researches of Moreira (2019) and Sarue (2018) are also examples of studies that met the criteria defined in the research but are not aligned with the field of education: the first deals with the institutional arrangements of the Justice System. The second investigates the theme of urban governance from the implementation of Porto Maravilha in the city of Rio de Janeiro. Finally, the study by Motta (2019) was also excluded from the analysis because it deals with the discretionary action of police teachers who work in Pacifying Police Units (UPPs) in the community Cidade de Deus, being a study aligned with the area of Public Security. The cases exemplified here reinforce the predominance of research on implementing policies in fields other than education and highlights the scientific gap of studies on implementing educational policies.

In the end, we considered for the qualitative analysis only ten articles (i.e., 0.57% of the total articles published in 20 A1 journals in the years 2018 and 2019); these ten articles were fully read the analysis of the theoretical approaches employed in the studies.

Selection of theses and dissertations

To survey the theses and dissertations, we searched the CAPES database, which functions as a repository of the theses and dissertations defended and published by Post-Graduation Programs in the country. Due to the platform’s format, this selection was made by a different procedure than the one adopted for selecting articles. In the portal, a filter option allows the search for the papers by keywords. In this search, we elected the descriptor implementation and the temporal cut-off so that the platform would provide only the papers published in 2018 and 2019.

The third filter was to determine in which areas of knowledge the search would be conducted, among which we defined all the options that contained the words Education or Teaching. They were: 1) Education; 2) Adult Education; 3) Education in Urban Peripheries; 4) Special Education; 5) Rural Education; 6) Teaching; 7) Science and Mathematics Teaching; 8) Professionalizing Education; 9) Learning education. After applying these criteria, we obtained 998 published papers, among which we selected those that mentioned the word “implementation” in the title or abstract.

Table 2
Number of theses and dissertations published in the CAPES catalog

Theses		Dissertations		Total
2018	2019	2018	2019
112	92	467	327	998

• Source: Prepared by the authors.

With the application of this new filter, we selected those that dealt with some educational policy, regardless of what it was. The number of studies decreased significantly: in 2018 we found 60 dissertations and 17 theses, and in 2019 we found 24 dissertations and six theses, totaling 107 papers. Of these, we looked at the bibliographic references to identify the presence of at least one of the reference authors (Box 1). With this last criterion adopted, we selected 37 works, which ended up being reduced to 29, because seven had no authorization for disclosure. One was excluded from the analysis because it dealt with the institutional arrangement of implementation of the Guandu-RJ Watershed Committee, being, therefore, a dissertation outside the area of education. However, it was developed in the Agricultural Education Graduate Program of the Federal Rural University of Rio de Janeiro (UFRRJ). Thus, we analyzed a total of 29 studies, 17 dissertations, four theses published in 2018, six dissertations, and two theses published in 2019. The table below presents the regional distribution of the analyzed works.

Table 3
Regional distribution of theses and dissertations selected and published in 2018 and 2019

South			Southeast		Midwest		Northeast
Institution	No	Institution	No	Institution	No	Institution	No
UFRGS	1	UFRJ	1	UFGD	1	UESC	1
UNOESC	1	UNIRIO	1	UNICENTRO	1	UECE	1
UFFS	1	UERJ	2
		PUC-Rio	3
		PUC-SP	1
		UNICID	2
		UNICAMP	5
		UNIMES	1
		UNINOVE	1
		UFJF	5
3	3	10	22	2	2	2	2

• Source: Prepared by the authors.

It calls attention to the unequal distribution of theses and dissertations on the theme in the Brazilian regions. We observed that 29 works were produced in 17 Graduate Programs, the majority (22 works) in Programs of institutions of southeastern Brazil, three in the southern region, two in the northeast region, and two in the central-western region, not existing works produced by institutions of the northern region in this selection. In addition, we note that six institutions are private and 11 are public. Once we selected the ten articles and the 29 theses and dissertations, we proceeded to analyze the theoretical foundations of the studies, presented and discussed in the next section.

THEORETICAL CONTRIBUTIONS OF STUDIES ON THE IMPLEMENTATION OF EDUCATIONAL PUBLIC POLICIES

Analysis of the articles

The ten articles selected for analysis were published in seven journals, being the Journal of Public Policy of UFMA having the highest number of selected articles. Box 2presents the articles.

Box 2
Selected articles: authors and journals of origin

The ten articles deal with different thematic scopes; some do not refer directly and explicitly to the policy implementation phase. This is the case, for example, with the work of Torres, Camelo, and Castro (2019), which investigates failures of coordination in elementary education policy through the creation of indicators of permanence/migration of students between the different education networks to estimate the effect of this indicator on student performance in the Prova Brasil. To this end, the study mobilizes references that discuss the Brazilian State’s federative model and the ways the collaboration regime is operationalized in various contexts. In this sense, the work is not directly and explicitly aligned with issues related to the implementation phase of the policy (e.g., the institutional arrangements of implementation, practices, interactions, and the influence of discretionary action of implementing agents at the mid-level and street level, among others), situating itself more in the field of policy evaluation, since it aims to verify the effects of the migration/permanence policy on student learning.

Collaborative actions between federated entities are also a theme discussed in the works of Batista (2018) and Veloso (2019), although the analytical lenses are different. The first one approaches the collaboration regime to investigate the implementation of the Articulated Actions Plan (AAR) in the municipalities of Rio Grande do Sul State. The study focuses on the implementation of AAR, having as a theoretical reference the policy cycle, as defined in the works of Stephen Ball and colleagues, not mobilizing the concepts and authors of reference of this study to analyze the implementation of that policy. The authors found in the search (Martha Arretche and Fernando Luiz Abrucio) appear only in the introduction of the text, in the discussion on federative relations. The second article investigates the Professional Learning policy, pointing out a mismatch between the federal policy for the professional qualification of young people and the demands and realities of local contexts.

The author advocates that this gap can be mitigated if the Professional Learning policy is implemented based on the collaboration regime in a decentralized manner, where policy execution takes into account the decisions of subnational governments, thus ensuring greater policy effectiveness. The study is aligned with themes such as decentralization and local state capabilities in the light of Fernando Luiz Abrucio and Martha Arretche.

Unlike the works of Torres, Camelo, and Castro (2019), Batista (2018), and Veloso (2019), the articles by Laurentino and Alloufa (2019), de Lisboa, Mendes, and Lima (2019) and by Gomes and Passos (2018) do not have as background the federalist context in the provision of the educational service, as they investigate policies formulated by the federal government in federal educational institutions, not dispensing, for its implementation, collaborative actions between federated entities.

The work of Laurentino and Alloufa (2019) addresses the implementation of Pronatec (National Program for Access to Technical Education and Employment) in the Federal Institute of Education, Science, and Technology of Paraíba - IFPB - Picuí campus from the Public Policy Sequential Approach (or Public Policy Cycle), based on the contributions of Klaus Frey and Leonardo Secchi. The authors emphasize the relevance of this approach for the “analysis of the life of a policy,” since public action can be divided into several phases (problem identification, agenda formation, formulation of alternatives, decision-making, implementation, evaluation, and termination), being the focus of the study the implementation phase. This cutout is justified based on the contributions of Sonia Draibe, who highlights the importance of studying the implementation phase to understand and identify not only the possible failures of implementation processes but also the mistakes in decision-making in the policy formulation phase. Thus, the authors use the analysis technique Anatomy of the General Implementation Process, proposed by Sonia Draibe for the evaluation of systems and sub-processes of implementation, namely: Managerial and Decision Making System; Dissemination Processes; Selection Systems; Training Systems; Internal Monitoring and Evaluation Systems; Logistic and Operational Systems. These subsystems or subprocesses have interconnections and interdependencies, and it is possible to verify interference between them and in the policy design itself, contributing to the knowledge of the reasons that lead the policy to succeed or fail in its results.

The work of Lisboa, Mendes, and Lima (2019), and Gomes and Passos (2018), defines public policies as a cycle, with implementation being one of the phases of this cycle, in which both works focus on the discretion of the implementing agents. On the one hand, the work of Lisboa, Mendes, and Lima (2019) investigates the implementation of the Policy for the Attendance of People with Specific Educational Needs (NAPNEs) in Brazilian Federal Institutes based on the classical approaches to policy implementation — top-down and bottom-up —, presenting the discretionary element of the actions of the implementing agents of street level and middle level in the implementation of the policy from authors such as Gabriela Lotta, Maria das Graças Rua, Lars Tummers and Victor Bekkers, Estela Najberg, Nelson Barbosa, Luciana Lima, and Luciano D’ascenzi. Meanwhile, Gomes and Passos (2018) analyze the National Student Assistance Program (PNAES) implementation in Federal Institutes, discussing the concept of the discretionary nature of the implementing agents in light of Sonia Draibe, Pedro Luiz Barros Silva and Marcus André Barreto de Melo. In particular, the study focuses on translating the national policy into local regulations (of each researched FI).

The other three studies have as their analytical scope the concept of governance. Within the scope of the New Public Management and the idea of Network Governance, the study of Tripodi and Sousa (2018) discusses the privatizing logic that has guided the formulation and implementation of public policies for education in Brazil from the category of state permeability. The concept of network governance is defined based on the works of Roderick Arthur and William Rhodes and the contributions of Eli Diniz and Jan Kooiman. The concept of state permeability, on the other hand, is approached in the light of Fernando Henrique Cardoso, Marcus André Melo and Eduardo Marques. According to the authors, the concept of permeability refers to two movements of State action, which take place concomitantly in the implementation of educational policies. The first refers to the initiatives of the public power to assimilate, adapt and implement knowledge and management logics of the private sector, and the second is the transfer of the provision of educational services from the public sector to the private sector. The idea of state permeability is articulated with the administrative trend of New Public Management (NPM) and Network Governance, the latter understood, based on the ideas of Jan Kooiman, as “a set of government efforts carried out in various ways and also by different political or social actors, public or private, occurring between various levels of public administration” (Tripodi and Sousa, 2018, p. 238). In this sense, the regulation of policies starts to be exercised by actors that are located outside the state apparatus. It is this dispersion of regulation that underlies the idea of network governance.

If the work of Tripodi and Sousa (2018) focused on a theoretical discussion, the study of Tripodi, Senna, and Rodrigues (2018) made use of the concept of state permeability for empirical analysis, namely: to investigate the various forms of partnerships established between the State and civil society (called third sector or non-state public) for the provision of Early Childhood Education in a sample of Brazilian municipalities. As in Tripodi and Sousa (2018), the study of Tripodi, Senna, and Rodrigues (2018) discusses the concept of state permeability in the light of Fernando Henrique Cardoso and Eduardo Marques in the context of NPM but does not discuss the idea of network governance. The authors mobilize the concepts of state permeability and institutional arrangements to map trends of supply modes of early childhood education in the daycare stage, explicit in Municipal Education Plans, formulated and/or reformulated in 2015.

The second article anchored in the concept of governance is the work of Martins and Junior (2018), which starts from the hypothesis that the formulation and implementation of the National Policy of Museum Education (PNEM) are outlined by actors who organize themselves in networks and associations. For the authors, the formulators and implementers actors coincide in the execution of PNEM precisely because they act in networks — a concept defined from the contributions of Michael McGuire and Robert Agranoff and Paulo Calmon and Arthur Costa as “a set of relationships between heterogeneous and interdependent actors, aimed at acting in the same subsystem of public policies” (MARTINS and JUNIOR, 2018, p. 320). These actors, organized into networks, act within a governance structure with formal and informal rules and several forms of interpretation and implementation. This study focuses on analyzing the characteristics of the network’s governance forms that formulate and implement the PNEM.

The studies of Tripodi and Sousa (2018) and Martins and Junior (2018) used similar definitions of network governance, although the reference authors are different. The third article that brings the concept of governance as an analytical category is the work of Vieira, Plank, and Vidal (2019). To understand the reasons that contribute to the good performance of learning in Ceará State, the work drew on the concepts of multiscale governance, defined by Paul Stephenson, Roderick Rhodes, and Bob Jessop, among others, and path dependency, addressed by Paul Pierson, James Mahoney, Scott Page, and Ian Greener. One of the central aspects is the collaboration regime established in providing educational services between the state government of Ceará and its municipalities. Collaborative forms of collaboration between the federative entities in education provision have a long history. They have, over time, created specific conditions that have led the state to implement its forms of government and governance with its municipalities. These specificities explain, to some extent, the good learning outcomes achieved in Ceará in recent years.

The following box seeks to synthesize the theoretical approaches found in the selected articles.

Box 3
Theoretical approaches of the articles

Analysis of theses and dissertations

The survey of the theses and dissertations analyzed reveals the need to produce abstracts and to define keywords that effectively reflect the research theme, the object of investigation and the theoretical scope used as an analytical lens of the empirical data. This is because the imprecision of the data can cause biases in the analysis and the very collection of data from studies on literature review, state of the art, state of knowledge, and meta-research.

This finding was made when we found that four studies did not address implementing public educational policies. In fact, the studies met the criteria established for the search but did not fit as studies on implementing educational public policies. One study carried out a meta-research on the concept of accountability; the other analyzed the academic production of the collaboration regime in Brazil; the third investigated the normative documents on policies aimed at encouraging reading; and the fourth focused on the reflections of large-scale assessments in teachers’ pedagogical practices. From the reading of the abstracts, it can be said that the first three papers were dedicated to surveying research, articles, or texts of specific policies (respectively), not addressing, in fact, the implementation of educational policies. The fourth work discusses the concept of large-scale assessment in educational legislation and teachers’ reactions to its application without discussing the implementation of educational assessment policies. The four papers were thus excluded from the selection.

It can then be said that a total of 25 theses and dissertations focused on the public policy implementation phase, with five (Sousa, 2019; Faria, 2018; Marquet, 2018; Nascimento, 2018; Mesko, 2018) using the policy cycle developed by Stephen Ball as an analytical lens.

In particular, Mesko’s dissertation (2018) uses the contributions of the policy cycle to deepen the discussion about the New Public Management and the managerialist model of the Brazilian State and the emergence of governance networks that occur from privatization initiatives - endogenous and exogenous - of Basic Education. Based on the work of Stephen Ball and Deborah Youdell⁴, Mesko (2018, p. 75-76) defines endogenous privatization as the incorporation into the public sector of “conceptions, ideas, and practices originating from the private sector,” while exogenous privatization is understood as “a new way of privatization in which consultancies and social organizations (civil society) design programs in public education policies and public education systems.”

This complex relationship between market, state, and public sector has been called by Ball (2013) new philanthropy. Based on this interaction, the Education Commitment Program of São Paulo — the object of investigation of Mesko’s research (2018) — is implemented.

The theme of managerialism and NGP is also present in the study of Silva (2018), which addressed the implementation of the All for Pernambuco Program. It was characterized as a managerialist-oriented policy and also in the dissertation of Marcelino (2018), where the State Reform and the managerialist model of management appear as a backdrop to situate the development of the Program to Support the Qualification of the Federal University of Juiz de Fora (Proquali/UFJF). In other words, in these two works, the issue of State Reform, managerialism, and NPM is not the object of investigation, but they do appear to describe the context in which the Programs are developed. In both works, the author of reference is Fernando Luiz Abrucio, who is also used in the dissertations by Lins (2018) and Matos (2018) to address the theme of federalism. The work of Moreira (2018, p. 71) uses the contributions of Martha Arretche to address the issue of decentralization of the educational supply in the Brazilian federative context, highlighting the limits of municipal administrations “due to the low economic capacity, significant dependence on fiscal transfers and weak administrative tradition.”

In general, we noticed a certain predominance of studies that use the concept of discretionary action of the implementing agents as a theoretical foundation. Some studies are dedicated to the analysis of the discretionary action of street-level and mid-level implementing bureaucrats (Mota, 2018; Camargo, 2019; Silva, 2019; Gomes Moreira, 2018), while others focus only on the discretionary action of street-level agents (Almeida, 2019; May, 2018; Gusto, 2018). These studies drew primarily on the contributions of Michael Lipsky and Gabriela Lotta.

These theses and dissertations are aligned with the bottom-up theoretical approach, which understands the policy as a continuum in which the action of the implementer transforms the rules of the policy formulation. From its practices and interactions, it is understood that the implementing agent modifies the policy formulation in the execution of its tasks. Although rules and norms are provided for in the formulation phase, the implementing agents mobilize their own beliefs and values when making decisions in the implementation processes, making these agents executors and policymakers.

The literature categorizes these agents as street-level bureaucrats — those who are in direct and immediate contact with policy users, such as doctors, teachers, and police officers — and as mid-level bureaucrats - defined as those who connect the formulation and implementation phases, being links between high-level agents and street-level agents. When addressing the implementation of financial autonomy in schools in Ceará through the PDDE, Gomes Moreira (2018), for example, discusses his research problem from the phases of the classic cycle of political science, emphasizing the bottom-up and top-down approaches from Hill, locating this discussion, although in the description of the evaluation phase and not the implementation. When analyzing the empirical evidence and discussing the financial autonomy of schools, the author describes the spaces of decision-making and the reinterpretations of the policy, mobilizing the discussion about the constraints on the decisions of bureaucrats from Gabriela Lotta.

Inserted this theoretical perspective, Mota (2018), Camargo (2019), Silva (2019), Almeida (2019), May (2018), and Gusto (2018), in addition to Gomes Moreira (2018), have developed studies on the implementation of educational policies in Basic Education. Among the selected studies, none focused on Higher Education policies based on this theoretical foundation.

The dissertation of Antoniolli (2019), although it does not use the discretionary nature of the actions of implementing bureaucracies as an analytical lens, makes use of the concept of autonomy to address the actions of implementing bureaucrats in the National School Feeding Programme (PNAE). In light of the contributions of Martha Arretche, the research understands the implementing agents as those responsible for implementing the policy, translating their conceptions and references into concrete intervention actions. Thus, even without using the concept of discretionary action, the research of Antoniolli (2019) deals with the same thematic scope as the other works.

Among the selected theses and dissertations, two works monitor or evaluate implementation. One of them is the one by Santos (2018, p. 8), which, although it aims to bring up “the monitoring of public policies to support the analysis of the implementation of the EI public policy,” it does so from other theoretical references⁵ different from those defined in this study. The author works on the concept of “implementation monitoring” based on Veronica Cruz, Heloisa Höfling, and Luciana Lago. Ribeiro (2018) also analyzed the implementation of a public policy focused on Early Childhood Education, discusses the concepts of analysis and evaluation, and proposes to work with the “process evaluations” based on the considerations of Sonia Draibe.

An example of a study that brought a different approach from those proposed in the descriptors presented is the dissertation by Antunes (2018), which sought to analyze the implementation of the National Student Assistance Program (PNAES) in two Federal Institutes of Education, Science and Technology (IFs). The author articulated it to the analysis of public policies (policy analysis) and worked with the logic of backward mapping from Elmore, to classify the implementation design in the two cases studied. This perspective (forward mapping and backward mapping) would be another option to the bottom-up approach traditionally developed by Michael Lipsky.

The work of Mercês (2018) can be an example of overlapping references since it focuses on the perception of school managers about the implementation of continuing and in-service teacher training policies from different — and not necessarily complementary — sources:

In conclusion, the policy cycle approach presented by Mainardes, the discussion on difficulties with implementation raised by Condé, the top-down and bottom-up currents mentioned by Segatto and the idea of the power of interaction present in Lotta bring valuable contributions to this study. (MERCÊS, 2018, p. 59)

However, these references are timidly taken up in the data analysis, which is based more on studies about teacher training. The study of Mengalli (2018), in turn, brings an analysis of the use of monitoring and measurement software as tools for implementing educational public policies. The thesis brings in the title the descriptor “implementation,” which appears several times throughout the text about an action — sometimes of the state, others of the local manager and even of the public — that effectively monitors the materialization of public policy. However, the author does not resort to any mentioned authors to discuss or conceptualize this implementation process. Nonetheless, it mobilizes the concept of social networks from Martins and Lotta: “social networks can be a robust alternative for understanding and evaluating education policies at the local level” (MARTINS and LOTTA, 2010, p. 858, apud MENGALLI, 2018, p. 66).

The dissertation of Nascimento (2018) provided important contributions to the scientific production on implementing educational public policies in that it makes an extensive and rigorous literature review on the subject, mobilizing several concepts, including institutional arrangements for implementation and implementing bureaucracies of street level and mid-level and several reference authors, such as Gabriela Lotta, Michael Lipsky, Roberto Pires, Paul Sabatier, Steven Maynard-Moody, Michael Musheno, Richard Matland, Martha Arretche, and Fernando Luiz Abrucio. Having as investigative scope the institutional implementation arrangement of the Institutional Program for Teaching Initiation Scholarship (PIBID), Nascimento (2018) analyzes the normative policy documents fulfilling three of the five steps proposed in the analytical model of Gomide and Pires (2014). This model points out that the analysis of the institutional arrangement of policy implementation should follow the following steps: 1) Identification of the policy objectives; 2) Identification of the actors involved directly or indirectly in the policy; 3) Identification of the processes, mechanisms, and spaces in the policy management; 4) Evaluation of the state capabilities produced by the arrangement; 5) Evaluation of the effects of the implementation arrangements.

Having presented the theoretical approaches of the selected theses and dissertations, the following box seeks to synthesize the trends of the approaches found in the analyzed material.

Box 4
Theoretical approaches of theses and dissertations

CONCLUDING REMARKS

Based on an extensive survey conducted in international and national publications, Lotta and colleagues (2018) searched in the most important journals in the field of Public Administration, Public Policy, and Political Science for articles that addressed “implementation,” “bureaucracies,” and “bureaucrats” associated with “public policies.” The time frame of 2003-2017 was the same in the search for international and national journals. In addition to the huge difference in the volume of international (218) and national (21) publications selected, it should be noted that the international production was categorized into 11 thematic groups, while the national production predominantly addressed three areas: street-level bureaucrats (including the use of their discretion and their values); governance structures (institutional arrangements and federative layers); others (case studies of specific policies and programs). The authors consider the gap and fragmentation in academic production on public policy implementation, especially in the field of Political Science and Administration: “The analysis of this meager national literature demonstrates that, in light of what has happened internationally in the field, there is still much fragmentation in studies on implementation” (LOTTA et al., 2018, p. 798).

In view of the research on the theme carried out in the field of education, we noted that this production is even scarcer, as we seek to present in this work, from the search for published articles and theses and dissertations defended in the period 2018-2019. Regarding the trends of theoretical approaches, we highlight that among the theses and dissertations, there was a greater concentration of studies on implementing agents and, among the articles, a greater concentration of studies on governance structures. This concentration coincides with that Lotta and colleagues (2018) found in the survey that had a broader scope in production areas. The third group mentioned by the authors, which included analyses of specific policies and programs, was not applied in our analysis because these studies were diluted in all the other categories since the vast majority of papers were dedicated to analyzing one policy or program.

It is worth highlighting the presence (more voluminous among the theses and dissertations) of works based on the theoretical and methodological framework of the policy cycle by Stephen Ball and colleagues, which is very present in the educational field. We also highlight the works that focused on the study of evaluation or monitoring of implementation, proposing an intersection between the phases of the classic policy cycle, where the evaluation of the process is present. In these works, the main reference was Sonia Draibe.

Moreover, it is crucial to consider the approach “under construction” to the specific literature of the field of Political Science, on implementation, by researchers from the educational area. Among the theses and dissertations, despite the filters applied, we found studies that did not clearly define their analytical framework to analyze the policy or program in focus or, even, the presentation of a theoretical framework at the beginning of the work that was not mobilized in the analysis of the collected empirical data.

Lastly, as in other studies that involve bibliographic surveys, we recognize that the results found are limited to the selection criteria and the referential categorization defined by the authors. We consider that, although they may not express a generalization of the trends of academic production on implementation of educational policies, their results shed light on a field still little explored. This gap brings some implications for the research, one of them being the construction of theoretical and methodological support for the study of public policies in the field of education. If, so far, the studies have been developed from references coming from other areas of knowledge - especially Political Science - one of the challenges posed is the construction of a theoretical and methodological framework in the educational field. Thus, in addition to advancing the existing gap, the field has the challenge of training researchers who build a theoretical and methodological contribution to policy implementation in the field of education itself.

Preprint

DOI: https://preprints.scielo.org/index.php/scielo/preprint/view/3618

REFERÊNCIAS

• ANTONIOLLI, E. 2019. Programa Nacional de Alimentação Escolar (PNAE): uma análise da implementação em municípios do Rio Grande do Sul (RS) Dissertação de Mestrado. Chapecó. Universidade Federal da Fronteira Sul.
• ANTUNES, E. 2018. Assistência estudantil nos institutos federais: da política à implementação Dissertação de Mestrado. Campinas. Universidade Estadual de Campinas.
• BALL, S. & BOWE, R. 1992. Subject departments and the “implementation” of National Curriculum policy: an overview of the issues. Journal of Curriculum Studies, London, v. 24, n. 2, p. 97-115.
• BALL, S. & OLMEDO, A. 2013. A “nova” filantropia, o capitalismo social e as redes de políticas globais em educação. In: PERONI, V. (Org.),Redefinições das Fronteiras entre o Público e o Privado: implicações para a democratização da educação Brasília, Liver Libro.
• BATISTA, N. 2018. A gestão democrática da educação básica no âmbito do plano de ações articuladas municipal: tensões e tendências. Revista Brasileira de Educação v. 23, pp. 1-24. doi: 10.1590/S1413-24782018230008
  » https://doi.org/10.1590/S1413-24782018230008
• CAMARGO, F. 2019. O direito à educação de alunos com deficiência: aspectos da implementação da política de educação inclusiva em Corumbá/MS Tese de doutorado. Rio de Janeiro. Pontifícia Universidade Católica do Rio de Janeiro.
• GIUSTO, S. 2018. A implementação do PNAIC na rede de ensino do município de São Paulo em territórios vulneráveis Dissertação de Mestrado. São Paulo. Universidade Cidade de São Paulo.
• FARIA, C. 2018. Políticas públicas de educação infantil no município de Duque de Caxias - RJ: de 2007 a 2017 Dissertação de Mestrado. Rio de Janeiro. Universidade do Estado do Rio de Janeiro.
• GOMES MOREIRA, A. 2018. A implementação da autonomia financeira da escola em municípios cearenses Tese de Doutorado. Fortaleza. Universidade Estadual do Ceará.
• GOMES, A. & PASSOS, G. 2018. A implementação do Programa Nacional de Assistência Estudantil (PNAES) nos Institutos Federais.Revista de Políticas Públicas, v. 22, n. 1, p. 415-442.
• GOMIDE, A. & PIRES, R. 2014. Capacidades estatais e democracia: a abordagem dos arranjos institucionais para análise de políticas públicas. In: GOMIDE, A. & PIRES, R. (Eds.). Capacidades estatais e democracia: arranjos institucionais de políticas públicas Brasília: Ipea, pp. 15-28.
• LAURENTINO, T. & ALLOUFA, J. 2019. Avaliação da implementação do Programa Nacional de Acesso ao Ensino Técnico e Emprego (Pronatec): um estudo de caso no município de Picuí-PB. Ensaio: aval. pol. públ. educ 27(105). doi: 10.1590/S0104-40362019002701468.
  » https://doi.org/10.1590/S0104-40362019002701468
• LAVALLE, A. & RODRIGUES, M. & GUICHENEY, H. 2019. Agência local e indução federal: a operação da política municipal de habitação em Recife e Curitiba. Rev. Sociol. Polit 27(71 ). doi: 10.1590/1678-987319277103.
  » https://doi.org/10.1590/1678-987319277103
• LINS, C. P. 2018. A função do coordenador pedagógico na implementação do pacto nacional pela alfabetização na idade certa em escolas da rede municipal de Dourados/Ms (2012-2017) Dissertação de Mestrado. Dourados. Universidade Federal da Grande Dourados.
• DOS SANTOS LISBOA, R. & MENDES, V. & LIMA, W. 2019. Implementação da política de atendimento às pessoas com necessidades educacionais específicas nos institutos federais brasileiros.Revista de Políticas Públicas V. 23, n. 2, p. 566-585.
• LOTTA, G. 2015. Burocracia e Implementação de Políticas de Saúde: os agentes comunitários na Estratégia Saúde da Família Rio de Janeiro: Editora Fiocruz.
• LOTTA, G. 2018. Burocracia, redes sociais e interação: uma análise da implementação de políticas públicas.Revista de Sociologia e Política, v.. 26, p. 145-173.
• LOTTA, G. & NUNES, A. & CAVALCANTI, S. & FERREIRA, D. & BONAT, J. 2018. Por uma agenda brasileira de estudos sobre implementação de políticas públicas.Revista do Serviço Público, v. 69(4), pp. 779-810.
• LOTTA, G. & BAUER, M. & JOBIM, R. & MERCHÁN, C. 2021. Efeito de mudanças no contexto de implementação de uma política multinível: análise do caso da Reforma do Ensino Médio no Brasil. Rev. Adm. Pública V. 55(2), pp. 395-413.
• MAINARDES, J. 2018. Metapesquisa no campo da política educacional: elementos conceituais e metodológicos. Educar em Revista, Curitiba, v.34, n.72, p. 303-319.
• MAY, S. 2018. Programa de melhoria na qualidade de ensino - excelência: uma análise da implementação da política em duas escolas estaduais do ensino médio na cidade de Cacoal, Rondônia Dissertação de Mestrado. Juiz de Fora. Universidade Federal de Juiz de Fora.
• MARCELINO, I. 2018. O programa de apoio à qualificação da Universidade Federal de Juiz de Fora na perspectiva do modelo de gestão por competências Dissertação de Mestrado. Juiz de Fora. Universidade Federal de Juiz de Fora.
• MARQUET, M. 2018. Políticas de inclusão escolar: o contexto do município de Capão da Canoa/RS (2007-2017) Dissertação de Mestrado. Canoa. Universidade Federal do Rio Grande do Sul.
• MARTINS, R. & LOTTA, G. 2010. Capital social e redes sociais como alternativa para análise de políticas públicas de educação: o caso de Icapuí-CE.Ensaio: avaliação e políticas públicas em educação, v. 18, p. 846-860.
• MARTINS, L.C. & JUNIOR, O.G. 2018. Redes e ativismo em políticas públicas: a construção da Política Nacional de Educação Museal (PNEM). Revista de Políticas Públicas da UFMA, v. 22, p. 307-330.
• MATOS, S. 2018. Censo Escolar e o desafio frente às inconsistências de informações no sistema do Educacenso Dissertação de Mestrado. Juiz de Fora. Universidade Federal de Juiz de Fora.
• MENGALLI, N. 2018. Estudos em softwares de monitoramento e mensuração: subsídios para a implementação de políticas públicas na área educacional Tese de Doutorado. São Paulo. Pontifícia Universidade Católica de São Paulo.
• MERCÊS, S. 2018. A hora-atividade como espaço para a formação continuada de professores no âmbito das escolas estaduais da regional Ouro Preto do Oeste (RO) Dissertação de Mestrado. Juiz de Fora. Universidade Federal de Juiz de Fora.
• MESKO, A. 2018. O programa “Educação - compromisso de São Paulo” e as estratégias de implementação das políticas empresariais na gestão escolar Dissertação de Mestrado. Campinas. Universidade Estadual de Campinas.
• MOREIRA, T. 2019. Disputas Institucionais e Interesses Corporativos no Sistema de Justiça: Impasses na criação da Defensoria Pública nos Estados. Dados - Revista de Ciências Sociais, Rio de Janeiro, v. 62, n. 4, p. 1-43.
• MOTA, M. O. 2018. Entre a meritocracia e a equidade: o prêmio nota dez na percepção e atuação de agentes implementadores Tese de Doutorado. Rio de Janeiro. Pontifícia Universidade Católica do Rio de Janeiro.
• MOTTA, L. 2019. Conhecer, classificar e intervir: Práticas e discursos de policiais-professores sobre os jovens vulneráveis na Cidade de Deus.Dilemas-Revista de Estudos de Conflito e Controle Social, v. 12, n. 3, p. 627-646.
• NASCIMENTO, J. A. 2018. Estudo de revisão do arranjo institucional de implementação de políticas: o caso do Programa Institucional de Bolsa de Iniciação à Docência (PIBID) Dissertação de Mestrado. Rio de Janeiro. Pontifícia Universidade Católica do Rio de Janeiro.
• OLIVEIRA, M. 2017. Correção de fluxo em uma escola da rede pública municipal do Rio de Janeiro: percepções e discricionariedade dos agentes implementadores Dissertação de Mestrado. Rio de Janeiro. Pontifícia Universidade Católica do Rio de Janeiro.
• OLIVEIRA, A. C. 2019. Implementação das políticas educacionais: tendências das pesquisas publicadas (2007-2017). Revista de Estudios Teóricos y Epistemológicos en Política Educativa v. 4, p.1 - 16.
• RIBEIRO, R. 2018. Uma análise do Programa “Mais Infância”: concepções e ações envolvidas na implementação da política pública para a Educação Infantil em Niterói (2013-2016) Dissertação de Mestrado. Rio de Janeiro. Universidade do Estado do Rio de Janeiro.
• SARUE, B. 2018. Quando projetos urbanos acontecem? Uma análise a partir do Porto Maravilha no Rio de Janeiro. Dados - Revista de Ciências Sociais, Rio de Janeiro, v. 61, n.3, p. 581-616.
• SILVA, F. 2018. A formação continuada no Pacto pela Alfabetização na Idade Certa - PNAIC - no contexto da superintendência regional de São João Del- Rei: impactos e desafios da formação em serviço Dissertação de Mestrado. Juiz de Fora. Universidade Federal de Juiz de Fora.
• SILVA, S. 2019. Políticas de educação integral no estado do Pará: aspectos de sua implementação na rede estadual de ensino Dissertação de Mestrado. São Paulo. Universidade Cidade de São Paulo.
• SOUSA, M. 2019. Discursos sobre o gerencialismo e a performatividade: a trajetória da política de avaliação educacional na rede municipal de ensino de São Paulo (1989 a 2016) Tese de Doutorado. Campinas. Universidade Estadual de Campinas.
• TELLO, C. & MAINARDES, J. 2015. Revisitando o enfoque das epistemologias da política educacional. Práxis Educativa, Ponta Grossa, v. 10, n. 1, p. 153-178. doi: 10.5212/PraxEduc.v.10i1.0007.
  » https://doi.org/10.5212/PraxEduc.v.10i1.0007
• TORRES, H. & CAMELO, R. & CASTRO, M. H. Dificuldades de coordenação e Políticas educacionais no brasil: o caso do ensino fundamental. Dados - Revista de Ciências Sociais, Rio de Janeiro, v. 62, n. 4, p. 1-31, 2019.
• TRIPODI, Z. & SENA, V. & RODRIGUES, E. 2018. Permeabilidade estatal e atores privados na oferta da educação infantil: o que dizem os Planos Municipais de Educação (2014-2024)?. Revista Brasileira de Educação, v. 23, pp. 1-25. DOI: 10.1590/S1413-24782018230064.
  » https://doi.org/10.1590/S1413-24782018230064
• TRIPODI, Z. & SOUSA, S. 2018. Do governo à governança: permeabilidade do estado a lógicas privatizantes na educação. Cadernos de Pesquisa, São Paulo, v.48, n.167, p. 228-253.
• VELOSO, J. 2019. Aprendizagem Profissional e o poder público municipal para além do incentivo ao mero cumprimento de cotas.Revista Brasileira de Políticas Públicas, v. 9, n. 1.
• VIEIRA, S. & PLANK, D. & VIDAL, E. 2019. Política Educacional no Ceará: processos estratégicos. Educação & Realidade, Porto Alegre, v. 44, n. 4, p. 1-24.

• 1
  To learn more, please access https://www.reippe.com/
• 2
  At the time of defining the descriptors and reference authors, we did not find any study from the field of Political Science on implementation of educational policies, although there were, in the field of Education, some studies already published on implementation of educational policies, such as the works of Oliveira (2017) and Mota (2018).
• 3
  The evaluation of this four-year period was invalidated by CAPES in 2020, after the study had already selected the journals. With this invalidation, we re-evaluated and found that all selected journals were also evaluated as A1 in the 2013-2016 quadrennium.
• 4
  Available at: https://observatorioeducacion.org/sites/default/files/ball_s._y_youdell_d._2008_la_privati zacion_encubierta_en_la_educacion_publica.pdf Accessed on: 14 Sep. 2022.
• 5
  The study of Santos (2018) cites the reference author Martha Arretche when discussing the trajectory of inequalities in Brazil, but does not use her to address the issue of public policy implementation or the perception of implementing agents about the policy.

• 20
  The translation of this article into English was funded by the Fundação de Amparo à Pesquisa do Estado de Minas Gerais - FAPEMIG, through the program of supporting the publication of institutional scientific journals.

Publication Dates

• Publication in this collection
  20 Mar 2023
• Date of issue
  2023

History

• Received
  04 Mar 2022
• Accepted
  28 June 2022