The use of multiple sources of information in public policy: the Brazilian federal bureaucracy

Abstract

Policy-making is a complex process involving stimuli and conditioning. Understanding the sources of information bureaucrats and public organizations use is essential to characterize policy-making and discuss the opportunities and limits of Evidence-Based Public Policies (EBPP) in the Brazilian context. The different sources of information available to bureaucrats in the country are investigated, such as the state’s internal and external sources and scientific and experiential sources. This paper presents the results of a survey with 2,180 officials of the direct federal administration conducted at the end of 2019. The findings indicate variation in the types of information bureaucrats mobilized depending on policy area and the type of policy work they perform.

Keywords:
evidence-based public policy; bureaucracy; policy-making; policy work

Resumo

A produção de políticas públicas é um processo complexo que envolve um conjunto de estímulos e condicionantes. Compreender o que, de fato, tem informado os burocratas e as organizações públicas é um elemento essencial para caracterizar a produção de políticas públicas, assim como para a discussão das potencialidades e dos limites das Políticas Públicas Baseadas em Evidências (PPBE) no contexto brasileiro. Neste estudo, investigamos as diferentes fontes de informação disponíveis aos burocratas brasileiros; dentre as quais, não somente as científicas, mas também as experienciais, as internas e as externas ao Estado. Esta pesquisa traz resultados de um survey respondido por 2.180 servidores da Administração Federal Direta no final de 2019. Os resultados indicam que há variação na utilização de fontes de informação a serem mobilizadas pelos burocratas de acordo com o tipo de trabalho desempenhado e, também, com a área de política pública na qual atuam.

Palavras-chave:
políticas públicas baseadas em evidências; burocracia; produção de política pública; trabalho na política pública

Resumen

La elaboración de políticas públicas es un proceso complejo que implica un conjunto de estímulos y limitaciones. Comprender lo que de hecho ha informado a los burócratas y las organizaciones públicas es un elemento esencial para caracterizar la elaboración de políticas, así como para discutir las potencialidades y los límites de las políticas públicas basadas en la evidencia (PPBE) en el contexto brasileño. Se investigan las diferentes fuentes de información de que disponen los burócratas brasileños, incluyendo las científicas, pero también las experienciales, internas y externas al Estado. Esta investigación trae los resultados de una encuesta contestada por 2.180 funcionarios de la administración federal directa a finales de 2019. Los resultados indican que hay variaciones en la elección de los tipos de información que deben movilizar los burócratas en función del tipo de trabajo realizado, así como del ámbito de la política pública en el que operan.

Palabras clave:
políticas públicas basadas en la evidencia; burocracia; elaboración de política pública; trabajo en las políticas públicas

1. INTRODUCTION

The relevance of studies that inform public policies is a subject that has been discussed since the birth of the field of public policy analysis (Lasswell & Lerner, 1951; Weiss, 1979). With the advent of the global COVID-19 crisis, this debate about the importance, use and credibility of science in supporting public policy decisions has intensified and thus it is an opportune moment to deepen our reflections about the Evidence-Based Public Policy (EBPP) approach. On one hand, this approach revisits precepts of instrumental rationality in the making of public policy decisions based on scientific evidence (Davies, Nutley & Smith, 2000); on the other, it catalyzes critiques which have emerged more recently from analytical currents, such as the argumentative and post-structuralist currents, which provide the foundation of an argument for the need to have a better understanding of various sources of information beyond scientific evidence.

As the critics of EBPP have pointed out, the access to and utilization of scientific evidence is often scarce or even non-existent, and this constitutes a limit to the objective rationality of the actors who may be affected by selection and interpretation bias in relation to evidence, and even their own analytical instruments may have a meager capacity to generate explanations within contexts of great complexity. Studies and evaluations require a joint analytical effort to make the production of knowledge viable, as well as making it applicable, accessible, and relevant in informing the decision-making process. Thus, other types of “evidence” are often utilized, such as previous personal or organizational experiences, informational instruments produced for policy communities by the media or by the social networks, or impressions garnered from policy beneficiaries and internal and external policy makers, etc.

Recognizing these critiques and the relevance of deepening our understanding of the informational resources used in the production of Brazilian policies, we begin with Pinheiro’s proposal (2020b) of a moderate model for the conceptualization of evidence, which recognizes the contributions of different epistemological currents in the study of public policies and argues that the significance of a term is given by the context of its use. In this article we will seek to empirically explore this model relying on survey data collected from federal bureaucrats who responded to the question, “What do federal bureaucrats use as information sources in their decision-making in a variety of contexts and works developed in the production of public policies?” In addition to the descriptive exploration of this research question, we will seek to study hypotheses for future analyses which may be able to explain the various uses that bureaucrats make of information.

The results indicate two contributions to EBPP studies. The first relies on the existence of multiple sources of information used by federal bureaucrats. EBPP studies in general have a prescriptive nature and are not concerned with understanding how decisions are in fact made and which information is used, scientific or not. Perfecting the decision-making process with the more frequent use of scientific sources assumes that we understand first of all how bureaucrats gather information. The second contribution points out the importance of the context and the work developed in public policy as a differentiating element in the use of various types of information. This is an innovative perspective from the point of view of new studies in this area and groundbreaking in terms of Brazil.

This article consists of six sections: the introduction, a review of the literature, a description of the analytical model utilized, the applied methodology, a discussion of the results, and the final considerations.

2. THE EVIDENCE-BASED PUBLIC POLICY LITERATURE

The field of public policy analysis was originally proposed, within the Anglo-Saxon context, as a science oriented towards perfecting of the use of rationality in the public policy decision-making process (Lasswell & Lerner, 1951). In the decades since, this literature has expanded in its thematic areas as well as its analytical approaches, which have come to question the idea of a perfect separation between the technical and the political and full objective rationality devoid of subjective interference. They reveal, by contrast, the various cognitive strategies utilized by individuals such as satisficing and incrementalism, which have created shortcuts and demonstrate that there is room for subjectivity in reducing the scope of possible solutions for public problems (Lindblom, 1959; Simon, 1956).

More recent works by authors based on constructivist perspectives, such as the “argumentative turn movement,” support the argument that the field of public policy, like the nature of human action, is oriented by ideas, values, and customs and the opposition or separation between technical and policy knowledge is unreal (DeLeon, 2008; Fischer & Gottweis, 2012; Lejano, 2006; Spink, 2019; Yanow, 2000). Thus, public policy in real life is not an instrumental decision-making process guided by scientific research and knowledge, but rather an interactive, discursive and negotiated arena, in which scientific evidence is just one of the frames of validation of possible meanings (Williams, 2010).

A half century after the first formulations about policy making, the EBPP movement’s central argument is the defense of the use of evidence - understood as “[...^ the result of systematic studies devoted to the growing accumulation of knowledge” (Davies, Nutley & Smith, 2000, p. 3) - to inform public policy decision-making. Given the increased complexity of public issues, accompanied by a growing multiplicity of informational resources that are available to capture and deal with them, EBPP has gained force in the past two decades among academics, policy makers and practitioners, because it sustains the logic of the rationalization and efficiency of public policy production based on scientific evidence (Davies et al., 2000).

Recent studies have resumed the critiques of public policies in terms of the capacity of technical influence which ignores public power and the relevance of aspects such as values, principles and judgement in the decision-making process (Cairney, 2019; Parkhurst, 2017). In this case, they question the viability of EBPP in contexts in which there is an absence of or little availability of scientific evidence and limits to the objective rationality of the actors and their analytical instruments in explaining contexts characterized by a high degree of complexity (Cairney, 2019; Parkhurst, 2017).

Pinheiro (2020a, 2020b) is aware of the fact that the definition of evidence itself is debatable. On one hand, using the rationalist paradigm, we find the idea of evidence as the result of rigorous and systematic scientific production which reveals the ‘truth’. On the other hand, factors derived from the formulations of constructivist paradigms mentioned above have come to be recognized as relevant in terms of public policy decision-making and production, such as the historical contingency of social phenomena, and the interests, values and motivations of actors, and the interactive reflexivity among actors and among actors and objects.

Given the absence of a specialized literature concerning the systematic characterization of evidence in public policy, and considering the accumulation of public policy studies briefly mentioned above, Pinheiro (2020b) proposes a moderate model, which he places between the two extremes. That is to say, between the radical perspective of the rationalist model - which does not consider the inherent complexity of the decision-making process, which is characterized by non-linearity, uncertainties, and multiple causality - and a radical perspective of the constructivist model, which makes general statements and the pragmatic use of evidence to analyze and evaluate public policies impossible.

Based on American pragmatism and the linguistic philosophy of Wittgenstein, Pinheiro’s moderate model (2020a, 2020b)¹ seeks to extract in a critical manner what has been learned from the rationalist and constructivist models, and it proposes using them in a contextual framework which is the characterizing element of an informative instrument.

The contextual framework that conditions the use of informative instruments and therefore their conformity and recognition as evidence in the eyes of users is composed of factors which provide specificity in the bureaucratic decision-making process. Pinheiro (2020a, 2020b) emphasizes three recurring factors which are interwoven: “[...^ i) politics - the temporality of politics, its ideological commitments, and its disputes over power and democracy; ii) epistemologies - the evaluation of policy, uncertainty, the reflexivity of social knowledge, etc. (Mulgan, 2005, p. 224); and iii) normative, institutional and organizational factors” (Pinheiro, 2020b, p. 23).

In light of the growing debate concerning EBPP, recent Brazilian and international empirical studies have generally been based on survey data provided by bureaucrats and have been dedicated to exploring what informs public policy based on an analysis of the practical field of how bureaucrats act in governmental organizations (Cherney, Head, Povey, Ferguson & Boreham, 2015; Macedo, Viana & Nascimento, 2019; Oliver, Innvar, Lorenc, Woodman & Thomas, 2014; Veselý, Ochrana & Nekola, 2018). Even though there is no explicit dialogue with the proposing of the moderate model, we assert that these studies have produced analytical support for exploring the model’s arguments and empirically operationalizing the concept of its contextual framework.

Veselý et al. Nekola (2018) portray, for example, the broad diversity of informative instruments in the Czech Republic and point out the preeminence of experience and resources produced by public organizations themselves as the main sources of information which inform the actions of Czech bureaucrats to the detriment of scientific sources. Veselý et al. (2018) add the relevance of understanding what these actors consider to be evidence and what role it should play in their daily practices. Thus, this allows us to assume that the use of evidence is a dynamic process in which actors have their own conceptions of what constitutes evidence, which corroborates Pinheiro’s arguments (2020a, 2020b).

Studies which analyze the context in which bureaucrats act through the concept of policy work, that is the work that they perform in carrying out public policies, demonstrate that in practice analytical work generally occurs in association with other forms of work, such as negotiation, communication, translation and even the democratization of state actions (Colebatch, Hoppe & Noordegraaf, 2010; Meltsner, 1976; Olejniczak, Raimondo & Kupiec, 2016). From this perspective, the various types of work involved in carrying out public policy demand a variety of abilities and resources, including those of an informational nature (Howlett & Wellstead, 2011; Koga & Viana, 2020; Wu, Ramesh & Howlett, 2015). In keeping with the relational perspective of bureaucratic studies (Cavalcante and Lotta, 2015; Pires, Lotta & Oliveira, 2018) and the recent critiques of EBPP (Cairney, 2019), policy work studies support the idea that the work of implementing public policy, including analytical work, is performed through the interaction of policy stakeholders and does not occur in an isolated or exclusive manner; therefore bureaucrats are influenced and informed by other forms of knowledge acquired from these other actors (Colebatch et al., 2010).

In Brazil, in a study by the National School of Public Administration (Escola Nacional de Administração Pública [Enap^, 2018), conducted in 2017 based on the same sample used in our study, civil service bureaucrats of the Federal Administration itself based their public policy work on a wide variety of sources of information. These included normative sources - such as judicial-legal norms and control organization recommendations - as well as media sources - the traditional media, social media and social networks - which were used in a recurring manner; while public policy monitoring and evaluation data as well as statistical data and surveys were the least used sources.

Building on the data analysis conducted by Enap (2018), Macedo et al. (2019) seeks to explore the correlations between various international sources of information used by the Federal Administration strictly speaking and variables such as (1) the demographic characteristics of bureaucracy, (2) the types of work in public policy, (3) different bodies of the Federal Administration, (4) the types of positions in the Federal Administration, and (5) the areas of public policy. Some of the main findings were that among the work performed by bureaucrats (relational, analytical, managerial and administrative), work of a relational nature was the only type of work that did not place much emphasis on sources of a normative nature, and that independent of the area of public policy, normative sources of information were dominant. These results suggest, therefore, that while there is variation in the types of information sources utilized, there is a predominance of the utilization of normative sources in various contexts of public policy work.

We argue that the empirical studies presented above suggest different forms of characterization and comparison with contextual frameworks. Among the various forms of characterization of a contextual framework, we have chosen two frameworks which we deem to be capable of displaying variation in the political, epistemological and normative dimensions, which are the three types of sources that make up the contextual framework proposed by Pinheiro (2020b). This treats the characterization of the contextual framework as an area of public policy and public policy work. We understand that bureaucrats exhibit characteristics of these three dimensions simultaneously in the performance of their work. Policy areas portray power struggles in the field, epistemological underpinnings, and institutional characteristics that are distinct for each one. In terms of bureaucrats, even though we understand that they are characterized by individual sociodemographic and functional variables, we believe that in this case there is a greater difficulty to simultaneously reflect these three dimensions to constitute an integrated context of epistemic, policy and normative behavior. Thus, we believe that work variables in public policy are a better representation of the diversity of contextual frameworks.

In this article, which seeks to simplify the understanding of our findings, we use “sources or types of information” to refer to the various types of sources of information used by the federal bureaucracy. Thus, the term “evidence” will be reserved for the scientific concept of evidence, which is the term typically used by our cited EBPP studies. In other words, information here refers to the genus, while evidence is a species. This characterization is in keeping with other studies in the field, such as the concept of fake news as a type of species of information in which there is no authenticity and whose intentional objective is to deceive (Recuero & Gruzd, 2019; Shu, Silva, Wang, Jang & Liu, 2017). Thus, this concept includes individual experience, and the normative production of the state, science, civil society and even sources of false information, even though this last category will not be addressed here.

3. METHODOLOGY

In this study we apply a designed survey based on the instruments used in the above- mentioned studies, especially Enap (2018). The data was collected through the application of a self-administered, online survey (the LimeSurvey Platform), sent by email to a preselected sample of recipients among the 96,543 civil servants who make up the Federal Administration strictly speaking. The sample was determined by a simple stratification technique. The first sample contained 6,055 civil servants. We then performed two more rounds of selection in exactly the same manner and arrived at a final number of 18,165 selected civil servants². We obtained 2,180 valid, complete responses, which represents a 12% response rate for the triple sample (36% of the original sample).

The survey applied was based on the variables proposed in the analytical model described in the previous section³. As already mentioned, this study focuses on an analysis of the relationship between the selected variables which characterize the three main elements of the model: type of information and contextual framework (“public policy work type” and “public policy area”). The comparison between the data obtained by Enap in 2017 (Enap, 2018) and this study in 2019 in terms of “work type” variable was made to analyze the stability of the results in the characterization of the types of work as well as the contextual frameworks.

The division of the areas of public policy is not unanimous in the literature. Field studies adopt various divisions (Cavalcante & Lotta, 2021; Filgueiras, Koga & Viana, 2020; Paula, Palotti, Cavalcante & Alves, 2017). In this study we use the division proposed by Cavalcante, Camões and Knopp (2015), organized using criteria which are external to this study based on the normative guidelines of the Federal Budgeting Secretariat, using the 2013 Annual Budget Bill. Updates have been made to the number of ministries and two subdivisions have been added, resulting in six areas of public policy: Central, Economic, Infrastructure, Social, Environmental and Control, as will be detailed in the results section. Table 1 lists the study sample with the divisions for each of the public policy areas analyzed.

TABLE 1
GOVERNMENT AREAS AND RELATED MINISTRIES

Government Area	Ministry/Superior Government Body	N	% Valid
Central	Ministry of Foreign Relations	103	4.8%
	Ministry of Justice and Public Safety	95	4.4%
	Ministry of Defense	84	3.9%
	General Secretariat of the President	20	0.9%
	Cabinet of the President	19	0.9%
	Institutional Security Cabinet of the President	13	0.6%
	President’s Administration Secretariat	06	0.3%
	President’s Personal Cabinet	05	0.2%
Control	Comptroller General of the Union	129	6%
	Attorney General of the Union	19	0,9%
Economics	Ministry of Economics	513	23.7%
	Ministry of Agriculture, Livestock and Supply	168	7.8%
	Ministry of Tourism	29	1.3%
Infrastructure	Ministry of Science, Technology, Innovation and Communications	197	9.1%
	Ministry of Regional Development	98	4.5%
	Ministry of Infrastructure	75	3.5%
	Ministry of Mines and Energy	60	2.8%
Environment	Ministry of the Environment	108	5%
Social	Ministry of Citizenship	203	9.4%
	Ministry of Health	105	4.8%
	Ministry of Education	99	4.6%
	Ministry of Women, the Family and Human Rights	19	0.9%

• Source: Elaborated by the authors.

We conducted two exploratory factor analyses to obtain the variable data which makes up this survey’s sections of “Public Policy Work” and “Informational Instruments” given that we seek to identify the latent dimensions for work and information type. These analyses suggest the existence of four factors in both cases using the application of the varimax rotation. To analyze the relationship among these factors, we estimated the factor scores based on Thurstone’s method (1935), which basically recalculates factor loads, considering the correlations between the original variables.

4. RESULTS

We initially analyze two groups of data in this section. The first deals with the types of work performed by federal bureaucrats, and the second deals with types of information instruments used by these bureaucrats. Then, we will report the correlations among the variables which make up these two groups and the public policy areas mentioned above.

The six policy areas described in Table 1 were defined based on the ministries in which the bureaucrats work. In addition to the four areas used by Cavalcante et al. (2015), we broke down two further areas: the Environment, pointed out in recent studies as acting and having abilities which are different from the infrastructure area in which they were originally included (Abers, Oliveira & Pereira, 2016; Filgueiras et al., 2020), and the area of Control, which is also a producer of information, and as has been indicated in previous studies (Enap, 2018;Macedo et al., 2019) and the results of this study, makes it relevant to study it separately from the Central area where it was included by Cavalcante et al. (2015).

4.1. Types of work

To map the work performed by public policy civil servants, we have adapted a question from the Enap survey (2018): “How often have you performed the following activities related to public policy at work during the past 12 months?”, modifying some of the items of the question in order to qualify the responses to achieve our objectives.

As presented in Graph 1, works related with providing advice and analysis are performed with greater frequency (“elaborating reports and technical notes”, “providing superiors with advice” and “collecting and analyzing public policy data”). Next in terms of frequency we find “administrative activities” and “control activities (“supervising compliance with public policy norms”, “meeting the demands of control bodies”, “elaborating, generating and supervising contracts”). Less frequently there are the activities “raising and negotiating financial resources” and “commissioning and validating public policy evaluation studies”.

GRAPH 1
FUNCTIONS PERFORMED IN THE PUBLIC POLICY ENVIRONMENT (PERCENT VALID)

Table 2 present the results of the factor analysis which explains a cumulative 55.5% of the data and reveals the existence of the four common activity profiles. The first profile which we term “analytical/control”, combines the functions of providing advice and control (elaborating reports and technical notes, collecting and analyzing data, elaborating normative texts). The second profile focuses on relational activities (representing the governmental body, meeting the needs of society, organizing events, and supervising the team⁴) associated with commissioning studies and raising financial resources. A third type involves the “managing and supervising of contracts” of public policy norms. Finally, the fourth profile is exclusively dedicated to administrative activities.

TABLE 2
FACTOR ANALYSIS OF THE FUNCTIONS PERFORMED IN PUBLIC POLICY (2019)

Function/Activity	Analytical/Control	Relational	Managing/Supervising Contracts	Administrative
Elaborating reports and technical notes	0.733
Collecting and analyzing public policy data	0.679	0.360
Elaborating normative texts (ex. bills)	0.535	0.370
Meeting governmental body demands	0.435		0.378	0.407
Providing superiors with advice	0.536	0.363		0.426
Commissioning and validating public policy evaluation studies	0.397	0.537	0.330
Raising and negotiating financial resources		0.726	0.331
Representing their governmental body in meetings	0.458	0.600		0.323
Consulting and meeting the demands of society	0.467	0.490
Organizing events		0.653		0.441
Managing team	0.307	0.339		0.354
Elaborating, generating and supervising contracts		0.313	0.727
Supervising compliance with public policy norms	0.415		0.467
Performing administrative activities				0.528
KMO: 0.897; χ² = 10015.460; df = 91; p < 0.001
Total Explained Variance - 55.5%
SS loadings	2.672	2.497	1.322	1.281

• Source: Elaborated by the authors.

This study’s data reveals the permanence of the patterns of types of work observed in the survey applied by Enap (2018) in 2017 - which supports our argument in favor of utilizing the four types as representations of contextual frameworks in terms of Pinheiro’s moderate model (2020b). Again we have three well defined profiles grouping together analytical, control, and relational and administrative activities respectively. There are some slight differences in terms of the raising financial resources function, which ceased to be a well-defined profile together with contracts, appearing in 2019 in the relational activities and the supervisory function, which in 2017 became associated with the analytical profile and now may be found in another profile, together with managing contracts. Thus, the relationship between analysis and control/supervision seems to sustain itself, given that the supervising function presents a load factor that is also close to that of the analytical profile.

It should be noted, in terms of analytical work, that the maintenance of the association of this type of work with activities related to meeting the needs of the control entities needs to be deepened to indicate a possible characterization of a mode of producing public policy information in the Federal Administration which is related to the meeting the needs of the control area. This hypothesis has been constructed through a dialogue with the Brazilian literature, indicating a growing influence of control in management and policy making in Brazil (Filgueiras, 2018; Grin, 2020; Nogueira & Gaetani, 2018). Cavalcante and Silva (2020), analyzing the specific case of the Federal Accounting Court (TCU), have identified a type of institutional change process which is denominated conversion in which the application of the actions of the FAC and the control bureaucrats occur not due to the virtue of normative changes, but rather due to the strengthening of its capacities and policy influence.

The growing demand for accountability and the increase in distrust in Brazilian representative institutions, associated with other exogenous forces such as the emergence of a global movement with an emphasis on measuring performance (Humphrey & Owen, 2000), have served as a correction in political representation which will place Brazilian institutions of control in a “[...^ position with a diffuse counter-majoritarian public interest which backs their interpretation of norms, strict control of the decision-making process, and the implementation of government policy,” (Filgueiras, 2018, p. 375).

In light of this debate, the results presented above lead to the questioning of what will be the effect on the production process and the use of public policy with the onset of this trend of strengthening the control function and an accentuated asymmetry in abilities, including those of analysis between management and control.

4.2. Types of Information

Graph 2 lists the types of information that are most frequently utilized. As this graphic demonstrates, civil servants most often use types of information which are of an endogenous nature, or in other words, they usually use information, opinions and knowledge produced by the bureaucracy itself which endorse its work on public policies. This statement corresponds, to some extent, to what is expected in the operation of specialized systems such as a government bureaucracy. “Laws and norms” stand out as the most utilized type of information used by civil servants: 82.11% of the respondents declare that they use them “always” or “frequently”.

Next comes “consulting colleagues at work” and “personal experience”, with 75.87% and 64.59% of the respondents declaring respectively that they apply them “always” or “frequently”. This is relevant data, because it demonstrates the importance of interpersonal relationships and individual career paths in understanding the practices employed by bureaucrats in selecting their sources of information.

However, the subsequent presence of “technical notes produced by the federal government” and “governmental information systems”, which come in at 62.66% and 49.82% respectively, makes it possible to affirm that an important part of the federal bureaucracy has incorporated information and knowledge of a factual-analytical nature in its work routines (even though it continues to be endogenous).

Another aspect that should be pointed out is the frequent use of “legal reports and decisions” and “control body recommendations”, which come in at 50.18% and 49.45% respectively. Following the same trend as the data concerning their functions, the incidence of these two items shows that a considerable portion of the bureaucracy bases its work on works linked to control and supervision, which leads us to question whether these sources include indirect references to scientific sources of information.

We should also note that an important portion of bureaucrats are inclined to use scientific sources, but this practice is mostly protected by the production of knowledge from data that comes from the information systems of the executive branch of the federal government itself.

The data obtained suggests that conventional sources play a smaller role. “Research articles, chapters or books” are mentioned as being used “always” or “frequently” by 30.46% of the respondents. “Scientific research reports (ex. research consulting products, IPEA discussion texts, etc.)” appear with less frequency, only 19.77% use them “always” or “frequently”. When we just consider civil servants who answered “always”, the figure for these two items is substantially lower: just 12.20% affirm that they always refer to scientific publications and 6.47% refer to research reports.

The numbers above reveal that scientific production has a low direct influence on the work of public policy civil servants. The contrast between the low incidence of direct scientific influence and the strong presence of analytical documents and statistics produced by the state itself gives credence to the “two-communities” theory (Caplan 1979; Dunn 1980) which is the hypothesis that bureaucrats and academics form separate communities which operate under different rules and use different language, and that there are not many links or much communication between them.

It would be hasty, however, to affirm that federal bureaucrats do not use scientific evidence in their work. Technical notes based on governmental data and legal reports are highly systematic sources of information and they possess great analytical capacity and empirical verification (when they are not directly or indirectly anchored by scientific works). What happens is that this use is broadly supported by the production structure of the federal bureaucracy and not that of universities. This is another instance of a “do it yourself” culture in the bureaucracy and a corresponding distance from the academic-scientific universe. On the other hand, we can speculate in line with the suggestions of Newman, Cherney and Head (2016), that contributions from the scientific field filter down to bureaucrats, diluted in the midst of all this internal production of knowledge. One hypothesis that could be explored is that the actions of this control, as mentioned in the previous section, can intermediate the process of the absorption of scientific evidence, which is occasionally used in its auditing evaluations and processes.

GRAPH 2
TYPES OF INFORMATION USED AT WORK ON PUBLIC POLICY (SIMPLE PERCENTAGE)

The last two lines of Graph 2 are least employed by the federal bureaucracy. There is very little information generated by society itself that is used by the federal bureaucracy. “Interest group information (ex.: unions, companies, social movements, NGOs, etc.)”, “consumer experiences and opinions or comments and ombudsman suggestions” and “participative recommendations (ex.: public policy boards, conferences, etc.)” are the types of information that are least often used routinely by civil servants, with around 10% of the respondents declaring that they use them “always” or “frequently”.

The “type of information” variable was also submitted to a technical factor analysis again to detect correlations and common profiles. Table 3 presents the results of this analysis, which explains a cumulative 63.2% of the data and reveals four common profiles for this variable.

TABLE 3
FACTOR ANALYSIS OF THE TYPES OF INFORMATION USED IN PUBLIC POLICY WORK

Type of information	Internal	External	Scientific	Experiential
Laws and norms	0.793
Technical notes produced by the federal government	0.835
Legal reports and decisions	0.869
Control body recommendations	0.794
Governmental information systems and databases	0.496
Best practices and initiatives produced by states and cities	0.333	0.556
Participative recommendations		0.727	0.361
Consumer experiences and opinions		0.741
Interest group information (ex.: NGOs)		0.784
International organization recommendations		0.592	0.413
Journalistic articles		0.419		0.302
Research articles, chapters or books		0.303	0.8
Scientific research reports		0.461	0.822
Personal experience				0.651
Consulting colleagues at work	0.32			0.731
KMO: 0.869; χ² = 14009.490; df = 105; p < 0.001
Total Explained Variance - 63.2%
SS loadings	3.331	2.988	1.816	1.339

• Source: Elaborated by the authors.

Factor analysis of the types of information suggests the existence of specific profiles. In Table 3 we can see that these profiles are mainly characterized by the specific origin of the type of information. Thus, the “internal” profile contains information produced by the federal bureaucracy’s structure itself (laws and norms, technical notes, legal reports, etc.). In turn, the “external” profile involves the types of information that are produced by other governmental and social actors, in an outside-in movement (best practices of states and cities, international organization recommendations, interest group information, consumer experiences and opinions, etc.). The “scientific” profile is third, and it consists of information produced by the specialized field of academics and researchers (research articles, books and reports). The “experiential” profile is fourth, and it is made up of “personal experience” acquired by the civil servant and his or her colleagues over their careers.

The data presented in Graph 2and Table 3 has a greater incidence of use for a specific type of information, that produced by the state itself. Considering that a bureaucracy is by nature a rational-legal organization, as observed by Weber (1963), a possible explanation for the configuration of this portrait of self-absorption may be the need on the part of the bureaucracy, for a type of seal of approval or validation of its process of absorbing knowledge from external sources. That is, there is a need to transform and translate external knowledge into a valid and palatable informational instrument for the use of public administration. Perhaps the instruments considered internal - reports, technical notes and recommendations - serve this purpose in part intentionally or unintentionally. In other words, it is possible that the scientific works or those proposed by various policy stakeholders, as mentioned above in the case of control bodies, indirectly influence the production from these internal sources. This is a hypothesis that needs to be explored in future studies.

4.3. Relationships between types of information, types of function, and areas of public policy

In order to identify possible correlations among the function/activity profiles and the information source profiles found in the factor analyses, we opted to apply Pearson’s correlation coefficient⁵. Table 4 describes the coefficients (which range from -1 to 1) for pairs of factors and their respective significance. To assist the interpretation of force in relation to these factors, we adopted the following positive or negative parameters: 0.9, very strong correlation; 0.7 to 0.9, strong correlation; 0.5 to 0.7, moderate correlation; 0.3 to 0.5, weak correlation; 0 to 0.3, very weak correlation.

TABLE 4
PEARSON CORRELATIONS BETWEEN TYPES OF INFORMATION AND TYPES OF WORK

Types of Information	Types of work
	Relational	Analytical/Control	Contract/Supervision	Administrative
Internal	-0.063*	0.484**	0.054*	0.268**
External	0.443**	0.093**	0.149**	0.038
Scientific	0.286**	0.142**	0.046	-0.087**
Experiential	0.107**	0.071**	-0.012	0.136**

• * p < 5%; ** p < 1% Note: The correlations between the factors that make up types of information and types of work are equal to zero given that factor analyses assume independent factors.
• Source: Elaborated by the authors.

Among the strongest correlations, we can highlight the association between the relational work and the utilization of external information (r=0.443, p-valor <0.001) or scientific (r = 0.286, p-value <0.001). Information that comes from the machine of government is more correlated to analytical/control functions (r = 0.484, p-value <0.001) or more administrative functions (r = 0.268, p-value <0.001). These correlations corroborate the hypothesis that the type of work exercised within the context of public administration implies a more recurrent use of a type of information; in this case it is related to being more or less insulated and seeking sources that are more or less internal within the state.

In addition, we can also highlight the role played by a more relational profile. Previous studies have pointed out that this profile consists mainly of higher commissioned positions within the Federal Administration (Cavalcante & Lotta, 2015; Enap, 2018) and with greater influence on policy making (Kidjie & Palotti, 2020). The findings of Table 4 demonstrate that this profile is also correlated with searching for more diverse sources of information, such as those outside of the federal government (best practices in other federal entities, instances of social participation, civil society and international organizations, among others) and scientific sources of information.

In the case of the relationships between types of information and public policy areas, we opted for a descriptive approach through a comparison of medians⁶. The results can be seen in Graph 3 which demonstrates that there are differences among the six public policy sectors in terms of the sources of information that they utilize.

We can observe the intense use of internal sources in practically every sector of public policy, especially within the areas of control and social policy. External and scientific sources are employed less frequently in most of the sectors, with the exception of the more recurrent use of external sources in the environmental sector and scientific sources in the infrastructure sector. Finally, sources related to experience are recurrent throughout all sectors, with a more elevated frequency for the central area and a lower frequency in the social sector.

In fact, the specialized literature has already demonstrated that the various areas of public policy operate in distinct manners in terms of their modes of operation and governance arrangements, such as their mobilization of capacities (Cavalcante & Lotta, 2021; Filgueiras et al., 2020; Pires & Gomide, 2016). This is why it is to be expected that they also present differences in terms of the sources of information that they utilize and their analytical abilities as well as their ability to absorb scientific evidence.

GRAPH 3
A COMPARISON OF TYPES OF INFORMATION BY PUBLIC POLICY AREA

The greater use of external sources and even scientific sources by the environmental area, which is characterized as being more subject to international regulations, external financing evaluation standards, and interactions with NGOs, corroborates the findings of other studies (Abers, 2016; Koga, Filgueiras, Nascimento, Borali & Lima, 2020). The preeminence of scientific sources in the area of infrastructure also is in keeping with empirical studies which indicate a significantly greater level of education among bureaucrats in this area compared to other federal bureaucracies, as well as organizational support for internal training and learning identified within the infrastructure area. (Farias, 2017; Paula et al., 2017). These two factors explain the greater use of scientific sources found among bureaucrats in other countries (Cherney et al., 2015; Landry, Lamari & Amara, 2003).

In order to deepen our understanding of the relationship between control and management in terms of the use of sources of information, it is worthwhile to also analyze their use of internal sources by bureaucrats in the control area to the detriment of other sources, even though in the case of scientific sources, they are slightly above the central, social and economic areas. Two hypotheses that deserve to be explored in future studies are the effects of the growing trend of adopting methodologies which involve the exploring and cross-referencing of databases in control evaluations, as well as the occasional occupying of the knowledge broker position, that is to say being the mediator, intermediary and validator of various sources of information including scientific evidence.

5. FINAL CONSIDERATIONS

Understanding the sphere of action and the consequences of various strategies employed in obtaining information resources is an important element in getting to know the decision-making and behavioral process within the context of public administration. The use of online surveys makes it possible to broaden the scope of this analysis at a relatively low cost.

In this study we have sought to analyze the perception that bureaucrats in the executive branch of the federal administration have of which sources of information they use and we may understand their use as a consequence of their contexts and the works developed in the production of public policy. Considering the critiques of the Evidence-Based Public Policy (EBPP) approach and dialoguing with Pinheiro’s moderate model (2020b) and the findings of other field researchers (Macedo et al., 2019; Veselý et al., 2018), we have begun with the assumption that other sources of information in addition to scientific sources may be used by federal bureaucrats to inform the production of public policy.

A first result is the strong analytical and advisory nature observed in the functions performed by federal bureaucrats. They are very relevant activities related to researching and even producing reliable information that can be consulted. However, one thing that stands out is the fact that this occurs through meeting the demands of control bodies, given that the specialized literature expects the use of evidence to be mobilized with the preponderant goal of supporting public policy production rather than responding to control bodies.

When we observe the types of information used by federal civil servants, another item that stands out is the extensive list of information that comes from the state itself, which reflects the endogenous nature of knowledge which is mobilized to inform the decision-making process and is typical of government bureaucracies which mobilize rational-legal knowledge. However, this result raises the question of the dynamics of information produced by the state internally, given that this type of informational instrument could be acting as an intermediary with other sources of knowledge. Once again, the control entities appear, because they are cited as relevant sources of knowledge. Bearing in mind the debate about the growing influence of control in policy making, it may be questioned whether the findings regarding the configuration of a particular way of using scientific evidence are intermediated by the validation of the control bodies. This point deserves to be investigated especially when there are occasional implications which exacerbate the asymmetries of capacities between management and control, especially in regard to the analytical dimension.

In terms of scientific sources, which are notable for their relatively low usage, we cannot reach conclusions due to the arguments raised above regarding their indirect use through internal sources or even other information sources. Moreover, this study’s results indicate that in various areas, such as infrastructure, and works, such as relational works, there is a differentiated use of this type of source.

It is important to emphasize that there are some limits to our findings. The method employed to observe possible effects due to context on the mobilization of sources of information - correlation analysis and the comparison of medians - should be understood as a first analytical effort which is subject to sample errors and the possible omitted variables and spurious correlations. Future investigations which apply other methods or approaches, such as structural equation modeling or interviewing bureaucrats, will be able to analyze these differences and seek a deeper understanding of the reasons and effects.

In any event, the exploration of the perception of the use of various sources of information within different contexts and public policy works demonstrates that, as expected, there is heterogeneity within the federal bureaucracy. These are the principal contributions of this study to the literature that discusses what informs public policy from a functional perspective of public administration. Works about the use of scientific evidence, especially those that follow the EBPP movement, should seek to understand how in fact the production of public policy functions. The level of insularity of bureaucratic work, as well as the educational level of the individuals, organizational incentives, and their types of social interactions are relevant factors that need to be tested in future studies which seek to explain the choices bureaucrats make in terms of their use of evidence in the production process.

REFERÊNCIAS

• Abers, R. N. (2016). Conflitos, mobilizações, e participação insti- tucionalizada: A relação entre a sociedade civil e a construção de grandes obras de infraestrutura (Textos para Discussão n. 2231). Rio de Janeiro, RJ: Ipea.
• Abers, R. N., Oliveira, M. S., & Pereira, A. K. (2016). Inclusive Development and the Asymmetric State: Big Projects and Local Communities in the Brazilian Amazon. Journal of development studies, 53, 1-16.
• Cairney, P. (2019). The UK Government’s Imaginative Use of Evidence to Make Policy. British Politics, 14(1), 1-22.
• Caplan, N. (1979). The Two-Communities Theory and Knowledge Utilization. American behavioral scientist, 22(3), 459-470.
• Cavalcante, P., Camoes, M. R. S., & Knopp, M. (2015). Burocracia de médio escalão nos setores governamentais: semelhanças e diferenças. In P. Cavalcante, & G. Lotta (Orgs.), Burocracia de Médio Escalão: perfil, trajetória e atuação(pp. 57-90). Brasília, DF: Enap.
• Cavalcante, P., & Lotta, G. (2015). Burocracia de médio escalão: perfil, trajetória e atuação Brasília, DF: Enap.
• Cavalcante, P., & Lotta, G. (2021, fevereiro). Are Governance Modes Alike? An Analysis Based on Bureaucratic Relationships and Skills. International Journal of Public Administration Recuperado de https://doi.org/10.1080/01900692.2021.1874983
  » https://doi.org/10.1080/01900692.2021.1874983
• Cavalcante, P; & Silva, M. (2020). Reformas do Estado no Brasil: Trajetórias, inovações e desafios Brasília, DF: Ipea.
• Cherney, A., Head, B., Povey, J., Ferguson, M., & Boreham, P. (2015, maio). Use of Academic Social Research by Public Officials: Exploring Preferences and Constraints That Impact on Research Use. Evidence & Policy: A Journal of Research, Debate and Practice, 11(2), 169-88.
• Colebatch, H. K., Hoppe, R., & Noordegraaf, M. (2010). Working for policy Amsterdam, Netherlands: Amsterdam University Press.
• Davies, H. T. O., Nutley, S., & Smith, P. (2000). What Works? Evidence-Based Policy and Practice in Public Services Bristol, UK: The Policy Press.
• DeLeon, P. (2008, junho). The Historical Roots of the Field. In R. E. Goodin, M. Moran, & M. Rein (Eds.), The Oxford Handbook of Public Policy Recuperado dehttp://oxfordhandbooks.com/view/10.1093/oxfordhb/9780199548453.001.0001/oxfordhb-9780199548453-e-002
  » http://oxfordhandbooks.com/view/10.1093/oxfordhb/9780199548453.001.0001/oxfordhb-9780199548453-e-002
• Dunn, W. (1980). The two-communities metaphor and models of knowledge use: an exploratory case survey. Knowledge: Creation, Diffusion, Utilization, 1(4), 515-536.
• Escola Nacional de Administração Pública. (2018, junho). Capacidades estatais para produção de políticas públicas: resultados do survey sobre serviço civil no Brasil(Cadernos Enap, 56). Brasília, DF: Autor. Recuperado de http://repositorio.enap.gov.br/handle/1/3233
  » http://repositorio.enap.gov.br/handle/1/3233
• Farias, J. S. (2017). O apoio setorial à absorção e à aplicação do conhecimento na área de infraestrutura brasileira: um estudo sob a abordagem exploration e exploitation. In J.M. Paula, P. L. M. Palotti, P. L. C., Cavalcante , & P. A. Alves. (Orgs.), Burocracia federal de infraestrutura econômica: reflexões sobre capacidades estatais(pp. 353-390). Brasília, DF: Enap.
• Filgueiras, F. (2018). Burocracias do controle, controle da burocracia e accountability no Brasil. In R. Pires, G. Lotta, & V. Oliveira (Eds.), Burocracia e políticas públicas no Brasil: intersecções analíticas Brasília, DF: Ipea , Enap.
• Filgueiras, F., Koga, N., & Viana, R. (2020). State Capacities and Policy Work in Brazilian Civil Service. Revista de Sociologia e Política, 28(74), e004.
• Fischer, F., & Gottweis, H. (2012). The Argumentative Turn Revisited: Public Policy as Communicative Practice Durham, North Carolina: Duke University Press.
• Grin, E. J. (2020). Controle externo e policy making? A atuação do TCU junto à administração pública federal (Texto para Discussão). Rio de Janeiro, RJ: Ipea .
• Hair, J. F., Anderson, R. E., Tatham, R. L., & Grablowsky, B. J. (1979). Multivariate data analysis with readings Tulsa, Oklahoma: Petroleum Publishing Company.
• Howlett, M., & Wellstead, A. M. (2011). Policy Analysts in the Bureaucracy Revisited: The Nature of Professional Policy Work in Contemporary Government. Politics & Policy, 39(4), 613-633. Recuperado de https://doi.org/10.1111/j.1747-1346.2011.00306.x
  » https://doi.org/10.1111/j.1747-1346.2011.00306.x
• Humphrey, C., & Owen, D. (2000). Debating the ‘Power’ of Audit. International Journal of Auditing, 4(1), 29-50. Recuperado de https://doi.org/10.1111/1099-1123.00302
  » https://doi.org/10.1111/1099-1123.00302
• Kidjie, S., & Palotti, P. L. M. (2020). How Do Middle Managers Matter in Policymaking? Examining Managerial Work and Influence in Brazil. International Journal of Public Administration Recuperado de https://doi.org/10.1080/01900692.2020.1766498
  » https://doi.org/10.1080/01900692.2020.1766498
• Koga, N. M., Filgueiras, F., Nascimento, M. I. B., Borali, N., & Lima, V. (2020). Policy capacity and governance conditions for implementing sustainable development goals in Brazil. Revista do Serviço Público, 71(esp.), 38-77.
• Koga, N. M., & Viana, R. (2020). O trabalho na política pública da burocracia federal brasileira: entre a técnica e a política. In L. Lima, & L. Schabbach (Orgs.), Políticas públicas: questões teórico-metodológicas emergentes Porto Alegre, RS: Editora da UFRGS/CEGOV.
• Landry, R., Lamari, M., & Amara, N. (2003). The Extent and Determinants of the Utilization of University Research in Government Agencies. Public Administration Review, 63(2), 192-205. Recuperado de https://doi.org/10.1111/1540-6210.00279
  » https://doi.org/10.1111/1540-6210.00279
• Lasswell, H. D., & Lerner, D. (1951). The policy sciences: recent developments in scope and method Stanford, California: Stanford University Press.
• Lejano, R. P. (2006). Framework for Policy Analysis: Merging text and context New York, NY: Taylor & Francis Goup.
• Lindblom, C. E. (1959, Spring). The Science of “Muddling Through”. Public Administration Review, 19(2), 79-88. Recuperado de https://doi.org/10.2307/973677
  » https://doi.org/10.2307/973677
• Macedo, A. S., Viana, R., & Nascimento, M. (2019). Capacidades analíticas no processo de produção de políticas públicas: quais fontes de evidências utilizam o serviço civil da Administração Pública Federal?Administração Pública e Gestão Social, 4(11), 1-22.
• Meltsner, A. (1976). Policy analysts in the bureaucracy Berkeley, California: University of California Press.
• Newman, J., Cherney, A., & Head, B. W. (2016). Do Policy Makers Use Academic Research? Reexamining the “Two Communities” of Research Utilization. Public Administration Review, 76(1), 24-32. Recuperado dehttps://doi.org/10.1111/puar.12464
  » https://doi.org/10.1111/puar.12464
• Nogueira, R. A., & Gaetani, F. (2018, dezembro). A questão do controle no debate da governança pública (Boletim de Análise Político-Institucional, 19). Brasília, DF: IPEA.
• Olejniczak, K., Raimondo, E., & Kupiec, T. (2016). Evaluation units as knowledge brokers: testing and calibrating an innovative framework. Evaluation, 22(2), 168-189.
• Oliver, K., Innvar, S., Lorenc, T., Woodman, J. & Thomas, J. (2014). A Systematic Review of Barriers to and Facilitators of the Use of Evidence by Policymakers. BMC Health Services Research, 14(2), 1-12. Recuperado de https://doi.org/10.1186/1472-6963-14-2
  » https://doi.org/10.1186/1472-6963-14-2
• Parkhurst, J. (2017). The politics of evidence: from evidence-based policy to the good governance of evidence London, UK: Routledge.
• Paula, J. M., Palotti, P. L. M., Cavalcante, P. L. C., & Alves, P. A. (2017). Burocracia federal de infraestrutura econômica: Reflexões sobre capacidades estatais Brasília, DF: Enap , Ipea.
• Pinheiro, M. M. S. (2020a). Políticas públicas baseadas em evidências (PPBEs): delimitando o problema conceitual (Texto para Discussão, n. 2554). Rio de Janeiro, RJ: Ipea.
• Pinheiro, M. M. S. (2020b). Políticas públicas baseadas em evidências: uma avaliação crítica (Boletim de Análise Político-Institucional, 1). Brasília, DF: Ipea.
• Pires, R., Lotta, G., & Oliveira, V. E. (2018). Burocracia e Políticas Públicas no Brasil: intersecções analíticas Brasília, DF: Ipea , Enap.
• Pires, R. R. C., & Gomide, A. Á. (2016). Governança e capacidades estatais: uma análise comparativa de programas federais. Revista de Sociologia e Política, 24(58), 121-143. Recuperado de https://doi.org/10.1590/1678- 987316245806
  » https://doi.org/10.1590/1678- 987316245806
• Recuero, R., & Gruzd, A. (2019). Cascatas de Fake News Políticas: um estudo de caso no Twitter. Galáxia, 41, 31-47. Recuperado dehttps://doi.org/10.1590/1982-25542019239035
  » https://doi.org/10.1590/1982-25542019239035
• Shu, K., Silva, A., Wang, S., Jang, J., & Liu, H. (2017). Fake news detection on social media: a data mining perspective ACM SIGKDD Explorations, 19(1), 22-36. Recuperado de https://doi.org/10.1145/3137597.3137600
  » https://doi.org/10.1145/3137597.3137600
• Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Review, 63(2), 129-138.
• Spink, P. (2019). Beyond public policy: A public action languages approach Northampton, UK: Edward Elgar Publishing.
• Thurstone, L.L (1935). The vectors of mind Chicago, Illinois: University of Chicago Press.
• Veselý, A., Ochrana, F., & Nekola, M. (2018). When Evidence is not Taken for Granted: The Use and Perception of “Evidence” in the Czech Republic Ministries. NISPAcee Journal of Public Administration and Policy, 11(2), 219-234. Recuperado de https://doi.org/10.2478/nispa-2018-0020
  » https://doi.org/10.2478/nispa-2018-0020
• Weber, M. (1963). Burocracia. In H. H. Gerth, & C. Wright Mills (Orgs.). Ensaios de Sociologia(pp. 230-281). Rio de Janeiro, RJ: Zahar.
• Weiss, C. (1979). The Many Meanings of Research Utilization. Public Administration Review, 39(5), 426-31.
• Williams, A. (2010). Is Evidence-Based Policy Making Really Possible? Reflections for Policymakers and Academics on Making Use of Research in the Work of Policy. In H. K. Colebatch, R. Hoppe, & M.Noordegraaf, M. (Eds.), Working for policy Amsterdam, Netherlands: Amsterdam University Press .
• Wu, X., Ramesh, M., & Howlett, M. (2015). Policy capacity: a conceptual framework for understanding policy competences and capabilities. Policy and Society, 34(3-4), 165-171. Recuperado dehttps://doi.org/10.1016/j.polsoc.2015.09.001
  » https://doi.org/10.1016/j.polsoc.2015.09.001
• Yanow, D. (2000). Conducting Interpretive Policy Analysis Newbury Park, California: Sage Publications.

• 1
  Pinheiro (2020a, 2020b) proposes his moderate model based on a deeper understanding of the history of the EBPP movement and the production of an official linguistic philosophical analysis through the use of these references in the field of Philosophy.
• 2
  The sample process for this survey had to deal with a decisive technical impediment which was the unavailability of updated data for the work unit and email contact of the Federal Public Administration’s civil servants in 2019. Thus, we decided to use the same Brazilian civil service sample selected by Enap in 2018. The utilization of the previous sample solved our civil servant data problem - above all in terms of their contacts - which was indispensable in terms of conducting our study. This also enabled us to compare some of the results between the two surveys, given that some of the 2017 survey questions were repeated in this study.
• 3
  In its six subject areas, the survey sought to identify: i) the respondents’ functional profile, mapping their type of link to the administration, their career and their ministry, among other things; ii) the area they work in within the field of public policy; iii) their main daily activities and functions at work; iv) the patterns of access and utilization for various types of information (with a focus on the area which we designate as “scientific evidence”) and the individual and organizational contexts which condition the use and non-use of this information, among other aspects; v) the knowledge, abilities and values of the respondents; and vi) their sociodemographic profiles (sex, race/ethnicity, level of education, etc.).
• 4
  The manage team function was placed in the relational profile for theoretical motives. Even though it is divided into three different factors and has a slightly elevated factor load for the administrative factor, we believe that it has a more direct relationship with the relational profile in the bureaucrats’ day to day activities.
• 5
  The Pearson correlation coefficient was chosen because this test makes it possible to measure the strength (positive or negative) of the relationship between two variables in terms of the type of work and type of information. Its application is possible in this case because the variables are normally distributed pair by pair.
• 6
  The Pearson correlation coefficient cannot be applied in this case because one of the variables “public policy area” is categorical. Thus, instead of using another correlation test, we opted to use a descriptive approach through a comparison of medians, which proved to be sufficient in pointing out the differences among the six areas in relation to each type of information source.

• [Translated version] Note: All quotes in English translated by this article’s translator.

• 7
  This method consists of subdividing into subpopulations (strata), then employing a simple random sample algorithm from each stratum. The size of the sample for each stratum depended on various factors: i) the size of the stratum; ii) the homogeneity of the elements within each stratum; iii) the selection cost for each unit within the stratum; and iv) the importance of the estimates for these strata.
• 8
  The interview may be viewed in its entirety at https://www.ipea.gov.br/portal/index.php?option=com_content& view=article&id=37256&Itemid=448

APPENDIX - DETAILING OF THE RESEARCH METHODS EMPLOYED - Data sample and collection

As mentioned above, the data collection occurred through the application of an online survey (LimeSurvey Platform), sent by email to a preselected sample of the target public for this study.

The sampling process had to deal with a decisive technical impediment which was the unavailability of updated data for civil servant emails and control bodies for the Federal Public Administration in 2019. In this way, we opted to use exactly the same sample selected by a previous study of the Brazilian civil service by Enap (2018). On one hand, the utilization of the previous sample solved our lack of data. It also permitted some degree of comparison between the results of the two surveys. On the other hand, this consists of outdated data for civil servants. In addition, the sample parameters of 2017 (total number of them and changes in their composition, etc.) were altered during these two years.

In the 2017 study, the sample selection was based on a universe of 96,543 civil servants within the executive branch of the administration and was performed using a simple stratification technique⁷. In other words, a previously established quantity of civil servants was selected - by lot - for each of the 24 bodies of the executive branch in the Siape database in 2017 (for more detail of the stratified composition, see Enap, 2018). The first collected sample contained 6,055 civil servants. Two more rounds of selection were thus performed using exactly the same method, and thus we arrived at a final number of 18.165 civil servants. These civil servants were invited to participate in this study’s 2019 survey.

After two months of online collection, we obtained 2,180 complete and valid responses, which represented of 12 % of the triple sample (36% of the original sample).

As mentioned above, the applied survey⁸ was basically designed based on the variables proposed in the literature. Over six thematic sessions, we sought to identify through the survey: i) the functional profile of the respondents; ii) their area within public policy iii) the main activities and functions performed by these civil servants in their day to day work; v) their knowledge, abilities and values; and vi) their socio-demographic profiles.

Factor analyses and correlations

As related above, exploratory factor analyses were conducted for the fields “public policy work” and “information instruments”, given that we were seeking to identify the latent dimensions in these types of works and information. In both analyses, three criteria for the retention of the number of factors were compared to arrive at the best decision: the Kaiser criterion, which retains the number of factors when they attain an eigenvalue greater than 1; the cumulative variance criterion, which retains the number of factors when it attains the capacity to explain 60% of the variation in the data; and the Parallel Analysis criterion, which simulates a projection in order to suggest the maximum number of factors that can be retained.

For the “public policy work” field, the Kaiser criterion of cumulative variance suggests the retention of three factors, while the Parallel Analysis criterion suggests the retention of up to six factors. Comparing the results of these criteria, we may conclude that a decision to retain between three and six factors is plausible. The choice to retain four factors was arrived at through a theoretical expectation based on previous studies in which the underlying reality of the data could be satisfactorily defined with four factors. The retention of more than four factors (five or six) resulted in a minimally perceptible gain in the simulated eigenvalues.

For the “informational instruments” field, both the Kaiser criteria and the cumulative variance suggested the retention of three factors. At the same time, the Parallel Analysis suggested the retention of at most four factors. Observing the correlation index, we noted a suggestive pattern for the existence of four factors, Thus, we decided to extract this number.

Based on the assumption that the generated factors are independent, a varimax rotation was chosen. This rotation is of an orthogonal type and assumes this independence. Even though there was no loss of any variables, we chose the criteria to form the factors based on the variables with a factor load above 0.3. Hair et al. (1998, p. 112) presents a table which indicates that factor loads above 0.3 have practical significance for sample sizes above 300. Our EFAs are in keeping with this criterion.

Finally, to analyze the relationship among the factors, we estimated their factor scores through Thurstone’s method (Thurstone, 1935), which basically recalculates factor loads considering the correlations among the original variables.

TABLE A
VARIABLE FREQUENCIES FOR THE “PUBLIC POLICY WORK” FIELD

Function Performed	Always	Frequently	Occasionally	Rarely	Never
Elaborating reports and technical notes	33.15%	30.54%	14.91%	9.75%	11.64%
Providing superiors with advice	25.15%	23.39%	17.97%	10.79%	22.70%
Collecting and analyzing data	19.01%	27.07%	19.11%	13.19%	21.62%
Performing administrative activities	25.55%	20.39%	16.21%	15.33%	22.52%
Managing the team	27.00%	17.74%	16.95%	10.26%	28.05%
Supervising compliance with norms	19.29%	16.59%	13.52%	12.12%	38.47%
Meeting control body demands	13.87%	18.63%	27.79%	18.58%	21.12%
Consulting and meeting the demands of groups of society	9.86%	16.55%	20.77%	17.42%	35.40%
Elaborating, generating and supervising contracts	12.08%	13.80%	14.88%	12.46%	46.78%
Representing their body in meetings	9.04%	15.15%	18.67%	15.75%	41.40%
Elaborating normative texts (ex. bills)	5.28%	11.31%	17.77%	17.07%	48.57%
Organizing events	4.47%	10.12%	19.75%	20.94%	44.73%
Commissioning and validating evaluation studies	3.69%	7.54%	13.05%	19.22%	56.50%
Raising and negotiating financial resources	3.13%	7.91%	8.85%	11.10%	69.01%

• Note: “How often have you performed the following activities related to public policy at work during the past 12 months?” Scale: Always (5), Frequently (4), Occasionally (3), Rarely (2), Never (1). Source: Elaborated by the authors.

TABLE B
VARIABLE FREQUENCIES FOR THE “INFORMATIONAL INSTRUMENTS” FIELD

Type of Information	Always	Frequently	Occasionally	Rarely	Never	Don’t Know
Laws and norms	60.46%	21.65%	10.46%	3.90%	2.29%	1.24%
Consulting colleagues at work	41.28%	34.59%	16.51%	3.58%	2.34%	1.70%
Personal experience	34.72%	29.86%	19.59%	7.80%	5.32%	2.71%
Federal government technical notes	32.75%	29.91%	20.60%	10%	5.14%	1.61%
Governmental information systems and databases	25.92%	23.90%	20.28%	14.36%	13.35%	2.20%
Legal reports and decisions	24.68%	25.50%	20.73%	15.83%	10.96%	2.29%
Control body recommendations	23.94%	25.50%	21.83%	14.86%	11.28%	2.57%
Research articles or books	12.20%	18.26%	27.71%	22.94%	16.56%	2.34%
Journalistic articles	10.05%	14.36%	22.25%	23.76%	26.88%	2.71%
Scientific research reports	6.47%	13.30%	22.80%	25.73%	28.07%	3.62%
International organization recommendations	5.83%	12.39%	18.81%	23.17%	35.87%	3.94%
Best practices of states and cities	4.72%	6.79%	17.43%	26.33%	40.23%	4.50%
Consumer experience and opinions	4.72%	7.25%	19.50%	27.16%	37.16%	4.22%
Participative recommendations	3.81%	7.61%	17.57%	27.11%	39.45%	4.45%
Information generated by interest groups	3.17%	7.80%	17.29%	26.38%	41.61%	3.76%

• Nota: “How often have you used the following types of information at work during the past 12 months?”
• Scale: Always (5), Frequently (4), Occasionally (3), Rarely (2), Never (1).
• Source: Elaborated by the authors.

Publication Dates

• Publication in this collection
  21 Jan 2022
• Date of issue
  Sep-Oct 2021

History

• Received
  23 Dec 2020
• Accepted
  30 Aug 2021