Information systems and intergovernmental relations in Brazilian social policies: a study on users’ adaptations to the local context

Abstract

The goal of the paper is to identify barriers to the adoption of information systems in Brazilian federal government social policies at the local level. The study focuses on users of information systems from other intergovernmental organizations that helped the federal government in the implementation of social public policies, especially at the local level. The federal government has been using information and communication technologies (ICT) to improve federal coordination of various social programs. This means that technology has been another component in the implementation of these policies. This paper adopts the assumption that users of information systems can change the context of technology use, not only the technology or information system itself. These changes, which were not foreseen by the system designers or policymakers, may explain the success or failure of the adoption of these systems in social policies coordinated by the Brazilian federal government. Case studies of two social programs in different areas (education and social assistance) were carried out in two Brazilian states (Pará and São Paulo). The focus of the article was to identify how users of information systems alter the context of information systems use. The findings helped to identify two dimensions that can cause difficulties or changes in the use of intergovernmental information systems at the local level: the technical and administrative capabilities of local governments and the rigidity of information systems and sharing of data and information to subnational participants of federal government social policies.

Keywords:
Intergovernmental relations; Information systems; Social policies

Resumo

O objetivo neste artigo foi identificar barreiras na adoção de sistemas de informação em políticas sociais do governo federal para coordenar e monitorar a implementação dessas políticas no nível local. A análise foi construída a partir de um estudo sobre os usuários desses sistemas de informação que envolvem diferentes níveis federativos, especialmente no âmbito municipal. O governo federal brasileiro vem utilizando as tecnologias de informação e comunicação (TIC) para melhorar a coordenação federativa de diversos programas sociais. Isso significa dizer que a tecnologia é mais um componente na implementação dessas políticas, que são desenhadas para serem compartilhadas com os entes subnacionais, e se tornam, consequentemente, um dos aspectos que merecem ser estudados. Neste trabalho adota-se como pressuposto que os usuários de sistemas de informação podem mudar o contexto do uso da tecnologia e não apenas a tecnologia ou o sistema de informação em si. Essas mudanças de uso que não estavam previstas na formulação do programa podem explicar fatores de sucesso ou fracasso da adoção desses sistemas em políticas sociais coordenadas pela instância federal. Assim, foram realizados estudos de caso de dois programas sociais de diferentes áreas (educação e assistência social) em dois estados brasileiros (Pará e São Paulo). Portanto, o foco da pesquisa foi orientado para identificar como os usuários de sistemas da informação dos programas escolhidos alteram o contexto de uso desses sistemas. As conclusões do estudo identificaram duas dimensões que podem trazer dificuldades ou mudanças quanto ao uso de sistemas de informação intergovernamentais no âmbito local: capacidades técnicas e administrativas dos municípios brasileiros; e, rigidez dos sistemas de informação e compartilhamento dos dados e informações com os entes subnacionais participantes das políticas sociais.

Palavras-chave:
Relações intergovernamentais; Sistemas de informação; Políticas sociais

Introduction

The topic of intergovernmental relations has been widely discussed in Brazilian literature, especially in the context of the decentralization of public policies in the country. This movement gains special attention from the changes that occurred at the end of the dictatorship, such as: (i) the promulgation of the 1988 Federal Constitution, which considerably increased the duties and powers of states and municipalities; and (ii) the state reform movements in the country, which, among other goals, aimed to improve public management by increasing the efficiency and effectiveness of actions carried out by public organizations (Abrucio, 2005; Arretche, 1996; Diniz, Faleiros, Porto, & Ribeiro, 2013; Farah, 2001). In this context, the intense use of information and communication technologies (ICT) in public administration activities aids their fulfillment, which involves the rationalization of internal processes, better public services, and the availability of information on the Internet (Cunha & Miranda, 2013; Ribeiro, 2009; Vaz, 2005).

The use of ICT by the federal government has also been adopted in social policies in order to facilitate their implementation and monitoring. One example is Cadastro Único, created to manage the data of beneficiaries of various social programs in a single registry information system. Among other duties, municipalities are responsible for entering information on citizens who benefit from these programs, while the federal government makes financial resources available and manages such information systems.

Among the studies that discuss the implications of the use of ICT in these social programs as a tool to articulate intergovernmental relations and ensure their effectiveness (Diniz et al., 2013; Licio, 2012; Silva, 2011), few are dedicated to evaluating the design and implementation of information systems in public policies from the perspective of users. In the field of information systems, studies addressing the perspective of users are more common and usually point out ways to overcome the gap between designers and users of these systems (Diniz, Bailey, & Sholler, 2014).

Thus, this paper fills part of this identified gap in the field of public administration by studying information systems in decentralized social policies in Brazil, focusing on the end users of these systems. This study, therefore, addresses the perspective of users of information systems related to social policies of the Ministry of Education (MEC) and the Ministry of Social and Agrarian Development (MDSA). These users work at public schools and the Reference Centers for Social Assistance (CRAS).

The objective of this article is to identify barriers to the adoption of information systems in federal government social policies, analyzing how these systems are used by local public agents. The multilevel framework proposed by Pozzebon, Diniz, and Jayo (2009) was adopted as theoretical framework for analysis. This framework assists in analyzing the role of different stakeholders related to the context of adopting a particular technology and the negotiation process among these agents, whose objective is to guarantee that the chosen technology is embraced. In this article, the framework helps to identify actors that adopt interorganizational information systems in social policies, the implementation characteristics of these systems, as well as barriers in the use of these systems by users at the local level.

After identifying the implications of social policies and intergovernmental relations in information systems in the literature, two social program case studies were carried out in different federal government areas that have adopted the following systems: (a) PDDE Interativo (related to the Ministry of Education) and (b) Sisjovem (related to the Ministry of Social Development). Semi-structured interviews were conducted with local public agents involved with these public policies in the states of Pará and São Paulo and federal government public servants in Brasília. In addition, documents that deal with these information systems, such as laws, news, booklets, and guides, were analyzed.

This study points out that the users of these systems at the local level can alter the context of the use of information systems in order to meet the conditionalities imposed by the federal government to beneficiaries enrolled on these social programs. This means that local public agents – users of the systems that control the social policies – adopt strategies that differ from those originally planned by system designers at the federal level.

Two barriers to the adoption of interorganizational information systems of federal government social policies emerge from this study: (a) the lack of technical and administrative capabilities in the Brazilian municipalities and (b) the rigidity of information systems for sharing data and information with subnational entities. Understanding such issues can help the construction of federal government strategies, minimize existing problems, guarantee the proper implementation of social policies, and stimulate the adoption of strategies that improve intergovernmental relations.

This article contributes to the public administration field by identifying barriers to the use of information systems in decentralized social policies proposed by the Brazilian federal government. In addition, the results of the research can help to create preliminary hypotheses on ways to overcome the barriers to the adoption of information systems in the public sector in general. The findings indicate the need for more flexible systems that can be adapted to the local realities of users.

Next, the article presents a literature review that addresses the subject of intergovernmental relations and social policies in Brazil, followed by an approach on interorganizational information systems in the public sector, and the gap between system designers and users.

Theoretical reference

Intergovernmental relations and social policies in Brazil

The subject of intergovernmental relations has been much discussed in Brazilian literature, especially after the 1980s. Prior to this period, the relations of government organizations was very close to that of a unitary state, characterized by financial and decision-making centralization at the federal level and a subsidiary role for states and local governments. These entities were involved in the specific implementation of public policies that were determined by the federal government (Abrucio & Franzese, 2007; Arretche, 1999; Farah, 2001).

From the 1980s onwards, several factors responsible for changing intergovernmental relations in Brazil can be identified (Farah, 2001). One of them is the promulgation of the 1988 Federal Constitution, which is also one of the milestones of redemocratization. The Constitution modified the responsibilities of the country's subnational entities, making municipalities to become federated entities and giving greater autonomy to local governments (Abrucio & Franzese, 2007). This process was followed by the decentralization of financial resources and responsibilities in different government levels in Brazil. Other important aspects include the country's context of crisis and the implementation of state reform policies that were developed mainly in the 1990s. Such policies aimed to improve public management by increasing the efficiency and effectiveness of the actions carried out by public organizations, including the decentralization of public policies as a goal (Diniz et al., 2013).

It is not possible to affirm if the aims of decentralization have been completely achieved in Brazil, since there is a high concentration of resources at the federal level and the municipalities still strongly depend on financial transfers from federal government programs (Abrucio & Grin, 2015). According to Arretche (1999), there was a positive association in the 1980s and 1990s of this decentralization movement with better public services. The decentralization of policies had the objective of guaranteeing more efficient and transparent services, as well as the possibility of greater accountability. Souza (2002) suggests decentralization can accentuate the disparities between the different regions of the country if the federal government does not act to rebalance them. Therefore, in this model with a greater degree of decentralization, the federal government should act as a coordinator of national public policies in partnership with local public policies and aim to reduce regional inequalities.

In Brazil, several studies on intergovernmental relations and their implications for federalism in the country have been produced. These studies discuss the characteristics of Brazilian federalism and contemplate dilemmas such as recentralization and decentralization processes of public policies (Souza, 2002), and the street-level bureaucrat perspective (Lotta, 2010), among others.

Thus, different analyses have been carried out to explain the dynamics of the relationship between Brazilian levels of government. According to Franzese (2010), the literature has different interpretations about the results of the 1980s decentralization process in Brazil. While some studies conclude that intergovernmental relations are based on recentralization (Almeida, 2005), others highlight the centralization of these relations (Arretche, 1996). A third view poses the role of federal coordination within intergovernmental relations (Abrucio, 2005). According to Ramos and Abrucio (2012), federal coordination is essential to ensure government entities’ interdependence. Another important aspect is that these intergovernmental relations have made provision of public policies increasingly more complex in Brazil:

With varying degrees of autonomy, but with greater scope than in the past, there are municipal, state, and federal managers in education, health, social assistance, as well as other areas that are the focus of important public policies. Thus, coordination across different levels of government becomes more and more necessary. However, the decision of one federated entity is not always in line with others. Understanding this intricate game is a fundamental task for those who are dedicated to studying and working with government programs (Abrucio & Franzese, 2007, p. 1).

Discussions on the concept of federal coordination emerged from a context in which modern states had to meet a series of social demands through the expansion and provision of social policies (Abrucio, 2005). In federal countries, decision-making is shared and governmental entities are interdependent. Thus, in order to achieve common objectives, federal coordination is necessary. “Coordination between diverse levels of government must, first, balance existing forms of cooperation and competition, while remembering that federalism is intrinsically conflictual” (Abrucio, 2005, p. 44). According to Abrucio (2005), coordination among government levels can generate cooperative actions as they optimize the use of common resources, assist less capable or poorer local governments, integrate shared public policies, and avoid predatory financial behavior. On the other hand, if federal coordination is not well structured, public policy may result in the subordination of subnational governments to the federal government rather than partnership, which can affect the interdependence of governmental entities.

The instruments of federal coordination are therefore pillars for the implementation of public policies, acting as:

a form of integration, sharing, and determination in the federations, which are expressed in the legal rules that oblige actors to share decisions and tasks; in political forums and mechanisms for intergovernmental negotiation; in the functioning of representative institutions; and in the coordinating and/or inductor role of the federal government (Viana & Machado, 2009, pp. 810–811).

Thus, federal coordination is fundamental to intergovernmental relations in the context of public policies. The complex relationships of several entities and actors, including civil society demands articulation social policies in Brazil in areas such as education, health, and social assistance. Using the Brazilian health system as an example, Viana and Machado (2009) point out that the country's federal context is characterized by institutional arrangements and rules that look to incorporate territorial interests; an increasing number of actors involved in the implementation of health policy, such as local managers and health counselors; and the definition of responsibilities across different levels of government.

In order to align regional disparities with the need to articulate common national strategies, it is essential that federal and local governments join efforts, thus ensuring the implementation of social policies. In Brazil, to guarantee these common strategies, social policies generally have national coordination, inducing and encouraging subnational governments to assume several responsibilities that permit them to receive funds, fulfill goals, or be punished if they do not achieve defined objectives (Abrucio, 2005).

Decentralization of social policies was adopted in Brazil as a principle, especially from the 1980s (Arretche, 1996). However, from a practical point of view, managing decentralization and its complex intergovernmental relations requires efforts by federal entities to articulate institutional arrangements and meet the objectives of public policies in social areas. Social policies coordinated by the federal government and implemented with the support of subnational entities are examples of such arrangements. These policies are characterized by local governments cooperation at various stages, including their formulation and implementation phases.

According to Franzese (2010), the term intergovernmental relations relates to interactions across different governmental organizations of diverse types and levels in the federal system. According to the author, the need to implement country-wide social policies generates changes within the federal matrix, as well as in the relations and ways of coordinating the entities that compose it. As these relationships have many arrangements, this article focuses on the intergovernmental relations that occur in the process of implementing and operating social policies in Brazil — particularly in the information systems associated with these policies.

The information systems that support social policy implementation play a key role in the constant and efficient interaction between local governments and the federal government. Generally developed by the federal government, these information systems are operated by state and municipal entities with the purpose of adding and validating information. This aids the management of social program activities, such as the constructing indicators, selecting beneficiaries, and developing evaluation and monitoring mechanisms, among others.

The Bolsa Família program is an example of shared management among different levels of government entities, with local levels being responsible for both implementing and controlling the program as well as managing the registration of beneficiaries. For this program, the Ministry of Social Development implemented the Cadastro Único — an information system that depends on intergovernmental relations for its management, since local governments are responsible for including and managing the registration of families in vulnerable social and economic situations (Diniz et al., 2013). Municipalities must accept the rules defined by the federal body in order to access the financial resources of this program, including the mandatory use of the information system created for this public policy. Bolsa Família also defines the responsibilities of each level of government, including penalties for non-compliance with the rules that are defined by the Brazilian federal government.

Despite the difficulties in intergovernmental relations, information systems are considered an essential element for the improvement of social policies in Brazil: “information systems have been considered as a strategic tool to improve the efficiency of public services, generating greater savings for the state, transparency, and quality in meeting the demands of citizens” (Oliveira, Faleiros, & Diniz, 2015).

Therefore, the adoption of information systems becomes a fundamental component in the implementation of federal government social policies in Brazil. However, as there are always barriers to the adoption and use of technology, it is necessary to study how information systems are shared across federal entities at different levels in order to better understand intergovernmental relations in social policies.

Interorganizational information systems and intergovernmental relations

Theories dealing with interorganizational information systems (Robey, Im, & Wareham, 2008) help in the analysis of information systems used by different actors in countries organized under the federal system. Interorganizational systems can be defined as information systems shared by two or more organizations that are designed to link processes in the different organizational structures involved (Kumar & Dissel, 1996; Robey et al., 2008; Silveira, 2005). There are several examples of the use of these systems such as database sharing, communication networks, and other ways of compiling information resources. Although commonly studied in business sectors, interorganizational information systems are also present in public organizations. Studies on this topic discuss the benefits and difficulties of adopting these systems in business organizations and generally focus on three aspects: factors that influence the adoption of these systems by organizations, the impact of these factors on the governance of economic transactions, and their consequences in organizations (Robey et al., 2008).

Studies that directly discuss intergovernmental relations and the adoption of interorganizational information systems are under explored in the literature (Bovens & Zouridis, 2002; Licio, 2012; Silva, 2011). Whereas it is more common to find studies that analyze federalism and intergovernmental relations (Abrucio, 2005; Obinger, Leibfried, & Castles, 2005; Rocha & Faria, 2004) or the use of ICT in the public sector (Diniz, Barbosa, Junqueira, & Prado, 2009; Joena & Wagenaar, 2007; Thurston, 2012), the investigation on the role of information systems to articulate different government levels involved in a single social policy implementation is rather rare.

A few Brazilian studies discuss the implications of ICT use in intergovernmental relations and the effectiveness of implementing public policies (Diniz et al., 2013, 2015; Licio, 2012; Silva, 2011). In these studies, the authors highlight some challenges to the effective use of information systems by their users at the subnational level — which are usually local public managers — such as difficulties with ICT infrastructure and human resources, which many municipalities still face. These difficulties include: not having Internet access or employees capable of operating the information system, the low possibility of users customizing systems, and having few opportunities to participate in the definition or evaluation of these systems.

Such difficulties can result in penalties for subnational entities, such as the exclusion of a municipality from social programs for not meeting system requirements or public policy implementation failures, thus decreasing their effectiveness. In the words of Oliveira et al. (2015), “the indiscriminate proliferation of systems, without the proper training of their users, can have negative consequences such as the exclusion of actors from processes or citizens from accessing public services” (p. 27).

Silveira (2005) analyzes the benefits and difficulties of implementing virtual organizations in Brazilian public administration and points out that the success of intergovernmental coordination depends on cooperation among organizations and the management and resolution of conflicts related to differences of organizational cultures in each level of government, meaning that greatest difficulties of intergovernmental coordination are more often found in the relations among government organizations than in the technology itself.

The gap between designers and users in the implementation of information systems

Failures in implementing information systems are commonly related to differences in understanding between designers and users about the context of using the technology at the local level. These differences in understanding (i.e., gaps) can range from little adaptations in the information systems from users to their complete non use, causing economic and social losses.

From a case study on correspondent banking in Brazil, Diniz et al. (2014) identified that, unlike that commonly pointed out by the predominant theories in the literature on information systems, the success of technology projects for development (Information and Communication Technologies for Development — ICT4D) heavily relies on the initiative of local users. These authors highlight the four most used approaches in understanding the gap between the designers’ perceptions and the users’ experience of information systems in their context: the cognitive approach, the local configuration approach, the contextualist approach, and user agency. The main criticisms of these approaches are that they all focus on technological adaptations and do not consider how context can be altered to guarantee the success of system implementation. According to the authors:

Users are not passive recipients who sit haplessly in a sea of contextual intersections and whose only choices are to accept the design shortfalls or reject technologies. Rather, when the context is more fungible than the technology, users may alter the context of use and aid successful implementation (Diniz et al., 2014, p.16).

These authors argue that local users can change information system implementation by changing the context of use rather than altering the technology itself. Altering the context — which is generally complex and related to political, cultural, financial, organizational issues, among others — for users is more achievable than changing the technology, which requires specific technical knowledge that is often scarce at the local level and rare for the end-user information systems to have.

By studying the use of intergovernmental systems, specifically those associated with social policies, this article identifies barriers to the adoption of these systems at the local level by analyzing how users alter the use context of these information systems. For this, a theoretical lens that deals with this phenomenon from the relations between designers and users of information systems was adopted, focusing on the difficulties of using these systems as well as the solutions found by users that alter the social, economic, cultural, or political context in which they are immersed. The theoretical framework used in this study is explained below.

The multilevel framework as an analytical lens

In order to understand the ways local users adapt information systems developed at the federal level, the multilevel framework proposed by Pozzebon et al. (2009) was employed. This framework, which was updated by Pozzebon and Diniz (2012), is based on three theoretical perspectives (structuration, constructivism, and contextualism). These were combined and developed to be used in studies that aimed to analyze relationships of different social groups involved in technology implementation. Thus, when negotiating their visions and objectives, which are sometimes divergent, the groups involved in an implementation process make (or not) a system come into operation, although sometimes in a different way from what was originally planned.

Table 1 summarizes the main concepts involved in the creation of the multilevel framework used in this study. Contextualism, initially introduced by Pettigrew (1990), has been adopted in information systems research (Walsham, 1993) and considers the importance of the contextual elements involved in the changes brought about by the implementation of new systems. Constructivism is based on the social shaping of technology (Bijker & Law, 1992) and considers the definition of relevant social groups involved in the implementation of new technology (Sahay & Robey 1996); particular perceptions (convergent or divergent) of each group on the new system's objectives and functioning (or interpretive frames, as defined by Bartunek & Moch, 1987); and the negotiation mechanisms adopted by them to establish an agreement for implementation. Finally, structuration, originally proposed by Giddens (1984) and adapted to Orlikowski's (2000) concept of technology-in-practice, considers intelligent and competent improvisations (Ciborra, 1999) derived from intentional actions, intuition, and practical sensitivity, incorporating everyday experiences into system implementation processes.

Table 1
Main concepts and references in the multilevel framework.

	Main concepts	Relevant references
Contextualism	Actions and perceptions of human actors that consider the context in which they take place.	Pettigrew (1990) Walsham (1993)
Constructivism	Technology implementation as the outcome of social negotiation processes among networks of social actors, based on their competing or convergent interpretive frames (set of assumptions).	Bartunek & Moch (1987) Bijker & Law (1992) Sahay & Robey (1996)
Structuration	Technology-in-practice or how people redefine the meaning, properties, and applications of a given technology during and after implementation.	Giddens (1984) Ciborra (1999) Orlikowski (2000)

This multilevel framework allows us to understand and describe the perspective of each of the groups involved in the adoption of certain technology. It is based on the understanding that different social groups involved in a system implementation have different views of the technology, demanding the development of a negotiation process to make it work. This generates, at the end, a certain use of the technology that may be different from what was initially proposed by the team of designers. Thus, to do an analysis that follows the multilevel framework, data collection initially identifies the interest groups involved in the use of information system and their perceptions about the technology (i.e., the context dimension). After identifying the relevant social groups and their respective perceptions of the technology, their negotiation process is analyzed during the development and implementation of the information system (the process dimension). Finally, this adopted framework guides the description of the system's use as a result of the interaction process of the previously defined groups (the content dimension).

This article highlights the dimensions of context, negotiation process and content proposed in the multilevel framework to establish the differences between the way the public policy information system was originally designed and how users in local governments adopt it. This will enable an explanation on the barriers to ICT use in social policies shared across different entities as well as different perceptions about the technologies used in information systems in Brazil.

Methodology

In order to identify the barriers to the use of information systems in social policies that require cooperation between government agencies of different levels, this article's methodology was composed through the theoretical reference, covering the concepts and references around the following themes: intergovernmental relations, federal coordination, relationships between information systems designers and users, interorganizational information systems, and the multilevel framework. In addition, qualitative research and case studies were carried out in order to analyze two Brazilian social programs that used information systems, which were created by the federal government and implemented by local governments in different areas (social assistance and education). These information systems are summarized below:

• Sisjovem - an information system of ProJovem Adolescente program, designed by the MDSA, which has the objective of evaluating and controlling the execution of the municipalities’ program activities;

• PDDE Interativo - a planning and management information system of the MEC's School Development Plan program (PDE Escola), which aims to help schools establish their realities and plan priority actions to improve management.

Sisjovem is part of ProJovem Adolescente, a national program that targets adolescents between the ages of 15 and 17 who are in situations of extreme poverty, at personal or social risk, or from families that are beneficiaries of the Bolsa Família. It aims to help socially vulnerable youths by promoting family and social integration, citizen participation, and school permanence (Brazil, 2008). Program activities encourage social coexistence and general professional courses through groups called collectives, composed of between 15 and 30 young people. The collectives are accompanied by a social counselor and supervised by a CRAS professional with higher education.

Although the program was created in 2008, Sisjovem only started to be used from 2010. The information system was created to assist the management of the program and its main purpose is to control the execution of ProJovem Adolescente's activities, with adolescents having to attend city council collectives on a monthly basis. This helps to define the transfer of financial resources to municipalities, with the amount depending on the number of participants in the program (Diniz et al., 2013). In addition, the system stores information about registrations, participants’ school attendance, and the execution of program activities. In 2013, the system was present in 3674 municipalities spread across 26 Brazilian states and the Federal District.

While the main objective of Sisjovem is to help the management of ProJovem Adolescente, the PDDE Interativo system serves as a diagnostic and planning tool for schools that provide basic education in the country. It aims to develop school management within PDE Escola in order to improve the performance of schools, as measured by the Index of Basic Education Development (IDEB). Schools enter their needs and plans in PDDE Interativo, which is used by the federal government to identify schools that are below average in the IDEB and should be prioritized with regards to financial resources. Schools in 3834 municipalities of 26 states were using the system in 2013.

Mandatory adherence to the information system in order to receive federal government resources can be highlighted as a reason for choosing to study these programs. This means that local governments only receive resources from the federal government if they use the information system associated with the public policy. It should be noted that the use of PDDE Interativo is only mandatory for the schools prioritized in the program, which receive more resources from federal government due to the criteria associated with the IDEB average. Other schools are not obliged to use the system but may adopt it as an instrument for planning their activities.

Although the design of these policies counts on both state and local governments for their implementation, this study focuses on the analysis of end users (schools and the CRAS) and local governments. This is because they are some of the most important stakeholders in this process and are responsible for entering data and information in the systems, which guide decision-making at the federal level.

Regarding the way these systems are implemented, according to Oliveira et al. (2015), the process of building information systems for the implementation of social policies can be more hierarchical or more participatory. Previous studies by these authors on several information systems implemented by the federal government in social contexts have pointed out that Sisjovem had less state and local governments’ participation in its development and evaluation. Thus, decisions about the system's design and management were taken by the MDSA and then just communicated with the other organizations that implements the policy, such as city and social assistance councils. Conversely, PDDE Interativo's implementation process was more participatory, with frequent changes in the system being made after feedback from its users (Oliveira et al., 2015).

In order to produce analyses from different socioeconomic contexts in Brazil, semi-structured interviews were conducted with public managers at the three levels of government (federal, state, and municipal). In total, 27 interviews were conducted at the end of 2012 and the beginning of 2013 — three of them with the federal government, four with state managers, and the others with local governments, the CRAS, and the schools studied. The municipalities visited were Belém, Ananindeua, and Augusto Corrêa, in the state of Pará; and Diadema, São Paulo, and Juquitiba, in the state of São Paulo.

A qualitative analysis of the data was carried out in order to try to understand how the information systems should be implemented, how they are actually used by end users, and the barriers to their use. Therefore, the analysis of the interviews sought to identify if the way in which the managers of the federal government expected the information system to be implemented and used by the local managers is similar to reality; that is, if the local governments are implementing the systems as anticipated or changing their implementation in order to meet the requirements and conditionalities set out in social policy. The adoption of the multilevel framework allowed for the identification of the stakeholders involved in the social policies studied and their interests (the context dimension), as well as the effective use of these systems by local managers (the content dimension), including the different ways found by these managers to meet the social policy requirements proposed by the federal government.

Analysis of the results

The analysis of the results aims to identify, from the multilevel framework, the context and content dimensions in the adoption of information systems in federal government social policies in two programs of the MEC and MDSA: PDDE Interativo and Sisjovem. In addition, it is important to point out differences in the use of the systems by local managers in relation to what the federal government proposed, as well as to understand the alterations made by users of information systems in decentralized social policies. Thus, the article intends to identify elements that demonstrate if the practices related to the use of these systems locally follow what was proposed by the social policy formulator — in this case, the federal government — as well as the barriers to the adoption and use of the information systems analyzed. Such results can help in the understanding and construction of hypotheses about how the federal government can improve intergovernmental relations in social programs that use information systems.

Analysis of the context dimension

The main stakeholders involved in Sisjovem were:

• The Ministry of Social and Agrarian Development (MDSA) — MDSA created the public policy and the information system Sisjovem. It is also responsible for its management. The Ministry created the system with the objective of controlling the participation of young people in the program Projovem Adolescente and defines the financial resources to be transferred to municipalities according to the number of beneficiaries;

• The State Secretaries of Social Assistance — responsible for monitoring the program at the state level;

• Secretaries and social assistance departments in local governments — responsible for helping those in charge of the Reference Centers for Social Assistance (CRAS);

• CRAS — responsible for implementing the program, holding collective meetings, and entering data and information required by Sisjovem.

Regarding the expectations in relation to the use of the system by local governments, the federal government defined the ProJovem Adolescente program according to the provisions of Decree no. 6629/2008 (Brazil, 2008) and Portaria no. 848/2010 (Brazil, 2010). According to these legal provisions, which explain in detail the program responsibilities of each level of government, including the use of Sisjovem, local governments are responsible for completing and updating the databases of subsystems and applications of the Unified System for Social Assistance (SUAS) network. These form the national socio-educational service information system, which is updated at least every three months.

Thus, from the official documents about the program and the interviews carried out in São Paulo and Pará, it was possible to identify the federal government expectations regarding the users (CRAS managers) responsible for entering and updating data in Sisjovem. The local and state secretariats would help CRAS managers to use the Sisjovem. Federal government managers cited some benefits for the federal government that justify the adoption of an information system in this program, such as: increasing program monitoring and its efficiency (the system must be updated monthly on the Internet for the MDSA); facilitating fund transfers to the local governments, because the information system also indicates the number of participants in the program, which is essential information in defining the financial resources available to local governments; and evaluating the program, allowing the government to create indicators or analyze the program from the information available on Sisjovem.

In PDE Escola, which uses PDDE Interativo, the following groups were identified:

• The Ministry of Education (MEC) — MEC created and executed the policy through the Secretariat of Basic Education;

• State and municipal secretariats of education and their Analysis and Approval Committees — responsible for evaluating the plans developed by their schools;

• Schools — responsible for implementing the program at the local level and compiling data in the system.

In PDE Escola the federal government is responsible for formulating public policy and designing the information system. Secretariats and other public organizations of state and local education have the task of evaluating the plans added to it by schools. Finally, school managers are the main users and implementers of the system, as defined by the federal government.

In general terms, in the context dimension, systems management is complex and spread across different levels of government. A network is built around the information systems and responsibilities are distributed among different public organizations. In this sense, both in Sisjovem and PDDE Interativo, the stakeholders involved and the responsibilities defined by the federal government can be classified as follows: (i) the policy and system formulator, where the federal government is responsible for defining the rules of social policies and information systems use, as well as financial resources for program implementation; (ii) the auxiliary and subsidiary organizations — these are the state and local governments that help end users (previously defined by the federal government) to implement the information systems for social policies; and (iii) the end users (schools, hospitals or CRAS, for example) responsible for entering and validating information, as well as managing the systems at the local level.

Analysis of the content dimension

The objective of this dimension is to identify how the information systems are used. For information systems in social policies, the federal government defines the roles and responsibilities of the different governmental entities. The information systems should be implemented according to what was predicted when they were designed. However, in practice, the systems are used by the municipalities in a way that is a little different from what was expected by the policymaker.

In relation to Sisjovem, most local governments respondents stated that the municipal secretariats were using it in a centralized fashion, despite the federal government having established CRAS employees as responsible for doing so. According to them, the main difficulties that led to this situation are related to the lack of training or information needed for the CRAS employees to use the system. For example, in Belém, capital of Pará state, the local government stipulated — even though the MDSA did not request it — that the advisors of the ProJovem Adolescente program should have higher education. The large number of activities in the CRAS has also been cited as contributing to the municipal secretariats’ task centralization, since the managers offer multiple activities in these centers and have several assignments. One of the reasons cited for this overload is that the information systems are not integrated. MDSA and other public organizations have different systems that often require the same information. These systems must be used by CRAS employees; that is, they do the same work more than once.

Moreover, Internet access in the CRAS is limited and very slow, making it very difficult for managers to use the information system. In addition, the Sisjovem has been criticized for not always working, which negatively affects the local governments as they can miss deadlines for updating data and, consequently, do not receive financial resources from the federal government.

Regarding the ideal use of PDDE Interativo by schools, the MEC expects them to use the system to improve information for diagnoses as well as to plan actions to enhance school management. According to the rules formulated by the Ministry, the information system must be used collaboratively by various stakeholders in the school such as teachers, principals, and other employees. In practice, some schools do not use the PDDE Interativo in such a way. For example, in one of the municipalities of Pará, those who were more familiar with the statistical data of the school filled this type of information and other information was added by others. In other cases, the education secretariats were also using the information system in a centralized way. It was noticed that, although diagnosis and planning must be done by the school, there is a concern related to entering data correctly in order to avoid penalties for both local governments and schools. It was also mentioned that, even though most schools have Internet access, it is often slow and the information system sometimes ceases to function properly. Work overload in schools has also been pointed to as something that affects information system usage, since it requires employees to work overtime, including weekends.

Another issue pointed to by users of both information systems was the association of the system login and password with only one CPF (the Brazilian Individual Taxpayer Registry) and email address. In general, more than one person used the information systems with only one login. Some respondents reported that this generated access difficulties. For example, if someone uses an incorrect password or login, the system is locked and access to the system is denied. Thus, respondents suggested that more logins should be created to avoid such problems.

These are some examples of rules defined by the federal government that, in practice, generate barriers or difficulties for users at the local level. It was also observed that end users changed the way information systems were implemented, often to ensure that the federal government's requirements were complied with. This, in fact, guaranteed the full functioning of the information systems studied and, consequently, the implementation of federal government social policies.

Discussion of results

Based on interviews with managers at federal, state, and local (schools and the CRAS) levels and the analysis of information through the multilevel framework — bearing in mind that users at the local level can alter the context and use of information systems — it is possible to highlight two main barriers for local users of these federal government systems:

• Technical and administrative difficulties in the use of information systems by local managers;

• Rigidity of information systems and sharing of data and information with local entities.

The first point concerns the contextual changes made by local governments to meet the requirements of interorganizational information systems. These changes occurred due to technical difficulties, such as slow Internet or a lack of computers; and administrative issues, such as work overload or users who were not able to properly use the systems. Thus, these systems were expected to be used by schools or the CRAS employees, but this requirement was not always met. The research identified that employees of the municipal secretariats were performing this task rather than schools or CRAS employees. This inversion was frequently found in the CRAS and, to a lesser extent, in the schools.

Therefore, Brazilian federal government mandated the public agents of the CRAS and schools to be responsible for entering and managing data in the information systems. These managers should have the most up-to-date and correct information as they directly work with the programs’ target audience. However, as discussed above, in reality, this does not happen and users utilize other departments of the local government (such as municipal secretariats) to ensure that the information required by the federal government is sent through the information system and, consequently, the local government can receive financial resources or avoid penalties for not sending the requested information.

It is important to emphasize that the adoption and use of ICT by local governments is not equal, and, therefore, municipalities have different human and technological resources. Such differences may also mean that local governments access and participation conditions related to social policies of the federal government are not the same. According to the 2012 Basic Municipal Information Survey – MUNIC (Brazilian Institute of Geography and Statistics [IBGE], 2012), produced by the Brazilian Institute of Geography and Statistics (IBGE), all Brazilian local governments used computers (100%) and almost all of them had Internet access in 2012 (99.8%).

The 2015 ICT Electronic Government survey (Brazilian Internet Steering Committee [CGI.br], 2016) also showed that Brazilian municipalities already have computer and Internet access, but they have not yet explored all the possibilities of using ICT in both internal activities and interaction with society. For example, less than half of Brazilian local governments have an information technology department (41%) or use information systems to support decision-making (18%). In relation to interactions with citizens, 88% of local governments had a website in 2015. Among these, most of them allow documents or forms to be downloaded through their website (77%), while few of them provide more sophisticated online services such as issuing tax bills (33%), obtaining documents such as licenses, certificates, permits and others (31%), or scheduling appointments (16%).

Therefore, the first point that must be considered by the federal government is the technical and administrative capabilities of local governments. Strategies should be created to minimize differences and ensure that local governments can partake in social programs by meeting the requirements needed to receive financial resources from federal government.

The second point concerns the rigidity of the systems proposed by the federal government, which do not allow any local public agent discretion in the use of information systems. One of the risks of decentralized social policies is making local governments mere subordinates, disregarding the presuppositions of federal coordination, such as interdependence of government levels. In addition, respondents criticized the lack of integration of different social policy information systems, forcing users to enter the same information for each social program. They also questioned the fact that local agents are not always allowed to access the information that they enter in these systems — information that could be used in local government activities. It would also be important for local governments to be able to customize information systems in order to use federal social policy data for their local policies. In the words of Oliveira et al. (2015):

In standardizing the decision-making process of bureaucrats by proposing the “correct” or official interpretation of the law, information systems can over-reduce the scope of this discretion, introducing greater hierarchical discipline in public organizations. In some cases, street-level bureaucrats may lose their primary role, limiting themselves to simply helping public service users with forms.

Therefore, the federal government can observe points related to the importance of using the systems so that they also meet local specificities and needs, avoiding situations such as those occurring in Sisjovem: the system functions only as a formal monitoring instrument of those who attend the program and the local public agents cannot use such data to rethink or evaluate policy at the local level. In schools, PDDE Interativo only allows financial resources to be used for certain situations previously defined by Ministry of Education. Thus, if schools have other needs, they cannot discuss them with the federal government or propose alternative resource usages, inhibiting innovation and creativity in proposing initiatives.

Fig. 1 shows research results and some proposals to federal government's improvement of information systems for decentralized social policies.

Fig. 1
Implementation of information systems in intergovernmental relations.

Final considerations

The Brazilian federal government has been using online information systems to implement programs in several areas such as decentralized social policies. Generally, municipalities can join federal government programs if they adhere to a number of conditions. Among them is the use of information systems as a mandatory tool, guaranteeing the implementation, monitoring, and evaluation of social policies. Initially, this obligation is extremely important to the improvement of the implementation of public policies at the federal level and it induces local governments to ICT use. Thus, the use of these systems can generate positive political and managerial outcomes, including efficiency improvements in social policies as well as increased transparency and monitoring of government activities.

Despite the importance of the theme, there are still few studies that analyze the use of information systems in social policies. This works fills part of this gap by analyzing two federal government social programs and focusing on the users of these information systems at the local level. Based on the assumption that users can alter the context of technology use — not only the technology or the information systems — the interviews with local government managers of the states of São Paulo and Pará indicated a series of technical and administrative difficulties to the proper usage of the systems. In order to use the Sisjovem and PDDE Interativo and meet the requirements of the federal government, local government managers have made certain changes. For example, they pass on the task of entering information required in the information systems to other actors, which was not foreseen by the federal government.

In addition, from the interviews and official documents of the social programs (such as legislation and reports), it was noticed that the federal government defines detailed rules and designs information systems that cannot be adapted by local users’ needs. This means that local government managers cannot have alternatives to create new ways of using the systems, or adapt them to the local context. Assuming that the federal government has to guarantee federal coordination, it is essential to create conditions so that local public agents do not just enter data in the information systems, they are able to participate in the construction of polices and technologies, and can use data in their own decision-making processes.

It is important for future studies to identify the characteristics of these information systems that contribute to the improvement of social policies, besides expanding analysis on the participation of local governments in system design and implementation, as well as the evaluation of local public policies. In addition, the analysis of results from other perspectives should be mentioned as a limitation of this article. This includes more in-depth study of the interpretations and perceptions of local users about the operation and use of these systems, as well as those of other stakeholders who are not part of government spheres, such as civil society organizations and citizens. Such limitations create opportunities for future research that could further study the effects of technology use on the relationships between different governmental organizations.

This paper also corroborates the results found by Diniz et al. (2014) on the implementation of banking correspondents in Brazil, which demonstrate that users of information systems can alter the context of technology use. Thus, major theoretical contributions of this article include the use of the multilevel framework (Pozzebon et al., 2009) in intergovernmental information systems, as well as analysis focused on context changes by local users, as proposed by Diniz et al. (2014), analyzing these theoretical models from other technological tools. Moreover, this article explores a subject little studied in the field of public administration: the adoption of ICT in intergovernmental relations and the actions of local users that guarantee its success. This means that, although users (such as public schools and CRAS agents) are not able to modify information systems according to their needs, they can alter the use context — social, organizational, political, cultural, among others — thus ensuring the better functioning of information systems in Brazilian federal government social policies.

• Peer Review under the responsibility of Departamento de Administração, Faculdade de Economia, Administração e Contabilidade da Universidade de São Paulo – FEA/USP.

References

• Abrucio, F. L. (2005). A coordenação federativa no Brasil: A experiência do período FHC e os desafios do governo Lula. Revista de Sociologia e Política, Curitiba, 24, 41-67.
• Abrucio, F. L., & Franzese, C. (2007). Federalismo e políticas públicas: O impacto das relações intergovernamentais no Brasil. In: M. F. Infante Araújo, L. Beira. (Org.). Tópicos de Economia Paulista para Gestores Públicos (Vol. 1) (pp. 13–31). FUNDAP: São Paulo.
• Abrucio, F. L. (Org.), & Ramos, M. N. (Org.) (2012). Regime de Colaboração e Associativismo Territorial Arranjos de Desenvolvimento da Educação (Vol. 1) (p. 151). São Paulo: Editora Moderna/Fundação Santillana.
• Abrucio, F. L., & Grin, E. J. (2015). From decentralization to federative coordination: The recent path of intergovernmental relations in Brazil. In International Conference on Public Policy. Retrieved from http://www.icpublicpolicy.org/conference/file/reponse/1434043516.pdf
  » http://www.icpublicpolicy.org/conference/file/reponse/1434043516.pdf
• Almeida, M. (2005). Recentralizando a federação. Revista de Sociologia e Política, Curitiba
• Arretche, M. T. S. (1996). Mitos da descentralização: Mais democracia e eficiência nas políticas públicas? Revista Brasileira de Ciências Sociais, 31
• Arretche, M. T. S. (1999). Políticas sociais no Brasil: Descentralização em um Estado federativo. Revista Brasileira de Ciências Sociais, 14(40).
• Bartunek, J. M., & Moch, M. (1987). First order, second order, and third order change and organization development interventions: A cognitive approach. Journal of Applied Behavior Science, 23(4), 483-500.
• Bijker, W. E., & Law, J. (1992). Shaping technology/building society: Studies in socio-technical change. Cambridge, MA: MICT Press.
• Bovens, M., & Zouridis, S. (2002). From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review, 62(2), 174-184.
• Brazilian Institute of Geography and Statistics – IBGE. (2012). MUNIC 2012.. Retrieved from http://www.ibge.gov.br/home/estatistica/economia/perfilmunic/2012/
  » http://www.ibge.gov.br/home/estatistica/economia/perfilmunic/2012/
• Brazil. (2008). Decreto 6.629 Retrieved from http://www.planalto.gov.br/ccivil_03/_Ato2007-2010/2008/Decreto/D6629.htm
  » http://www.planalto.gov.br/ccivil_03/_Ato2007-2010/2008/Decreto/D6629.htm
• Brazil. (2010). Portaria no 848 Retrieved from www.mds.gov.br/acesso-ainformacao/legislacao/assistenciasocial/portarias/2010/PortariaMDS_no_848_de_28-12-2010.pdf
  » www.mds.gov.br/acesso-ainformacao/legislacao/assistenciasocial/portarias/2010/Portaria_MDS_no_848_de_28-12-2010.pdf
• Ciborra, C. (1999). Notes on improvisation and time in organizations. Accounting, Management & Information Technologies, 9(2), 77-94.
• Cunha, M. A. V., & Miranda, P. R. M. (2013). O uso de TIC pelos governos: Uma proposta de agenda de pesquisa a partir da produção acadêmica e da prática nacional. Revista O&S, 543-566.
• Diniz, E. H., Barbosa, A. F., Junqueira, A. R. B., & Prado, O. (2009). O governo eletrônico no Brasil: Perspectiva histórica a partir de um modelo estruturado de análise. Revista de Administração Pública, 43(1), 23-48.
• Diniz, E. H., Faleiros, S. M., Porto, L. C., & Ribeiro, M.M. (2013). Fluxos de informações entre entes federados para a construção de políticas sociais. Retrieved from http://participacao.mj.gov.br/pensandoodireito/wp-content/uploads/2013/11/Volume-49-IIEAESP-Fluxo-de-Pol%C3%ADticas-Sociais.pdf
  » http://participacao.mj.gov.br/pensandoodireito/wp-content/uploads/2013/11/Volume-49-IIEAESP-Fluxo-de-Pol%C3%ADticas-Sociais.pdf
• Diniz, E. H., Bailey, D. E., & Sholler, D. (2014). Achieving ICT4D project success by altering context not technology. Information Technologies & International Development, 10(4), 15-29.
• Farah, M. F. S. (2001). Parcerias, novos arranjos institucionais e políticas públicas no nível local de governo. Revista de Administração Pública, Rio de Janeiro, 35(1), 119-144.
• Franzese, C. (2010). Federalismo cooperativo no Brasil: Da Constituição de 1988 aos sistemas de políticas públicas Doctoral Thesis. São Paulo, Brazil: Getulio Vargas Foundation. Retrieved from http://bibliotecadigital.fgv.br/dspace/handle/10438/8219
  » http://bibliotecadigital.fgv.br/dspace/handle/10438/8219
• Giddens, A. (1984). The constitution of society. Berkeley, CA: University of California Press.
• Internet Steering Committee – CGI.br. (2016). In A. F. Barbosa (Ed.), Survey on the use of information and communication technologies in the Brazilian public sector – ICT Electronic Government 2015 São Paulo: CGI.br. Retrieved from http://cetic.br/media/docs/publicacoes/2/TIC_eGOV_2015_LIVROELETRONICO.pdf
  » http://cetic.br/media/docs/publicacoes/2/TIC_eGOV_2015_LIVRO_ELETRONICO.pdf
• Joena, F., & Wagenaar, P. (2007). Discretion and digital discipline. Public Administration, 85(1), 189-214.
• Kumar, K., & Dissel, H. G. V. (1996). Managing conflict and cooperation in interorganizational systems. MIS Quarterly, 20(3), 279-300.
• Licio, E. C. (2012). Para além da recentralização: Os caminhos da coordenação federativa do Programa Bolsa Família (2003–2010) Doctoral Thesis. Brasília, Brazil: University of Brasília. Retrieved from http://repositorio.unb.br/handle/10482/10759
  » http://repositorio.unb.br/handle/10482/10759
• Lotta, G. S. (2010). Implementação de Políticas Públicas: O impacto dos fatores relacionais e organizacionais sobre a atuação dos Burocratas de Nível de Rua do Programa Saúde da Família Doctoral Thesis. São Paulo, Brazil: University of São Paulo. Retrieved from http://www.teses.usp.br/teses/disponiveis/8/8131/tde-20102010-120342/pt-br.php
  » http://www.teses.usp.br/teses/disponiveis/8/8131/tde-20102010-120342/pt-br.php
• Obinger, H., Leibfried, S., & Castles, F. G. (2005). Introduction: Federalism and welfare state. In H. Obinger, S. Leibfried, Castles, & G. Francis (Eds.), Federalism and the welfare state: New world and European experiences (pp.1–64). Cambridge: Cambridge University Press.
• Oliveira, L. C. P., Faleiros, S. M., & Diniz, E. H. (2015). Sistemas de informação em políticas sociais descentralizadas: Uma análise sobre a coordenação federativa e práticas de gestão. RAP, 49(1).
• Orlikowski, W. J. (2000). Using technology and constituting structures: A practice lens for studying technology in organizations. Organization Science, 11(4), 404-428.
• Pettigrew, A. M. (1990). Longitudinal field research on change: Theory and practice. Organization Science, 1(3), 267-292.
• Pozzebon, M., & Diniz, E. H. (2012). Theorizing ICT and society in the Brazilian context: A multilevel, pluralistic and remixable framework. BAR, 9(3), 287-307.
• Pozzebon, M., Diniz, E. H., & Jayo, M. (2009). Chapter II: Adapting the structurationist view of technology for studies at the community/societal levels. In Y. K. Dwivedi, Y. K. Dwivedi, et al. (Eds.), Handbook of research on contemporary theoretical models in information systems (pp. 18–33). Hershey, PA: IGI Global.
• Ribeiro, M. M. (2009). Como os estados brasileiros promovem a transparência nos portais de compras eletrônicas? In Paper presented at Annals of II Congresso CONSAD Retrieved from http://consad.org.br/wp-content/uploads/2013/02/COMO-OS-ESTADOS-BRASILEIROS-PROMOVEM-A-TRANSPAR%C3%8ANCIA-NOS-PORTAIS-DE-COMPRAS-ELETR%C3%94NICAS4.pdf
  » http://consad.org.br/wp-content/uploads/2013/02/COMO-OS-ESTADOS-BRASILEIROS-PROMOVEM-A-TRANSPAR%C3%8ANCIA-NOS-PORTAIS-DE-COMPRAS-ELETR%C3%94NICAS4.pdf
• Robey, D., Im, G., & Wareham, J. D. (2008). Theoretical foundations of empirical research on interorganizational systems: Assessing past contributions and guiding future directions. Journal of the Association for Information Systems, 9
• Rocha, C. A., & Faria, C. A. (2004). Cooperação intermunicipal, reterritorialização da gestão pública e provisão de bens e serviços sociais no Brasil contemporâneo: Experiência dos consórcios de saúde em Minas Gerais. Cadernos Metrópole, 11, 73-105.
• Sahay, S., & Robey, D. (1996). Organizational context, social interpretation, and the implementation and consequences of geographic information systems. Accounting, Management and Information Technology, 6(4), 255-282.
• Silva, F. J. P. (2011). Inovações tecnológicas no serviço público brasileiro: O caso do Sistema de Gestão de Convênios e Contratos de Repasse – Portal SICONV Master’s Dissertation. Brasília, Brazil: The University of Brasília. Retrieved from http://repositorio.unb.br/handle/10482/9881
  » http://repositorio.unb.br/handle/10482/9881
• Silveira, H. (2005). Gestão da informação em organizações virtuais: Uma nova questão para a coordenação interorganizacional no setor público. Ciência da Informação, 34(2), 70-80.
• Souza, C. (2002). Governos e sociedades locais em contextos de desigualdades e de descentralização. Ciência e Saúde Coletiva, 7(3), 431-442.
• Thurston, A. (2012). Public records: Evidence for openness. London: Uni-versity of London. Retrieved from http://blogs.estadao.com.br/publicos/files/2012/08/Public-Records-as-Evidence-for-Openness-FINAL.doc.pdf
  » http://blogs.estadao.com.br/publicos/files/2012/08/Public-Records-as-Evidence-for-Openness-FINAL.doc_.pdf
• Vaz, J. C. (2005). Governança eletrônica: Para onde é possível caminhar? Retrieved from http://www.polis.org.br
  » http://www.polis.org.br
• Viana, A. L. A., & Machado, C. V. (2009). Descentralização e coordenação federativa: A experiência brasileira na saúde. Ciência Coletiva e Saúde, 14(3), 807-817.
• Walsham, G. (1993). Interpreting information systems in organizations Cambridge, MA: John Wiley and Sons.

Publication Dates

• Publication in this collection
  Jan-Mar 2018

History

• Received
  22 June 2016
• Accepted
  29 May 2017