Transparency and accountability of government algorithms: the case of the Brazilian electronic voting system

Abstract

In recent years a number of open data movements have emerged around the world, ensuring citizens more opportunities to access information, transparency being a factor associated with trust in public organizations and government. The transparency in algorithms translates into the knowledge of the steps performed and the criteria adopted to obtain a certain result. The objective of this study is to identify the characteristics of transparency and accountability of the Brazilian electronic voting system. Through the case study, recommendations and good practices of transparency in algorithms were examined with the particularities of the Brazilian electronic voting system, which provides control and oversight to society. This study advances the discussion on the influence of new technologies on democracy, placing the Brazilian electronic voting system within the limits of transparency and accountability of algorithms and the government in general.

Keywords:
Electronic governing; Digital governance; Electronic ballot; Good practices; Principles

Resumo

Nos últimos anos, uma série de movimentos de dados abertos tem surgido ao redor do mundo, assegurando aos cidadãos mais oportunidades para acessar informações, sendo a transparência um fator associado à confiança nas organizações públicas e no governo. A transparência em algoritmos traduz-se no conhecimento dos passos realizados e critérios adotados para a obtenção de determinado resultado. O objetivo deste estudo consiste em identificar as características de transparência e accountability do sistema eletrônico de votação brasileiro. Por meio do estudo de caso, foram confrontadas recomendações e boas práticas de transparência - que propiciam controle e fiscalização por parte da sociedade em algoritmos - com as particularidades do sistema de votação eletrônico brasileiro. Este estudo avança na discussão da influência das novas tecnologias na democracia, situando o sistema eletrônico de votação brasileiro nos limites da transparência e accountability de algoritmos e do governo em geral.

Palavras-chave:
Governo eletrônico; Governança digital; Urna eletrônica; Boas práticas; Princípios

Resumen

En los últimos años, han surgido varios movimientos de datos abiertos en todo el mundo, que garantizan a los ciudadanos más oportunidades para acceder a la información, y la transparencia es un factor asociado con la confianza en las organizaciones públicas y el gobierno. La transparencia en los algoritmos se traduce en el conocimiento de los pasos realizados y los criterios adoptados para obtener un resultado determinado. El objetivo de este estudio es identificar las características de transparencia y responsabilidad del sistema electoral electrónico brasileño. A través del estudio de caso, las recomendaciones y las buenas prácticas de transparencia se confrontaron con las particularidades del sistema electoral electrónico brasileño, que proporciona control y supervisión por parte de la sociedad en algoritmos. Este estudio avanza en la discusión sobre la influencia de las nuevas tecnologías en la democracia, colocando el sistema electoral electrónico brasileño dentro de los límites de transparencia y responsabilidad de los algoritmos y del gobierno en general.

Palabras clave:
Gobierno electrónico; Gobernanza digital; Urna electrónica; Buenas prácticas; Principios

INTRODUCTION

In recent years, a series of open data movements have appeared around the world (ATTARD, ORLANDI, SCERRI et al., 2015). Citizens have had more opportunities to search for and access information directly from the government, with transparency being a factor associated with the population’s trust in public organizations and the government (ALBU and FLYVERBOM, 2016).

Within this context, transparency is understood to be making information available, which permits external actors to monitor performance, from the perspective of citizen participation in the process, and as a result, their capacity to evaluate the actions of authorities (GRIMMELIKHUIJSEN and WELCH, 2012). Even though there is no consensus in these approaches in terms of what transparency is, this field of research agrees that it involves the extent to which stakeholders have access and a similar understanding of what information is necessary without loss, noise, delay or distortion (PAPENFUSS and SCHAEFER, 2015). Accountability consists of the duty of an individual or organization to be responsible in some way for the way they conduct their actions (HOOD, 2010). Transparency and accountability are constructs which are closely linked, with transparency being one of the components of the process of accountability in the public sector (VASCONCELLOS, LUNKES and TALIANI, 2018).

Recent discussions have focused on the transparency of algorithms which are being used to make decisions and are being incorporated into public systems, such as transport, health and policing (FINK, 2018). Algorithms consist of a structured group of information system commands, designed to process instructions and information to generate a result, with it being true that in information societies, the activities, choices and decisions previously decided by humans are now being more and more often handled by algorithms, which can advise and sometimes decide how data should be interpreted and what actions to adopt (MITTELSTADT, ALLO, TADDEO et al., 2016).

Algorithms have become more and more autonomous and invisible, making it difficult for the public to examine and identify their impartiality, and sometimes they introduce inadvertent biases, reinforcing historic discrimination, favoring a political orientation or reinforcing undesirable practices (JANSSEN and KUK, 2016). According to the World Wide Web Foundation (2017, p. 16), making an algorithm more accountable is “making sure that data can be evaluated, controlled and corrected.”

Fink’s studies (2018) about the transparency of the algorithms used by the American government indicate that its policies and practices related to the publishing of algorithms are inconsistent, with there being a need for more effective mechanisms to promote accountability and future studies which will consider the transparency of algorithms in the debate about the limits of accountability and the government in general. Ananny and Crawford (2018) identify limits to transparency and propose a starting point for a algorithmic accountability model. It should be noted that there is a prevalence of theoretical studies in this field (MACHADO, 2018), which reveals the importance of empirical research about algorithmic transparency.

The purpose of this study consists in identifying the characteristics of transparency and accountability in the Brazilian electronic voting system, which has been considered innovative in applying information and communications technology to public management (BALBE, 2014).

In this article we contextualize the importance of this subject in modern information societies from the perspective of new technologies and democracy, digital governance, and algorithmic transparency and accountability. This is followed by a description of this study’s methodological steps. Then the model and proposed analysis dimensions are presented and compared with the particular characteristics of the Brazilian electronic voting system. Finally, the results of this case study are presented, and an agenda for future research regarding algorithmic transparency and accountability in the public sector is suggested.

Digital Governance

New technologies have the potential to construct a new relationship between governments and citizens, making possible more efficient, democratic and transparent public administration (GUIMARÃES and MEDEIROS, 2005).

In this sense, the governance of the public sector can be defined, according to Lane (2000), as a group of theories about how governments articulate themselves to provide services to society. Public governance has been understood as a process through which institutions, organizations and citizens guide themselves, treating the interaction between the public sector and society and how they organize themselves to make collective decisions, in order to provide transparent mechanisms so that these decisions materialize (UNITED NATIONS, 2002).

Electronic or digital governance, in turn, refers to the way in which the internet can improve the capacity of the State to govern and formulate its policies, being defined as the utilization of innovative information and communications technologies by the public sector, such as the internet, to offer citizens quality services, reliable information, and more knowledge, to facilitate access to government processes and encourage their participation (UNITED NATIONS, 2002).

According to Grande, Araujo and Serna (2002), electronic governance has been studied by three analytic models denominated as (a) e-administration: related to the providing of services; (b) e-democracy: related to the incentives for consultation and the extension of democratic processes and (c) e-governance: related to making processes more dynamic in the elaboration of public policies. To Okot-Uma (2000, p. 5), electronic governance involves “[...] new styles of leadership, new ways of accessing [public] services, new ways to hear citizens [...] and new ways to organize and supply information.”

The pressure for public organizations to act with efficiency, effectiveness, transparency and control mechanisms and presenting accounts has led to the adoption of management models based on transparency focusing on results, with information and communications technology (ICT) instruments being essential to this strategy (BARBOSA, FARIA and PINTO, 2007).

Transparency and accountability

Transparency consists of the availability of information on an organization, which enables external actors to monitor its internal functioning or performance, which is more and more seen as a mark of good governance (RUIJER and MEIJER, 2016). Transparency is considered to be a condition of accountability - which is frequently associated with the concept of responsibility - and a fundamental requirement for good governance (GOEDE and NEUWIRTH, 2014).

The term accountability has been used in the American literature and a large portion of Brazilian literature to address issues related to displaying the accounts of those who govern and their democratic responsibilities (PRADO, 2009). Abrucio and Loureiro (2004) define democratic accountability as a construct of institutional mechanisms by which those who govern are held responsible for their acts by those they govern.

According to the United Nations, accountability is one of the prerequisites for democracy and good government in which those who govern and public servants are responsible for their actions and decisions. Accountability identifies who is responsible for what and what type of conduct is illegal (UNITED NATIONS, 2006).

Raupp and Pinho (2013) point out that the concept of accountability can be perceived in light of the dimensions of transparency, participation and the presenting of accounts.

Transparency can be understood as a rule of conduct to be followed by public agents, so that their actions are open to society making social control by citizens and institutions possible (FOX, 2007). Grimmelikhuijsen and Welch (2012) propose a broad definition of transparency which considers the relationship between the State and its citizens and makes possible the evaluation of state activities.

Ruediger (2003) assesses the importance of social participation affirming that the democratic potential of new technologies depends on greater demands by civil society for transparency and participation. Akutsu and Pinho (2002, p. 732) state that “[...] without organized civil society, public managers will not feel obliged to promote accountability.”

Accountability involves “[...] (objective and subjective) responsibility, control, transparency, the obligation to present accounts, justifications for actions which were or were not taken, rewards and/or punishment” (RAUPP and PINHO, 2014, p. 145). To Mainwaring and Welna (2003), accountability or the presenting of accounts by a public agent possesses three main elements: (i) answerability, which is transparency understood as making information available, (ii) responsiveness, understood as responding to demands for information and taking responsibility for these respective acts, and (iii) enforcement, the capacity of sanctions and coercion to ensure access to information.

Meijer (2014) teaches us that the instrumental value of transparency is frequently related to accountability, serving as a prerequisite for the information phase of any of its processes. In this particular, authors such as Fox (2007), Hood (2010) and Meijer (2014) propose specific analysis models to study the relationship between transparency and accountability.

In the Brazilian context, Campos (1990), in a seminal work, pointed out the difficulty of translating the concept of accountability to Portuguese. She related the need to protect citizens from bad bureaucratic conduct and found that the absence of public accountability mechanisms is due to the not very democratic relationship that exists between the State and society. Pinho and Sacramento (2009), revisiting Campos’s text, emphasize that the concept of accountability involves responsibility, control, transparency, and the obligation to present accounts. They conclude that an institutional and cultural evolution can be perceived which is in favor of increased accountability. Loureiro, Teixeira and Prado (2008) indicate the relevance of empirical studies of governmental transparency associated with the debate about accountability, in the context of constructing and improving democratic institutions.

In this study, we will adopt the scope of the definition proposed by Hood (2010): accountability consists of the duty of an individual or organization to respond in some way in regard to the manner in which they conduct their actions.

Transparency and Accountability of Governmental Algorithms

Algorithms are more and more present in governmental operations and, as a result, in governmental decisions, making them more effective and, at the same time, keeping information secret in “black boxes” (PASQUALE, 2015). Kroll (2015) states that important decisions about people are being taken by algorithms: the counting of votes, voter rolls are purged, the concession of financial assistance, the selection of tax payers for auditing, of passengers to be inspected and even the granting of credit.

Algorithms are not simple and objective instructions (WILLSON, 2017); they are based on society’s perceptions and understandings and not on tangible, global and fixed laws (EISCHEN, 2003). Algorithms have biases that can reflect discriminatory practices in society, due to the limitations of computational systems which may be perceived only after interaction with users (FRIEDMAN and NISSENBAUM, 1996).

Algorithms or some of their inputs may be secret, their implementation may be secret or the process may not be precisely described, which will make it difficult to monitor if a citizen or authority suspects that something is wrong and wishes to verify whether decisions are being made in accordance with current policy. Citizens and society have an interest in making these processes more transparent, given that bases for making these decisions are rarely made available to the public (KROLL, 2015).

The algorithms which support the electronic voting process can be divided into three groups: pre-voting, voting and post-voting. In the first, algorithms provide validation of electors who are apt to vote, as well as the validation of registered candidates, and the verification and guarantee of inviolability of electronic ballot boxes. In the second, algorithmic instructions validate the identity and credentials of the voter and count and store votes in a way that ensures and respects their secrecy. Finally, algorithms need to verify whether the ballot boxes have been adulterated, comparing registered votes with voting registers and the data load of each ballot box, permitting the recounting of votes when necessary, guaranteeing the integrity of voting in case of the need to switch ballot boxes, totaling the results and realizing the calculations inherent in voting legislation such as proportional representation (STOICA and GHILIC-MICU, 2016).

Thus, it is fundamental to make algorithms more transparent for the community in general, including their developers, users and those affected by their results, in order to identify and deal with biases, with it being true that this material gains relevance to the extent that algorithms are modified and become more complex, concentrating power in the hands of the people who understand their functioning (JANSSEN and KUK, 2016).

Algorithmic standardization has incited a series of preoccupations about responsibility, impartiality and the autonomy of this process, and this has been aggravated by the rigidity of computational systems responsible for a large variety of decisions in the lives of people (ZIEWITZ, 2015), who tend to accept that automated processes are true and precise (CITRON, 2007).

Transparency can correct the errors in any algorithmic process, promoting efficiency and allowing individuals to correct imprecise data collected about them. In this way, transparency also brings scrutiny which pressures agencies to improve their practices, so that transparent processes will be more precise, and therefore, efficient (ZARSKY, 2016).

Judith Donath, of Harvard University’s Berkman Klein Center for Internet & Society, laments the lack of transparency in systems and warns:

The danger of greater dependence on algorithms is that the decision-making process has become oracular: opaque and unquestionable. The solution is design. The process should not be a black box into which we feed data and an answer comes out, but rather a transparent process projected to not only produce a result, but to explain how it arrived at this result. Systems should be capable of producing clear and legible texts and graphics that can help users - readers, editors, doctors, patients, loan requesters, voters, etc. - understand how the decision was taken (ANDERSON and RAINIE, 2017, p. 22).

Besides providing access to information by being transparent, organizations need to guarantee that citizens understand what is happening and how this information is generated and used (NUNES, CAPPELLI and RALHA, 2017).

The concept of algorithmic accountability began to appear due to the possibility of the damage and discrimination that they can cause. The concept of accountability discussed above, when applied to algorithms, has often been confused with transparency. Transparency is an essential component of accountability, permitting citizens, consumers, journalists, monitoring organizations to verify and understand inputs, processes and outputs of a complex algorithmic system to identify evidence of damages and make provisions for reparations if necessary (DIAKOPOULOS, 2014). According to the World Wide Web Foundation (2017, p. 16), making an algorithm more accountable “[...] means guaranteeing that the damage can be evaluated, controlled and corrected.”

Research Method

This qualitative study is guided by the case study method (YIN, 2001), as a way to investigate the phenomenon of the interaction between new technologies and democracy, including the characteristics of transparency and accountability in the Brazilian electronic voting system.

The selection of the Brazilian electronic voting system as a case to be studied is due to the relevance of the public service involved - national and regional elections - as well as the interest of the community in perceiving the transparency of the system and monitoring how fairly it functions.

New information and communications technologies have led to the appearance of so-called electronic democracy, or e-democracy, whose most important manifestation is the electronic voting system (FREITAS and MACADAR, 2017). The Brazilian electronic ballot box has been considered an innovative experience in applying information and communications technology to public management (BALBE, 2014). However, the lack of transparency in the voting system is often reflected in the voter’s lack of confidence in the results of the voting (ACHIENG and RUHODE, 2013).

Our data collection was based on three sources, in order to trace a convergent line of research among a variety of collected pieces of evidence (YIN, 2001). The first refers to the perception and reports of specialists about the electronic voting system, collected from interviews granted to press outlets. The second consists of documental analysis - scientific articles, and reports and documents by governmental and non-governmental organizations, focused especially on transparency and accountability within the context of digital governance in the public sector. Finally, we used documental evidence from Brazilian voting legislation, which regulates the studied analysis unit.

In this way we employed the triangulation strategy, gathering information from distinct sources of evidence, in order to establish the convergence of our findings (FLICK, 2009; YIN, 2001). The triangulation of data sources is effective when data collection occurs regarding the same aspect of a phenomenon, seeking convergences and divergences (BRUNING, GODRI and TAKAHASHI, 2018)bem como atribuir maior confiabilidade e fidedignidade aos Estudos de Caso e discutem-se cinco modalidades de triangulação: (i. In this manner, the use of multiple sources assures greater credibility and reduces the possibility of research bias (EISENHARDT, 1989).

Box 1consolidates the cast of specialists (three linked to the Electoral Justice system and three external actors) considering perceptions and reports, in interviews granted to the press, which deal with the electronic voting system and its characteristics, especially in terms of aspects of transparency, security and reliability.

Box 1
Cast of Specialists

The selection of scientific articles was performed in three phases. The search, conducted in November 2018, was based on directories available through CAPES and Google Scholar, using the keywords “e-voting”, “algorithm”, “transparency”, “public services” and “accountability” in the title, subject or keywords, without restriction in terms of the year of publication since this is a recent subject. We identified 3,760 studies. The selection was refined taking into consideration works related to the transparency and accountability of new technologies applied to the public sector.

In addition, we made Google searches using the same keywords, which resulted in the selection of documents published by organizations dedicated to the study of algorithmic transparency in systems in general (ACM, 2017; DIAKOPOULOS, FRIEDLER, ARENAS et al., 2016; WORLD WIDE WEB FOUNDATION, 2017) and within the context of public administration (STATS NZ, 2018). The content analysis of these documents led to the construction of the constant analysis model of Figure 1, which contains seven dimensions described in Box 2.

Figure 1
Analysis Model of Algorithmic Transparency and Accountability

With the documental analysis of Brazilian legislation which regulates the national policy of digital governance (BRASIL, 2016a, 2018) and the electoral regulations which govern the electronic voting process (TSE, 2018), we identified and categorized the practices adopted by the Brazilian voting system which tend to make it more transparent and accountable. The possibility of analyzing the laws, decrees and resolutions for this type of research was indicated by May (2004). Richardson (2012, p. 228) observes that documental research “[...] does not have the objective of social phenomena, in terms of when and how they are produced, but rather the manifestations which register these phenomena and the ideals elaborated by them. Similar to content analysis, documental research requires the codification of information and the establishment of categories.”

The practices and characteristics of the transparency and accountability of the Brazilian electronic voting system were analyzed and categorized in light of the analysis model described in Figure 1, resulting in the consolidation presented in Box 4. We consider that the system practices or characteristics which are linked to the item of analysis have been met, and those that deal with one or more aspects of the item are considered to have been partially met. The non-existence of a practice related to the analyzed item was considered unmet. Finally, the evaluation items that are not linked to the evaluated unit of analysis (the Brazilian electronic voting system) were treated as “not applicable.”

The data analysis was based on technique of content analysis (BARDIN, 2016), pre-analyzing the documents to select relevant aspects, and analyzing them using inference and interpretation, characterizing the most significant thematic elements.

RESULTS AND DISCUSSION

In this section we will discuss our case study results in light of the analysis model displayed in Figure 1, whose dimensions are summarized in Box 2. The evaluation of the algorithms and systems used in digital public services with the assistance of the analysis model makes it possible to verify whether the analyzed object displays the expected behavior, free from errors, biases or discrimination.

Box 2
Dimensions of Analysis of Algorithmic Transparency

Box 3summarizes, with the data consolidated in Box 4, the dimensions, evaluated items and the degree of adhesion to the Brazilian electronic voting system analysis model. The analysis of Box 3 makes it possible to monitor the degree of adherence to best practices of algorithmic transparency by dimension, as high (A≥75%), average (50%≤A<75%) or low (A<50%).

Box 3
Overall View of the Results (Brazilian Electronic Voting System)

In the “consciousness” dimension we verified a high degree of adherence (100%) in the case evaluated by the analysis model, reflecting the preoccupation of the Electoral Justice system with promoting advertising campaigns to strengthen society’s trust in the electronic voting system (TSE, 2017a).

The “accountability”, “explanation” and “data origin, privacy and justice” dimensions have low degrees of total adherence (25%, 33% and 33%, respectively) to the analyzed items.

The efforts undertaken by the Electoral Justice system in publishing the functioning of the system, perceived in the consciousness dimension, contrast with the low adherence encountered in the accountability dimension. In fact, the difficulty that citizens have in understanding and inspecting the state’s actions in using the electronic voting system has led to the persistence of distrust in terms of its fair functioning, even though more than 20 years have passed since it was first developed.

The “access and correction”, “auditing” and “validation, precision and test” dimensions have average degrees of adherence (66% and 50%, respectively) to the analyzed items.

Box 4, presents the final configuration of the dimensions and their respective items of analysis contrasted with evidence of transparency identified in the Brazilian electronic voting system.

Box 4
Dimensions and Evidence of Transparency and Accountability of Algorithms of the Brazilian Electronic Voting System

Box 4
continuation

Box 4
continuation

In the “consciousness” dimension, Electoral Justice initiatives were identified in the promotion of advertising campaigns (TSE, 2020), in order to clarify for users the possibility of biases and potential damage caused by the system (ACM, 2017). The limitations of the system in terms of its functionalities (STATS NZ, 2018) are published by the Electoral Justice system showing the restrictions to voting in traffic (possible in cities with more than 100 thousand voters and outside of the State just for presidential elections) and locations where it is only possible to vote with the collection of biometric data. The delivery of an effective public benefit, prioritizing the needs of society (BRASIL, 2016b; STATS NZ, 2018), is emphasized by the Electoral Justice system, which points out that the use of information technology in the electoral process has made it possible to eliminate various types of fraud and human error that existed in the old electoral process, such as the duplication of votes and the switching of ballots during the counting, and has also saved time, energy and resources, and has impeded adulterations and guaranteed voting secrecy (BRASIL, 2016c).

In the “access and reparations” dimension, public agencies should encourage the adoption of mechanisms which permit questioning and correction by those who have been negatively affected by decisions based on information provided by algorithms (ACM, 2017; ALMEIDA and DONEDA, 2016). In this sense, the practice of publishing ballot box bulletins on the internet has been verified (TSE, 2017c), which makes it possible to contest any result sent to the Supreme Electoral Court which diverges from the original bulletin. In addition, the digital records of votes (RDV) make it possible to recount votes in an automated manner (BRASIL, 2016c, p. 28).

It can also recommend ways to pursue reparations for adverse effects, providing contacts so that the user knows how to proceed in these cases and the designation of a person responsible for the social impact of algorithmic decision-making systems (ALMEIDA and DONEDA, 2016; DIAKOPOULOS, FRIEDLER, ARENAS et al., 2016). In the electronic voting system, it is the Electoral Justice system that resolves any conflicts.

It should also have a contingency plan in case of unexpected behavior by the system (DIAKOPOULOS, FRIEDLER, ARENAS et al., 2016). In this case, the Electoral Justice system envisions the adoption of procedures which seek to correct problems presented in electronic ballot boxes during voting, counting or operational errors committed by table volunteers. These procedures are known as voting and counting contingency plans and they are not always capable of resolving all situations, which may lead to the possibility of totally manual voting (with the utilization of ballots and a conventional ballot box) or mixed voting (part electronic and part manual) (TSE, 2019c).

In the accountability dimension, the government has to provide timely and reliable information - i.e. the publication of the system’s source code (DIAKOPOULOS, FRIEDLER, ARENAS et al., 2016) - so that the citizen can supervise state action (BRASIL, 2018) and the public manager will be responsible for the decisions made by the algorithms (ACM, 2017). In the electronic voting system, the source programs and executables are subject to inspection (TSE, 2017a). In addition, the effectiveness of the dimension under analysis may be strengthened by collaboration with citizens in all of the cycle phases of public policies and in the creation and improvement of public services (BRASIL, 2018). It is relevant to point out that a large portion of the security and transparency mechanisms incorporated in electronic voting systems are the fruit of technical recommendations made by civil society entitites and suggestions from party bodies (CUNHA, 2009).

Bokslag and Vries (2016) affirm that the electronic voting process is much less transparent, especially for lay people. This is because it’s necessary to have an advanced knowledge of cryptography to prove that your vote was taken into consideration in the election results and that all of the votes were counted correctly. Just a small number of researchers will understand this, while the rest of the population will have to trust a system that they cannot understand.

Janino (2017), the Supreme Electoral Court’s Secretary of Information Technology, attributes the distrust of the Brazilian population with electronic voting to the population’s low level of education, which makes people incapable of understanding the underlying technology of the system. The specialist Rezende (2018) argues that only a printed ballot can give the voter a way to trust the system itself, without needing to believe the words of a technology specialist. This lack of understanding corroborates the low degree of adherence in the accountability dimension.

In the “explanation” dimension, it is understood that systems and institutions that use algorithmic decision making should provide explanations of the procedures followed (ACM, 2017) in technical terms (STATS NZ, 2018) and using common language (DIAKOPOULOS, FRIEDLER, ARENAS et al., 2016), which should include examples and infographics (STATS NZ, 2018). In this requisite, it has been verified that the Supreme Electoral Court has a variety of explanatory material about the electronic ballot box (TSE, 2019d), and electoral statistics (TSE, 2019e), in addition to making available technical documents such as digital summaries (hashes) which are generated in an electoral system sealing ceremony (TSE, 2019f). Icaza (2018), director of the Department of Electoral Cooperation and Observation of the OAS, believes that the Supreme Electoral Court’s explanations of the functioning of the electronic ballot boxes in the 2018 elections occurred with “total and absolute transparency”.

The “data origin, privacy and justice” dimension contemplates procedures to avoid the manipulation of the system by malicious actors, impeding discriminating effects, and protecting citizen secrecy and privacy, purpose for which the data was collected. If the process is not transparent, it is less likely that people will trust the decisions made and will not continue to share their personal information. (STATS NZ, 2018). In order to avoid malicious actors manipulating the system, only accredited entities and institutions can have access to the software code used by the Electoral Justice system (TSE, 2017a). The electronic ballot box should assure the secrecy and inviolability of the vote (BRASIL, 1997), with security being reinforced by the inexistence of a connection with the internet (BRASIL, 2016c). Almeida (2018) believes that it cannot be affirmed that the electronic ballot box is 100% reliable, but it possesses security mechanisms that are far superior to those involved in the manipulation of paper.

The “auditing” dimension consists of the possibility of verifying corrections of the algorithm inserted in the digital service. To make this feasible, models, data and decisions should be stored for later analysis, in case there is a suspicion of damage (ACM, 2017). It should also allow the research community to perform public audits (DIAKOPOULOS, FRIEDLER, ARENAS et al., 2016). Finally, it is recommended that decisions informed by algorithms have human supervision, and be preferentially performed by various interested actors (BRASIL, 2018) and supervised by governance groups to ensure the meeting of standards of ethics and privacy (STATS NZ, 2018). The ability to audit an electronic voting system is related to transparency and is an important in terms of the way the system is perceived by the public at large (FREITAS e MACADAR, 2017).

In the dimension under analysis, we have identified that entities and institutions accredited by the Supreme Electoral Court can monitor and audit the software used in Brazilian elections (TSE, 2017c). The system is submitted to public security tests which various entities can participate in, including university departments of Information Technology (TSE, 2017a). The elections are supervised in each location by electoral boards which consist of provisionary collegiate bodies made up of two or four citizens and a legal judge (BRASIL, 1965). After the election, political parties and other bodies can solicit from the Electoral Justice system copies of reports and archives generated by systems used in elections (TSE, 2017c). Weber (2018), a Justice of the Supreme Electoral Court, classifies Brazilian electronic ballot boxes as “totally reliable”, because they can be audited.

Finally, the “validation, precision and test” dimension prescribes that institutions should validate, test and document the models and methods used in these systems to avoid discriminatory damage (ACM, 2017). Sources of errors and uncertainties, in the algorithms and data sources, should be identified and stored in order to help in any procedure to mitigate damage (DIAKOPOULOS, FRIEDLER, ARENAS et al., 2016). In addition, the systems and algorithms that guide decisions should be periodically revised in order to verify whether they continue to fulfill the intended objectives (STATS NZ, 2018).

In this last dimension, it has been identified that the Supreme Electoral Court performs the Public Security Test (TPS), which brings together specialists “[...] who try to corrupt ballot boxes and their internal and external components with the objective of discovering vulnerabilities of the system in relation to any possible violation of their results and breaking of voting secrecy” (BRASIL, 2016c, p. 26).

On the eve of the election, the Supreme Electoral Court performs an audit of the systems through a so-called parallel vote, when certain ballot boxes are selected at random to verify their compliance through voting by fictitious voters (TSE, 2017a). After the introduction of biometric identification in elections, the specialist Rezende (2018) argues that the simulated parallel vote has lost its meaning, given that any “cheating program”, irregularly installed in a ballot box, could identify that it is being executed in a ballot box with a large percentage of biometric identification errors, frustrating the surveillance.

Digo Aranha (PAYÃO, 2017) points out that the improvement of the Brazilian voting system passes through not only the voting software security and its auditing processes, but also the implantation of mechanisms, such as the printed ballot, that allow the voter to verify whether the system registered his or her intentions correctly.

CONCLUSION

The field of algorithmic transparency studies, especially those linked to the public interest, has dealt with the importance of submitting to public scrutiny the “ingredients” and the steps taken in the preparation of the “recipe”. It is not enough for digital services to show gains in efficiency and greater accessibility; they need to be able to attest to their legitimacy.

The effect of this has been that the demand for transparency has knocked on the door of public services provided by digital means. This is because there is no doubt that transparency is an indispensable condition for the promotion of accountability (GOEDE and NEUWIRTH, 2014). However, studies that identify the criteria to be followed by public algorithms that seek to be transparent are still incipient.

The transparency and accountability of digital public services gain greater relevance when they involve new technologies and democracy (KROLL, 2015), given the wide adoption of electronic voting systems in various countries (STOICA and GHILIC-MICU, 2016).

Our study of the Brazilian electronic voting system in light of the proposed analysis model has demonstrated its characteristics of transparency and accountability, making it possible to identify its strengths and weaknesses. One of the main findings points to the contrast to the high degree of adherence to the model in the consciousness dimension (100%) and the low degree of adherence in the accountability dimension (25%). The effect of this is the difficulty that citizens have in understanding and monitoring the current utilization of the electronic voting system which has resulted in distrust in the fairness of its operations, which leads to a need for constant publication initiatives about the system’s functioning sponsored by the Electoral Justice system. A poll conducted in July 2018 by the digital company Avast found that there is little trust in the electronic ballot box system, given that 91.84% of Brazilian voters believe that the system can be violated (AVAST, 2018).

While some believe that the high level of distrust of the population in terms of the electronic voting system is due to a low level of education (JANINO, 2017), others believe there is a need to adopt mechanisms that allow voters to verify whether the system has registered their votes properly (PAYÃO, 2017), opening the possibility of an evaluation of state activities (GRIMMELIKHUIJSEN and WELCH, 2012).

Among the evaluated dimensions, the degree of adherence to auditing (66%) stands out, which permits data recovery in cases of suspected damage, which is indispensable for transparency and is important to how people view the system in general (FREITAS and MACADAR, 2017).

Transparency cannot be measured just by the mere publication of data, because it is not an end in itself, since it should fulfill a certain purpose (KLEIN, KLEIN and LUCIANO, 2018), which, in this case, is related to the need of the electoral process to promote broad participation, transparency and present the accounts of those involved and their democratic responsibility (PRADO, 2009). In the case of the electronic voting system, it has focused its accountability on the perspective of presenting governmental accounts and the responsibility of public agents to strengthen trust in the electoral process.

In 2018, the general elections held in Brazil counted on the participation of the Electoral Observation Mission of the Organization of American States (EOM/OAS), which noted the professionalism and the technical expertise with which the electoral process was organized, providing citizens with official information in a rapid manner and contributing to the certainty of the process (CHINCHILLA, 2018a). The mission recognized the efficiency, security and transparency of the Brazilian electronic voting system, and made recommendations for improvements (CHINCHILLA, 2018b).

Among the main recommendation of the mission were enlarging the size of the sample used in the parallel voting, increasing the presence of technicians from the political parties during the monitoring of the ballot boxes, and the creation of spaces for dialogue in which authorities and party representatives can work together to develop new measures which can increase the confidence of all actors in the system (CHINCHILLA, 2018b). The creation of these spaces for dialogue will help address the great need for participation to promote accountability (AKUTSU and PINHO, 2002; RUEDIGER, 2003).

Considering the growing importance of the transparency of governmental actions, the objective of this study has been to identify the mechanisms that promote the transparency and accountability of the Brazilian electronic voting system. The results obtained and the procedures adopted are relevant to the practices of public managers who desire to evaluate the transparency and quality of digital public services. The relevance of this empirical work is based on governmental transparency associated with accountability, understood as the problem of constructing and improving democratic institutions (LOUREIRO, TEIXEIRA and PRADO, 2008). The dimensions and items of analysis utilized make it possible to examine the transparency of algorithms and digital public services, which makes it possible to reapply them in the future and compare them with other digital systems and services. The theoretical contribution can be credited to an unprecedented combination of criteria designed to study the transparency of digital public services, vital for democracy, from the point of view of promoting accountability.

Future research can evaluate, in a longitudinal manner, the evolution of security and transparency characteristics of the Brazilian electronic voting system since its origin in 1996. A comparative analysis of the Brazilian system with systems used in elections in other countries would also be of interest. Finally, there is also the possibility of expanding the case analysis selected by using a triangulation of methods utilizing focus groups and Delphi techniques.

REFERÊNCIAS

• ABRUCIO, F. L.; LOUREIRO, M. R. Finanças públicas, democracia e accountability. In: BIDERMAN, C.; ARVATE, P. (Orgs.). Economia do setor público no Brasil. Rio de Janeiro: Elsevier, 2004. p. 75-102.
• ACHIENG, M.; RUHODE, E. The adoption and challenges of electronic voting technologies within the South African context. International Journal of Managing Information Technology, v. 5, n. 4, 2013.
• AKUTSU, L.; PINHO, J. A. G. Sociedade da informação, accountability e democracia delegativa: investigação em portais de governo no Brasil. I Revista de Administração Pública, Rio de Janeiro, v. 36, n. 5, p. 723-745, 2002.
• ALBU, O. B.; FLYVERBOM, M. Organizational Transparency: Conditions and Consequences. I Business & Society, v. 58, n. 2, p. 268-297, 2016. Disponível em:<https://doi.org/10.1177/0007650316659851>. Acesso em: 31 mar. 2020.
  » https://doi.org/10.1177/0007650316659851
• ALMEIDA, N. F. I Analista comenta história, funcionamento e confiabilidade da urna eletrônica. 2018. Disponível em:<Disponível em:https://www2.camara.leg.br/camaranoticias/radio/materias/PAINEL-ELETRONICO/563749-ANALISTA-COMENTA-HISTORIA,-FUNCIONAMENTO-E-CONFIABILIDADE-DA-URNA-ELETRONICA.html >. Acesso em:1 jul. 2019.
  » https://www2.camara.leg.br/camaranoticias/radio/materias/PAINEL-ELETRONICO/563749-ANALISTA-COMENTA-HISTORIA,-FUNCIONAMENTO-E-CONFIABILIDADE-DA-URNA-ELETRONICA.html
• ALMEIDA, V. A. F.; DONEDA, D. What Is Algorithm Governance? I IEEE Computing Society, v. 20, n. 4, p. 60-63, jul./ago. 2016. Disponível em: <http://dx.doi.org/10.1109/MIC.2016.79>. Acesso em: 15 abr. 2020.
  » https://doi.org/10.1109/MIC.2016.79
• ANANNY, M.; CRAWFORD, K. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. I New Media and Society, v. 20, n. 3, p. 973-989, 2018.
• ANDERSON, J.; RAINIE, L. Code-Dependent: Pros and Cons of the Algorithm Age. I Pew Research Center: Internet & Tecnology, Washington, D.C., 08 fev. 2017. Disponível em:<Disponível em:http://www.pewinternet.org/2017/02/08/code-dependent/ >. Acesso em:21 mar. 2019
  » http://www.pewinternet.org/2017/02/08/code-dependent/
• ASSOCIATION FOR COMPUTING MCHINE - ACM. I Principles for Algorithmic Transparency and Accountability. 2017. Disponível em:<Disponível em:https://www.acm.org/binaries/content/assets/public-policy/2017_joint_statement_algorithms.pdf >. Acesso em:10 nov. 2018.
  » https://www.acm.org/binaries/content/assets/public-policy/2017_joint_statement_algorithms.pdf
• ATTARD, J. et al. A systematic review of open government data initiatives. I Government Information Quarterly, v. 32, n. 4, p. 399-418, 2015.
• AVAST. 91,84% dos brasileiros acreditam que o sistema eletrônico de votação pode ser violado nas eleições. I Avast, Sala de Imprensa, São Paulo, 22 ago. 2018Disponível em:<Disponível em:https://press.avast.com/pt-br/9184-dos-brasileiros-acreditam-que-o-sistema-eletrônico-de-votação-pode-ser-violado-nas-eleições >. Acesso em:01 jul. 2019.
  » https://press.avast.com/pt-br/9184-dos-brasileiros-acreditam-que-o-sistema-eletrônico-de-votação-pode-ser-violado-nas-eleições
• BALBE, R. S. Uso de tecnologias de informação e comunicação na gestão pública: exemplos no governo federal. I Revista do Serviço Público, v. 61, n. 2, p. 189-209, 27 jan. 2014.
• BARBOSA, A. F.; FARIA, F. I.; PINTO, S. L. Governança eletrônica no setor público. In: KNIGHT, P. T. et al. (Orgs.). I E-Desenvolvimento no Brasil e no mundo: subsídios e Programa e-Brasil. São Caetano do Sul: Yendis Editora, 2007.
• BARDIN, L. I Análise de conteúdo. São Paulo: Edições 70, 2016.
• BOKSLAG, W.; VRIES, M. Evaluating e-voting: theory and practice. I ArXiv, 1602.02509v1, fev. 2016. Disponível em:<Disponível em:https://arxiv.org/pdf/1602.02509.pdf >. Acesso em: 30 mar. 2020.
  » https://arxiv.org/pdf/1602.02509.pdf
• BRASIL. Lei n^o 4.737. I Código Eleitoral, 1965. Disponível em:<Disponível em:http://www.planalto.gov.br/ccivil_03/leis/l4737.htm >. Acesso em: 30 mar. 2020.
  » http://www.planalto.gov.br/ccivil_03/leis/l4737.htm
• BRASIL. I Lei n^o 9.504. 1997. Disponível em:<Disponível em:http://www.planalto.gov.br/ccivil_03/leis/l9504.htm >. Acesso em:30 mar. 2020.
  » http://www.planalto.gov.br/ccivil_03/leis/l9504.htm
• BRASIL. I Decreto n^o 8.638, de 15 de janeiro de 2016. Institui a Política de Governança Digital no âmbito dos órgãos e das entidades da administração pública federal direta, autárquica e fundacional. 2016a. Disponível em: <Disponível em: http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2016/Decreto/D8638.htm >. Acesso em:18 mar. 2019.
  » http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2016/Decreto/D8638.htm
• BRASIL. I Decreto n^o 8.638, de 15 de janeiro de 2016. 2016b. Disponível em: <Disponível em: http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2016/Decreto/D8638.htm >. Acesso em:16 mar. 2019.
  » http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2016/Decreto/D8638.htm
• BRASIL. I Urna eletrônica: 20 anos a favor da democracia. Brasília: Tribunal Superior Eleitoral, 2016c.
• BRASIL. I Estratégia de Governança Digital: Transformação Digital - cidadania e governo. Brasília: Ministério do Planejamento, Desenvolvimento e Gestão, Secretaria de Tecnologia da Informação e Comunicação, 2018.
• BRUNING, C.; GODRI, L.; TAKAHASHI, A. R. W. Triangulação em Estudos de Caso: incidência, apropriações e mal-entendidos em pesquisas da área de Administração. I Administração: Ensino e Pesquisa, v. 19, n. 2, p. 277-307, 2018.
• CAMPOS, A. M. Accountability: quando poderemos traduzi-la para o português?. I Revista de administração pública, v. 24, n. 2, p. 30-50, 1990.
• CHINCHILLA, L. Relatório Preliminar MOE - 1^o Turno. Brasília: Organización de los Estados Americanos, 2018a. Disponível em:<http://www.oas.org/eomdatabase/MoeReport.aspx?Lang=es&Id=410&MissionId=490>.
  » http://www.oas.org/eomdatabase/MoeReport.aspx?Lang=es&Id=410&MissionId=490
• CHINCHILLA, L. I Relatório Preliminar MOE - 2^o Turno. Brasília: Organización de los Estados Americanos, 2018b. Disponível em:<http://www.oas.org/eomdatabase/MoeReport.aspx?Lang=es&Id=410&MissionId=490>.
  » http://www.oas.org/eomdatabase/MoeReport.aspx?Lang=es&Id=410&MissionId=490
• CITRON, D. K. Technological Due Process. I Washington University Law Review, v. 85, n. 6, 2008. Disponível em:<Disponível em:https://openscholarship.wustl.edu/law_lawreview/vol85/iss6/2 >. Acesso em: 15 abr. 2020.
  » https://openscholarship.wustl.edu/law_lawreview/vol85/iss6/2
• CUNHA, A. A. P. I A evolução dos mecanismos de transparência no desenvolvimento do projeto de votação eletrônica no Brasil: 1996-2008. Porto Alegre: UFRGS, 2009.
• DIAKOPOULOS, N. I Algorithmic Accountability Reporting: On the Investigation of Black Boxes. New York, NY: Tow Center for Digital Journalism, Columbia School of Journalism, 2014.
• DIAKOPOULOS, N. et al. I Principles for Accountable Algorithms and a Social Impact Statement for Algorithms. FAT/ML, 2016. Disponível em:<Disponível em:http://www.fatml.org/resources/principles-for-accountable-algorithms >. Acesso em: 10 nov. 2018.
  » http://www.fatml.org/resources/principles-for-accountable-algorithms
• EISCHEN, K. Opening the ‘Black Box’ of software the micro-foundations of informational technologies, practices and environments. I Information, Communication & Society, v. 6, n. 1, p. 57-81, jan. 2003.
• EISENHARDT, K. M. Building Theories from Case Study Research. I The Academy of Management Review, v. 14, n. 4, p. 532-550, Oct. 1989.
• FINK, K. Opening the government’s black boxes: freedom of information and algorithmic accountability. Information Communication and Society, v. 21, n. 10, p. 1453-1471, 2018.
• FLICK, U. Introdução à pesquisa qualitativa. 3. ed. Porto Alegre: Artmed, 2009. v. 84.
• FOX, J. The uncertain relationship between transparency and accountability. I Development in Practicel, v. 17, n. 4-5, p. 37-41, 2007.
• FREITAS, J. L.; MACADAR, M. A. The Brazilian Electronic Voting System: evolution and challenges. In: JOINT INTERNATIONAL CONFERENCE ON ELECTRONIC VOTING, 2., 2017. Anais...Disponível em:<Disponível em:https://www.researchgate.net/publication/320805527 >. Acesso em:15 abr. 2020.
  » https://www.researchgate.net/publication/320805527
• FRIEDMAN, B.; NISSENBAUM, H. Bias in computer systems. ACM Transactions on Information Systems, v. 14, n. 3, p. 330-347, 01 jul. 1996.
• GOEDE, M.; J. NEUWIRTH, R. Good governance and confidentiality: a matter of the preservation of the public sphere. Corporate Governance: the international journal of business in society, v. 14, n. 4, p. 543-554, 29 jul. 2014.
• GRANDE, J. I. C.; ARAUJO, M. C. R.; SERNA, M. S. La necesidad de Teoría sobre Gobierno Electrónico Una Propuesta Integradora. In: CONCURSO DE ENSAYOS Y MONOGRAFÍAS DEL CLAD SOBRE REFORMA DEL ESTADO Y MODERNIZACIÓN DE LA ADMINISTRACIÓN PÚBLICA, 16., 2002, Caracas, Anais...Caracas: CLAD, 2002. Disponível em:<Disponível em:http://siare.clad.org/fulltext/0043103.pdf >. Acesso em:15 abr. 2020.
  » http://siare.clad.org/fulltext/0043103.pdf
• GRIMMELIKHUIJSEN, S. G.; WELCH, E. W. Developing and Testing a Theoretical Framework for Computer-Mediated Transparency of Local Governments. Public Administration Review, v. 72, n. 4, p. 562-572, 2012.
• GUIMARÃES, T. A.; MEDEIROS, P. H. R. A relação entre governo eletrônico e governança eletrônica no governo federal brasileiro. Cadernos EBAPE.BR, Rio de Janeiro, v. 3, n. 4, p. 1-18, 2005.
• HOOD, C. Accountability and Transparency: Siamese Twins, Matching Parts, Awkward Couple? West European Politics, v. 33, n. 5, p. 989-1009, set. 2010.
• ICAZA, G. DE. Não há motivo para desconfiar da urna eletrônica, diz especialista da OEA. Folha de São Paulo, São Paulo, 23 set. 2018. Disponível em: <Disponível em: https://www1.folha.uol.com.br/poder/2018/09/nao-ha-motivo-para-desconfiar-da-urna-eletronica-diz-especialista-da-oea.shtml >. Acesso em:01 jul. 2019.
  » https://www1.folha.uol.com.br/poder/2018/09/nao-ha-motivo-para-desconfiar-da-urna-eletronica-diz-especialista-da-oea.shtml
• JANINO, G. Entrevista com Giuseppe Janino, secretário de tecnologia da informação do TSE. YouTube [Canal Jovem Pan - 3 em 1], 25. jul. 2017. (15m27s). Disponível em:<Disponível em:https://www.youtube.com/watch?v=Am0iFvYx2Vk >. Acesso em:01 jul. 2019.
  » https://www.youtube.com/watch?v=Am0iFvYx2Vk
• JANSSEN, M.; KUK, G. The challenges and limits of big data algorithms in technocratic governance. Government Information Quarterly, v. 33, n. 3, p. 371-377, 2016.
• KLEIN, R. H.; KLEIN, D. C. B.; LUCIANO, E. M. Identificação de mecanismos para a ampliação da transparência em portais de dados abertos: uma análise no contexto brasileiro. Cadernos EBAPE.BR, v. 16, n. 4, p. 692-715, 2018.
• KROLL, J. A. Accountable Algorithms. Princeton: Princeton University, 2015.
• LANE, J.-E. New Public Management. London: Routledge, 2000.
• LOUREIRO, M. R.; TEIXEIRA, M. A. C.; PRADO, O. Construção de instituições democráticas no Brasil contemporâneo: transparência das contas públicas. Organizações & Sociedade, v. 15, n. 47, p. 107-119, 2008.
• MACHADO, H. F. S. Algoritmos, regulação e governança: uma revisão de literatura. Revista de Direito Setorial e Regulatório, v. 4, n. 1, p. 39-62, 2018.
• MAINWARING, S.; WELNA, C. Democratic accountability in Latin America. Oxford: Oxford University Press, 2003.
• MAY, T. Pesquisa social: questões, métodos e processos. Porto Alegre: Artmed, 2004.
• MEIJER, A. Transparency. In: BOVENS, M.; GOODIN, R. E.; SCHILLEMANS, T. (Eds.). The Oxford Handbook of Public Accountability. Oxford: Oxford University Press, 2014. p. 21.
• MITTELSTADT, B. D. et al. The ethics of algorithms: Mapping the debate. Big Data & Society, v. 3, n. 2, p. 1-12, 2016. Disponível em: <http://dx.doi.org/10.1177/2053951716679679>. Acesso em: 15 abr. 2020.
  » https://doi.org/10.1177/2053951716679679
• NUNES, V. T. N.; CAPPELLI, C.; RALHA, C. G. Transparency in Information Systems. In: BOSCARIOLI, C.; ARAUJO, R. M.; MACIEL, R. S. P. (Eds.). I GranDSI-BR Grand Research Challenges in Information Systems in Brazil 2016-2026. Porto Alegre: Sociedade Brasileira de Computação, 2017.
• OKOT-UMA, R. W. ELECTRONIC GOVERNANCE: Re-inventing Good Governance. Commonwealth Secretariat London, p. 1-19, 2000. Disponível em:<Disponível em:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.197.6576&rep=rep1&type=pdf >. Acesso em:15 abr. 2020.
  » http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.197.6576&rep=rep1&type=pdf
• PAPENFUSS, U.; SCHAEFER, C. Improving public accountability by aligning reporting to organizational changes in public service provision - an empirical Internet study of all Austrian, German and Swiss towns and states from an agency-theory perspective Points for practitioners. International Review of Administrative Sciences, v. 76, n. 3, p. 555-576, 2015.
• PASQUALE, F. The black box society: The secret algorithms that control money and information. Harvard: Harvard University Press, 2015.
• PAYÃO, F. Urnas eletrônicas: falhas, vulnerabilidades e fraudes do mesário. Tecmundo, São Paulo, 18 set. 2017. Disponível em: <Disponível em: https://www.tecmundo.com.br/seguranca/122152-urnas-eletronicas-falhas-vulnerabilidades-fraudes-mesario.htm >. Acesso em:1 jul. 2019.
  » https://www.tecmundo.com.br/seguranca/122152-urnas-eletronicas-falhas-vulnerabilidades-fraudes-mesario.htm
• PRADO, O. Governo eletrônico, reforma do estado e transparência: o programa de governo eletrônico do Brasil. Rio de Janeiro: Ed. FGV, 2009.
• RAUPP, F. M.; PINHO, J. A. G. Accountability em câmaras municipais: uma investigação em portais eletrônicos. Revista de Administração, v. 48, n. 4, p. 770-782, 2013.
• RAUPP, F. M.; PINHO, J. A. G. Prestação de contas nos portais eletrônicos de assembleias legislativas: um estudo após a lei de acesso à informação. Revista Gestão e Planejamento, v. 15, n. 1, p. 144-161, 2014.
• REZENDE, P. A. D. Urna pode ser fraudada. Brasília, 10 fev.2018. Disponível em: <Disponível em: https://cic.unb.br/~rezende/trabs/entrevistaGP2.html >. Acesso em:01 jul. 2019.
  » https://cic.unb.br/~rezende/trabs/entrevistaGP2.html
• RICHARDSON, R. J. Pesquisa Social: métodos e técnicas. 3. ed. São Paulo: Atlas, 2012.
• RUEDIGER, M. A. Governança democrática na era da informação. Revista de Adminsitração Pública, v. 37, n. 6, p. 1257-1280, 2003.
• RUIJER, E.; MEIJER, A. National Transparency Regimes: Rules or Principles? A Comparative Analysis of the United States and The Netherlands. International Journal of Public Administration, v. 39, n. 11, p. 895-908, 2016.
• STATS NZ. Algorithm assessment report. 2018. Disponível em:<Disponível em:https://data.govt.nz/use-data/analyse-data/government-algorithm-transparency >. Acesso em: 30 mar. 2020.
  » https://data.govt.nz/use-data/analyse-data/government-algorithm-transparency
• STOICA, M.; GHILIC-MICU, B. E-Voting Solutions for Digital Democracy in Knowledge Society. Informatica Economica, v. 20, n. 3, p. 55-65, 2016.
• TRIBUNAL SUPERIOR ELEITORAL - TSE. Resolução n^o 23.550. Brasilília, DF: TSE, 2017a.
• TRIBUNAL SUPERIOR ELEITORAL - TSE. Resolução n^o 23.509. Brasilília, DF: TSE, 2017b.
• TRIBUNAL SUPERIOR ELEITORAL - TSE. Resolução n^o 23.554. Brasilília, DF: TSE, 2017c.
• TRIBUNAL SUPERIOR ELEITORAL - TSE. Legislação - Código Eleitoral Anotado e Legislação Complementar. Brasilília, DF: TSE, 2018. Disponível em:<Disponível em:http://www.tse.jus.br/legislacao> Acesso em:01 jul. 2019.
  » http://www.tse.jus.br/legislacao>
• TRIBUNAL SUPERIOR ELEITORAL - TSE. Procedimentos de contingência. Brasilília, DF: TSE, 2019a. Disponível em: <Disponível em: http://www.tse.jus.br/eleicoes/urna-eletronica/seguranca-da-urna/procedimentos-de-contingencia >. Acesso em:21 jul. 2019.
  » http://www.tse.jus.br/eleicoes/urna-eletronica/seguranca-da-urna/procedimentos-de-contingencia
• TRIBUNAL SUPERIOR ELEITORAL - TSE. Urna eletrônica. Brasilília, DF: TSE, 2019b. Disponível em: <Disponível em: http://www.tse.jus.br/eleicoes/urna-eletronica/urna-eletronica >. Acesso em: 21 jul. 2019.
  » http://www.tse.jus.br/eleicoes/urna-eletronica/urna-eletronica
• TRIBUNAL SUPERIOR ELEITORAL - TSE. Estatísticas eleitorais. Brasilília, DF: TSE, 2019c. Disponível em: <Disponível em: http://www.tse.jus.br/eleicoes/estatisticas/estatisticas-eleitorais >. Acesso em:21 jul. 2019.
  » http://www.tse.jus.br/eleicoes/estatisticas/estatisticas-eleitorais
• TRIBUNAL SUPERIOR ELEITORAL - TSE. Resumos digitais (hashes) dos sistemas eleitorais. Brasilília, DF: TSE, 2019d. Disponível em: <Disponível em: http://www.tse.jus.br/eleicoes/urna-eletronica/seguranca-da-urna/ hash/resumos-digitais-hash-dos-sistemas-eleitorais >. Acesso em: 21 jul. 2019.
  » http://www.tse.jus.br/eleicoes/urna-eletronica/seguranca-da-urna/ hash/resumos-digitais-hash-dos-sistemas-eleitorais
• TRIBUNAL SUPERIOR ELEITORAL - TSE. Campanhas publicitárias. Brasilília, DF: TSE, 2020. Disponível em: <Disponível em: http://www.tse.jus.br/imprensa/campanhas-publicitarias >. Acesso em:01 jul. 2019.
  » http://www.tse.jus.br/imprensa/campanhas-publicitarias
• UNITED NATIONS - UN. Benchmarking E-government: A Global Perspective - Assessing the Progress of the UN Member States. New York: UN, 2002. Disponível em:<Disponível em:https://publicadministration.un.org/publications/content/PDFs/E-Library%20Archives/2002%20Benchmarking%20E-government_A%20Global%20Perspective%20-%20Assessing%20the%20UN%20Member%20States%20(2002).pdf >. Acesso em:18 out. 2018.
  » https://publicadministration.un.org/publications/content/PDFs/E-Library%20Archives/2002%20Benchmarking%20E-government_A%20Global%20Perspective%20-%20Assessing%20the%20UN%20Member%20States%20(2002).pdf
• UNITED NATIONS - UN. Definition of basic concepts and terminologies in governance and public administration. New York: UN, 2006. Disponível em:<Disponível em:http://unpan1.un.org/intradoc/groups/public/documents/un/unpan022332.pdf >. Acesso em:18 mar. 2019.
  » http://unpan1.un.org/intradoc/groups/public/documents/un/unpan022332.pdf
• VASCONCELLOS, M. L. L.; LUNKES, R. J.; TALIANI, M. T. C. Thirty Years of Studies on Transparency, Accountability, and Corruption in the Public Sector: The State of the Art and Opportunities for Future Research Public Integrity, v. 20, n. 5, p. 512-533, 2018. Disponível em:<Disponível em:https://www.tandfonline.com/doi/full/10.1080/10999922.2017.1416537 > Acesso em: 01 abr. 2020.
  » https://www.tandfonline.com/doi/full/10.1080/10999922.2017.1416537
• WEBER, R. Presidente do TSE rebateu a críticas feitas por Bolsonaro no último domingo sobre a ausência do voto impresso. Último Segundo, São Paulo, 18 set.2018. Disponível em:<Disponível em:https://ultimosegundo.ig.com.br/politica/2018-09-18/urna-eletronica-rosa-weber.html >. Acesso em: 01 jul. 2019.
  » https://ultimosegundo.ig.com.br/politica/2018-09-18/urna-eletronica-rosa-weber.html
• WILLSON, M. Algorithms (and the) everyday. Information, Communication & Society, v. 20, n. 1, p. 137-150, 02 jan. 2017.
• WORLD WIDE WEB FOUNDATION. Algorithmic accountability - Applying the concept to different country contexts. Washington, DC: World Wide Web Foundation, 2017. Disponível em: <Disponível em: https://webfoundation.org/docs/2017/07/Algorithms_Report_WF.pdf >. Acesso em: 20 mar. 2019.
  » https://webfoundation.org/docs/2017/07/Algorithms_Report_WF.pdf
• YIN, R. K. Estudo de caso: planejamento e métodos. 2. ed. Porto Alegre: Bookman, 2001.
• ZARSKY, T. The Trouble with Algorithmic Decisions: An Analytic Road Map to Examine Efficiency and Fairness in Automated and Opaque Decision Making. Science, Technology, & Human Values, v. 41, n. 1, p. 118-132, 2016.
• ZIEWITZ, M. Governing Algorithms: Myth, Mess, and Methods. Science, Technology & Human Values, v. 1, n. 1, p. 1-14, jan. 2015

• [Translated version] Note: All quotes in English translated by this article’s translator.

Publication Dates

• Publication in this collection
  18 Dec 2020
• Date of issue
  Nov 2020

History

• Received
  22 Feb 2019
• Accepted
  13 Oct 2019