We provide the first replication study of political science research published in Brazil by attempting to replicate every quantitative article published in five major Brazilian journals between 2012 and 2016. We also tested whether replication rates varied between established fields, more traditional and where the use of quantitative data is more common, and emerging fields. Our results show that transparency and reproduction are still in a development stage in Brazilian Political Science. Of the 650 articles reviewed, we asked for data to 197 quantitative articles. From those, only 28% agreed to share datasets and computed codes. We were able to attempt a replication for only 14%, and successfully reproduce the results of less than 5%. We conclude by suggesting the adoption of transparency and replicability procedures that are standard in other scientific communities.
replication; transparency; reproduction; Brazilian Political Science; data sharing
Este é o primeiro exercício de replicação de pesquisas em ciência política publicado no Brasil. O objetivo foi replicar todos os artigos quantitativos publicados em cinco grandes revistas brasileiras entre 2012 e 2016. Também foram testadas as diferenças potenciais nas taxas de replicação entre campos de pesquisa estabelecidos, mais tradicionais e onde o uso de dados quantitativos é mais frequente, e emergentes. Solicitamos os dados para 197 artigos quantitativos. Desses 197, somente 28% dos autores concordaram em compartilhar os dados e os códigos. Foi possível tentar replicar apenas 14%, com 5% de sucesso. A conclusão sugere a adoção de procedimentos de transparência e replicabilidade já introduzidos em outras comunidades científicas.
replicação; transparência; reprodução; Ciência Política Brasileira; compartilhamento de dados
Este es el primer ejercicio de replicación de la investigación en ciencia política publicado en Brasil. El objetivo fue replicar todos los artículos cuantitativos publicados en cinco importantes revistas brasileñas entre 2012 y 2016. También se probaron las posibles diferencias en las tasas de replicación entre los campos de investigación establecidos, más tradicionales y donde el uso de datos cuantitativos es más frecuente, y los emergentes. Solicitamos datos para 197 artículos cuantitativos. De estos 197, solo el 28% de los autores acordó compartir datos y códigos. Fue posible intentar replicar solo el 14%, con un 5% de éxito. La conclusión sugiere la adopción de procedimientos de transparencia y replicabilidad ya introducidos en otras comunidades científicas.
replicación; transparencia; reproducción; Ciencias Políticas Brasileñas; compartir datos
Il s’agit du premier exercice de réplication de la recherche en science politique publiée au Brésil. L’objectif était de reproduire tous les articles quantitatifs publiés dans cinq grandes revues brésiliennes entre 2012 et 2016. Les différences potentielles de taux de réplication entre les domaines de recherche établis et plus traditionnels et où l’utilisation de données quantitatives est plus fréquente et les domaines émergents ont également été testés. Nous avons demandé des données pour 197 articles quantitatifs. Sur ces 197, seuls 28% des auteurs ont accepté de partager des données et des codes. Il était possible d’essayer de reproduire seulement 14%, avec 5% de succès. La conclusion suggère l’adoption de procédures de transparence et de reproductibilité déjà introduites dans d’autres communautés scientifiques.
réplication; transparence; la reproduction; Science politique brésilienne; partage de données
Scientific disciplines are facing a credibility crisis. In social sciences, significant published findings often cannot be replicated (Goodman, Fanelli and Ioannidis, 2016; Key, 2016KEY, Ellen M. (2016), “How are we doing? data access and replication in political science”. PS: Political Science and Politics, vol. 49, no 02, pp. 268-272.; Christensen et al., 2019CHRISTENSEN, Garret; FREESE, Jeremy; MIGUEL, Edward (2019), Transparent and reproducible social science research: how to do open science. Oakland, CA: University of California Press.). High-profile publications have been retracted and, in some cases, identified as fraudulent. Some scholars have speculated that at least half of all published research is simply wrong (Ionnadis, 2005IONNADIS, John. (2005), “Why most published research findings are false”. PLOS Medicine. Available at: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124.
https://journals.plos.org/plosmedicine/a... ). The reproduction of other scholars’ work has led to several high-profile reviews and at least one significant retraction of a fraudulent result1 1 . See Lacour and Green fraud case in https://www.chronicle.com/article/What-Social-Science-Can-Learn/230645 . All these (and other) problems suggest that much research is a waste of resources and human capital, degrade trust in science, and have led some to question public funding for research projects (Christensen and Miguel, 2018CHRISTENSEN, Garret; MIGUEL, Edward (2018), “Transparency, reproducibility, and the credibility of economics research”. Journal of Economic Literature, vol. 56, no 3, pp. 920-980.).
Yet, as Figueiredo Filho et al. (2019) pointed out, there is no consensus in the literature about the meaning of transparency, reproducibility, and replication, which leads these concepts to be treated as interchangeable. By and large, one assesses transparency by the research material available to the academic community. Reproducibility is determined by the capacity of any scholar to reproduce the same results from the research material made available. Finally, replication may have two different meanings. The first, used in this article, is a synonym of reproducibility. The second, according to King (1995)KING, Gary. (1995), “Replication, Replication”. PS: Political Science and Politics, vol. 28, no 03, pp. 444-452., is the attempt to further knowledge by introducing slight variations in a reproducible paper, e.g., by adding variables to the original model, testing the same hypothesis in different contexts, etc.
In response, scholars have investigated the causes of this credibility crisis and suggested several solutions. The causes include institutional mechanisms that encourage “fishing” or p-hacking2 2 . For more information about p-hacking concept see Head et al. (2015) and Chordia, Goyal and Saretto (2017). (Humphreys, De La Sierra and Van Der Windt, 2013) and lead to selection bias in publication choice (Gerber, Green and Nickerson, 2001). For example, journals are more likely to publish “significant” findings than null results (Ferguson and Heene, 2012FERGUSON, Christopher J.; HEENE, Moritz. (2012), “A vast graveyard of undead theories: publication bias and psychological science’s aversion to the null”. Perspectives on Psychological Science, vol. 07, no 06, pp. 555-561.). Knowing this, scholars fail to submit null results for publication. They may also hunt through the dataset for significant results. All these mechanisms push relevant null results aside, encourage poor scientific practices, and lead to a high rate of false positives3 3 . A false positive result consists in rejecting the null hypothesis when it was not the case of rejecting it. The opposite case is known as false negative, when the null hypothesis is not rejected when it should be. in published studies.
Addressing all these problems will require significant changes in scientific institutions. Scholars have proposed several reforms and best practices that have the potential to deal with these issues. These practices might include open access to academic production, data and research materials, computer codes (preferably in open source formats), open peer review, use of pre-registration and data analysis plans, reviewing research designs instead of results (Cruwell et al., 2019CRUWELL, Sophia et al. (2019), “Seven easy steps to open science: an annotated reading list”. Zeitschrift für Psychologie, vol. 227, no 4, pp. 237-248.; Martins, 2020MARTINS, Henrique Castro. (2020), “A Importância da Ciência Aberta (Open Science) na Pesquisa em Administração”. Rev. Adm. Contemp, vol. 24 no 01.), and others promoted by organizations and initiatives, such as the Iniciativa Brasileira de Reprodutibilidade (https://www.reprodutibilidade.bio.br/), the Project TIER (https://www.projecttier.org/), the Berkeley Institute for Transparency in the Social Sciences (https://www.bitss.org/) and the Open Science Foundation (https://osf.io/).
Research transparency is one of these new practices. By research transparency we mean providing all the resources needed for other scholars to replicate one’s findings, including source materials, computer code, and other relevant resources (King, Keohane and Verba, 1994; Figueiredo Filho et al., 2019; Bakken, 2019BAKKEN, Suzanne. (2019), “The journey to transparency, reproducibility, and replicability”. Journal of the American Medical Informatics Association, vol. 26, Issue 3, pp. 185-187.). Transparency permits studies’ replication and reproducibility, which are key factors for research’s credibility (Janz, 2016JANZ, Nicole. (2016), “Bringing the gold standard into the classroom: replication in university teaching”. International Studies Perspectives, vol. 17, issue 4, pp. 392-407.). In the same way that transparency is a condition for research reproduction, reproducibility is the ability to reproduce the same results of a study using the same data and methodology (NSF, 2015NSF (National Science Foundation). (2015), Reproducibility, Replicability, and Generalization in the Social, Behavioral, and Economic Sciences. Available at <https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_ Report_on_Replicability_SubcommitteeMay_2015.pdf>.
https://www.nsf.gov/sbe/SBE_Spring_2015_... )4 4 . Though Hamermesh (2007) separates simple from scientific replication, designating the last term to the expansion of scientific knowledge, we are more concerned with the first term, meaning transparency and reproducibility. Therefore, throughout this paper, we use replication and reproducibility interchangeably. .
Adopting transparency and reproducibility procedures would diminish the probability of mistakes (Simonsohn, 2013SIMONSOHN, Uri. (2013), “Just post it: the lesson from two cases of fabricated data detected by statistics alone”. Psychological Science, vol. 24, no 10, pp. 1875-1888.; King, 1995KING, Gary. (1995), “Replication, Replication”. PS: Political Science and Politics, vol. 28, no 03, pp. 444-452.), increase both the citations rate (Piwowar, Day and Fridsma, 2007) and the trustfulness, efficiency and cumulative advance of academic work (King, 1995KING, Gary. (1995), “Replication, Replication”. PS: Political Science and Politics, vol. 28, no 03, pp. 444-452.; King, 2006KING, Gary. (2006), “Publication, Publication”. PS: Political Science and Politics, vol. 39, no 01, pp. 119-125.; Ball and Medeiros 2012BALL, Richard; MEDEIROS, Norm. (2012), “Teaching integrity in empirical research: a protocol for documenting data management and analysis”. The Journal of Economic Education, vol. 43, no 02, pp. 182-189., Finifter 1975FINIFTER, Bernard M. (1975), “Replication and extension of social research through secondary analysis”. Social Information, vol. 14, no 02, pp. 119-153.; Goodman et al., 2015; Ebersol et al., 2016; Markowetz, 2015MARKOWETZ, Florian. (2015), “Five selfish reasons to work reproducibly”. Genome Biology, vol. 16, no 01, p. 274.; Elman, Kapiszewski and Lupia, 2018; Christensen and Miguel, 2018CHRISTENSEN, Garret; MIGUEL, Edward (2018), “Transparency, reproducibility, and the credibility of economics research”. Journal of Economic Literature, vol. 56, no 3, pp. 920-980.; Gleditsch, 2020GLEDITSCH, Kristian S. (2020), Houston, we have a problem: enhancing academic freedom and transparency in publishing through post-publication debate”. Political Studies Review. DOI: 10.1177/1478929919889309.). The transparency and reproducibility culture are starting to emerge in some important fields, such as business and management sciences (Martins, 2020MARTINS, Henrique Castro. (2020), “A Importância da Ciência Aberta (Open Science) na Pesquisa em Administração”. Rev. Adm. Contemp, vol. 24 no 01.), psychological sciences (Gilbert et al., 2016GILBERT, Daniel T. et al. (2016), “Comment on Estimating the Reproducibility of Psychological Science”. Science, vol. 351, no 6277, p. 1037.), biomedical sciences (Bakken, 2019BAKKEN, Suzanne. (2019), “The journey to transparency, reproducibility, and replicability”. Journal of the American Medical Informatics Association, vol. 26, Issue 3, pp. 185-187.) and social sciences (Christensen, Freese and Miguel, 2019), as well as in great part of developed countries’ scientific communities (Degterev, 2020DEGTEREV, Denis A. (2020), Replicability of Research on International Relations As a Global Trend. Herald of the Russian Academy of Sciences, vol. 90, pp. 36-44.). Some journals already require the release of all replication materials, including data and computer code, as a condition of publication5 5 . More than 1,100 journals have implemented at least one of the Transparency and Openness Promotion Guidelines, according to the Center of Open Science website. Some of the journals are American Journal of Political Science, European Political Science, Italian Political Science Review, Quarterly Journal of Political Science, State Politics and Policy Quarterly, Public Administration Review, Political Science Research and Methods, Political Behavior, Law and Human Behavior, Journal of Economic Behavior & Organization, Political Science Research and Methods. The complete list can be accessed in https://www.cos.io/our-services/top-guidelines? . These requirements have significantly increased the replicability of research in those countries. However, little is known about replicability outside the United States and Western Europe.
Even with all this evidence showing that transparency helps to ensure academic research trustfulness, scientists are, in general, still suspicious about sharing their data with the community (Janz and Freese, 2019JANZ, Nicole; FREESE, Jeremy. (2019), “Good and bad replications in political science: how replicators and original authors (should) talk to each other”. Available in https://www.mzes.uni-mannheim.de/openscience/wp-content/uploads/2019/01/Janz-Freese_-Good-and-Bad-Replications-1.pdf
https://www.mzes.uni-mannheim.de/opensci... ) or face many difficulties to guarantee free and usable access to them. The lack of sufficient resources (including technological frames and abilities, but also time and money) to make research data public available is one of the most cited reasons for scientists not to share their data (Tenopir et al., 2011TENOPIR, Carol et al. (2011), “Data sharing by scientists: practices and perceptions”. PloS One, vol. 6, no 6, p. e21101.). Besides, there is an important concern about the mistaken use of the shared data, specially as far as plagiarism is concerned (Huang et al., 2012HUANG, Xiaolei et al. (2012), “Willing or unwilling to share primary biodiversity data: results and implications of an international survey”. Conservation Letters, vol. 05, no 05, pp. 399-406.). These reasons help us to understand why the rate of data sharing continues low, evidenced by empirical studies in different scientific fields, like less than 6% in behavioral sciences (Hildebrandt and Prenoveau, 2020HILDEBRANDT, Tom; PRENOVEAU, Jason M. (2020), “Rigor and reproducibility for data analysis and design in the behavioral sciences”. Behavior Research and Therapy, vol. 126. doi:10.1016/j.brat.2020.103552), 10-13% in medical sciences (Savage and Vickers, 2009SAVAGE, Caroline J.; VICKERS, Andrew J. (2009), “Empirical study of data sharing by authors publishing in Plos Journals”. PloS One, vol. 4, no 9, p. e7078.; Thelwall et al., 2020THELWALL, Mike et al. (2020), “Is useful research data usually shared? an investigation of genome-wide association study summary statistics”. Plos One, vol. 15, no 2, p. e0229578.), 14% in psychological sciences (Hardwicke et al., 2020HARDWICKE, Tom E. et al. (2020), Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014-2017). Available in https://osf.io/preprints/metaarxiv/9sz2y/
https://osf.io/preprints/metaarxiv/9sz2y... .), 36% in ecology and evolutionary sciences (Roche et al., 2015ROCHE, Dominique G. et al. (2015), “Public data archiving in ecology and evolution: how well are we doing?”. PLoS Biol, vol. 13, no 11, p. e1002295.) and 40% in biological sciences (HUANG et al., 2012HUANG, Xiaolei et al. (2012), “Willing or unwilling to share primary biodiversity data: results and implications of an international survey”. Conservation Letters, vol. 05, no 05, pp. 399-406.).
Using this conceptual framework, we report herein on the first research transparency and reproducibility analysis of political science in Brazil. We attempted to reproduce results of 197 articles reporting quantitative findings published between 2012 e 2016 in the five leading Brazilian political science and general social science journals, the Brazilian Political Science Review, the Revista de Ciência Política, Dados – Revista de Ciências Sociais, Opinião Pública, and Revista Brasileira de Ciências Sociais. Besides examining overall replication rates, we also observed differences in replicability across established, where the use of quantitative data is more common, and emerging subfields, as described below, as well as differences in the availability of data from publications. We tested for differences in methodological approach, transparency, and replicability between research in more and less established subfields in Brazilian Political Science.
Our results show that transparency and reproduction are still in a development stage in Brazilian Political Science. Of the 650 articles reviewed, we asked for data to 197 quantitative articles. From those, only 28% agreed to share datasets and computed codes. We were able to attempt a replication for only 14%, and successfully reproduce the results of less than 5%. Established subfield papers had higher transparency and reproduction rates than those for emerging fields, partly because they were more likely to use quantitative methods. For such articles, we found significantly higher rates of response, data provision, and replication success than with quantitative papers from emerging fields.
This study is divided into four sessions including this introduction. The next session presents the research design that includes the explanation of our hypotheses, criteria for classification of papers into fields (established or emerging) and methodology (quantitative or non-quantitative), details about the request for replication data and coding methodology for our dependent variables. The following session gives details about the replication procedure and the most important results this research has found. The last session is a conclusion of the study and it also brings some suggestions to improve transparency and replicability into the Brazilian social sciences community.
Conducting a replication study is not as trivial as some might think. Replications are difficult to conduct, time-consuming, and hard to publish (Janz, 2016JANZ, Nicole. (2016), “Bringing the gold standard into the classroom: replication in university teaching”. International Studies Perspectives, vol. 17, issue 4, pp. 392-407.). It is important to follow strict procedures to ensure that its design does not cause a failure in replicating. It seems to be the case of an important replication study conducted and reported by the Open Science Collaboration that attempted to replicate 100 published studies from psychological sciences fields and reported a surprisingly low reproducibility rate6 6 . Research Articles, August 28th, 2015, aac4716. . Gilbert et al. (2016)GILBERT, Daniel T. et al. (2016), “Comment on Estimating the Reproducibility of Psychological Science”. Science, vol. 351, no 6277, p. 1037. reported that this study’s design had errors in sampling, power and biases that would harm its conclusions.
To avoid this kind of error, we used a simpler, but accurate procedure. This study used the same dataset as used in original studies and tried to replicate the reported results without any change in samples or methodologies. We analyzed every paper published in the leading Brazilian Political Science journals, and coded each one according to its methodological approach, response to the data request, and research replicability7 7 . Since we covered the five leading journals for five years, increasing either number of journals or years should not make a difference for our analysis. . We tried to replicate every paper that used a quantitative approach. The focus on “quantitative studies” is simply because such designs should make replication easier. When datasets and computer codes are archived appropriately, other analysts should be able to reproduce every figure or table very quickly. Rich and detailed historical studies may also be replicated, but the effort required to do so for five years of articles was beyond the capacity of our project.
Our research was intended primarily as a measurement exercise to establish the extent of replication norms in Brazil. However, we also hypothesized that research in established subfields would be more replicable than research in emerging fields, reflecting the same biases discussed above. In established fields, null results are more likely to be published since they may challenge established scholarship. Similarly, the existence of a larger number of faculty focusing on core questions may increase scholars’ care in testing hypotheses, knowing others may question their work. In contrast, in emerging fields, manuscripts are less likely to be published unless they have significant results. Consider a study on soccer games and votes for incumbents. If there is a null result, the paper is unlikely to be published and may be dismissed as a bad idea. If there is a significant result, a journal may be inclined to publish a “clever” piece that may get media attention. In this way, emerging fields are more likely to suffer false-positive bias due to the institutional problems discussed above8 8 . This example is provided for illustration purposes only and is not intended to comment on any research project. .
We examined all 650 articles published between 2012 and 2016 in the five leading Brazilian political science and general social science journals: Brazilian Political Science Review, Revista Brasileira de Ciência Política, DADOS, Opinião Pública, and Revista Brasileira de Ciências Sociais. Figures 1 and 2 show the distribution of papers by journal and year, respectively.
Identifying Articlefrom Established Subfields
To explore our hypotheses regarding mature and emerging areas of study, we first defined four research subfields: Electoral Politics, Legislative Politics, Political Parties, and Emerging/Other. The first three are among the most established not only in Brazilian Political Science, but also in the international academic community. The other categories included all emerging subfields. Then, we collected the keywords from all papers, and we selected the ones related to the established fields. We pre-determined categories based on the paper keywords, as Appendix A shows. From the 180 keywords, 88 (49%) were assigned to Electoral Politics, 58 (32%) to Legislative Politics, and 34 (19%) to Political Parties. In addition, we preregistered these categories at the Open Science Foundation. For example, we categorize keywords like “candidates”, “candidacies” and “election” as Electoral Politics; keywords like “House of Representatives”, “coalitions” and “Congress” as Legislative Politics and “party system”, “party funding” and “partisanship” were included in the Party Politics category.
We classified each paper into one or more of these four categories. From all 650 papers, 154 (24%) used one or more of the selected keywords. Among those, 60 (9%), 37 (6%), and 16 (2%) included keywords from one of the subgroups Electoral Politics (E), Legislative Politics keywords (L), and Political Parties (P) keywords, respectively. An additional 15 (2%) had keywords from both Electoral Politics and Legislative Politics (E/L), 21 (3%) papers included both Electoral Politics and Political Parties keywords (E/P), and two had both Legislative Politics and Political Parties keywords (L/P). Finally, three papers had all three keywords (E/L/P). The pieces that did not have any of the selected keywords were classified in the “Emerging” area of study.
Established subfields represented 23.7% of all 650 papers. The sum of each subfield is not equal to this percentage because a paper can be classified into more than one subfield. Note that the Electoral Studies subfield is slightly more popular than the other two established subfields. Appendix B has a detailed classification of each paper.
Identifying Quantitative Works
To implement our research design, we first needed to identify which papers had a quantitative approach that allowed replication using only dataset and computer codes. Our methodological decision to only try to replicate quantitative studies relied on the research’s limitations to both analyze and reproduce qualitative data9 9 . For more details on the challenges of implementing reproduction and replication of qualitative studies see Elman and Kapiszewski (2014), Tuval-Mashiach (2017) or https://www.qualtd.net/ and https://openworking.tudl.tudelft.nl/2019/02/11/what-does-reproducibility-mean-for-qualitative-research/ .
We sought the main results reported for each paper and evaluated if it included any quantitative analysis. We created a dummy variable named Quantitative and assumed value 1 if, after a basic overview of the study, we identified quantitative results among the study’s main findings. The presence of charts and tables would indicate the use of a quantitative methodology. This selection cannot be considered very strict for a couple of reasons. First, we did the work manually, since there was no way to collect the articles’ metadata automatically10 10 . We thank an anonymous reviewer for pointing us that there is a R package that can extract metadata from journals today. See more details in https://fmeireles.com/blog/rstats/rscielo-um-scraper-para-extrair-dados-scielo. . Second, in order to enlarge the number of observations, we decided to include studies that had relevant quantitative results but did not necessarily use a statistical inference analytical method. Therefore, we considered a study with only a descriptive analysis as quantitative.
This decision brought up a discussion about differences within the group of quantitative studies. It would be more challenging to replicate research that used complex econometric models than another that did only a descriptive analysis. We are aware of this situation, but, for this research, we chose not to explore this heterogeneity. It would be necessary to develop an objective criterion to classify each quantitative methodology by their complexity and separate them into groups according to different replication difficulties. These procedures were beyond our time and work resources.
Even with a soft parameter for Quantitative, the number of quantitative papers was only 197, or 30% of all pieces examined. Notably, the proportion of quantitative articles in established subfields was significantly higher than in emerging subfields. Although they were less than a quarter of the potentially replicable papers, the established subfield papers included more than half of the quantitative research.
Table 1 Quantitative papers by Maturity
Requesting Replication Materials
For each paper reporting quantitative results, we sent an e-mail message to the authors and requested a replication dataset and a computer code. The message explained the research goals and assured author privacy in disclosure of our research, using the model e-mail below:
“Subject: Request for Data for Replication Study Without Identification.
Dear Mr(s). (Author’s name),
We write on behalf of Professors George Avelino (FGV, coordinator of the Center of Politics and Economics of the Public Sector), and Scott Desposato (the University of California, San Diego, director of the Center for Latin American Studies at the same university). We are developing an analysis of the replicability of the Political Science articles published in the leading Brazilian journals in the field, and your paper is in our database.
We would like to request both the dataset and the computer codes required to reproduce the results in your article (Title of the article), published in the journal (Name of the journal) on year (Year), volume (Volume number), (Journal number).
Your name, article name, or any other identifying information about you or your article, will NOT be disclosed in our final report.
Thank you in advance for your contribution to our project. Regards,”
The procedure followed the schedule below:
Phase I (responses)
· March 5th, 2018: sent the original message;
· March 26th, 2018: sent the first follow-up message to those who did not answer the original message and to address any questions;
· April 20th, 2018: sent a second follow-up message to those who did answer the first follow-up message and to answer any remaining questions;
· April 27th, 2018: deadline for receiving answers.
Phase II (completeness)
· May 7th, 2018: sent a second message asking for the complete material for those who answered our first e-mail, but who provided incomplete replication materials;
· May 21st, 2018: final deadline for answers for incomplete material.
In total, scholars had almost eight weeks answer our first message, and 11 weeks to provide replication materials. In general, the ones who actually answered our request did it very shortly. A more extended period might have increased our answer rate, but we received no additional answers after the final deadline.
Besides the proportion of studies that used a quantitative approach, we examined four transparency and replicability measures applied to each article that we characterized as having some quantitative analysis. Our four measures were:
· Responsiveness: Did the author(s) answer our email, even with a refusal to collaborate?
· Agreement: Did the author(s) agree to share data?
· Completeness: Did the author(s) provide enough data and code to enable the research replication attempt?
· Replicability: Were results replicable?
For Responsiveness, we first coded the papers by combining the answers in the two research phases (the first message and the follow-up) and the lack thereof: 1) Answered in phase 1; 2) Answered in phase 2; 3) Did not answer. Then, we coded Responsiveness as “1” for those that answered our request either in Phase 1 or Phase 2. In total, 83 of the 197 papers (42%) answered our request. They were divided nearly equally between established and emerging fields.
The second variable, Agreement, measures whether the author agreed to share the data or not. The variable had six categories: 1) Answered with data; 2) No response; 3) Data is not available; 4) Cannot share data; 5) Refusal and 6) Other. We coded the variable Agreement as a “1” for articles where the author provided data. This variable was not among our pre-registered dependent variables, but it seemed appropriate to distinguish the authors who answered positively from those who did not. Among the 83 responses, 56 (67.5%) provided some analysis data. From these, 36 papers (almost 65% of the 56) were from established fields.
As far as Completeness was concerned, we coded the following categories: 1) Complete; 2) No useful data; 3) Only code; 4) Only data source, and 5) Only dataset. We coded Completeness as a “1” for articles in which authors sent at least one dataset and some computer code. Only half of the 56 papers that provided data did it completely. More than two-thirds of these papers came from established fields (19).
Finally, Replicability had three categories: 1) Completely replicable (all results were reproduced, including sign and statistical significance); 2) Partially replicable (not all results were completely reproduced, or signs/significance did not match the ones reported in the paper) or 3) Non-Replicable (no result could be reproduced). We coded Replicability as “1” for articles in which data that the author(s) provided produced the same results in terms of values, signs, and statistical significance. From those 28 papers with complete data, we were able to fully replicate only 10 (35.7%), seven of which were from established fields.
Table 2 presents how dependent variables were coded according to the situations described above. Only studies that were coded “1” for Replicability were considered successfully replicable. Table 3 brings descriptive variables by field type for our sample (quantitative studies) and dependent variables.
Dependent Variables Coding
We attempted to replicate all papers for which authors shared data and, in some cases, code. The first decision was to define which quantitative results to replicate, since many papers had multiple charts and tables with data not related to the core results. Given the magnitude of the task, we chose to focus on the paper’s prominent results. To identify a prominent result, we identified quantitative findings that were most discussed, especially in the abstract, introduction, and conclusion of the study. This procedure constrained our analysis to a smaller number of charts and tables. It also meant we ignored difficulties in replicating secondary analyses.
In replicating, we found problems with data, codes, or results. Figure 4 presents the distribution of these problems. Of the 28 for which we could attempt a replication, we found problems with almost two-thirds of them (18). The most frequent had to do with computer codes, which represented about one-third (10). We also had problems with results and data in eight (28.6%) and seven papers (25%), respectively. Finally, seven articles (21.5%) presented more than one type of problem.
The types of coding problems involved either a non-working or an incomplete statistical computing code. In some cases, we could not calculate a specific result displayed in the paper. In others, there were syntax errors that made a complete replication impossible. We frequently received code lacking one or more calculations, and in many cases, poorly documented, with analyses and data management left unexplained. In one case, the data analysis was computationally intensive and eventually produced results like those published, but it did not reproduce the central figures. Interestingly, many code sets led to results that were not part of the article, suggesting some fishing for results. Finally, there were some minor syntax errors that we were able to correct.
Data problems involved cases in which materials that the authors shared lacked a critical variable or even a key dataset. For example, one author provided a dataset which seemed to be a complete dataset (data frame and code) and which could replicate some of the published regressions. However, further results in the computer code required an additional dataset, which the authors did not share. In some cases, the authors shared a web address where we could find the raw data, but they did not provide specific files with instructions for its coding and cleaning. In other cases, the authors provided a complex code with no instructions on how to execute it.
Some cases were more problematic since we could not open the dataset, or the data differed from the published results. In other cases, authors shared many different files, but none was useful or even required by the accompanying computer code. One author shared a zip file with multiple “xls” files from which we could not replicate results. These papers were classified as “no useful data” and did not proceed to the replication phase.
In some other cases, although we were able to produce results, they did not match those published. We classified those cases as results problems. Some codes failed to define the data subset used for analysis, and the number of observations displayed in the tables did not match with our results. In such cases, even descriptive statistics could not be reproduced. Other papers had problems with the statistical results. For example, our reproduction of regression models yielded different coefficient values for one or more variables.
Data was basically provided in Excel, Stata and R formats, which are well known statistical software. Except for minor corrections in scripts (including adaptations to software updates), replicators were oriented to not modify the code structure. Dataset and code organization is an important part regarding data transparency. In the case an author shared files in a different format or in a way that was not possible to understand how to use it, they were asked to provide more information. In cases that it happened, authors did not answer for our help request and the data were considered as “non-useful”.
Successful replications had several shared features. Most notably, the authors provided a well-organized set of data and codes. In one case, the authors did not send the computer code, but their results were clearly described in the text of their article and were straightforward to reproduce. In another case, the authors provided the dataset in a “xls” format file and a Stata file with few lines. Those lines described what table or graph it was related to in the paper. This type of information was critical to enable replication.
Established versus Emerging Quantitative Subfields
Of the 650 papers examined, 197 (30.3%) used a quantitative approach. Of these, only 56 (28.4% of quantitative papers and 8.6% overall) responded positively to our request for data. In half of these cases, the data was incomplete, which did not allow replication. Most of these were cases in which the author shared only codes or only a dataset, but not both. In the end, we completed a replication attempt on 28 papers (14.2% of quantitative papers and 4.3% overall), and only ten of them were fully replicable. In those last cases, the authors provided all data and code to reproduce the core results in the paper. In other words, we were able to successfully replicate only 5% of quantitative and 1.5% of all papers.
We hypothesized that established quantitative subfields would have stronger replication norms and higher rates of replicability than emerging fields. We tested our hypothesis by comparing our measures of responsiveness, agreement, data provision, and reproducibility for both established and emerging fields. Table 4 results from T-tests for a difference of means between these two groups among the five dependent variables.
T-Tests on Dependent Variables by Maturity (Emerging – Established)
For all our dependent variables, the proportions are higher for the established than for the emerging fields. Authors of research in established areas were nearly twice as likely to respond to our request with some data (36% versus 20.6% in emerging fields). Additionally, 52.8% of authors from those fields provided complete data, versus 45% of those in emerging ones. Finally, of those that provided data, over half of the results from established fields could be replicated (52.8%), versus a slightly lower rate for emerging areas (45%).
The difference of means test presented in Table 3 showed that established subfields papers have a significantly higher probability of adopting a quantitative methodology, answering a request for data, providing the data with more quality, and ultimately being replicable. When including control variables in an OLS regression, the results are similar.
Table 5 reports the results of OLS regressions of our five dependent variables. Independent variable of interest is Established, a dummy variable that indicates papers from established subfields. Control variables include the number of authors and dummies for journals and year of publication.
OLS Regressions Results
Established variable remains positive and significant for all dependent variables. The control variables do not have any clear patterns11 11 . We did not preregister hypotheses about control variables. . The number of authors had a positive and significant effect only on whether articles included quantitative analysis and whether any of them answered our e-mail. Using Brazilian Political Science Review as baseline, the reported effect of journals shows that both Revista Brasileira de Ciência Política and Revista Brasileira de Ciências Sociais seem to have a lower rate of quantitative papers.
Additionally, authors from the same journals were generally less likely to agree to share their data and to share complete data. Finally, results for year of publication have 2012 as baseline and do not strongly suggest an increasing replicability in more recent papers. Only Agreement (2015 and 2016) and Completeness (2016) reported significant higher averages for more recent papers.
Around the world, the credibility crisis in science poses a challenge to scholars’ efforts to advance knowledge both in the natural and the social disciplines. As Figueiredo et al. (2019)FIGUEIREDO FILHO, Dalson et al. (2019), “Seven reasons why: a user’s guide to transparency and reproducibility”. Brazilian Political Science Review [Online], vol. 13, no 02. (15/12/2019), pp. 1-37. Available in: http://dx.doi.org/10.1590/1981-3821201900020001. ISSN 1981-3821.
http://dx.doi.org/10.1590/1981-382120190... discussed, norms of transparency and replicability are becoming part of standard scientific practice in the United States, Western Europe, and other regions. The scientific community cannot neglect credibility issues anymore. Addressing this crisis will require significant changes in the procedures of scientific institutions, journals, and scholars. There are reforms and best practices that should spread to all academic communities. Especially noteworthy is a general norm of research transparency.
This study is the first look at transparency and replicability in Brazil. It covers only a small sample of the academic work produced in the country and it has its limitations in terms of generalization and comparison with other countries’ standards. Even so, the evidence presented here can trigger new research in this field that would help the Brazilian academic community to follow this critical path.
Let us make it clear: our findings do not mean most research is invalid, or conclusions are incorrect. It is just a check on how easy it is to replicate or reproduce the main results. Strictly, if one considers the time lag between submission and publication, our data may even not represent the field accurately during the period12 12 . We thank an anonymous reviewer for calling our attention to this point. . Besides, we only check whether scholars provide the data we requested within two months of our request and whether we can easily use it to generate results identical to those in publication. With more time and more explanation of our purposes, perhaps we would have a higher response rate. With some sources from which we did receive data but failed to replicate results, with more time, maybe we could have reverse engineered findings and asked for more assistance from authors in understanding what they provided. With more time, we could have attempted to replicate results by collecting the original data ourselves from their sources. Again, non-reproducible does not mean invalid in any way. We believe it primarily reflects a lack of transparency norms in Brazilian political science and the need for a new standard according to which it should be easy to replicate and verify any published findings.
The difference in the results from established and emerging fields was also somewhat expected. In general, the more scholars engage in a dialogue around common themes and questions, and the more the field often uses quantitative analysis, the higher their need for transparency procedures to enhance the collective advancement of knowledge.
Admittedly, introducing replication practices in qualitative research poses a different challenge, which would go beyond the scope of this paper. Despite the methodological differences involved, this would be a great improvement for the Brazilian political science community, due to its large number of qualitative works. Also, introducing transparency practices for qualitative research would help to keep pace with the increasing use of mixed methods research designs, which usually requires teams made with members from different methodological backgrounds. Within this picture, trying to define transparency and replication practices in qualitative research, without compromising its small “n” and interpretive characteristics, is a crucial step. (Sukumar and Metoyer, 2019SUKUMAR, Porna Talkad; METOYER, Ronald. (2019), “Replication and transparency of qualitative research from a constructivist perspective”. OSF Preprints (https://osf.io/6efvp)
https://osf.io/6efvp... ; Moravcsik, 2014a, 2014b, 2019).
Our replicability analysis in Brazil suggests that political scientists in that country need to adopt transparency practices. Finally, we offer several recommendations for scholars and journals to improve replicability and maturity in these fields.
Authors can publish a Pre-Analysis Plan or a Registered Report before the beginning of their data collection or analysis (Casey, Glennerster and Miguel, 2012; Miguel et al., 2014MIGUEL, Edward et al. (2014), “Promoting Transparency in Social Science Research”. Science, vol. 343, no 166, pp. 30-31.), even on qualitative studies (Pineiro and Rosenblatt, 2016PINEIRO, Rafael; ROSENBLATT, Fernando. (2016), “Pre-Analysis Plans for Qualitative Research”. Revista de Ciência Política (Santiago) [online], vol. 36, no 3, pp. 785-796.). This procedure will strengthen research transparency, mitigating some of the credibility problems we found. Explicitly specifying the research question and defining our analysis plan will make researchers pay more attention to their study design and focus on theory and research questions. These procedures are relatively easy to adopt, since there are platforms like the Open Science Framework, where one can pre-register research plans and share data, codes, and results.
Scientific institutions can teach and encourage their members to implement these procedures (Steinhardt, 2020). Undergraduate, Graduate and Ph.D programs should have replication classes and should teach the best practices for data management13 13 . See https://www.academia.edu/42204205/Transparency_and_Reproducibility_in_Empirical_Research_2020.1_ for a good example of course structure provided by Professor Dalson Britto Figueiredo Filho of Universidade Federal de Pernambuco. . These classes would increase the students’ chances to get their research proposals funded, since a growing number of funding institutions began to require the inclusion of a data management plan in the submitted proposals. For instance, FAPESP, a well-known Brazilian research funding institution, began to require the inclusion of a data management plan (http://www.fapesp.br/gestaodedados/). Additionally, data management courses could also provide an easy way for students to get their first publication by replicating other papers (King, 2006KING, Gary. (2006), “Publication, Publication”. PS: Political Science and Politics, vol. 39, no 01, pp. 119-125.). Finally, a recent paper by Christensen et al. (2019)CHRISTENSEN, Garret; FREESE, Jeremy; MIGUEL, Edward (2019), Transparent and reproducible social science research: how to do open science. Oakland, CA: University of California Press. estimates that papers that make data and coding available receive additional scholarly citations.
Journals can support the publication of null results and of review processes that focus on research questions and design, rather than provocative results. For example, the Center for Open Science recommends that journals include peer review of research plans as well as results, known as Registered Reports:
Registered Reports is a publishing format used by over 200 journals that emphasizes the importance of the research question and the quality of methodology by conducting peer-review before data collection. High-quality protocols are then provisionally accepted for publication if the authors follow through with the registered methods. This format rewards best practices in adhering to the hypothetical-deductive model of the scientific method. It eliminates a variety of questionable research practices, including low statistical power, selective reporting of results, and publication bias, while allowing complete flexibility to report serendipitous findings. (https://cos.io/rr/)
Registering a research plan is not an instrument to restrict scientists on their flexibility to conduct their research or a safeguard for journals against their responsibility for the credibility of their publications. Unpredictable events may occur during data collection, imposing changes in analysis strategy. A pre-registered report informs readers about such unforeseen problems faced by researchers, and solutions implemented, enhancing research transparency (Christensen, Freese and Miguel, 2019). In other scenarios, readers never learn the real reason behind the methodological choice, opening the way for credibility questioning. For journals, pre-registered analysis plans are an opportunity to mitigate publication bias and reduce incentives for fishing and p-hacking.
Officially, releasing replication data is another essential practice to improve science transparency. Using and sharing computer codes allow research verification and replication (King, 2003KING, Gary. (2003), “The Future of the Replication Movement”. International Studies Perspectives, vol. 04, no 1, pp. 72-107.; Elman, Kapiszewski and Lupia, 2018; Dafoe, 2014DAFOE, Allan. (2014), “Science deserves better: the imperative to share complete replication files”. Political Science and Politics, vol. 47, no 1, pp. 60-66.). Publishing replication materials increases incentives for careful analysis and accurate reporting. Verification also allows for investigating potential cases of fraud. Indeed, it would have been impossible to uncover recent cases of data fabrication in the United States scandals without access to underlying data. Replication materials can also support academic teaching and learning.
To keep in touch with the international trend, the Brazilian scientific community should incorporate transparency practices. ABCP (Associação Brasileira de Ciência Política), as well as other associations and Journals, should adopt standards of replication, requiring and archiving data and replication code for published research, subject to the privacy and copyright restrictions.
They could endorse or even implement together a general data repository to concentrate all scientific work. A well-known benchmark is The Berkeley Initiative for Transparency in the Social Sciences (BITSS).
The Berkeley Initiative for Transparency in the Social Sciences (BITSS) is another new institution that has emerged in recent years to promote dialogue and build consensus around transparency practices. BITSS has established an active training program for the next generation of economists and other social scientists, as well as an award to recognize emerging leaders in this area, the Leamer-Rosenthal Prize for Open Social Science (Christensen and Miguel, 2018CHRISTENSEN, Garret; MIGUEL, Edward (2018), “Transparency, reproducibility, and the credibility of economics research”. Journal of Economic Literature, vol. 56, no 3, pp. 920-980.:61-62).
Adopting these recommendations will increase the maturity and credibility of social science in Brazil and benefit scholarship and society. Associations and journals can both encourage and facilitate such practices. For example, ABCP and universities can offer courses to teach their members how to track their research procedures, how to prepare a dataset, and how to use computer programs that facilitate this task. In addition, to further transparency, ABCP could lead an association with an internationally known data storage, such as Dataverse (https://dataverse.org/), which could increase the rewards for registering dataset and codes by making them available to the international scientific community. Universities need to nurture a reproducibility and replication culture within their departments and courses to ensure that a “gold standard” of credibility in academia is strengthened (Janz, 2016JANZ, Nicole. (2016), “Bringing the gold standard into the classroom: replication in university teaching”. International Studies Perspectives, vol. 17, issue 4, pp. 392-407.). They can also provide online storage for datasets and code sharing.
To further the publication of datasets that interest a broad audience, journals’ editors can create a section dedicated to papers presenting datasets to the community. Journals can also publicly require datasets and codes to submission approval (Dafoe, 2014DAFOE, Allan. (2014), “Science deserves better: the imperative to share complete replication files”. Political Science and Politics, vol. 47, no 1, pp. 60-66.). Finally, journals need to encourage replication publications, but not implement a reverse publication bias, focusing only on failed replications attempts (Janz and Freese, 2019JANZ, Nicole; FREESE, Jeremy. (2019), “Good and bad replications in political science: how replicators and original authors (should) talk to each other”. Available in https://www.mzes.uni-mannheim.de/openscience/wp-content/uploads/2019/01/Janz-Freese_-Good-and-Bad-Replications-1.pdf
https://www.mzes.uni-mannheim.de/opensci... ). Recently, the Brazilian Political Science Review adopted a policy that supported transparency. From their author’s guideline for submission:
Authors submitting empirical papers should provide, when and if the manuscript is approved, the database used in the analysis, the code dictionary describing the variables, and the code for replication or the series of steps of the analysis. The database must be in an easily accessible format for conventional statistical software – such as R, Stata, SPSS, and Excel. The code dictionary should indicate the name of the variable, its description, and the source of the data. Where appropriate, the code for replication of the analyzes should be extensively commented. When the article does not use statistical software that allows the creation of a replication code, the author should describe the procedures, step-by-step, for replication to be possible.
In a good note, a more recent survey with the editors of all other four journals investigated in this research shows they are all intending to introduce transparency and replicability procedures soon. This result shows that concerns with those procedures are starting to prevail in the Brazilian Political Science, making it closer to the international community. A future analysis will tell whether this is an excessive optimistic vision or not.
The scientific community is increasingly adopting transparency norms. Most likely, they will become mandatory for international publication and international research funding. The sooner Brazil’s scientific community incorporates these transparency standards, the better. In short, implement transparency procedures – like research pre-registration and sharing of codes and data – is beneficial for everyone. It mitigates credibility suspicions, values scientific work, and, above all, allows research activity to concentrate on what is most important: the questions they seek to answer.
- BALL, Richard; MEDEIROS, Norm. (2012), “Teaching integrity in empirical research: a protocol for documenting data management and analysis”. The Journal of Economic Education, vol. 43, no 02, pp. 182-189.
- BAKKEN, Suzanne. (2019), “The journey to transparency, reproducibility, and replicability”. Journal of the American Medical Informatics Association, vol. 26, Issue 3, pp. 185-187.
- CASEY, Katherine; GLENNERSTER, Rachel; MIGUEL, Edward (2012), “Reshaping institutions: evidence on aid impacts using a pre-analysis plan”. The Quarterly Journal of Economics, vol. 127, no 4, pp. 1755-1812.
- CHRISTENSEN, Garret; MIGUEL, Edward (2018), “Transparency, reproducibility, and the credibility of economics research”. Journal of Economic Literature, vol. 56, no 3, pp. 920-980.
- CHRISTENSEN, Garret; FREESE, Jeremy; MIGUEL, Edward (2019), Transparent and reproducible social science research: how to do open science. Oakland, CA: University of California Press.
- CHRISTENSEN, Garret et al. (2019), “A Study of the Impact of Data Sharing on Article Citations Using Journal Policies as a Natural Experiment”. PLoS ONE, vol. 14, no 12, e0225883. https://doi.org/10.1371/journal.pone.0225883
- CHORDIA, Tarun; GOYAL, Amit; SARETTO, Alessio. (2017), “P-hacking: evidence from two million trading strategies”. Swiss Finance Institute Research Paper, n. 17-37.
- CRUWELL, Sophia et al. (2019), “Seven easy steps to open science: an annotated reading list”. Zeitschrift für Psychologie, vol. 227, no 4, pp. 237-248.
- DAFOE, Allan. (2014), “Science deserves better: the imperative to share complete replication files”. Political Science and Politics, vol. 47, no 1, pp. 60-66.
- DEGTEREV, Denis A. (2020), Replicability of Research on International Relations As a Global Trend. Herald of the Russian Academy of Sciences, vol. 90, pp. 36-44.
- ELMAN, Colin; KAPISZEWSKI, Diana. (2014), “Data Access and Research Transparency in the qualitative tradition”. PS: Political Science & Politics, vol. 47, no 01, pp. 43-47.
- FERGUSON, Christopher J.; HEENE, Moritz. (2012), “A vast graveyard of undead theories: publication bias and psychological science’s aversion to the null”. Perspectives on Psychological Science, vol. 07, no 06, pp. 555-561.
- FIGUEIREDO FILHO, Dalson et al. (2019), “Seven reasons why: a user’s guide to transparency and reproducibility”. Brazilian Political Science Review [Online], vol. 13, no 02. (15/12/2019), pp. 1-37. Available in: http://dx.doi.org/10.1590/1981-3821201900020001 ISSN 1981-3821.
- FINIFTER, Bernard M. (1975), “Replication and extension of social research through secondary analysis”. Social Information, vol. 14, no 02, pp. 119-153.
- GERBER, Alan S.; GREEN, Donald P.; NICKERSON, David. (2001), “Testing for Publication Bias in Political Science”. Political Analysis, vol. 09, no 04, pp. 385-392.
- GILBERT, Daniel T. et al. (2016), “Comment on Estimating the Reproducibility of Psychological Science”. Science, vol. 351, no 6277, p. 1037.
- GLEDITSCH, Kristian S. (2020), Houston, we have a problem: enhancing academic freedom and transparency in publishing through post-publication debate”. Political Studies Review. DOI: 10.1177/1478929919889309.
- GOODMAN, Alissa et al. (2015), “Ten simple rules for the care and feeding of scientific data”. PLOS Computational Biology, vol. 10, no 04, pp. 01-05.
- GOODMAN, Steven N.; FANELLI, Danielle; IOANNIDIS, John P. A. (2016), “What does research reproducibility mean?”. Science Translational Medicine, vol. 08, no 341, pp. 341ps12.
- HAMERMESH, Daniel (2007), “Replication in Economics”. NBER Working Paper, n. 13026.
- HARDWICKE, Tom E. et al. (2020), Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014-2017). Available in https://osf.io/preprints/metaarxiv/9sz2y/
- HEAD, Megan L. et al. (2015), “The Extent and Consequences of P-Hacking in Science”. PLOS Biology 13(3): e1002106. https://doi.org/10.1371/journal.pbio.1002106
- HILDEBRANDT, Tom; PRENOVEAU, Jason M. (2020), “Rigor and reproducibility for data analysis and design in the behavioral sciences”. Behavior Research and Therapy, vol. 126. doi:10.1016/j.brat.2020.103552
- HUANG, Xiaolei et al. (2012), “Willing or unwilling to share primary biodiversity data: results and implications of an international survey”. Conservation Letters, vol. 05, no 05, pp. 399-406.
- HUMPHREYS, Macartan; DE LA SIERRA, Raul Sanchez; VAN DER WINDT, Peter. (2013), “Fishing, commitment, and communication: a proposal for comprehensive nonbinding research registration”. Political Analysis, vol. 21, no 01, pp. 1-20.
- IONNADIS, John. (2005), “Why most published research findings are false”. PLOS Medicine. Available at: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
- JANZ, Nicole. (2016), “Bringing the gold standard into the classroom: replication in university teaching”. International Studies Perspectives, vol. 17, issue 4, pp. 392-407.
- JANZ, Nicole; FREESE, Jeremy. (2019), “Good and bad replications in political science: how replicators and original authors (should) talk to each other”. Available in https://www.mzes.uni-mannheim.de/openscience/wp-content/uploads/2019/01/Janz-Freese_-Good-and-Bad-Replications-1.pdf
- KEY, Ellen M. (2016), “How are we doing? data access and replication in political science”. PS: Political Science and Politics, vol. 49, no 02, pp. 268-272.
- KING, Gary. (1995), “Replication, Replication”. PS: Political Science and Politics, vol. 28, no 03, pp. 444-452.
- KING, Gary. (2003), “The Future of the Replication Movement”. International Studies Perspectives, vol. 04, no 1, pp. 72-107.
- KING, Gary. (2006), “Publication, Publication”. PS: Political Science and Politics, vol. 39, no 01, pp. 119-125.
- KING, Gary; KEOHANE, Robert O.; VERBA, Sidney. (1994), Designing social inquiry: scientific inference in qualitative research. Princeton: Princeton University Press.
- MARKOWETZ, Florian. (2015), “Five selfish reasons to work reproducibly”. Genome Biology, vol. 16, no 01, p. 274.
- MARTINS, Henrique Castro. (2020), “A Importância da Ciência Aberta (Open Science) na Pesquisa em Administração”. Rev. Adm. Contemp, vol. 24 no 01.
- MIGUEL, Edward et al. (2014), “Promoting Transparency in Social Science Research”. Science, vol. 343, no 166, pp. 30-31.
- MORAVCSIK, Andrew. (2014a), “Transparency: The Revolution in Qualitative Research”. PS: Political Science and Politics, vol. 47, no 1, pp. 48-53.
- MORAVCSIK, Andrew. (2014b), “Trust but verify: the transparency revolution and qualitative international relations”. Security Studies, vol. 23, no 4, pp. 663-688.
- MORAVCSIK, Andrew. (2019), Transparency in Qualitative Research. SAGE Research Methods (https://www.princeton.edu/~amoravcs/library/TransparencyinQualitativeResearch.pdf
- NSF (National Science Foundation). (2015), Reproducibility, Replicability, and Generalization in the Social, Behavioral, and Economic Sciences. Available at <https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_ Report_on_Replicability_SubcommitteeMay_2015.pdf>.
» https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_ Report_on_Replicability_SubcommitteeMay_2015.pdf
- PINEIRO, Rafael; ROSENBLATT, Fernando. (2016), “Pre-Analysis Plans for Qualitative Research”. Revista de Ciência Política (Santiago) [online], vol. 36, no 3, pp. 785-796.
- PIWOWAR, Heather A.; DAY, Roger S.; FRIDSMA, Douglas B. (2007), “Sharing Detailed Research Data Is Associated with Increased Citation Rate”. PloS One, vol. 2, no 3, p. e308.
- ROCHE, Dominique G. et al. (2015), “Public data archiving in ecology and evolution: how well are we doing?”. PLoS Biol, vol. 13, no 11, p. e1002295.
- SAVAGE, Caroline J.; VICKERS, Andrew J. (2009), “Empirical study of data sharing by authors publishing in Plos Journals”. PloS One, vol. 4, no 9, p. e7078.
- SIMONSOHN, Uri. (2013), “Just post it: the lesson from two cases of fabricated data detected by statistics alone”. Psychological Science, vol. 24, no 10, pp. 1875-1888.
- SUKUMAR, Porna Talkad; METOYER, Ronald. (2019), “Replication and transparency of qualitative research from a constructivist perspective”. OSF Preprints (https://osf.io/6efvp)
- TENOPIR, Carol et al. (2011), “Data sharing by scientists: practices and perceptions”. PloS One, vol. 6, no 6, p. e21101.
- THELWALL, Mike et al. (2020), “Is useful research data usually shared? an investigation of genome-wide association study summary statistics”. Plos One, vol. 15, no 2, p. e0229578.
- TUVAL-MASHIACH, Rivka. (2017), “Raising the curtain: the importance of transparency in qualitative research”. Qualitative Psychology, vol. 4, no 2, pp. 126-138.
. See Lacour and Green fraud case in https://www.chronicle.com/article/What-Social-Science-Can-Learn/230645
. For more information about p-hacking concept see Head et al. (2015)HEAD, Megan L. et al. (2015), “The Extent and Consequences of P-Hacking in Science”. PLOS Biology 13(3): e1002106. https://doi.org/10.1371/journal.pbio.1002106.
https://doi.org/10.1371/journal.pbio.100... and Chordia, Goyal and Saretto (2017).
. A false positive result consists in rejecting the null hypothesis when it was not the case of rejecting it. The opposite case is known as false negative, when the null hypothesis is not rejected when it should be.
. Though Hamermesh (2007)HAMERMESH, Daniel (2007), “Replication in Economics”. NBER Working Paper, n. 13026. separates simple from scientific replication, designating the last term to the expansion of scientific knowledge, we are more concerned with the first term, meaning transparency and reproducibility. Therefore, throughout this paper, we use replication and reproducibility interchangeably.
. More than 1,100 journals have implemented at least one of the Transparency and Openness Promotion Guidelines, according to the Center of Open Science website. Some of the journals are American Journal of Political Science, European Political Science, Italian Political Science Review, Quarterly Journal of Political Science, State Politics and Policy Quarterly, Public Administration Review, Political Science Research and Methods, Political Behavior, Law and Human Behavior, Journal of Economic Behavior & Organization, Political Science Research and Methods. The complete list can be accessed in https://www.cos.io/our-services/top-guidelines?
. Research Articles, August 28th, 2015, aac4716.
. Since we covered the five leading journals for five years, increasing either number of journals or years should not make a difference for our analysis.
. This example is provided for illustration purposes only and is not intended to comment on any research project.
. For more details on the challenges of implementing reproduction and replication of qualitative studies see Elman and Kapiszewski (2014)ELMAN, Colin; KAPISZEWSKI, Diana. (2014), “Data Access and Research Transparency in the qualitative tradition”. PS: Political Science & Politics, vol. 47, no 01, pp. 43-47., Tuval-Mashiach (2017)TUVAL-MASHIACH, Rivka. (2017), “Raising the curtain: the importance of transparency in qualitative research”. Qualitative Psychology, vol. 4, no 2, pp. 126-138. or https://www.qualtd.net/ and https://openworking.tudl.tudelft.nl/2019/02/11/what-does-reproducibility-mean-for-qualitative-research/
. We thank an anonymous reviewer for pointing us that there is a R package that can extract metadata from journals today. See more details in https://fmeireles.com/blog/rstats/rscielo-um-scraper-para-extrair-dados-scielo.
. We did not preregister hypotheses about control variables.
. We thank an anonymous reviewer for calling our attention to this point.
. See https://www.academia.edu/42204205/Transparency_and_Reproducibility_in_Empirical_Research_2020.1_ for a good example of course structure provided by Professor Dalson Britto Figueiredo Filho of Universidade Federal de Pernambuco.
Hypotheses and data analysis plan were pre-registered at the Open Science Foundation except as noted. Analysis of existing datasets was exempted by the UCSD Institutional Review Board (IRB). The authors thank the Berkeley Institute for Transparency in the Social Sciences for funding and support. George Avelino and Ivan Mardegan acknowledge the support from São Paulo Research Foundation FAPESP (Grants #: 2013/156158-1 and 2017/20855-1). The authors also thanks Raduan Meira for research assistance.
A – Keywords Classification
|2010 Brazilian elections||Electoral Politics|
|2010 elections||Electoral Politics|
|2014 Brazilian elections||Electoral Politics|
|2014 elections||Electoral Politics|
|Biracial electoral races||Electoral Politics|
|Brazilian elections||Electoral Politics|
|Brazil’s presidential election in 2014||Electoral Politics|
|Candidates for deputy||Electoral Politics|
|Career patterns||Electoral Politics|
|Coattail effect||Electoral Politics|
|Correct voting||Electoral Politics|
|Economic vote||Electoral Politics|
|Economic voting in Brazil||Electoral Politics|
|Election campaigns||Electoral Politics|
|Election expenses||Electoral Politics|
|Election studies||Electoral Politics|
|Elections 2014||Electoral Politics|
|Electoral Arena||Electoral Politics|
|Electoral behavior||Electoral Politics|
|Electoral behavior||Electoral Politics|
|Electoral behavior in Brazil||Electoral Politics|
|Electoral campaign||Electoral Politics|
|Electoral campaigns||Electoral Politics|
|Electoral competition||Electoral Politics|
|Electoral coordination||Electoral Politics|
|Electoral costs||Electoral Politics|
|Electoral effects||Electoral Politics|
|Electoral financing||Electoral Politics|
|Electoral fraud||Electoral Politics|
|Electoral geography||Electoral Politics|
|Electoral governance||Electoral Politics|
|Electoral information||Electoral Politics|
|Electoral instability||Electoral Politics|
|Electoral justice||Electoral Politics|
|Electoral justice system||Electoral Politics|
|Electoral performance||Electoral Politics|
|Electoral realignment||Electoral Politics|
|Electoral recruitment||Electoral Politics|
|Electoral rules||Electoral Politics|
|Electoral strategies||Electoral Politics|
|Electoral Studies||Electoral Politics|
|Electoral success||Electoral Politics|
|Electoral systems||Electoral Politics|
|Electoral television slots||Electoral Politics|
|Electoral volatility||Electoral Politics|
|Evangelical vote||Electoral Politics|
|Female candidacies||Electoral Politics|
|Geographical scattering of votes||Electoral Politics|
|Invalid votes||Electoral Politics|
|Issue voting||Electoral Politics|
|Local elections||Electoral Politics|
|Municipal elections||Electoral Politics|
|Negative campaign||Electoral Politics|
|Online campaign||Electoral Politics|
|Political campaigns||Electoral Politics|
|Political career||Electoral Politics|
|Political careers||Electoral Politics|
|Political competition||Electoral Politics|
|Political cycle||Electoral Politics|
|Political preference||Electoral Politics|
|Political preferences||Electoral Politics|
|Political recruitment||Electoral Politics|
|Pre-election surveys||Electoral Politics|
|Presidential elections||Electoral Politics|
|Presidential elections 2010||Electoral Politics|
|Progressive ambition||Electoral Politics|
|Retrospective voting||Electoral Politics|
|Strategic voting||Electoral Politics|
|Televised presidential campaigns||Electoral Politics|
|Vote for federal and state deputies||Electoral Politics|
|Vote for governors||Electoral Politics|
|Vote for president||Electoral Politics|
|Vote for president in Brazil||Electoral Politics|
|Vote for the Senate||Electoral Politics|
|Voting behavior||Electoral Politics|
|Voting machine||Electoral Politics|
|Young voters||Electoral Politics|
|Brazilian National Congress||Legislative Politics|
|Brazilian Senators||Legislative Politics|
|Chamber of Deputies||Legislative Politics|
|Coalition government||Legislative Politics|
|Coalition presidentialism||Legislative Politics|
|Constituent Assembly||Legislative Politics|
|Executive Legislative relations||Legislative Politics|
|Executive-Legislative relations||Legislative Politics|
|Executive-Legislative relationship||Legislative Politics|
|Federal deputies||Legislative Politics|
|Federal deputy||Legislative Politics|
|Federal Senate||Legislative Politics|
|Formation of cabinets||Legislative Politics|
|Government coalitions||Legislative Politics|
|House of Representatives||Legislative Politics|
|Legislative Assemblies||Legislative Politics|
|Legislative behavior||Legislative Politics|
|Legislative branch||Legislative Politics|
|Legislative Consulting||Legislative Politics|
|Legislative debate||Legislative Politics|
|Legislative elections||Legislative Politics|
|Legislative elections 2010||Legislative Politics|
|Legislative forces||Legislative Politics|
|Legislative information||Legislative Politics|
|Legislative Power||Legislative Politics|
|Legislative procedure||Legislative Politics|
|Legislative process||Legislative Politics|
|Legislative production||Legislative Politics|
|Legislative recruitment||Legislative Politics|
|Legislators’ attitudes||Legislative Politics|
|National congress||Legislative Politics|
|National Congress (Brazil)||Legislative Politics|
|Parliamentary Arena||Legislative Politics|
|Parliamentary behavior||Legislative Politics|
|Parliamentary career||Legislative Politics|
|Parliamentary committee||Legislative Politics|
|Parliamentary elite||Legislative Politics|
|Party blocs||Legislative Politics|
|Party coalitions||Legislative Politics|
|Senator profiles||Legislative Politics|
|Senators’ profiles||Legislative Politics|
|State Assemblies||Legislative Politics|
|State Legislative Assemblies||Legislative Politics|
|State legislator||Legislative Politics|
|The Young Parliament||Legislative Politics|
|Brazilian party system||Political Parties|
|Brazilian Social Democracy Party||Political Parties|
|Cartel party||Political Parties|
|Catch-all Party||Political Parties|
|Media and partisanship||Political Parties|
|Partisan bases||Political Parties|
|Partisan image||Political Parties|
|Partisan migration||Political Parties|
|Party constitutions||Political Parties|
|Party democracy||Political Parties|
|Party feelings||Political Parties|
|Party Fund||Political Parties|
|Party funding||Political Parties|
|Party identification||Political Parties|
|Party nationalizations||Political Parties|
|Party organization||Political Parties|
|Party politics||Political Parties|
|Party politics and the environment||Political Parties|
|Party preference||Political Parties|
|Party programs||Political Parties|
|Party strategies||Political Parties|
|Party system||Political Parties|
|Party systems||Political Parties|
|Party unity||Political Parties|
|Political Parties||Political Parties|
|Political parties and the environment||Political Parties|
|Political party||Political Parties|
|Religious partnerships||Political Parties|
|Small parties||Political Parties|
|State party systems||Political Parties|
B – Paper Maturity Classification
|¿Por qué la Gente Vota a la Izquierda? Clivajes, Ideología y Voto Retrospectivo en Bolivia y Uruguay en Perspectiva Comparada||E|
|A Cor dos Eleitos: Determinantes da Sub-Representação Política dos Não-Brancos no Brasil||E|
|A Dimensão Geográfica das Eleições Brasileiras||E|
|A Lógica Social do Voto Correto no Brasil||E|
|À Margem das Margens? A Precisão das Pesquisas Pré-Eleitorais Brasileiras em 2010||E|
|A Study on the Impact of Campaign Finance, Political Capital and Gender on Electoral Performance||E|
|Análise da Não-Resposta em Surveys Políticos||E|
|Candidatos Corruptos São Punidos? Accountability na Eleição Brasileira de 2006||E|
|Classificando Ocupações Prévias à Entrada na Política: Uma Discussão Metodológica e um Teste Empírico||E|
|COBERTURA dA IMPRENSA e ELEIÇÕES PRESIDENCIAIS DE 2006 Efeitos Realmente Limitados?||E|
|Como os Brasileiros Escolhem os Governadores? Desvendando as Razões do Voto para os Executivos Estaduais no Brasil em 2014||E|
|Determinantes do Voto no Primeiro Turno das Eleições Presidenciais Brasileiras de 2010: Uma Análise Exploratória||E|
|Dinheiro, Tempo e Memória Eleitoral: Os Mecanismos que Levam ao Voto nas Eleições para Prefeito em 2012||E|
|Does the Electoral Rule Matter for Political Polarization? The Case of Brazilian Legislative Chambers||E|
|Economia, Ideologia e Eleições na América Latina||E|
|Elecciones 2009 en Uruguay: Permanencia de Lealtades Políticas y Accountability Electoral||E|
|Electoral Governance in Brazil||E|
|Eleição de Dilma ou Segunda Re-eleição de Lula? Uma Análise Espacial do Pleito de 2010||E|
|Eleições Municipais Comparadas: A Escolha do Chefe do Executivo no Brasil e no Uruguai e o Impacto Sobre os Sistemas Partidários Locais (2000-2005)||E|
|Ensino Superior e Voto: Análise do Comportamento Eleitoral da Comunidade Uspiana em 2010||E|
|Entrando na Arena? Gênero e a Decisão de Concorrer a um Cargo Eletivo||E|
|Estruturas de Poder nas Redes de Financiamento Político nas Eleições de 2010 no Brasil||E|
|Fazendo Eleitores e Eleições: Mobilização Política e Democracia no Brasil Pós-Estado Novo||E|
|Financiamento Eleitoral, Representação Política e Gênero: Uma Análise das Eleições de 2006||E|
|Grupos Sociais e Preferência Política O Voto Evangélico no Brasil||E|
|How Important is Twitter to Local Elections in Brazil? A Case Study of Fortaleza City Council||E|
|Ideologia e Comportamento Político no Eleitorado Brasileiro||E|
|Impact of Electronic Voting Machines on Blank Votes and Null Votes in Brazilian Elections in 1998||E|
|INCENTIVOS da DINÂMICA POLÍTICA SOBRE a CORRUPÇÃO Reeleição, Competitividade e Coalizões nos Municípios Brasileiros||E|
|Investigating Elite Behavior through Field Experiment in Brazil: Do Candidates Answer More to Core or Swing Voters?||E|
|O Ciclo de Política como Campo Estratégico: O Caso do Benefício de Prestação Continuada||E|
|O Federalismo como Experiência: Campos Sales e as Tentativas de Estabilização da República||E|
|Os Custos Eleitorais do Bolsa Família: Reavaliando seu Impacto Sobre a Eleição Presidencial de 2006||E|
|Os Determinantes da Avaliação da Economia na Eleição Presidencial Brasileira em 2014||E|
|Partidos, Competição Política e Fraude Eleitoral: A Tônica das Eleições na Primeira República||E|
|Poder Judiciário e Competição Política: As Eleições de 2010 e a Lei da “Ficha-Limpa”||E|
|Political Competition and the Diffusion of Conditional Cash Transfers in Brazil||E|
|Propensão ao Risco e Ambição Progressiva: O Caso dos Deputados Federais nos Estados Unidos||E|
|PT no Purgatório: Ambivalência Eleitoral no Primeiro Turno das Eleições Presidenciais de 2010||E|
|The Capital Mistake: Local Information and National Electoral Reforms||E|
|The Impacts of Conditional Cash Transfers in Four Presidential Elections (2002–2014)||E|
|TRANSFERÊNCIA de RENDA e ELEIÇÕES A Trajetória Latino-Americana||E|
|Vale a Pena Ser um Bom Prefeito? Comportamento Eleitoral e Re-eleição no Brasil||E|
|Vencedores e Perdedores nas Eleições Presidenciais de 2014: O Efeito da Derrota nas Urnas sobre a Satisfação e o Apoio em Relação à Democracia no Brasil||E|
|Capital Familiar e Carreira Política no Brasil: Gênero, Partido e Região nas Trajetórias para a Câmara dos Deputados||E/L|
|Corporate Dependence in Brazil’s 2010 Elections for Federal Deputy||E/L|
|Determinantes dos Padrões de Carreira Política dos Deputados Federais Paulistas entre as Legislaturas 49ª (1991-1995) e 53ª (2007-2011)||E/L|
|Eleições Presidenciais, Federalismo e Política Social||E/L|
|Estrutura da Competição pela Presidência e Consolidação do Sistema Partidário no Brasil||E/L|
|Federalismo, Coalizões de Governo e Escolhas de Carreira dos Deputados Federais||E/L|
|O Custo da Política Subnacional: A Forma como o Dinheiro é Gasto Importa? Relação entre Receita, Despesas e Sucesso Eleitoral||E/L|
|O Desempenho das Mulheres nas Eleições Legislativas de 2010 no Rio Grande do Sul||E/L|
|Os Senadores-Empresários: Recrutamento, Carreira e Partidos Políticos dos Empresários no Senado Brasileiro (1986-2010)||E/L|
|Perfil Profissional e Distribuição Regional dos Senadores Brasileiros em Dois Séculos de História||E/L|
|Uma Avaliação Empírica da Competição Eleitoral para a Câmara Federal no Brasil||E/L|
|Uso Estratégico de Eleições Alternadas? Efeitos da Candidatura para Prefeito sobre a Votação dos Concorrentes ao Cargo de Deputado Federal no Brasil||E/L|
|Brazilian Congress, 2014 Elections and Governability Challenges||E/L/P|
|Migração Partidária na Câmara dos Deputados de 1987 a 2009||E/L/P|
|Nacionalização Partidária e Estratégias Eleitorais no Presidencialismo de Coalizão||E/L/P|
|A Concentração Eleitoral no Brasil (1994-2014)||E/P|
|A Estabilidade e a Efetividade da Preferência Partidária no Brasil||E/P|
|Articulações Intrapartidárias e Desempenho Eleitoral no Brasil||E/P|
|As Lógicas do Recrutamento Político||E/P|
|Competição Partidária e Voto nas Eleições Presidenciais no Brasil||E/P|
|Comportamento Eleitoral e Estratégia Partidária nas Eleições Presidenciais no Brasil (2002 – 2010)||E/P|
|Contexto e Competição na Política Paulista||E/P|
|Da Polícia à Política: Explicando o Perfil dos Candidatos das Forças Repressivas de Estado à Câmara dos Deputados||E/P|
|Diversidade Sexual, Partidos Políticos e Eleições no Brasil Contemporâneo||E/P|
|Does Size Matter? Electoral Performance of Small Parties in Brazil||E/P|
|Identificação Partidária e Voto. As Diferenças entre Petistas e Peessedebistas||E/P|
|Mercado Futuro: A Economia Política da (Re)Partidarização da Imprensa no Brasil||E/P|
|Nas Brumas do HGPE: A Imagem Partidária nas Campanhas Presidenciais Brasileiras (1989 a 2010)||E/P|
|Reforma Política no Brasil: Indagações sobre o Impacto no Sistema Partidário e na Representação||E/P|
|Sentimentos Partidários e Antipetismo: Condicionantes e Covariantes||E/P|
|Socialismo Moreno, Conservadorismo Pálido? Cor e Recrutamento Partidário em São Paulo e Rio de Janeiro nas Eleições de 2012||E/P|
|The 2014 Elections and the Brazilian Party System||E/P|
|Topografia do Brasil Profundo: Votos, Cargos e Alinhamentos nos Municípios Brasileiros||E/P|
|VOTO ESTRATÉGICO e COORDENAÇÃO ELEITORAL Testando a Lei de Duverger no Brasil||E/P|
|As Organizações Sociais de Saúde e o Desempenho do SUS: Um Estudo sobre a Atenção Básica em São Paulo||L|
|Bicameralismo e Poder Executivo no Brasil: Revisão de Projetos Presidenciais entre 1989-2010||L|
|Comissões Permanentes, Estrutura de Assessoramento e o Problema Informacional na Câmara dos Deputados do Brasil||L|
|Cooperação, Liderança e Impasse entre o Legislativo e o Executivo na Produção Legislativa do Congresso Nacional do Brasil||L|
|Crime e Congresso Nacional: Uma Análise da Política Criminal Aprovada de 1989 a 2006||L|
|Deputados Brasileiros no Twitter: Um Estudo Quantitativo dos Padrões de Adoção e Uso da Ferramenta||L|
|Determinantes do Controle Horizontal em Parlamentos Reativos: O Caso do Brasil (1988-2005)||L|
|Facciosismo, Rachas Intrapartidários e Capacidade Decisória na Democracia Brasileira de 1946||L|
|Financiamento de Campanha e Apoio Parlamentar à Agenda Legislativa da Indústria na Câmara dos Deputados||L|
|Governo e Oposição no Senado Brasileiro (1989-2010)||L|
|Governos Minoritários no Presidencialismo Latino-Americano: Determinantes Institucionais e Políticos||L|
|Impacto da Educação Cívica sobre o Conhecimento Político: A Experiência do Programa Parlamento Jovem de Minas Gerais||L|
|Influência Majoritária em Eleições Proporcionais: Os Efeitos Presidenciais e Governatoriais sobre as Eleições para a Câmara dos Deputados Brasileira (1994-2010)||L|
|O Efeito das Emendas ao Orçamento no Comportamento Parlamentar e a Dimensão Temporal: Velhas Teses, Novos Testes||L|
|O Poder da Caneta: A Medida Provisória no Processo Legislativo Estadual||L|
|O Poder no Executivo: Uma Análise do Papel da Presidência e dos Ministérios no Presidencialismo de Coalizão Brasileiro (1995-2010)||L|
|Os Sem-Voto do Legislativo Brasileiro: Quem São os Senadores Suplentes e Quais os Seus Impactos sobre o Processo Legislativo||L|
|Oversized Government Coalitions in Latin America||L|
|Parlamentares, Representação Política e Redes Sociais Digitais: Perfis de Uso do Twitter na Câmara dos Deputados||L|
|Partidos Legislativos e Governo de Coalizão: Controle Horizontal das Políticas Públicas||L|
|Políticas Públicas e a Descentralização Legislativa da Assembleia Constituinte Brasileira, 1987-1988||L|
|The Institutional Presidency from a Comparative Perspective: Argentina and Brazil since the 1980s||L|
|The Legislative Work in an Authoritarian Regime: the Case of the São Paulo Administrative Department||L|
|Unboxing the Active Role of the Legislative Power in Brazil||L|
|Variedades de Governos de Coalizão no Presidencialismo. Chile, 1990-2010||L|
|What is the Best Strategy to Obtain Legislative Support? Survey Evidence from Brazilian State Assemblies||L|
|Avaliando Vínculos entre Partidos e Deputados nas Assembleias Legislativas Brasileiras||L/P|
|Senado Brasileiro: Casa Federativa ou Partidária?||L/P|
|¿Son los Chilenos Igualmente Solidarios?La Influencia de los Recursos Personales en las Donaciones de Dinero||Other|
|A Afeição dos Cidadãos pelos Políticos Mal-Afamados: Identificando os Perfis Associados à Aceitação do ‘Rouba, mas Faz’ no Brasil||Other|
|A Comparative Analysis of Brazil’s Foreign Policy Drivers Towards the USA: Comment on Amorim Neto (2011)||Other|
|A Conciliação e os Meios Alternativos de Solução de Conflitos no Império Brasileiro||Other|
|A Construção do Campo de Internet e Política: Análise dos Artigos Brasileiros Apresentados entre 2000 e 2014||Other|
|A Democracia Disjuntiva no Contexto Brasileiro: Algumas Considerações a Partir do Trabalho das Delegacias de Polícia||Other|
|A Economia Solidária e a Revitalização do Paradigma Cooperativo||Other|
|A Identificação de Enquadramentos Através da Análise de Correspondências: Um Modelo Analítico Aplicado à Controvérsia das Ações Afirmativas Raciais na Imprensa||Other|
|A Macrodinâmica Social Brasileira: Mudanças, Continuidades e Desafios||Other|
|A Política de Segurança Pública no Brasil: Uma Análise dos Gastos Estaduais (1999-2010)||Other|
|A Psicologia Política das Classes Sociais no Brasil: Atributos das Atitudes Políticas por Estratificação e Mobilidade Social||Other|
|A Reconfiguração do Sindicalismo Brasileiro nos Anos 2000: As Bases Sociais e o Perfil Político-Ideológico da Conlutas||Other|
|A Redução da Maioridade Penal Diminui a Violência? Evidências de um Estudo Comparado||Other|
|A Segregação dos Jovens em Paris||Other|
|Acesso Individual e Coletivo de Moradores de Favelas à Justiça||Other|
|Ações Afirmativas no Vestibular da UFPR entre 2005 e 2012: De Política Afirmativa Racial a Política Afirmativa de Gênero||Other|
|Administrando o Debate Público: O Globo e a Controvérsia em Torno das Cotas Raciais||Other|
|Aferindo a Capacidade de Influência das Conferências de Políticas Públicas sobre os Programas das Respectivas Políticas Setoriais||Other|
|Análise Espacial da Burocracia da Assistência Social nos Municípios Brasileiros: Notas para uma Reflexão sobre a Capacidade de Implementação dessa Política||Other|
|Associativismo e Representação Política Feminina no Brasil||Other|
|Atitudes, Cognição e Participação Política: Padrões de Influência dos Ambientes de Socialização sobre o Perfil Político dos Jovens||Other|
|Avaliação de Impacto das Condicionalidades de Educação do Programa Bolsa Família (2005 e 2009)||Other|
|Beyond Brazilian Coalition Presidentialism: The Appropriation of the Legislative Agenda||Other|
|Boa Noite, e Boa Sorte: Determinantes da Demissão de Ministros Envolvidos em Escândalos de Corrupção no Primeiro Governo Dilma Rousseff||Other|
|‘Break-In Parties’ and Changing Patterns of Democracy in Latin America||Other|
|Cadenas Globales de Valor y Desarrollo: Perspectivas Críticas desde el Sur Global||Other|
|Capital Social e Políticas Públicas na Região Metropolitana de Porto Alegre: Comparando Novo Hamburgo e Estância Velha||Other|
|Capital Social y Participación, Una Perspectiva desde el Cono Sur de América: Porto Alegre, Montevideo y Santiago de Chile||Other|
|Cartografia do Pensamento Político Brasileiro: Conceito, História, Abordagens||Other|
|Classe Social e Deslocamentos de Renda no Brasil||Other|
|Classes e Práticas Sociais||Other|
|Classificando Regimes Políticos Utilizando Análise de Conglomerados||Other|
|Conciliação e Tensões entre Trabalho e Família para Mulheres Titulares do Programa Bolsa Família||Other|
|Confiança na Polícia em Minas Gerais: O Efeito da Percepção de Eficiência e do Contato Individual||Other|
|Confiança nas Forças Armadas Brasileiras: Uma Análise Empírica a Partir dos Dados da Pesquisa Sips – Defesa Nacional||Other|
|Construindo a América Do Sul: Identidades e Interesses na Formação Discursiva da UNASUL||Other|
|Cooperação e Coordenação na Implementação de Políticas Públicas: O Caso da Saúde||Other|
|Cooperation for Development, Brazilian Regional Leadership and Global Protagonism||Other|
|Coordenação Federativa e a “Armadilha da Decisão Conjunta”: As Comissões de Articulação Intergovernamental das Políticas Sociais no Brasil||Other|
|Corrupção e Participação Política no Brasil: Diagnósticos e Consequências||Other|
|Corruption and Political Participation in the Americas and the Caribbean||Other|
|Cultura Política, Mudanças Econômicas e Democracia Inercial. Uma Análise Pós-Eleições de 2014||Other|
|Democracy, Redistribution, and Equality||Other|
|Desempenho Econômico do Rio de Janeiro: Trajetórias Passadas e Perspectivas Futuras||Other|
|Desenho Institucional e Promoção da Justiça em Espaços Participativos: Implicações em um Estudo de Caso||Other|
|Desigualdade de Rendimentos do Trabalho no Curto e no Longo Prazo: Tendências de Idade, Período e Coorte||Other|
|Desigualdades de Gênero na Carreira Acadêmica No Brasil||Other|
|Diferenciação Institucional e Desigualdades no Ensino Superior||Other|
|Educação e Rendimentos dos Ricos no Brasil||Other|
|Empirical Determinants of Government Efficiency: A Study Based on Objective Indicators||Other|
|Engajamento Cívico e Capital Social: Um Modelo Interativo para o Efeito da Confiança Interpessoal||Other|
|Entre cooperação e Centralização Federalismo e Políticas Sociais no Brasil Pós-1988||Other|
|Entre o Altruísmo e o Familismo: A Agenda Parlamentar Feminina e as Políticas Família-Trabalho (Brasil, 2003-2013)||Other|
|Equidade na Educação Superior no Brasil: Uma Análise Multinomial das Políticas Públicas de Acesso||Other|
|Escola Pública e Segmentos Populares em um Contexto de Construção Institucional da Democracia||Other|
|Estado e Desigualdade de Renda no Brasil Fluxos de Rendimentos e Estratificação social||Other|
|Estrutura de Classes, Educação e Queda da Desigualdade de Renda (2002-2011)||Other|
|Estrutura Social e Segregação em São Paulo: Transformações na Década de 2000||Other|
|Explicando os Efeitos de Programas de Socialização Política: A Experiência do Parlamento Jovem no Brasil||Other|
|Fatores Familiares e Desempenho Escolar: Uma Abordagem Multidimensional||Other|
|Federalismo e Bons Governos: Uma Análise Política da Gestão Fiscal dos Municípios||Other|
|Flying in Clear Skies: Technical Arguments Influencing ANAC Regulations||Other|
|Foreign Aid and the Governance of International Financial Organizations: The Brazilian-Bloc Case in the IMF and the World Bank||Other|
|Framing Effects on Foreign Policy: Experimental Evidence from Emerging Countries and the Argentine- Brazilian Rivalry||Other|
|Gênero e Representação Política nos Conselhos Gestores no Brasil||Other|
|Institucionalização Partidária e Consistência Programática: A Resposta da Esquerda a Ataques Especulativos na América Latina||Other|
|Integración Sociopolítica y Bienestar Subjetivo en México||Other|
|Internet and Politics Studies in Brazil: Mapping the Characteristics and Disparities of the Research Field||Other|
|Juventude por Cor e Renda no Acesso ao Ensino Superior Somando Desvantagens, Multiplicando Desigualdades?||Other|
|Juventude Violenta: Processos, Retrocessos e Novos Percursos||Other|
|Liberalism, Marxism and Democratic Theory Revisited: Proposal of a Joint Index of Political and Economic Democracy||Other|
|Manda Quem Pode: A Elite Governante da Cultura em Portugal (1976-2011)||Other|
|Mapping and Explaining the Use of the Left-Right Divide||Other|
|Mecanismos Institucionais de Regulação Federal e seus Resultados nas Políticas de Educação e Saúde||Other|
|Medindo o Acesso à Justiça Cível no Brasil||Other|
|Mensurando Alteridades, Estabelecendo Direitos: Práticas e Saberes Governamentais na Criação de Fronteiras Étnicas||Other|
|Mercantilização no Feminino: A Visibilidade do Trabalho das Mulheres no Brasil||Other|
|Metamorfoses da Questão Geracional: O Problema da Incorporação dos Jovens na Dinâmica Social||Other|
|Migrações e Trabalho no Brasil Fatores Étnico-Nacionais e Raciais||Other|
|Mobilização do Direito como Repertório de Ação Coletiva e Crítica Institucional no Campo Ambiental Brasileiro||Other|
|Modelando a Participação Social: Uma Análise da Propensão à Inserção em Instituições Participativas, a Partir de Características Socioeconômicas e Políticas||Other|
|Mortes Prematuras e Conflito Armado pelo Domínio das Favelas no Rio de Janeiro||Other|
|Mudança de Fronteiras Étnicas e Participação Política de Descendentes de Imigrantes em São Paulo||Other|
|Mudanças e Continuidades no Movimento de Direitos Humanos: Padrões Organizacionais, Relacionais e Discursivos||Other|
|O Que Fazem os Conselhos e Quando o Fazem? Padrões Decisórios e o Debate dos Efeitos das Instituições Participativas||Other|
|O Que Pensa Quem “Bate à Porta” de uma Casa Que Só “Apanha”?: Percepções e Orientações dos Visitantes sobre o Congresso Nacional||Other|
|O Rio de Janeiro e o Estado Nacional (1946-2010)||Other|
|O Sistema Oligárquico Representativo da Primeira República||Other|
|O Topo da Distribuição de Renda no Brasil: Primeiras Estimativas com Dados Tributários e Comparação com Pesquisas Domiciliares (2006-2012)||Other|
|Omissões e Seletividades da Ciência Política Brasileira: Lacunas Temáticas e seus Problemas Sócio- Epistêmicos||Other|
|Os Alunos do Ensino Médio e Sciences Po: Entre a Meritocracia e a Percepção das Desigualdades||Other|
|Os Impactos da Geração de Empregos sobre as Desigualdades de Renda: Uma Análise da Década de 2000||Other|
|Para Além da Renda: Uma Análise da Pobreza Multidimensional no Brasil||Other|
|Parcerias entre a Escola e a Comunidade em Portugal: Uma Análise a partir da Avaliação Externa das Escolas||Other|
|Perceptions on Justice, the Judiciary and Democracy||Other|
|Personality, Political Attitudes and Participation in Protests: The Direct and Mediated Effects of Psychological Factors on Political Activism||Other|
|Pesquisa e Produção de Conhecimento Sobre a América Latina na Ciência Política Brasileira||Other|
|Políticas de Ação Afirmativa e o Experimento de Listas: O Caso das Cotas Raciais na Universidade Brasileira||Other|
|Pós-Graduação em Relações Internacionais no Brasil: Anotações sobre sua Institucionalização||Other|
|Posição Socioeconômica, Idade e Condição de Saúde no Brasil||Other|
|Preto no Branco? Mensuração, Relevância e Concordância Classificatória no País da Incerteza Racial||Other|
|Privação Relativa e Ativismo em Protestos no Brasil: Uma Investigação sobre o Horizonte do Possível||Other|
|Proposta de um Indicador Socioeconômico para os Alunos das Escolas Públicas dos Municípios Brasileiros||Other|
|Protesto Político na América Latina: Tendências Recentes e Determinantes Individuais||Other|
|Qualificando a Adesão à Democracia: Quão Democráticos são os Democratas Brasileiros?||Other|
|Quatro Décadas de Mobilidade Social no Brasil||Other|
|Quem é Classe Média no Brasil? Um Estudo sobre Identidades de Classe||Other|
|Racionalidade Online: Provimento de Razões em Discussões Virtuais||Other|
|Renda, Relações Sociais e Felicidade no Brasil||Other|
|Saúde, Religião e Trânsito Religioso: Estudo Pró-Saúde||Other|
|Sobre a “Crise” do Estado de Bem-Estar: Retração, Transformação Fáustica ou o Quê?||Other|
|Sofisticação Política e Opinião Pública no Brasil: Revisitando Hipóteses Clássicas||Other|
|Solidariedade e Expressão Jurídica: Valores Políticos de Vereadores sobre Direitos Sociais||Other|
|State Transfers, Taxes and Income Inequality in Brazil||Other|
|SUPREMO Relator: Processo Decisório e Mudanças na Composição Do STF nos Governos FHC e Lula||Other|
|Tempo na ou da Justiça Criminal Brasileira: Uma Discussão Metodológica||Other|
|Território e Desigualdades de Renda em Regiões Metropolitanas do Brasil||Other|
|The Bigger, the Better: Coalitions in the GATT/WTO||Other|
|The Elusive New Middle Class in Brazil||Other|
|The Stratification of Diversity: Measuring the Hierarchy of Brazilian Political Science||Other|
|The Three Achilles’ Heels of Brazilian Political Science||Other|
|Transições Ocupacionais e Mercados de Trabalho Intermetropolitanos: Os Casos de Rio de Janeiro e São Paulo||Other|
|Trazendo de Volta a Mensuração: Fundamentos Metodológicos do Índice de Democracia Eleitoral1||Other|
|Trust and Political Information: Attitudinal Change in Participants in the Youth Parliament in Brazil||Other|
|Uma Nova Abordagem Empírica para a Hierarquia de Status no Brasil||Other|
|Votando na Prateleira: A Politização do Consumo na América Latina||Other|
|As Transformações nas Formas de Militância no Interior do PT: Maior Inclusão e Menor Intensidade||P|
|O Eleitor Antipetista: Partidarismo e Avaliação Retrospectiva||P|
|O Perfil do Financiamento dos Partidos Brasileiros (2006-2012): O Que as Tipologias Dizem?||P|
|O Sistema Partidário Brasileiro: Um Debate Com a Literatura Recente||P|
|Política Estadual e Desigualdade: Por Que alguns Estados Redistribuem Mais do Que Outros?||P|
|Profissionalização Política, Processo Seletivo e Recursos Partidários: Uma Análise da Percepção dos candidatos do PT, PMDB, PSDB e DEM nas Eleições para Deputado Federal de 2010||P|
|Unidade Partidária e Política Externa na América Latina||P|
C – Pre-Analysis Plan
Publication in this collection
14 May 2021
Date of issue
23 Jan 2020
5 July 2020
20 July 2020