Transparency and Replication in Brazilian Political Science: A First Look* * Hypotheses and data analysis plan were pre-registered at the Open Science Foundation except as noted. Analysis of existing datasets was exempted by the UCSD Institutional Review Board (IRB). The authors thank the Berkeley Institute for Transparency in the Social Sciences for funding and support. George Avelino and Ivan Mardegan acknowledge the support from São Paulo Research Foundation FAPESP (Grants #: 2013/156158-1 and 2017/20855-1). The authors also thanks Raduan Meira for research assistance.

Transparência e Replicação na Ciência Política Brasileira: Um Primeiro Olhar

Transparencia y Replicación en la Ciencia Política Brasileña: Una Primera Mirada

Transparence et Réplication dans la Science Politique Brésilienne: Premier Regard

George Avelino Scott Desposato Ivan Mardegan About the authors

ABSTRACT

We provide the first replication study of political science research published in Brazil by attempting to replicate every quantitative article published in five major Brazilian journals between 2012 and 2016. We also tested whether replication rates varied between established fields, more traditional and where the use of quantitative data is more common, and emerging fields. Our results show that transparency and reproduction are still in a development stage in Brazilian Political Science. Of the 650 articles reviewed, we asked for data to 197 quantitative articles. From those, only 28% agreed to share datasets and computed codes. We were able to attempt a replication for only 14%, and successfully reproduce the results of less than 5%. We conclude by suggesting the adoption of transparency and replicability procedures that are standard in other scientific communities.

replication; transparency; reproduction; Brazilian Political Science; data sharing

RESUMO

Este é o primeiro exercício de replicação de pesquisas em ciência política publicado no Brasil. O objetivo foi replicar todos os artigos quantitativos publicados em cinco grandes revistas brasileiras entre 2012 e 2016. Também foram testadas as diferenças potenciais nas taxas de replicação entre campos de pesquisa estabelecidos, mais tradicionais e onde o uso de dados quantitativos é mais frequente, e emergentes. Solicitamos os dados para 197 artigos quantitativos. Desses 197, somente 28% dos autores concordaram em compartilhar os dados e os códigos. Foi possível tentar replicar apenas 14%, com 5% de sucesso. A conclusão sugere a adoção de procedimentos de transparência e replicabilidade já introduzidos em outras comunidades científicas.

replicação; transparência; reprodução; Ciência Política Brasileira; compartilhamento de dados

RESUMEN

Este es el primer ejercicio de replicación de la investigación en ciencia política publicado en Brasil. El objetivo fue replicar todos los artículos cuantitativos publicados en cinco importantes revistas brasileñas entre 2012 y 2016. También se probaron las posibles diferencias en las tasas de replicación entre los campos de investigación establecidos, más tradicionales y donde el uso de datos cuantitativos es más frecuente, y los emergentes. Solicitamos datos para 197 artículos cuantitativos. De estos 197, solo el 28% de los autores acordó compartir datos y códigos. Fue posible intentar replicar solo el 14%, con un 5% de éxito. La conclusión sugiere la adopción de procedimientos de transparencia y replicabilidad ya introducidos en otras comunidades científicas.

replicación; transparencia; reproducción; Ciencias Políticas Brasileñas; compartir datos

RÉSUMÉ

Il s’agit du premier exercice de réplication de la recherche en science politique publiée au Brésil. L’objectif était de reproduire tous les articles quantitatifs publiés dans cinq grandes revues brésiliennes entre 2012 et 2016. Les différences potentielles de taux de réplication entre les domaines de recherche établis et plus traditionnels et où l’utilisation de données quantitatives est plus fréquente et les domaines émergents ont également été testés. Nous avons demandé des données pour 197 articles quantitatifs. Sur ces 197, seuls 28% des auteurs ont accepté de partager des données et des codes. Il était possible d’essayer de reproduire seulement 14%, avec 5% de succès. La conclusion suggère l’adoption de procédures de transparence et de reproductibilité déjà introduites dans d’autres communautés scientifiques.

réplication; transparence; la reproduction; Science politique brésilienne; partage de données

INTRODUCTION

Scientific disciplines are facing a credibility crisis. In social sciences, significant published findings often cannot be replicated (Goodman, Fanelli and Ioannidis, 2016; Key, 2016KEY, Ellen M. (2016), “How are we doing? data access and replication in political science”. PS: Political Science and Politics, vol. 49, no 02, pp. 268-272.; Christensen et al., 2019CHRISTENSEN, Garret; FREESE, Jeremy; MIGUEL, Edward (2019), Transparent and reproducible social science research: how to do open science. Oakland, CA: University of California Press.). High-profile publications have been retracted and, in some cases, identified as fraudulent. Some scholars have speculated that at least half of all published research is simply wrong (Ionnadis, 2005IONNADIS, John. (2005), “Why most published research findings are false”. PLOS Medicine. Available at: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124.
https://journals.plos.org/plosmedicine/a...
). The reproduction of other scholars’ work has led to several high-profile reviews and at least one significant retraction of a fraudulent result1 1 . See Lacour and Green fraud case in https://www.chronicle.com/article/What-Social-Science-Can-Learn/230645 . All these (and other) problems suggest that much research is a waste of resources and human capital, degrade trust in science, and have led some to question public funding for research projects (Christensen and Miguel, 2018CHRISTENSEN, Garret; MIGUEL, Edward (2018), “Transparency, reproducibility, and the credibility of economics research”. Journal of Economic Literature, vol. 56, no 3, pp. 920-980.).

Yet, as Figueiredo Filho et al. (2019) pointed out, there is no consensus in the literature about the meaning of transparency, reproducibility, and replication, which leads these concepts to be treated as interchangeable. By and large, one assesses transparency by the research material available to the academic community. Reproducibility is determined by the capacity of any scholar to reproduce the same results from the research material made available. Finally, replication may have two different meanings. The first, used in this article, is a synonym of reproducibility. The second, according to King (1995)KING, Gary. (1995), “Replication, Replication”. PS: Political Science and Politics, vol. 28, no 03, pp. 444-452., is the attempt to further knowledge by introducing slight variations in a reproducible paper, e.g., by adding variables to the original model, testing the same hypothesis in different contexts, etc.

In response, scholars have investigated the causes of this credibility crisis and suggested several solutions. The causes include institutional mechanisms that encourage “fishing” or p-hacking2 2 . For more information about p-hacking concept see Head et al. (2015) and Chordia, Goyal and Saretto (2017). (Humphreys, De La Sierra and Van Der Windt, 2013) and lead to selection bias in publication choice (Gerber, Green and Nickerson, 2001). For example, journals are more likely to publish “significant” findings than null results (Ferguson and Heene, 2012FERGUSON, Christopher J.; HEENE, Moritz. (2012), “A vast graveyard of undead theories: publication bias and psychological science’s aversion to the null”. Perspectives on Psychological Science, vol. 07, no 06, pp. 555-561.). Knowing this, scholars fail to submit null results for publication. They may also hunt through the dataset for significant results. All these mechanisms push relevant null results aside, encourage poor scientific practices, and lead to a high rate of false positives3 3 . A false positive result consists in rejecting the null hypothesis when it was not the case of rejecting it. The opposite case is known as false negative, when the null hypothesis is not rejected when it should be. in published studies.

Addressing all these problems will require significant changes in scientific institutions. Scholars have proposed several reforms and best practices that have the potential to deal with these issues. These practices might include open access to academic production, data and research materials, computer codes (preferably in open source formats), open peer review, use of pre-registration and data analysis plans, reviewing research designs instead of results (Cruwell et al., 2019CRUWELL, Sophia et al. (2019), “Seven easy steps to open science: an annotated reading list”. Zeitschrift für Psychologie, vol. 227, no 4, pp. 237-248.; Martins, 2020MARTINS, Henrique Castro. (2020), “A Importância da Ciência Aberta (Open Science) na Pesquisa em Administração”. Rev. Adm. Contemp, vol. 24 no 01.), and others promoted by organizations and initiatives, such as the Iniciativa Brasileira de Reprodutibilidade (https://www.reprodutibilidade.bio.br/), the Project TIER (https://www.projecttier.org/), the Berkeley Institute for Transparency in the Social Sciences (https://www.bitss.org/) and the Open Science Foundation (https://osf.io/).

Research transparency is one of these new practices. By research transparency we mean providing all the resources needed for other scholars to replicate one’s findings, including source materials, computer code, and other relevant resources (King, Keohane and Verba, 1994; Figueiredo Filho et al., 2019; Bakken, 2019BAKKEN, Suzanne. (2019), “The journey to transparency, reproducibility, and replicability”. Journal of the American Medical Informatics Association, vol. 26, Issue 3, pp. 185-187.). Transparency permits studies’ replication and reproducibility, which are key factors for research’s credibility (Janz, 2016JANZ, Nicole. (2016), “Bringing the gold standard into the classroom: replication in university teaching”. International Studies Perspectives, vol. 17, issue 4, pp. 392-407.). In the same way that transparency is a condition for research reproduction, reproducibility is the ability to reproduce the same results of a study using the same data and methodology (NSF, 2015NSF (National Science Foundation). (2015), Reproducibility, Replicability, and Generalization in the Social, Behavioral, and Economic Sciences. Available at <https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_ Report_on_Replicability_SubcommitteeMay_2015.pdf>.
https://www.nsf.gov/sbe/SBE_Spring_2015_...
)4 4 . Though Hamermesh (2007) separates simple from scientific replication, designating the last term to the expansion of scientific knowledge, we are more concerned with the first term, meaning transparency and reproducibility. Therefore, throughout this paper, we use replication and reproducibility interchangeably. .

Adopting transparency and reproducibility procedures would diminish the probability of mistakes (Simonsohn, 2013SIMONSOHN, Uri. (2013), “Just post it: the lesson from two cases of fabricated data detected by statistics alone”. Psychological Science, vol. 24, no 10, pp. 1875-1888.; King, 1995KING, Gary. (1995), “Replication, Replication”. PS: Political Science and Politics, vol. 28, no 03, pp. 444-452.), increase both the citations rate (Piwowar, Day and Fridsma, 2007) and the trustfulness, efficiency and cumulative advance of academic work (King, 1995KING, Gary. (1995), “Replication, Replication”. PS: Political Science and Politics, vol. 28, no 03, pp. 444-452.; King, 2006KING, Gary. (2006), “Publication, Publication”. PS: Political Science and Politics, vol. 39, no 01, pp. 119-125.; Ball and Medeiros 2012BALL, Richard; MEDEIROS, Norm. (2012), “Teaching integrity in empirical research: a protocol for documenting data management and analysis”. The Journal of Economic Education, vol. 43, no 02, pp. 182-189., Finifter 1975FINIFTER, Bernard M. (1975), “Replication and extension of social research through secondary analysis”. Social Information, vol. 14, no 02, pp. 119-153.; Goodman et al., 2015; Ebersol et al., 2016; Markowetz, 2015MARKOWETZ, Florian. (2015), “Five selfish reasons to work reproducibly”. Genome Biology, vol. 16, no 01, p. 274.; Elman, Kapiszewski and Lupia, 2018; Christensen and Miguel, 2018CHRISTENSEN, Garret; MIGUEL, Edward (2018), “Transparency, reproducibility, and the credibility of economics research”. Journal of Economic Literature, vol. 56, no 3, pp. 920-980.; Gleditsch, 2020GLEDITSCH, Kristian S. (2020), Houston, we have a problem: enhancing academic freedom and transparency in publishing through post-publication debate”. Political Studies Review. DOI: 10.1177/1478929919889309.). The transparency and reproducibility culture are starting to emerge in some important fields, such as business and management sciences (Martins, 2020MARTINS, Henrique Castro. (2020), “A Importância da Ciência Aberta (Open Science) na Pesquisa em Administração”. Rev. Adm. Contemp, vol. 24 no 01.), psychological sciences (Gilbert et al., 2016GILBERT, Daniel T. et al. (2016), “Comment on Estimating the Reproducibility of Psychological Science”. Science, vol. 351, no 6277, p. 1037.), biomedical sciences (Bakken, 2019BAKKEN, Suzanne. (2019), “The journey to transparency, reproducibility, and replicability”. Journal of the American Medical Informatics Association, vol. 26, Issue 3, pp. 185-187.) and social sciences (Christensen, Freese and Miguel, 2019), as well as in great part of developed countries’ scientific communities (Degterev, 2020DEGTEREV, Denis A. (2020), Replicability of Research on International Relations As a Global Trend. Herald of the Russian Academy of Sciences, vol. 90, pp. 36-44.). Some journals already require the release of all replication materials, including data and computer code, as a condition of publication5 5 . More than 1,100 journals have implemented at least one of the Transparency and Openness Promotion Guidelines, according to the Center of Open Science website. Some of the journals are American Journal of Political Science, European Political Science, Italian Political Science Review, Quarterly Journal of Political Science, State Politics and Policy Quarterly, Public Administration Review, Political Science Research and Methods, Political Behavior, Law and Human Behavior, Journal of Economic Behavior & Organization, Political Science Research and Methods. The complete list can be accessed in https://www.cos.io/our-services/top-guidelines? . These requirements have significantly increased the replicability of research in those countries. However, little is known about replicability outside the United States and Western Europe.

Even with all this evidence showing that transparency helps to ensure academic research trustfulness, scientists are, in general, still suspicious about sharing their data with the community (Janz and Freese, 2019JANZ, Nicole; FREESE, Jeremy. (2019), “Good and bad replications in political science: how replicators and original authors (should) talk to each other”. Available in https://www.mzes.uni-mannheim.de/openscience/wp-content/uploads/2019/01/Janz-Freese_-Good-and-Bad-Replications-1.pdf
https://www.mzes.uni-mannheim.de/opensci...
) or face many difficulties to guarantee free and usable access to them. The lack of sufficient resources (including technological frames and abilities, but also time and money) to make research data public available is one of the most cited reasons for scientists not to share their data (Tenopir et al., 2011TENOPIR, Carol et al. (2011), “Data sharing by scientists: practices and perceptions”. PloS One, vol. 6, no 6, p. e21101.). Besides, there is an important concern about the mistaken use of the shared data, specially as far as plagiarism is concerned (Huang et al., 2012HUANG, Xiaolei et al. (2012), “Willing or unwilling to share primary biodiversity data: results and implications of an international survey”. Conservation Letters, vol. 05, no 05, pp. 399-406.). These reasons help us to understand why the rate of data sharing continues low, evidenced by empirical studies in different scientific fields, like less than 6% in behavioral sciences (Hildebrandt and Prenoveau, 2020HILDEBRANDT, Tom; PRENOVEAU, Jason M. (2020), “Rigor and reproducibility for data analysis and design in the behavioral sciences”. Behavior Research and Therapy, vol. 126. doi:10.1016/j.brat.2020.103552), 10-13% in medical sciences (Savage and Vickers, 2009SAVAGE, Caroline J.; VICKERS, Andrew J. (2009), “Empirical study of data sharing by authors publishing in Plos Journals”. PloS One, vol. 4, no 9, p. e7078.; Thelwall et al., 2020THELWALL, Mike et al. (2020), “Is useful research data usually shared? an investigation of genome-wide association study summary statistics”. Plos One, vol. 15, no 2, p. e0229578.), 14% in psychological sciences (Hardwicke et al., 2020HARDWICKE, Tom E. et al. (2020), Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014-2017). Available in https://osf.io/preprints/metaarxiv/9sz2y/
https://osf.io/preprints/metaarxiv/9sz2y...
.), 36% in ecology and evolutionary sciences (Roche et al., 2015ROCHE, Dominique G. et al. (2015), “Public data archiving in ecology and evolution: how well are we doing?”. PLoS Biol, vol. 13, no 11, p. e1002295.) and 40% in biological sciences (HUANG et al., 2012HUANG, Xiaolei et al. (2012), “Willing or unwilling to share primary biodiversity data: results and implications of an international survey”. Conservation Letters, vol. 05, no 05, pp. 399-406.).

Using this conceptual framework, we report herein on the first research transparency and reproducibility analysis of political science in Brazil. We attempted to reproduce results of 197 articles reporting quantitative findings published between 2012 e 2016 in the five leading Brazilian political science and general social science journals, the Brazilian Political Science Review, the Revista de Ciência Política, Dados – Revista de Ciências Sociais, Opinião Pública, and Revista Brasileira de Ciências Sociais. Besides examining overall replication rates, we also observed differences in replicability across established, where the use of quantitative data is more common, and emerging subfields, as described below, as well as differences in the availability of data from publications. We tested for differences in methodological approach, transparency, and replicability between research in more and less established subfields in Brazilian Political Science.

Our results show that transparency and reproduction are still in a development stage in Brazilian Political Science. Of the 650 articles reviewed, we asked for data to 197 quantitative articles. From those, only 28% agreed to share datasets and computed codes. We were able to attempt a replication for only 14%, and successfully reproduce the results of less than 5%. Established subfield papers had higher transparency and reproduction rates than those for emerging fields, partly because they were more likely to use quantitative methods. For such articles, we found significantly higher rates of response, data provision, and replication success than with quantitative papers from emerging fields.

This study is divided into four sessions including this introduction. The next session presents the research design that includes the explanation of our hypotheses, criteria for classification of papers into fields (established or emerging) and methodology (quantitative or non-quantitative), details about the request for replication data and coding methodology for our dependent variables. The following session gives details about the replication procedure and the most important results this research has found. The last session is a conclusion of the study and it also brings some suggestions to improve transparency and replicability into the Brazilian social sciences community.

RESEARCH DESIGN

Conducting a replication study is not as trivial as some might think. Replications are difficult to conduct, time-consuming, and hard to publish (Janz, 2016JANZ, Nicole. (2016), “Bringing the gold standard into the classroom: replication in university teaching”. International Studies Perspectives, vol. 17, issue 4, pp. 392-407.). It is important to follow strict procedures to ensure that its design does not cause a failure in replicating. It seems to be the case of an important replication study conducted and reported by the Open Science Collaboration that attempted to replicate 100 published studies from psychological sciences fields and reported a surprisingly low reproducibility rate6 6 . Research Articles, August 28th, 2015, aac4716. . Gilbert et al. (2016)GILBERT, Daniel T. et al. (2016), “Comment on Estimating the Reproducibility of Psychological Science”. Science, vol. 351, no 6277, p. 1037. reported that this study’s design had errors in sampling, power and biases that would harm its conclusions.

To avoid this kind of error, we used a simpler, but accurate procedure. This study used the same dataset as used in original studies and tried to replicate the reported results without any change in samples or methodologies. We analyzed every paper published in the leading Brazilian Political Science journals, and coded each one according to its methodological approach, response to the data request, and research replicability7 7 . Since we covered the five leading journals for five years, increasing either number of journals or years should not make a difference for our analysis. . We tried to replicate every paper that used a quantitative approach. The focus on “quantitative studies” is simply because such designs should make replication easier. When datasets and computer codes are archived appropriately, other analysts should be able to reproduce every figure or table very quickly. Rich and detailed historical studies may also be replicated, but the effort required to do so for five years of articles was beyond the capacity of our project.

Hypotheses

Our research was intended primarily as a measurement exercise to establish the extent of replication norms in Brazil. However, we also hypothesized that research in established subfields would be more replicable than research in emerging fields, reflecting the same biases discussed above. In established fields, null results are more likely to be published since they may challenge established scholarship. Similarly, the existence of a larger number of faculty focusing on core questions may increase scholars’ care in testing hypotheses, knowing others may question their work. In contrast, in emerging fields, manuscripts are less likely to be published unless they have significant results. Consider a study on soccer games and votes for incumbents. If there is a null result, the paper is unlikely to be published and may be dismissed as a bad idea. If there is a significant result, a journal may be inclined to publish a “clever” piece that may get media attention. In this way, emerging fields are more likely to suffer false-positive bias due to the institutional problems discussed above8 8 . This example is provided for illustration purposes only and is not intended to comment on any research project. .

We examined all 650 articles published between 2012 and 2016 in the five leading Brazilian political science and general social science journals: Brazilian Political Science Review, Revista Brasileira de Ciência Política, DADOS, Opinião Pública, and Revista Brasileira de Ciências Sociais. Figures 1 and 2 show the distribution of papers by journal and year, respectively.

Figure 1
Papers by Journal

Figure 2
Papers by Year

Identifying Articlefrom Established Subfields

To explore our hypotheses regarding mature and emerging areas of study, we first defined four research subfields: Electoral Politics, Legislative Politics, Political Parties, and Emerging/Other. The first three are among the most established not only in Brazilian Political Science, but also in the international academic community. The other categories included all emerging subfields. Then, we collected the keywords from all papers, and we selected the ones related to the established fields. We pre-determined categories based on the paper keywords, as Appendix A shows. From the 180 keywords, 88 (49%) were assigned to Electoral Politics, 58 (32%) to Legislative Politics, and 34 (19%) to Political Parties. In addition, we preregistered these categories at the Open Science Foundation. For example, we categorize keywords like “candidates”, “candidacies” and “election” as Electoral Politics; keywords like “House of Representatives”, “coalitions” and “Congress” as Legislative Politics and “party system”, “party funding” and “partisanship” were included in the Party Politics category.

We classified each paper into one or more of these four categories. From all 650 papers, 154 (24%) used one or more of the selected keywords. Among those, 60 (9%), 37 (6%), and 16 (2%) included keywords from one of the subgroups Electoral Politics (E), Legislative Politics keywords (L), and Political Parties (P) keywords, respectively. An additional 15 (2%) had keywords from both Electoral Politics and Legislative Politics (E/L), 21 (3%) papers included both Electoral Politics and Political Parties keywords (E/P), and two had both Legislative Politics and Political Parties keywords (L/P). Finally, three papers had all three keywords (E/L/P). The pieces that did not have any of the selected keywords were classified in the “Emerging” area of study.

Established subfields represented 23.7% of all 650 papers. The sum of each subfield is not equal to this percentage because a paper can be classified into more than one subfield. Note that the Electoral Studies subfield is slightly more popular than the other two established subfields. Appendix B has a detailed classification of each paper.

Figure 3
Papers by Subfields and Maturity

Identifying Quantitative Works

To implement our research design, we first needed to identify which papers had a quantitative approach that allowed replication using only dataset and computer codes. Our methodological decision to only try to replicate quantitative studies relied on the research’s limitations to both analyze and reproduce qualitative data9 9 . For more details on the challenges of implementing reproduction and replication of qualitative studies see Elman and Kapiszewski (2014), Tuval-Mashiach (2017) or https://www.qualtd.net/ and https://openworking.tudl.tudelft.nl/2019/02/11/what-does-reproducibility-mean-for-qualitative-research/ .

We sought the main results reported for each paper and evaluated if it included any quantitative analysis. We created a dummy variable named Quantitative and assumed value 1 if, after a basic overview of the study, we identified quantitative results among the study’s main findings. The presence of charts and tables would indicate the use of a quantitative methodology. This selection cannot be considered very strict for a couple of reasons. First, we did the work manually, since there was no way to collect the articles’ metadata automatically10 10 . We thank an anonymous reviewer for pointing us that there is a R package that can extract metadata from journals today. See more details in https://fmeireles.com/blog/rstats/rscielo-um-scraper-para-extrair-dados-scielo. . Second, in order to enlarge the number of observations, we decided to include studies that had relevant quantitative results but did not necessarily use a statistical inference analytical method. Therefore, we considered a study with only a descriptive analysis as quantitative.

This decision brought up a discussion about differences within the group of quantitative studies. It would be more challenging to replicate research that used complex econometric models than another that did only a descriptive analysis. We are aware of this situation, but, for this research, we chose not to explore this heterogeneity. It would be necessary to develop an objective criterion to classify each quantitative methodology by their complexity and separate them into groups according to different replication difficulties. These procedures were beyond our time and work resources.

Even with a soft parameter for Quantitative, the number of quantitative papers was only 197, or 30% of all pieces examined. Notably, the proportion of quantitative articles in established subfields was significantly higher than in emerging subfields. Although they were less than a quarter of the potentially replicable papers, the established subfield papers included more than half of the quantitative research.

Table 1
Quantitative papers by Maturity

Requesting Replication Materials

For each paper reporting quantitative results, we sent an e-mail message to the authors and requested a replication dataset and a computer code. The message explained the research goals and assured author privacy in disclosure of our research, using the model e-mail below:

“Subject: Request for Data for Replication Study Without Identification.

Dear Mr(s). (Author’s name),

We write on behalf of Professors George Avelino (FGV, coordinator of the Center of Politics and Economics of the Public Sector), and Scott Desposato (the University of California, San Diego, director of the Center for Latin American Studies at the same university). We are developing an analysis of the replicability of the Political Science articles published in the leading Brazilian journals in the field, and your paper is in our database.

We would like to request both the dataset and the computer codes required to reproduce the results in your article (Title of the article), published in the journal (Name of the journal) on year (Year), volume (Volume number), (Journal number).

Your name, article name, or any other identifying information about you or your article, will NOT be disclosed in our final report.

Thank you in advance for your contribution to our project. Regards,”

The procedure followed the schedule below:

Phase I (responses)

· March 5th, 2018: sent the original message;

· March 26th, 2018: sent the first follow-up message to those who did not answer the original message and to address any questions;

· April 20th, 2018: sent a second follow-up message to those who did answer the first follow-up message and to answer any remaining questions;

· April 27th, 2018: deadline for receiving answers.

Phase II (completeness)

· May 7th, 2018: sent a second message asking for the complete material for those who answered our first e-mail, but who provided incomplete replication materials;

· May 21st, 2018: final deadline for answers for incomplete material.

In total, scholars had almost eight weeks answer our first message, and 11 weeks to provide replication materials. In general, the ones who actually answered our request did it very shortly. A more extended period might have increased our answer rate, but we received no additional answers after the final deadline.

Coding

Besides the proportion of studies that used a quantitative approach, we examined four transparency and replicability measures applied to each article that we characterized as having some quantitative analysis. Our four measures were:

· Responsiveness: Did the author(s) answer our email, even with a refusal to collaborate?

· Agreement: Did the author(s) agree to share data?

· Completeness: Did the author(s) provide enough data and code to enable the research replication attempt?

· Replicability: Were results replicable?

For Responsiveness, we first coded the papers by combining the answers in the two research phases (the first message and the follow-up) and the lack thereof: 1) Answered in phase 1; 2) Answered in phase 2; 3) Did not answer. Then, we coded Responsiveness as “1” for those that answered our request either in Phase 1 or Phase 2. In total, 83 of the 197 papers (42%) answered our request. They were divided nearly equally between established and emerging fields.

The second variable, Agreement, measures whether the author agreed to share the data or not. The variable had six categories: 1) Answered with data; 2) No response; 3) Data is not available; 4) Cannot share data; 5) Refusal and 6) Other. We coded the variable Agreement as a “1” for articles where the author provided data. This variable was not among our pre-registered dependent variables, but it seemed appropriate to distinguish the authors who answered positively from those who did not. Among the 83 responses, 56 (67.5%) provided some analysis data. From these, 36 papers (almost 65% of the 56) were from established fields.

As far as Completeness was concerned, we coded the following categories: 1) Complete; 2) No useful data; 3) Only code; 4) Only data source, and 5) Only dataset. We coded Completeness as a “1” for articles in which authors sent at least one dataset and some computer code. Only half of the 56 papers that provided data did it completely. More than two-thirds of these papers came from established fields (19).

Finally, Replicability had three categories: 1) Completely replicable (all results were reproduced, including sign and statistical significance); 2) Partially replicable (not all results were completely reproduced, or signs/significance did not match the ones reported in the paper) or 3) Non-Replicable (no result could be reproduced). We coded Replicability as “1” for articles in which data that the author(s) provided produced the same results in terms of values, signs, and statistical significance. From those 28 papers with complete data, we were able to fully replicate only 10 (35.7%), seven of which were from established fields.

Table 2 presents how dependent variables were coded according to the situations described above. Only studies that were coded “1” for Replicability were considered successfully replicable. Table 3 brings descriptive variables by field type for our sample (quantitative studies) and dependent variables.

Table 2
Dependent Variables Coding

Table 3
Descriptive Variables

Replication Procedure

We attempted to replicate all papers for which authors shared data and, in some cases, code. The first decision was to define which quantitative results to replicate, since many papers had multiple charts and tables with data not related to the core results. Given the magnitude of the task, we chose to focus on the paper’s prominent results. To identify a prominent result, we identified quantitative findings that were most discussed, especially in the abstract, introduction, and conclusion of the study. This procedure constrained our analysis to a smaller number of charts and tables. It also meant we ignored difficulties in replicating secondary analyses.

In replicating, we found problems with data, codes, or results. Figure 4 presents the distribution of these problems. Of the 28 for which we could attempt a replication, we found problems with almost two-thirds of them (18). The most frequent had to do with computer codes, which represented about one-third (10). We also had problems with results and data in eight (28.6%) and seven papers (25%), respectively. Finally, seven articles (21.5%) presented more than one type of problem.

Figure 4
Papers Replicated by Replication Problems

The types of coding problems involved either a non-working or an incomplete statistical computing code. In some cases, we could not calculate a specific result displayed in the paper. In others, there were syntax errors that made a complete replication impossible. We frequently received code lacking one or more calculations, and in many cases, poorly documented, with analyses and data management left unexplained. In one case, the data analysis was computationally intensive and eventually produced results like those published, but it did not reproduce the central figures. Interestingly, many code sets led to results that were not part of the article, suggesting some fishing for results. Finally, there were some minor syntax errors that we were able to correct.

Data problems involved cases in which materials that the authors shared lacked a critical variable or even a key dataset. For example, one author provided a dataset which seemed to be a complete dataset (data frame and code) and which could replicate some of the published regressions. However, further results in the computer code required an additional dataset, which the authors did not share. In some cases, the authors shared a web address where we could find the raw data, but they did not provide specific files with instructions for its coding and cleaning. In other cases, the authors provided a complex code with no instructions on how to execute it.

Some cases were more problematic since we could not open the dataset, or the data differed from the published results. In other cases, authors shared many different files, but none was useful or even required by the accompanying computer code. One author shared a zip file with multiple “xls” files from which we could not replicate results. These papers were classified as “no useful data” and did not proceed to the replication phase.

In some other cases, although we were able to produce results, they did not match those published. We classified those cases as results problems. Some codes failed to define the data subset used for analysis, and the number of observations displayed in the tables did not match with our results. In such cases, even descriptive statistics could not be reproduced. Other papers had problems with the statistical results. For example, our reproduction of regression models yielded different coefficient values for one or more variables.

Data was basically provided in Excel, Stata and R formats, which are well known statistical software. Except for minor corrections in scripts (including adaptations to software updates), replicators were oriented to not modify the code structure. Dataset and code organization is an important part regarding data transparency. In the case an author shared files in a different format or in a way that was not possible to understand how to use it, they were asked to provide more information. In cases that it happened, authors did not answer for our help request and the data were considered as “non-useful”.

Successful replications had several shared features. Most notably, the authors provided a well-organized set of data and codes. In one case, the authors did not send the computer code, but their results were clearly described in the text of their article and were straightforward to reproduce. In another case, the authors provided the dataset in a “xls” format file and a Stata file with few lines. Those lines described what table or graph it was related to in the paper. This type of information was critical to enable replication.

Established versus Emerging Quantitative Subfields

Of the 650 papers examined, 197 (30.3%) used a quantitative approach. Of these, only 56 (28.4% of quantitative papers and 8.6% overall) responded positively to our request for data. In half of these cases, the data was incomplete, which did not allow replication. Most of these were cases in which the author shared only codes or only a dataset, but not both. In the end, we completed a replication attempt on 28 papers (14.2% of quantitative papers and 4.3% overall), and only ten of them were fully replicable. In those last cases, the authors provided all data and code to reproduce the core results in the paper. In other words, we were able to successfully replicate only 5% of quantitative and 1.5% of all papers.

We hypothesized that established quantitative subfields would have stronger replication norms and higher rates of replicability than emerging fields. We tested our hypothesis by comparing our measures of responsiveness, agreement, data provision, and reproducibility for both established and emerging fields. Table 4 results from T-tests for a difference of means between these two groups among the five dependent variables.

Table 4
T-Tests on Dependent Variables by Maturity (Emerging – Established)

For all our dependent variables, the proportions are higher for the established than for the emerging fields. Authors of research in established areas were nearly twice as likely to respond to our request with some data (36% versus 20.6% in emerging fields). Additionally, 52.8% of authors from those fields provided complete data, versus 45% of those in emerging ones. Finally, of those that provided data, over half of the results from established fields could be replicated (52.8%), versus a slightly lower rate for emerging areas (45%).

The difference of means test presented in Table 3 showed that established subfields papers have a significantly higher probability of adopting a quantitative methodology, answering a request for data, providing the data with more quality, and ultimately being replicable. When including control variables in an OLS regression, the results are similar.

Table 5 reports the results of OLS regressions of our five dependent variables. Independent variable of interest is Established, a dummy variable that indicates papers from established subfields. Control variables include the number of authors and dummies for journals and year of publication.

Table 5
OLS Regressions Results

Established variable remains positive and significant for all dependent variables. The control variables do not have any clear patterns11 11 . We did not preregister hypotheses about control variables. . The number of authors had a positive and significant effect only on whether articles included quantitative analysis and whether any of them answered our e-mail. Using Brazilian Political Science Review as baseline, the reported effect of journals shows that both Revista Brasileira de Ciência Política and Revista Brasileira de Ciências Sociais seem to have a lower rate of quantitative papers.

Additionally, authors from the same journals were generally less likely to agree to share their data and to share complete data. Finally, results for year of publication have 2012 as baseline and do not strongly suggest an increasing replicability in more recent papers. Only Agreement (2015 and 2016) and Completeness (2016) reported significant higher averages for more recent papers.

CONCLUSION

Around the world, the credibility crisis in science poses a challenge to scholars’ efforts to advance knowledge both in the natural and the social disciplines. As Figueiredo et al. (2019)FIGUEIREDO FILHO, Dalson et al. (2019), “Seven reasons why: a user’s guide to transparency and reproducibility”. Brazilian Political Science Review [Online], vol. 13, no 02. (15/12/2019), pp. 1-37. Available in: http://dx.doi.org/10.1590/1981-3821201900020001. ISSN 1981-3821.
http://dx.doi.org/10.1590/1981-382120190...
discussed, norms of transparency and replicability are becoming part of standard scientific practice in the United States, Western Europe, and other regions. The scientific community cannot neglect credibility issues anymore. Addressing this crisis will require significant changes in the procedures of scientific institutions, journals, and scholars. There are reforms and best practices that should spread to all academic communities. Especially noteworthy is a general norm of research transparency.

This study is the first look at transparency and replicability in Brazil. It covers only a small sample of the academic work produced in the country and it has its limitations in terms of generalization and comparison with other countries’ standards. Even so, the evidence presented here can trigger new research in this field that would help the Brazilian academic community to follow this critical path.

Let us make it clear: our findings do not mean most research is invalid, or conclusions are incorrect. It is just a check on how easy it is to replicate or reproduce the main results. Strictly, if one considers the time lag between submission and publication, our data may even not represent the field accurately during the period12 12 . We thank an anonymous reviewer for calling our attention to this point. . Besides, we only check whether scholars provide the data we requested within two months of our request and whether we can easily use it to generate results identical to those in publication. With more time and more explanation of our purposes, perhaps we would have a higher response rate. With some sources from which we did receive data but failed to replicate results, with more time, maybe we could have reverse engineered findings and asked for more assistance from authors in understanding what they provided. With more time, we could have attempted to replicate results by collecting the original data ourselves from their sources. Again, non-reproducible does not mean invalid in any way. We believe it primarily reflects a lack of transparency norms in Brazilian political science and the need for a new standard according to which it should be easy to replicate and verify any published findings.

The difference in the results from established and emerging fields was also somewhat expected. In general, the more scholars engage in a dialogue around common themes and questions, and the more the field often uses quantitative analysis, the higher their need for transparency procedures to enhance the collective advancement of knowledge.

Admittedly, introducing replication practices in qualitative research poses a different challenge, which would go beyond the scope of this paper. Despite the methodological differences involved, this would be a great improvement for the Brazilian political science community, due to its large number of qualitative works. Also, introducing transparency practices for qualitative research would help to keep pace with the increasing use of mixed methods research designs, which usually requires teams made with members from different methodological backgrounds. Within this picture, trying to define transparency and replication practices in qualitative research, without compromising its small “n” and interpretive characteristics, is a crucial step. (Sukumar and Metoyer, 2019SUKUMAR, Porna Talkad; METOYER, Ronald. (2019), “Replication and transparency of qualitative research from a constructivist perspective”. OSF Preprints (https://osf.io/6efvp)
https://osf.io/6efvp...
; Moravcsik, 2014a, 2014b, 2019).

Our replicability analysis in Brazil suggests that political scientists in that country need to adopt transparency practices. Finally, we offer several recommendations for scholars and journals to improve replicability and maturity in these fields.

Authors can publish a Pre-Analysis Plan or a Registered Report before the beginning of their data collection or analysis (Casey, Glennerster and Miguel, 2012; Miguel et al., 2014MIGUEL, Edward et al. (2014), “Promoting Transparency in Social Science Research”. Science, vol. 343, no 166, pp. 30-31.), even on qualitative studies (Pineiro and Rosenblatt, 2016PINEIRO, Rafael; ROSENBLATT, Fernando. (2016), “Pre-Analysis Plans for Qualitative Research”. Revista de Ciência Política (Santiago) [online], vol. 36, no 3, pp. 785-796.). This procedure will strengthen research transparency, mitigating some of the credibility problems we found. Explicitly specifying the research question and defining our analysis plan will make researchers pay more attention to their study design and focus on theory and research questions. These procedures are relatively easy to adopt, since there are platforms like the Open Science Framework, where one can pre-register research plans and share data, codes, and results.

Scientific institutions can teach and encourage their members to implement these procedures (Steinhardt, 2020). Undergraduate, Graduate and Ph.D programs should have replication classes and should teach the best practices for data management13 13 . See https://www.academia.edu/42204205/Transparency_and_Reproducibility_in_Empirical_Research_2020.1_ for a good example of course structure provided by Professor Dalson Britto Figueiredo Filho of Universidade Federal de Pernambuco. . These classes would increase the students’ chances to get their research proposals funded, since a growing number of funding institutions began to require the inclusion of a data management plan in the submitted proposals. For instance, FAPESP, a well-known Brazilian research funding institution, began to require the inclusion of a data management plan (http://www.fapesp.br/gestaodedados/). Additionally, data management courses could also provide an easy way for students to get their first publication by replicating other papers (King, 2006KING, Gary. (2006), “Publication, Publication”. PS: Political Science and Politics, vol. 39, no 01, pp. 119-125.). Finally, a recent paper by Christensen et al. (2019)CHRISTENSEN, Garret; FREESE, Jeremy; MIGUEL, Edward (2019), Transparent and reproducible social science research: how to do open science. Oakland, CA: University of California Press. estimates that papers that make data and coding available receive additional scholarly citations.

Journals can support the publication of null results and of review processes that focus on research questions and design, rather than provocative results. For example, the Center for Open Science recommends that journals include peer review of research plans as well as results, known as Registered Reports:

Registered Reports is a publishing format used by over 200 journals that emphasizes the importance of the research question and the quality of methodology by conducting peer-review before data collection. High-quality protocols are then provisionally accepted for publication if the authors follow through with the registered methods. This format rewards best practices in adhering to the hypothetical-deductive model of the scientific method. It eliminates a variety of questionable research practices, including low statistical power, selective reporting of results, and publication bias, while allowing complete flexibility to report serendipitous findings. (https://cos.io/rr/)

Registering a research plan is not an instrument to restrict scientists on their flexibility to conduct their research or a safeguard for journals against their responsibility for the credibility of their publications. Unpredictable events may occur during data collection, imposing changes in analysis strategy. A pre-registered report informs readers about such unforeseen problems faced by researchers, and solutions implemented, enhancing research transparency (Christensen, Freese and Miguel, 2019). In other scenarios, readers never learn the real reason behind the methodological choice, opening the way for credibility questioning. For journals, pre-registered analysis plans are an opportunity to mitigate publication bias and reduce incentives for fishing and p-hacking.

Officially, releasing replication data is another essential practice to improve science transparency. Using and sharing computer codes allow research verification and replication (King, 2003KING, Gary. (2003), “The Future of the Replication Movement”. International Studies Perspectives, vol. 04, no 1, pp. 72-107.; Elman, Kapiszewski and Lupia, 2018; Dafoe, 2014DAFOE, Allan. (2014), “Science deserves better: the imperative to share complete replication files”. Political Science and Politics, vol. 47, no 1, pp. 60-66.). Publishing replication materials increases incentives for careful analysis and accurate reporting. Verification also allows for investigating potential cases of fraud. Indeed, it would have been impossible to uncover recent cases of data fabrication in the United States scandals without access to underlying data. Replication materials can also support academic teaching and learning.

To keep in touch with the international trend, the Brazilian scientific community should incorporate transparency practices. ABCP (Associação Brasileira de Ciência Política), as well as other associations and Journals, should adopt standards of replication, requiring and archiving data and replication code for published research, subject to the privacy and copyright restrictions.

They could endorse or even implement together a general data repository to concentrate all scientific work. A well-known benchmark is The Berkeley Initiative for Transparency in the Social Sciences (BITSS).

The Berkeley Initiative for Transparency in the Social Sciences (BITSS) is another new institution that has emerged in recent years to promote dialogue and build consensus around transparency practices. BITSS has established an active training program for the next generation of economists and other social scientists, as well as an award to recognize emerging leaders in this area, the Leamer-Rosenthal Prize for Open Social Science (Christensen and Miguel, 2018CHRISTENSEN, Garret; MIGUEL, Edward (2018), “Transparency, reproducibility, and the credibility of economics research”. Journal of Economic Literature, vol. 56, no 3, pp. 920-980.:61-62).

Adopting these recommendations will increase the maturity and credibility of social science in Brazil and benefit scholarship and society. Associations and journals can both encourage and facilitate such practices. For example, ABCP and universities can offer courses to teach their members how to track their research procedures, how to prepare a dataset, and how to use computer programs that facilitate this task. In addition, to further transparency, ABCP could lead an association with an internationally known data storage, such as Dataverse (https://dataverse.org/), which could increase the rewards for registering dataset and codes by making them available to the international scientific community. Universities need to nurture a reproducibility and replication culture within their departments and courses to ensure that a “gold standard” of credibility in academia is strengthened (Janz, 2016JANZ, Nicole. (2016), “Bringing the gold standard into the classroom: replication in university teaching”. International Studies Perspectives, vol. 17, issue 4, pp. 392-407.). They can also provide online storage for datasets and code sharing.

To further the publication of datasets that interest a broad audience, journals’ editors can create a section dedicated to papers presenting datasets to the community. Journals can also publicly require datasets and codes to submission approval (Dafoe, 2014DAFOE, Allan. (2014), “Science deserves better: the imperative to share complete replication files”. Political Science and Politics, vol. 47, no 1, pp. 60-66.). Finally, journals need to encourage replication publications, but not implement a reverse publication bias, focusing only on failed replications attempts (Janz and Freese, 2019JANZ, Nicole; FREESE, Jeremy. (2019), “Good and bad replications in political science: how replicators and original authors (should) talk to each other”. Available in https://www.mzes.uni-mannheim.de/openscience/wp-content/uploads/2019/01/Janz-Freese_-Good-and-Bad-Replications-1.pdf
https://www.mzes.uni-mannheim.de/opensci...
). Recently, the Brazilian Political Science Review adopted a policy that supported transparency. From their author’s guideline for submission:

Authors submitting empirical papers should provide, when and if the manuscript is approved, the database used in the analysis, the code dictionary describing the variables, and the code for replication or the series of steps of the analysis. The database must be in an easily accessible format for conventional statistical software – such as R, Stata, SPSS, and Excel. The code dictionary should indicate the name of the variable, its description, and the source of the data. Where appropriate, the code for replication of the analyzes should be extensively commented. When the article does not use statistical software that allows the creation of a replication code, the author should describe the procedures, step-by-step, for replication to be possible.

In a good note, a more recent survey with the editors of all other four journals investigated in this research shows they are all intending to introduce transparency and replicability procedures soon. This result shows that concerns with those procedures are starting to prevail in the Brazilian Political Science, making it closer to the international community. A future analysis will tell whether this is an excessive optimistic vision or not.

The scientific community is increasingly adopting transparency norms. Most likely, they will become mandatory for international publication and international research funding. The sooner Brazil’s scientific community incorporates these transparency standards, the better. In short, implement transparency procedures – like research pre-registration and sharing of codes and data – is beneficial for everyone. It mitigates credibility suspicions, values scientific work, and, above all, allows research activity to concentrate on what is most important: the questions they seek to answer.

REFERENCES

  • BALL, Richard; MEDEIROS, Norm. (2012), “Teaching integrity in empirical research: a protocol for documenting data management and analysis”. The Journal of Economic Education, vol. 43, no 02, pp. 182-189.
  • BAKKEN, Suzanne. (2019), “The journey to transparency, reproducibility, and replicability”. Journal of the American Medical Informatics Association, vol. 26, Issue 3, pp. 185-187.
  • CASEY, Katherine; GLENNERSTER, Rachel; MIGUEL, Edward (2012), “Reshaping institutions: evidence on aid impacts using a pre-analysis plan”. The Quarterly Journal of Economics, vol. 127, no 4, pp. 1755-1812.
  • CHRISTENSEN, Garret; MIGUEL, Edward (2018), “Transparency, reproducibility, and the credibility of economics research”. Journal of Economic Literature, vol. 56, no 3, pp. 920-980.
  • CHRISTENSEN, Garret; FREESE, Jeremy; MIGUEL, Edward (2019), Transparent and reproducible social science research: how to do open science. Oakland, CA: University of California Press.
  • CHRISTENSEN, Garret et al. (2019), “A Study of the Impact of Data Sharing on Article Citations Using Journal Policies as a Natural Experiment”. PLoS ONE, vol. 14, no 12, e0225883. https://doi.org/10.1371/journal.pone.0225883
    » https://doi.org/10.1371/journal.pone.0225883
  • CHORDIA, Tarun; GOYAL, Amit; SARETTO, Alessio. (2017), “P-hacking: evidence from two million trading strategies”. Swiss Finance Institute Research Paper, n. 17-37.
  • CRUWELL, Sophia et al. (2019), “Seven easy steps to open science: an annotated reading list”. Zeitschrift für Psychologie, vol. 227, no 4, pp. 237-248.
  • DAFOE, Allan. (2014), “Science deserves better: the imperative to share complete replication files”. Political Science and Politics, vol. 47, no 1, pp. 60-66.
  • DEGTEREV, Denis A. (2020), Replicability of Research on International Relations As a Global Trend. Herald of the Russian Academy of Sciences, vol. 90, pp. 36-44.
  • ELMAN, Colin; KAPISZEWSKI, Diana. (2014), “Data Access and Research Transparency in the qualitative tradition”. PS: Political Science & Politics, vol. 47, no 01, pp. 43-47.
  • FERGUSON, Christopher J.; HEENE, Moritz. (2012), “A vast graveyard of undead theories: publication bias and psychological science’s aversion to the null”. Perspectives on Psychological Science, vol. 07, no 06, pp. 555-561.
  • FIGUEIREDO FILHO, Dalson et al. (2019), “Seven reasons why: a user’s guide to transparency and reproducibility”. Brazilian Political Science Review [Online], vol. 13, no 02. (15/12/2019), pp. 1-37. Available in: http://dx.doi.org/10.1590/1981-3821201900020001 ISSN 1981-3821.
    » http://dx.doi.org/10.1590/1981-3821201900020001
  • FINIFTER, Bernard M. (1975), “Replication and extension of social research through secondary analysis”. Social Information, vol. 14, no 02, pp. 119-153.
  • GERBER, Alan S.; GREEN, Donald P.; NICKERSON, David. (2001), “Testing for Publication Bias in Political Science”. Political Analysis, vol. 09, no 04, pp. 385-392.
  • GILBERT, Daniel T. et al. (2016), “Comment on Estimating the Reproducibility of Psychological Science”. Science, vol. 351, no 6277, p. 1037.
  • GLEDITSCH, Kristian S. (2020), Houston, we have a problem: enhancing academic freedom and transparency in publishing through post-publication debate”. Political Studies Review. DOI: 10.1177/1478929919889309.
  • GOODMAN, Alissa et al. (2015), “Ten simple rules for the care and feeding of scientific data”. PLOS Computational Biology, vol. 10, no 04, pp. 01-05.
  • GOODMAN, Steven N.; FANELLI, Danielle; IOANNIDIS, John P. A. (2016), “What does research reproducibility mean?”. Science Translational Medicine, vol. 08, no 341, pp. 341ps12.
  • HAMERMESH, Daniel (2007), “Replication in Economics”. NBER Working Paper, n. 13026.
  • HARDWICKE, Tom E. et al. (2020), Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014-2017). Available in https://osf.io/preprints/metaarxiv/9sz2y/
    » https://osf.io/preprints/metaarxiv/9sz2y/
  • HEAD, Megan L. et al. (2015), “The Extent and Consequences of P-Hacking in Science”. PLOS Biology 13(3): e1002106. https://doi.org/10.1371/journal.pbio.1002106
    » https://doi.org/10.1371/journal.pbio.1002106
  • HILDEBRANDT, Tom; PRENOVEAU, Jason M. (2020), “Rigor and reproducibility for data analysis and design in the behavioral sciences”. Behavior Research and Therapy, vol. 126. doi:10.1016/j.brat.2020.103552
  • HUANG, Xiaolei et al. (2012), “Willing or unwilling to share primary biodiversity data: results and implications of an international survey”. Conservation Letters, vol. 05, no 05, pp. 399-406.
  • HUMPHREYS, Macartan; DE LA SIERRA, Raul Sanchez; VAN DER WINDT, Peter. (2013), “Fishing, commitment, and communication: a proposal for comprehensive nonbinding research registration”. Political Analysis, vol. 21, no 01, pp. 1-20.
  • IONNADIS, John. (2005), “Why most published research findings are false”. PLOS Medicine. Available at: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
    » https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
  • JANZ, Nicole. (2016), “Bringing the gold standard into the classroom: replication in university teaching”. International Studies Perspectives, vol. 17, issue 4, pp. 392-407.
  • JANZ, Nicole; FREESE, Jeremy. (2019), “Good and bad replications in political science: how replicators and original authors (should) talk to each other”. Available in https://www.mzes.uni-mannheim.de/openscience/wp-content/uploads/2019/01/Janz-Freese_-Good-and-Bad-Replications-1.pdf
    » https://www.mzes.uni-mannheim.de/openscience/wp-content/uploads/2019/01/Janz-Freese_-Good-and-Bad-Replications-1.pdf
  • KEY, Ellen M. (2016), “How are we doing? data access and replication in political science”. PS: Political Science and Politics, vol. 49, no 02, pp. 268-272.
  • KING, Gary. (1995), “Replication, Replication”. PS: Political Science and Politics, vol. 28, no 03, pp. 444-452.
  • KING, Gary. (2003), “The Future of the Replication Movement”. International Studies Perspectives, vol. 04, no 1, pp. 72-107.
  • KING, Gary. (2006), “Publication, Publication”. PS: Political Science and Politics, vol. 39, no 01, pp. 119-125.
  • KING, Gary; KEOHANE, Robert O.; VERBA, Sidney. (1994), Designing social inquiry: scientific inference in qualitative research. Princeton: Princeton University Press.
  • MARKOWETZ, Florian. (2015), “Five selfish reasons to work reproducibly”. Genome Biology, vol. 16, no 01, p. 274.
  • MARTINS, Henrique Castro. (2020), “A Importância da Ciência Aberta (Open Science) na Pesquisa em Administração”. Rev. Adm. Contemp, vol. 24 no 01.
  • MIGUEL, Edward et al. (2014), “Promoting Transparency in Social Science Research”. Science, vol. 343, no 166, pp. 30-31.
  • MORAVCSIK, Andrew. (2014a), “Transparency: The Revolution in Qualitative Research”. PS: Political Science and Politics, vol. 47, no 1, pp. 48-53.
  • MORAVCSIK, Andrew. (2014b), “Trust but verify: the transparency revolution and qualitative international relations”. Security Studies, vol. 23, no 4, pp. 663-688.
  • MORAVCSIK, Andrew. (2019), Transparency in Qualitative Research. SAGE Research Methods (https://www.princeton.edu/~amoravcs/library/TransparencyinQualitativeResearch.pdf
    » https://www.princeton.edu/~amoravcs/library/TransparencyinQualitativeResearch.pdf
  • NSF (National Science Foundation). (2015), Reproducibility, Replicability, and Generalization in the Social, Behavioral, and Economic Sciences. Available at <https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_ Report_on_Replicability_SubcommitteeMay_2015.pdf>.
    » https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_ Report_on_Replicability_SubcommitteeMay_2015.pdf
  • PINEIRO, Rafael; ROSENBLATT, Fernando. (2016), “Pre-Analysis Plans for Qualitative Research”. Revista de Ciência Política (Santiago) [online], vol. 36, no 3, pp. 785-796.
  • PIWOWAR, Heather A.; DAY, Roger S.; FRIDSMA, Douglas B. (2007), “Sharing Detailed Research Data Is Associated with Increased Citation Rate”. PloS One, vol. 2, no 3, p. e308.
  • ROCHE, Dominique G. et al. (2015), “Public data archiving in ecology and evolution: how well are we doing?”. PLoS Biol, vol. 13, no 11, p. e1002295.
  • SAVAGE, Caroline J.; VICKERS, Andrew J. (2009), “Empirical study of data sharing by authors publishing in Plos Journals”. PloS One, vol. 4, no 9, p. e7078.
  • SIMONSOHN, Uri. (2013), “Just post it: the lesson from two cases of fabricated data detected by statistics alone”. Psychological Science, vol. 24, no 10, pp. 1875-1888.
  • SUKUMAR, Porna Talkad; METOYER, Ronald. (2019), “Replication and transparency of qualitative research from a constructivist perspective”. OSF Preprints (https://osf.io/6efvp)
    » https://osf.io/6efvp
  • TENOPIR, Carol et al. (2011), “Data sharing by scientists: practices and perceptions”. PloS One, vol. 6, no 6, p. e21101.
  • THELWALL, Mike et al. (2020), “Is useful research data usually shared? an investigation of genome-wide association study summary statistics”. Plos One, vol. 15, no 2, p. e0229578.
  • TUVAL-MASHIACH, Rivka. (2017), “Raising the curtain: the importance of transparency in qualitative research”. Qualitative Psychology, vol. 4, no 2, pp. 126-138.

NOTES

  • 1
  • 2
    . For more information about p-hacking concept see Head et al. (2015)HEAD, Megan L. et al. (2015), “The Extent and Consequences of P-Hacking in Science”. PLOS Biology 13(3): e1002106. https://doi.org/10.1371/journal.pbio.1002106.
    https://doi.org/10.1371/journal.pbio.100...
    and Chordia, Goyal and Saretto (2017).
  • 3
    . A false positive result consists in rejecting the null hypothesis when it was not the case of rejecting it. The opposite case is known as false negative, when the null hypothesis is not rejected when it should be.
  • 4
    . Though Hamermesh (2007)HAMERMESH, Daniel (2007), “Replication in Economics”. NBER Working Paper, n. 13026. separates simple from scientific replication, designating the last term to the expansion of scientific knowledge, we are more concerned with the first term, meaning transparency and reproducibility. Therefore, throughout this paper, we use replication and reproducibility interchangeably.
  • 5
    . More than 1,100 journals have implemented at least one of the Transparency and Openness Promotion Guidelines, according to the Center of Open Science website. Some of the journals are American Journal of Political Science, European Political Science, Italian Political Science Review, Quarterly Journal of Political Science, State Politics and Policy Quarterly, Public Administration Review, Political Science Research and Methods, Political Behavior, Law and Human Behavior, Journal of Economic Behavior & Organization, Political Science Research and Methods. The complete list can be accessed in https://www.cos.io/our-services/top-guidelines?
  • 6
    . Research Articles, August 28th, 2015, aac4716.
  • 7
    . Since we covered the five leading journals for five years, increasing either number of journals or years should not make a difference for our analysis.
  • 8
    . This example is provided for illustration purposes only and is not intended to comment on any research project.
  • 9
    . For more details on the challenges of implementing reproduction and replication of qualitative studies see Elman and Kapiszewski (2014)ELMAN, Colin; KAPISZEWSKI, Diana. (2014), “Data Access and Research Transparency in the qualitative tradition”. PS: Political Science & Politics, vol. 47, no 01, pp. 43-47., Tuval-Mashiach (2017)TUVAL-MASHIACH, Rivka. (2017), “Raising the curtain: the importance of transparency in qualitative research”. Qualitative Psychology, vol. 4, no 2, pp. 126-138. or https://www.qualtd.net/ and https://openworking.tudl.tudelft.nl/2019/02/11/what-does-reproducibility-mean-for-qualitative-research/
  • 10
    . We thank an anonymous reviewer for pointing us that there is a R package that can extract metadata from journals today. See more details in https://fmeireles.com/blog/rstats/rscielo-um-scraper-para-extrair-dados-scielo.
  • 11
    . We did not preregister hypotheses about control variables.
  • 12
    . We thank an anonymous reviewer for calling our attention to this point.
  • 13
    . See https://www.academia.edu/42204205/Transparency_and_Reproducibility_in_Empirical_Research_2020.1_ for a good example of course structure provided by Professor Dalson Britto Figueiredo Filho of Universidade Federal de Pernambuco.
  • *
    Hypotheses and data analysis plan were pre-registered at the Open Science Foundation except as noted. Analysis of existing datasets was exempted by the UCSD Institutional Review Board (IRB). The authors thank the Berkeley Institute for Transparency in the Social Sciences for funding and support. George Avelino and Ivan Mardegan acknowledge the support from São Paulo Research Foundation FAPESP (Grants #: 2013/156158-1 and 2017/20855-1). The authors also thanks Raduan Meira for research assistance.

APPENDIX

A – Keywords Classification

B – Paper Maturity Classification

C – Pre-Analysis Plan

Publication Dates

  • Publication in this collection
    14 May 2021
  • Date of issue
    2021

History

  • Received
    23 Jan 2020
  • Reviewed
    5 July 2020
  • Accepted
    20 July 2020
Instituto de Estudos Sociais e Políticos (IESP) da Universidade do Estado do Rio de Janeiro (UERJ) R. da Matriz, 82, Botafogo, 22260-100 Rio de Janeiro RJ Brazil, Tel. (55 21) 2266-8300, Fax: (55 21) 2266-8345 - Rio de Janeiro - RJ - Brazil
E-mail: dados@iesp.uerj.br