Have We Been Transparent Enough? Challenges in Replicability and Credibility in Business Research

“Science should be ‘show me’, not ‘trust me’!” Through this phrase, Stark (2018, 613) discusses the need to repeat scientific studies to confer robustness to research results. The replicability of works is viewed as the golden standard for scientific research, given that transparency is the central requisite needed to replicate or reproduce studies. (Janz, 2015; Marques, 2019).

Therefore, transparency in business studies is something that is crucial to their credibility, and it also has various impacts, as noted by The Declaration on Research Assessment (DORA - https://sfdora.org/), and Committee on Publication Ethics (COPE - https://publicationethics.org/data) (Mendes-Da-Silva, 2018). But, how many of us have assured the transparency of our research results, by sharing data, materials, and/or code (duly commented) used in qualitative, statistical or econometric analyses (Gandrud, 2018)?

Transparency and causality, in turn, are the keywords most often cited in the past few years in academia (Marques, 2019). At the same time, to establish causality we need to be able to repeat experiments and tests that attest to the robustness of our results. To Aguinis, Cascio and Ramani (2017), the area of business is not immune to crises of replicability and reproducibility in the scientific community. Thus, there is a high level of preoccupation with the problem of producing irreproducible results, and by extension the credibility of research results. In this respect, researchers, research stimulation agencies and journals need to work together to make published research more reliable, and improve the returns of the money invested in scientific research (Nature, 2016).

Also, according to Aguinis et al. (2017) reproducibility signifies that someone who is not an author of a given study is capable of obtaining the same results by using the data utilized by the authors. Replicability signifies that someone who is not a participant in the authorship of a published study is capable of obtaining the same empirical models, with substantially similar data, through the application of the same steps, even if this is within a different context with different data.

It is evident that it is not necessarily easy to affirm that the results of a study are believable and useful if they are unreproducible and non-replicable. Unfortunately, as Aguinis et al. (2017) points out, there is a proliferation of suggestive evidence that there is a certain deficit of replicability in research that is presented as something reasonably disseminated.

Scientific knowledge advances through corroboration, when researchers verify the results obtained through the studies of other colleagues. No work of research, therefore can be considered the final word. However, there is a substantial part of research that does not sufficiently contemplate and prioritize the continuity of the knowledge construction process. Even the rapid detection of errors in published articles, including non-intentional ones, could be handled better if the data (and the codes used in the analysis) were available, according to Allison, Brown, George, and Kaiser (2016).

These procedures can help in making necessary adjustments and retractions in articles if errors are detected even after publication. Journals, scientists, institutions and financers have a role in the resolution of reproducibility. The RAC has sought to adopt policies and measures to substantially improve the transparency and robustness of what we publish. In addition, we hope to collaborate with the promotion of raising the awareness of the business researcher community.

Replicability of quantitative and qualitative research

Qualitative and quantitative approaches are frequently seen as opposite paradigms of the research methodology, with a focus on generalizing the validity of quantitative methods taking precedence over the supposedly less robust nature of qualitative research. The proponents of quantitative research may emphasize the replicability of results, but generally tend to negate the challenges faced by the positivist paradigm. These challenges are essentially centered around its potentially reductionist nature, at least according to some defenders of the contributions of qualitative research.

According to the arguments of Kandori (2018), replicability can be seen as an aspect that induces credibility in business research. This should make the community reflect on aspects of transparency that can permit research replicability, no matter whether they’re of a quantitative or qualitative nature. Quantitative research in business is usually more emphatically supported by the use of data and analysis tools, which facilitate the replication of studies with objective approaches. And this is of value in following a smooth road that can lead to the identification of causal relationships, as discussed by Poirier (1988) and Makridakis, Assimakopoulos, and Spiliotis (2018).

Aguinis et al. (2017) point out that the growing pressure on researchers to publish in top journals can induce behavior that implies a reduction in the replicability of conducted research. That is: shortcuts to obtain more impressive results, and these will supposedly will more likely be accepted for publication. Examples of such shortcuts: (a) the selection of certain variables (favorable to the results) which are included in the empirical model to be tested; (b) the use of certain control variables; (c) removing outliers; and (d) proposing hypotheses after knowing the results, aka HARKing (Hypothesizing After Results are Known). To Aguinis et al. (2017), resorting to these shortcuts requires basically countless attempts to obtain statistical models with greater predictive capacity, but researchers frequently omit these procedures. In this respect, Friedman and Sunder (1994, p. 85), affirm that “a reasonable number of researchers torture the data until it confesses”.

Quantitative methodology, at least to a certain extent, can be seen as relatively subjective. Bearing in mind that even an empirical study is subject to black box thinking, there are biases due to the researcher’s point of view, especially when considering potentially contestable data used in a study (Lundy, 1996). That being said, while methods of quantitative analyses are under explicit scrutiny, qualitative researchers have a task to perform: prove that their selection of methods is equal, or even more valuable than quantitative methods in the eyes of the peers in the scientific community.

If on one hand quantitative studies possess aspects that merit attention in terms of replicability, qualitative studies are no different, though the care that needs to be taken is specific. Thus, works that are based on case studies, grounded theory or action research, for example, can benefit from verifications before their submission to a journal as detailed and prescribed by Aguinis and Solarino (2019) when they point to a group of transparency criteria for qualitative research which assures the replicability of the study, and therefore, its credibility. It is expected that the efforts of authors, editors and reviewers affect the quality of these journals, and it is already possible to see initiatives in the international community related to classifying journals according to the typical reproducibility of the studies published in them, such as https://replicationindex.com/tag/replicability/.

The criteria of research transparency can vary according to their modality, i.e. whether they are quantitative or qualitative, and according to the type of replication that is sought (Aguinis & Solarino, 2019; Bergh, Sharp, Aguinis, & Li, 2017; Goffin, Åhlström, Bianchi, & Richtnér, 2019; Hoorani, Nair, & Gibbert, 2019; Moravcsik, 2014; Plakoyiannaki, Wei, & Prashantham, 2019; Tsang & Kwan, 1999). Thus, according to Aguinis and Solarino (2019), in terms of qualitative research, most of the transparency criteria tend to be required, especially if by chance the replication is to be exact, since in this modality the finality would be verifying whether the findings of a previous study can be reproduced using the same data and the same methods. If we are considering an empirical replication, we expect to employ the same method, but with different data. That is, the intention of empirical replication is to verify the external validity (i.e. generalizability) of the results in various contexts. An alternative method, if the researcher is interested in dealing with the same population with different procedures, would be a conceptual replication.

In terms of the apparent emphasis on reproducibility in research within the context of quantitative or data driven approaches, there is also a debate in regard to the reproducibility of qualitative research in the area of business. To researchers like Moravcsik (2014), transparency is a pre-condition for the advance of qualitative research. In this respect, Aguinis and Solarino (2019) analyzed 52 articles published in currently one of the most prestigious journals in the business research community, the Strategic Management Journal, in which they revealed that none of these articles was sufficiently transparent to the point of making it possible to replicate them, no matter whether this was in the exact, empirical or conceptual sense.

In this same work, the authors offered recommendations based on 12 criteria of transparency, and how to measure them. Aguinis and Solarino (2019) also argue that these criteria can be used to evaluate qualitative studies that have already been published, or they can also be used to guide future studies based on qualitative methods. The preoccupation with preserving the transparency of qualitative research, with the objective of making it possible to replicate these studies has occupied recent space in the literature, which suggests that this is an aspect worthy of the attention of the business researcher community (Goffin et al., 2019; Hoorani et al., 2019; Plakoyiannaki et al., 2019).

It is possible to note the efforts that have been made in other areas of knowledge in the sense of presenting an advance in the direction of the reproduction of results including the sharing of methods (including test codes), for example: https://ctuning.org/ae/artifacts.html. In the RAC we have already published, since July 2018, works with shared data and/or materials (Mendes-Da-Silva, 2019), and even one of these articles shared the respective codes for an econometric analysis (Ermel, 2018; Ermel & Martelanc, 2018). Initiatives of this type make it possible to replicate and reproduce studies, so that the results can be continually verified and improved (Marques, 2019). In this way, the advance in knowledge produced in the business area can be considered effective, and less subject to the need for redundant efforts. This becomes particularly relevant when we are submitting to a research environment whose resources are increasingly scarce, and this is not just in financial terms.

This special edition and final words

In terms of keeping informed as researchers of what we need to know, we still need to make an effort in our research, and the RAC in this edition offers the community new works that make up a set of research results on a theme that is of special interest to Brazil and other countries in South America: technology and the management of the Amazon region (Athayde et al., 2019).

I would like to direct my sincere and honest recognition to the professional work performed by Professors Emílio José Montero Arruda Filho (UNAMA, Brazil), Cristiana Fernandes de Muylder (FUMEC University, Brazil), Airton Cardoso Cançado (Federal University of Tocantins, Brazil), Ruby Roy Dholakia (University of Rhode Island College of Business, USA), and Angela Paladino (Faculty of Business and Economics of the University of Melbourne, Australia), who position and present the four other articles that have been selected for this special edition through their guest article entitled Technology Perspectives and Innovative Scenarios Applied in the Amazon Region.

In the second article, entitled: Innovation and the Diffusion of Technology in Agriculture in Floodplains in the State of Amazonas, by Jonas Fernando Petry, Sabrina Arcanjo Sebastião, Erik Garcia Martins, and Paulo Berti de Azevedo Barros, the authors affirm that their objective is to investigate innovation and the diffusion of technology within the context of agriculture in the floodplains in the interior of the State of Amazonas (Brazil). The results suggest that the competitive environment on the supply side, as well as the influence of agribusiness industries, technical assistance, the development policy agenda, university projects, and field practices, favor the diffusion of technologies.

The third article by Mauro Margalho Coutinho, Mário Vasconcellos Sobrinho, Sue Anne Collares Maestri de Oliveira and Ana Margarida Santiago, is entitled Coproduction between Government and Civil Society to Establish Smart Cities in the State of Pará. It discusses the possibilities of constructing the concept of smart cities in the state of Pará (Brazil), based on the coproduction of public services by civil society and the government. They present two coproduction approaches within the context of telemedicine. The authors seek to develop their work around citizens and NGOs willing to coproduce with the government based on the establishment of partnerships.

The fourth article of this special RAC edition is Go Global or Stay Local? Understanding How Fiscal Incentives Reshape Supply by authors Ricardo Silveira Martins, Janaina Siegler, Armando Souza-Junior, Barbara Flynn, and Guilherme Silveira Martins, who propose investigating the way in which companies reshape their supply networks and their production networks through fiscal incentives in free economic zones. Six production networks located in Manaus (Brazil) are analyzed using the structure of a global production network in terms of their factory types and the nature of their supply network relationships. The authors find that incentives can affect the shape of the production network from an immersion perspective, implying a change in a company’s strategic role.

The fifth and last article of this edition is entitled Interorganizational Relationships in the Amazon Biotech Industry Based on Entrepreneurs´ Perceptions, by Rosana Zau Mafra, Dimas José Lasmar, and Dalton Chaves Vilela Júnior. In this work they argue that various modalities of relationships have been adopted in restructuring industries as a survival and development strategy. The authors believe that the biotech industry requires partnerships, given the large technical-scientific investment needed. This, however, has been difficult to implement within the current context. In the Amazon, where the aggregation of value in the use of biodiversity resources plays an important role in sustainable regional development, studies point to historical difficulties in interactions between the biotech industry, academia and the government. The authors emphasize that their objective is to analyze the perceptions of companies in terms of interorganizational relationships in the regional biotech industry. They conclude that their results highlight the need for strategic action to bring together the actors in this industry.

Finally, in terms of the possibility of using shortcuts to realize quantitative research and the supposed difficulty of replicating and reproducing works of a qualitative nature, we would like to point out that within the context of the role to be played by the academic community, it appears imperative to reflect on the maintaining of standards in transparency in terms of what we produce and publish. In this respect, I would invite the community to submit works which have the central purposes of evaluating the transparency of research that has been published here, which also indicate paths that will enable our researchers to guarantee the replicability of business research, no matter whether it employs a quantitative or qualitative approach (Aguinis, Cascio, & Ramani, 2017; Aguinis & Solarino, 2019). As Stark says (2018), science should be ‘help me if you can’, not ‘catch me if you can’.

Referências

• Aguinis, H., Cascio, W. F., & Ramani, R. S. (2017). Science's reproducibility and replicability crisis: International business is not immune. Journal of International Business Studies, 48(6), 653-663. https://doi.org/10.1057/s41267-017-0081-0
  » https://doi.org/10.1057/s41267-017-0081-0
• Aguinis, H., & Solarino, A. M. (2019). Transparency and replicability in qualitative research: The case of interviews with elite informants. Strategic Management Journal, 40(8), 1291-1315. https://doi.org/10.1002/smj.3015
  » https://doi.org/10.1002/smj.3015
• Allison, D. B., Brown, A. W., George, B. J., & Kaiser, K. A. (2016). Reproducibility: A tragedy of errors. Nature, 530, 27-29. https://doi.org/10.1038/530027a
  » https://doi.org/10.1038/530027a
• Athayde, S., Mathews, M., Bohlman, S., Brasil, W., Doria, C. R. C., Dutka-Gianelli, J., Fearnside, P. M., Loiselle, B., Marques, E. E., Melis, T. S., Millikan, B., Moretto, E. M., Oliver-Smith, A., Rossete, A., Vacca, R., & Kaplan, D. (2019). Mapping research on hydropower and sustainability in the Brazilian Amazon: Advances, gaps in knowledge and future directions. Current Opinion in Environmental Sustainability, 37, 50-69. https://doi.org/10.1016/j.cosust.2019.06.004
  » https://doi.org/10.1016/j.cosust.2019.06.004
• Bergh, D. D., Sharp, B. M., Aguinis, H., & Li, M. (2017). Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings. Strategic Organization, 15(3), 423-436. https://doi.org/10.1177/1476127017701076
  » https://doi.org/10.1177/1476127017701076
• Ermel, M. D. A. (2018). “Data for:” "Perfil da participação do advisor nas fusões e aquisições no Brasil" Published at RAC. Mendeley Data, v1. Recuperado de http://dx.doi.org/10.17632/rfybj4xc6w.1
  » http://dx.doi.org/10.17632/rfybj4xc6w.1
• Ermel, M. D. A., & Martelanc, R. (2018). Antecedentes e consequências da participação de advisor em fusões e aquisições. Revista de Administração Contemporânea, 22(6), 859-884. Recuperado de http://www.scielo.br/pdf/rac/v22n6/1982-7849-rac-22-06-0859.pdf https://doi.org/10.1590/1982-7849rac2018170282
  » http://www.scielo.br/pdf/rac/v22n6/1982-7849-rac-22-06-0859.pdf
• Friedman, D., & Sunder, S. (1994). Experimental methods: A primer for economists New York, NY: Cambridge University Press.
• Gandrud, C. (2018). Reproducible research with R and R Studio (2nd ed.). NY: Chapman and Hall/CRC.
• Goffin, K., Åhlström, P., Bianchi, M., & Richtnér, A. (2019). Perspective: State‐of‐the‐art: The quality of case study research in innovation management. Journal of Product Innovation Management, 36(5), 586-615. https://doi.org/10.1111/jpim.12492
  » https://doi.org/10.1111/jpim.12492
• Hoorani, B. H., Nair, L. B., & Gibbert, M. (2019). Designing for impact: The effect of rigor and case study design on citations of qualitative case studies in management. Scientometrics, 1-22. https://doi.org/10.1007/s11192-019-03178-w
  » https://doi.org/10.1007/s11192-019-03178-w
• Janz, N. (2015). Bringing the gold standard into the classroom: Replication in University Teaching. International Studies Perspectives, 17(4), 1-16. https://doi.org/10.1111/insp.12104
  » https://doi.org/10.1111/insp.12104
• Kandori, M. (2018). Replicability of experimental data and credibility of economic theory. Japanese Economic Review, 69(1), 4-25. https://doi.org/10.1111/jere.12175
  » https://doi.org/10.1111/jere.12175
• Lundy, P. (1996). Limitations of quantitative research in the study of structural adjustment. Social Science & Medicine, 42(3), 313-324. https://doi.org/10.1016/0277-9536(95)00153-0
  » https://doi.org/10.1016/0277-9536(95)00153-0
• Makridakis, S., Assimakopoulos, V., & Spiliotis, E. (2018). Objectivity, reproducibility and replicability in forecasting research. International Journal of Forecasting, 34(4), 835-838. https://doi.org/10.1016/j.ijforecast.2018.05.001
  » https://doi.org/10.1016/j.ijforecast.2018.05.001
• Marques, F. (2019). Mecanismos de autocorreção da ciência. Pesquisa FAPESP, 280, 8-10. Recuperado de https://revistapesquisa.fapesp.br/2019/06/07/mecanismos-de-autocorrecao-da-ciencia/
  » https://revistapesquisa.fapesp.br/2019/06/07/mecanismos-de-autocorrecao-da-ciencia/
• Mendes-Da-Silva, W. (Ed.). (2018). Editorial: Promoção de transparência e impacto da pesquisa em negócios. Revista de Administração Contemporânea, 22(4), 639-649. Recuperado de http://www.scielo.br/pdf/rac/v22n4/1982-7849-rac-22-4-1982-7849rac2018180210.pdf https://doi.org/10.1590/1982-7849rac2018180210
  » http://www.scielo.br/pdf/rac/v22n4/1982-7849-rac-22-4-1982-7849rac2018180210.pdf
• Mendes-Da-Silva, W. (2019). Open data articles of Journal of Contemporary Administration (Jul2018-Jul2019) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.3345934
  » http://doi.org/10.5281/zenodo.3345934
• Moravcsik, A. (2014). Transparency: The revolution in qualitative research. PS: Political Science & Politics, 47(1), 48-53. https://doi.org/10.1017/S1049096513001789
  » https://doi.org/10.1017/S1049096513001789
• Nature. (2016). Editorial: Reality check on reproducibility. Nature, 533, 437, https://doi.org/10.1038/533437a
  » https://doi.org/10.1038/533437a
• Plakoyiannaki, E., Wei, T., & Prashantham, S. (2019). Rethinking qualitative scholarship in emerging markets: Researching, theorizing, and reporting. Management and Organization Review, 15(2), 217-234. https://doi.org/10.1017/mor.2019.27
  » https://doi.org/10.1017/mor.2019.27
• Poirier, D. J. (1988). Causal relationships and replicability. Journal of Econometrics, 39(1/2), 213-234. https://doi.org/10.1016/0304-4076(88)90046-2
  » https://doi.org/10.1016/0304-4076(88)90046-2
• Stark, P. B. (2018). Before reproducibility must come preproducibility. Nature, 557, 613. https://doi.org/10.1038/d41586-018-05256-0
  » https://doi.org/10.1038/d41586-018-05256-0
• Tsang, E. W., & Kwan, K. M. (1999). Replication and theory development in organizational science: A critical realist perspective. Academy of Management Review, 24(4), 759-780. https://doi.org/10.2307/259353
  » https://doi.org/10.2307/259353

Publication Dates

• Publication in this collection
  28 Oct 2019
• Date of issue
  Sep-Oct 2019