ENSAIO PESQUISA EM EDUCAÇÃO EM CIÊNCIAS EXPERIENCES WITH OPEN PEER REVIEW

INITIAL CLARIFICATIONS

In 2021, we announced to the academic community the introduction of new editorial practices at Ensaio Pesquisa em Educação em Ciências that are in line with the open science movement (Mendonça & Franco, 2021). Our consonance with the open science paradigm is justified by the social character of scientific knowledge production, both because the progress of science depends on the access to the reservoir of knowledge (journals, congress annals, books, etc.), and because academic works acquire scientific status when they are peer reviewed. The scrutiny of scientific productions is crucial to the development of knowledge because it is based on shared and public criteria that determine the acceptance or rejection of certain studies, preventing their biased disqualification through the use of arbitrary standards.

It is the access to knowledge, as well as the fair and transparent communication and appraisal of the processes and results of peer research that keeps researchers constantly exchanging, producing, and developing science. In this sense, we advocate that science be developed and communicated effectively, allowing for other people to contribute and collaborate in research efforts (Albagli, Clinio & Raytoch, 2014; Longino, 2002).

Despite this, we recognize that a full understanding of the processes involving the life of a journal (e.g., submission, review, editing of scientific articles) tends to be inaccessible to most authors and even to reviewers and editors. It is uncommon for reviewers to have access to the review of another reviewer, which could contribute to mutual training and more intense dialogue in the field. Situations in which editors share information about the internal and widely varied review processes that take place in their journals are also rare. Lastly, authors usually only have access to a system through which they submit their manuscripts and an email with the result of the review. With the goal of opening up this black box, we got ourselves involved with open science processes and have been reflecting about them in our editorials since 2021.

In this editorial, we aim to present the journal’s modus operandi in accord with open science editorial practices, seeking to evaluate the experiences lived in the last two years and outlining perspectives for their advancement. We outline the criteria that inform open peer review (henceforth OPR) and the preparation of opinion articles that resulted from the experiences of 2021 and 2022. At the end of the text, we present the future challenges in the practice of sharing supplementary research data, which began to be implemented at the end of 2022 and which we intend to expand in 2023.

In order to seek the best strategies for developing practices aligned with the ethos of open science, our starting point was to build the editorial team itself. The team is composed of newly graduated PhDs who work in the technical review of the articles, forwarding them to the associate editor and editor-in-chief. The associate editor oversees the publication process in collaboration with the editor-in-chief. In the end, the editor-in-chief finalizes the process and sends the accepted articles to publication.

The associate editor and editor-in-chief met biweekly in 2021 and 2022 to address the following topics relating to open science: open peer review, sharing of supplementary research data, preprints, authorship and collaboration criteria, ¹criteria for analysis of empirical and theoretical works, characteristics of constructive reviews, inclusive language in scientific reporting and scientific editing. Debates were established based on the reading and discussion of literature, exchanges with other scientific editorial teams, and participation in seminars promoted by scientific organizations including, among others, SciELO Brasil (Scientific Electronic Library Online) and ABEC Brasil (Brazilian Association of Scientific Editors).

LESSONS LEARNED ABOUT OPEN REVIEW AT ENSAIO

We identified in our studies that open peer review is a term that encompasses several peer review models aligned with the open science agenda. Briefly, opening up the review consists of disclosing, at any stage of the scientific review/communication and with different levels of access to the public, the identities and arguments of authors and reviewers. The main OPR models identified were (see Shintaku, Brito, Ferreira Jr., Barraviera, 2020; Pedri & Araújo, 2021; Ross-Hellauer & Görögh, 2019; Ross-Hellauer, Deppe, & Schmidt, 2017):

1. Open identities: authors and reviewers are aware of each other’s identity, which can be disclosed at the beginning or at the end of the review process.

2. Open reports: review reports are published alongside the articles and may or may not be signed by reviewers. Reviews may be published in the form originally prepared by reviewers. In addition, the rounds of review exchanges between reviewers and authors can be made publicly available on the journal’s website to provide greater transparency to the review process. Or still, review reports can support reviewers in the creation of an article that dialogues with the reviewed manuscript.

3. Open participation and open pre-review manuscripts: the wider community is able to contribute to the peer review process. It consists of a form of hybrid review, in which the manuscript, after an initial review by the editor-in-chief, is made available in its initial version on the journal’s ‘discussion’ website. The manuscript is openly accessible to the scientific community for commenting and to reviewers for evaluation of acceptance or rejection.

4. Open final-version commenting: the wider community can review or comment, publicly, on the journal’s website, on final ‘version of record’ publications.

5. Portable: reviews are shared among scientific journals, taking into account the work already done and the scope of publications most suitable to certain journals.

Ensaio has adopted the first and second OPR models described above. The first point to be considered in opening up identities in the journal’s editorial process is the authorship agreement regarding the disclosure and the interaction with reviewers. When submitting the manuscript for review, authors send a letter to the journal, responding to a few items regarding open science practices, among them the interest in disclosing their identity if the manuscript were accepted. This is, then, the primary condition for breaking the double-blind process/reviewing policy/principle at Ensaio. In addition, we chose to disclose author and reviewer identities at the end of the review process, which seemed more coherent to us as it provides a gradual introduction to OPR. It is worth noting that only manuscripts that were accepted and whose reviewers prepared constructive reviews (discussed below) are invited to publish the opinion article. These stages of the review process are described in the flowchart below (Figure 1):

Figure 1
Flowchart describing the OPR process at Ensaio

We understand that the double-blind model is the most commonly used form of peer review in the sciences in general and in Science Education in particular. In no way are we questioning the quality of many closed review processes. The literature shows that this methodology has several advantages and also points out the weaknesses of this type of review process (for further details see Mendonça & Franco, 2021). Ideally, there should be an equal distribution of intellectual authority in scientific communities, so that the position regarding research, dialogue, and constructive criticism, which all aim to improve research propositions, are central elements in the production and communication of scientific knowledge and not just the authority status of a particular researcher or research group. On the contrary, authors and reviewers generally prefer not to reveal their identities due to interpersonal and political conflicts (Shintaku, Brito, Ferreira Jr., Barraviera, 2020; Pedri & Araújo, 2021; Ross-Hellauer & Görögh, 2019, Ross-Hellauer, Deppe & Schmidt, 2017).

The main reason for maintaining the anonymity of authors is to avoid publication biases such as authority in the field, gender, and authorship from prestigious institutions. The anonymity of reviewers is protected, allowing them to review the article honestly without running the risk of suffering retaliation from the authors (Shintaku, Brito, Ferreira Jr., Barraviera, 2020; Pedri & Araújo, 2021; Ross-Hellauer & Görögh, 2019, Ross-Hellauer, Deppe, & Schmidt, 2017). Because of such controversies, we chose caution in opening identities, which is done at the end of the review process, except for rejected articles, considering the constraints involved as well as the lack of experience and familiarity of our field with this process.

The text of the opinion article should be based on the reviewer’s own review report, the authors’ response letter, and the original article accepted for publication. In this text, it is important to revisit the main points highlighted in the analysis of the manuscript under review and the way with which the authors dealt with such aspects in the response letter and in the original accepted article, that is, details of the review process are presented. Reviewers specializing in the themes addressed in the original article reviewed must also advance the discussion of the relationship between the original article and the field of knowledge from a certain perspective in order to make an original contribution of theoretical or methodological nature to the field of Science Education research and/or teaching.

With OPR, the Ensaio journal has sought to value reviewers more highly by inviting them to publish an article based on their review and providing greater transparency to the review process (because of the nature of the opinion article’s text) and to encourage self-training of reviewers due to the opportunity afforded to them to understand aspects of the reviews contributed by experienced reviewers, given that the social practices of scientific communities are learned from participation. Moreover, Varas Espinoza (2015) considers that the involvement of reviewers with the journal (i.e., participation both as a reviewer and as an author publishing in the journal) is an important factor for producing good reviews, directly affecting the quality of published research.

SOME CASES OF OPEN REVIEWS PUBLISHED IN ENSAIO

Throughout 2021 and 2022, we had the opportunity to publish seven opinion articles under the “Perspectives” section of Ensaio. All were the result of a collaborative process along which we learned and reflected together.

In our journal, this type of publication starts from an agreement between the persons involved in the publication of the articles: authors, who agree to the publication of opinion articles; reviewers; and editorial staff. Through a meeting with the editors, we presented to the invited reviewers the proposal for the creation of an opinion article as well as the criteria for preparing the text (as outlined in this editorial).

Authors participate less in this process, because the decision on whether to invite reviewers to produce opinion articles lies with the editors. In conversations with the authors of the accepted articles, we ensured them that the opinion article is built upon a fresh look at the original review and, even when they contributed substantive changes to the manuscripts, there is no emphasis on comparing the versions of the article that were reviewed, but rather on its final presentation and, even more, on the themes it provokes for reflection. We understand, however, that the process has also been productive and formative for authors, who have the chance to see their text be read and commented on with the publication of the opinion article, which greatly expands the dissemination and debate around their ideas.

In our experiences, we had three cases in which the opinion article was prepared jointly by two reviewers, namely the opinion articles The instant to see, the time to understand and the moment to conclude: notes on Psychoanalysis and Science Education, written by Wilson Elmer Nascimento and Alberto Villani and published in 2021; The subjective and the operational in science learning: an articulation between types of learning and types of content, written by Amanda Medeiros and Daniel Goulart, published in 2022; and Conscious/non-conscious dialectic in conception of the world: theoretical, methodological and practical implications for the teaching of nature sciences from a historical-critical perspective, written by Hélio da Silva Messeder Neto and Júlia Mazinini Rosa and published in 2022. In the others, only one of the reviewers was invited and agreed to produce a text with the aforementioned characteristics.

In general, responses from reviewers have been very positive; these are constructive reviews with expanded dialogues that in no way criticize the work of the authors in a pejorative way. On the contrary, they expand the boundaries of their texts to other important themes in the field. Reviewers have the opportunity to reflect both on their performance as evaluators and on the subject matter of the article, since they are experts in the theoretical and/or methodological perspectives of the studies who have the chance to develop hypotheses and arguments with their text in dialogue with the original article.

One example that illustrates these experiences is the opinion article Reflections on student engagement in Emergency Remote Learning, by Tobias Espinosa. This was the first opinion article published in our journal (Espinosa, 2021). The text resulted from a dialogue born of the review of the article Student engagement in a remote and emergency teaching of Physics, also published in 2021, by Helder de Figueiredo e Paula, Sérgio Luiz Talim, Cecília Siman Salema, and Vinícius Reis Camillo (Paula et al., 2021). This opinion article was significant for the editors because it reflected the entire formative process in preparation for an experience with open reviews and, at the same time, was able to achieve the goals we expected for that moment of our journal.

The original review brought contributions to the authors, who advanced in their analyses and theoretical constructions. In the opinion article, in turn, the reviewer discussed two aspects: i) different theoretical perspectives and indicators of the engagement construct; and ii) factors that may influence student engagement in Emergency Remote Learning. Throughout the text, the reviewer revisited proposals significant to the field, sought dialogues among different theories and methodologies, advancing those contributions already made in the article by Paula et al. (2021). Another important aspect, as we expected from the publication of our opinion articles, is that Espinosa’s text was based on the most relevant elements of his original review, which in addition to providing a good contribution to the authors, were expanded upon and publicized to other readers in the field. Throughout 2021 and 2022, other opinion articles were published, each emphasizing certain aspects and without following a pre-established formula or script. In all cases, we have witnessed the production of high-quality texts that have been positively appraised by the field.

LESSONS LEARNED ABOUT CONSTRUCTIVE REVIEWS

Peer evaluation or review is a key practice in the process of publishing articles in scientific journals. Backed by these reviews, we make the final editorial decision to accept or reject a manuscript and, in these two years of organizing our editorial work around the open peer review process, they are the ones that provide guidance to the associate editors in inviting reviewers to publish the opinion articles.

At various times we asked ourselves: What constitutes a quality review? What elements should be considered in preparing good reviews? In this section we discuss what our editorial team has been thinking about these issues.

Regardless of the editorial decision, a good review should contribute to advancing the quality of an article; thus, a question that should guide the reviewer during the review process is: “How could the article be better? Therefore, the review must be written in a constructive way, avoiding hostility” (Hohendorff, 2021, p.2). The same author points out that the best way for a review to be constructive is by indicating suggestions for modifications in the form of action, that is, stating clearly what must be done. The directness of the reviewers’ comments and corrections is also highlighted by Varas Espinoza (2015) as a critical aspect in producing a constructive review. This means that, instead of just saying that a paragraph, argument or idea needs improving, it is important that we indicate more specifically the aspects that need to be considered by the authors for restructuring the text.

In addition to directness and clarity, van Rooyen, Black and Godlee (1999) emphasize that reviewer’s suggestions and corrections must be substantiated by evidence-based explanations and describe other aspects that are key to structuring the text of the review report: discussing the structural aspects of the article; outlining the relevance and originality of the research; problematizing the theoretical, methodological, and analytical aspects of the study and discussion of study results. The modifications that we made to Ensaio’s review forms lead reviewers to contemplate the aspects outlined by van Rooyen, Black and Godlee (1999) in their reviews; they were also expanded to incorporate reflections on the implications of the study for research in science education and the ethical precautions of research.

Even though time is a factor that affects the circulation of knowledge and journals’ reviews, Cabezas Del Fierro and colleagues (2018) consider that this variable and the size of the text do not determine the quality of a review. From our studies, we understand that a good review must be constructive. For a review to have this characteristic, the comments and suggestions of reviewers must be made with clarity and to a good extent directed towards the structural aspects of the article and the central elements of scientific research.

FROM OPR TO OPEN DATA: OUR PERSPECTIVES

In the letter addressed to Ensaio by the authors, there is information about the sharing of data that are supplementary to those already presented in the body of the manuscript under review (e.g., raw data organized into .csv files for statistical analysis, interview transcripts) ensuring the anonymity of research participants. Exceptions are allowed in cases of legal and ethical issues and must be disclosed by the authors as a limitation to making the database available. The goal is to facilitate the review of the manuscript and, in case it is accepted, contribute to the preservation and reuse of content and the reproducibility of research (SciELO, 2018).

We would like to clarify to the community that supplementary data are understood as primary sources, which must be organized, systematized, and registered in data libraries, being partially or fully accessible (contingent on ethical and legal issues) to the wider community. Open data refers to textual or other materials, including products and/or components of ongoing or completed research which are made openly available through DOI licenses that allow users to download, copy, analyze, and reprocess the data. The sharing of databases allows for research to be redone and have its reproducibility evaluated, verifying whether its results are compatible with the data and sources of evidence used to reach the conclusions set forth in the article, errors to be found or even people to arrive at new findings (data reuse) (Clinio & Albagli, 2017; Albagli, Clinio & Raytoch, 2014).

Ensaio currently has a data repository (SciELO Dataverse) available from https://data.scielo.org/dataverse/brepec. The SciELO repository is multidisciplinary and aims at depositing, preserving, and disseminating research data from manuscripts submitted or accepted for publication or that were published in SciELO portfolio journals. SciELO journals operate jointly with the SciELO repository for indexing, preservation, and dissemination of data pertaining to the articles they publish, providing authors with a guarantee of stability in the data deposit location.

Ensaio now has an editor serving in its editorial board who is responsible for curating the data deposited in the repository in tandem with the technical analysis conducted by SciELO Data staff. For more details on the process for submitting data to the repository, refer to the documents available from https://scielo.org/pt/sobre-o-scielo/scielo-data-pt/.

In the 2022 issue, we introduced the open data process with the release of anonymized data from the article Three-tier test validation for mapping epistemological density profiles, written by Viviane Florentino de Melo and Amanda Amantes and available from https://data.scielo.org/dataset.xhtml?persistentId=doi:10.48331/scielodata.G7PNYG.

We would like to emphasize that the idea for publishing the data came from one of the reviewers, Professor Cláudio J. Cavalcanti, who keenly followed the implementation of the open science policy in the journal and suggested that the data be made available. Professor Cavalcanti was invited to prepare an opinion article about this process, which will also be published in this 2023 issue. The authors agreed to the reviewer’s suggestion, removing any form of identification of research participants; he revised the data — which alerted us to the need for a data editor in order not to overload the review process — and they published their data on the SciELO platform.

Continuing with this process, we hereby announce to the community that all manuscripts accepted from 2023 onwards will include a section at the end of the text containing one of the following statements on data availability (derived from https://wp.scielo.org/wp-content/uploads/Guia_TOP_pt.pdf) (SciELO, 2018):

1. Data not available: the dataset supporting the results of this study is not publicly available.

2. Data available:

2.1. The entire dataset supporting the results of this study was published in the article itself.

2.2. The entire dataset supporting the results of this study was made available on the SciELO Dataverse and can be accessed at [URL or DOI].

2.3. The entire anonymized dataset supporting the results of this study was made available on the SciELO Dataverse and can be accessed at [URL or DOI].

3. Data available upon request:

3.1. The entire dataset supporting the results of this study is available upon request to the corresponding author.

3.2. The dataset is not publicly available due to [detail reason for restriction; for example, containing information that compromises the privacy of research participants].

3.3. The entire dataset supporting the results of this study is available upon request to [name of organization].

As we come to the end of this editorial, we realize that recounting this process has allowed us to share some of the lessons learned about various aspects regarding the open science practices we have embraced since 2021 and to offer a few perspectives for our work in 2023 in preparing issue 25 of the journal. We take the opportunity to thank the authors who have chosen to share their research in our journal and the reviewers for the invaluable work they have done in creating constructive reviews. We would like to expand this dialogical network and, therefore, we invite the academic community in the field of Science Education to publish and also to review in our journal. We understand, mainly by virtue of our articulation with open science practices, that this connection as author and reviewer results in more consistent review processes, contributing to improving the quality of Ensaio’s publications, which to us seems a promising path for research in Science Education.

REFERÊNCIAS

• Albagli, S., Clinio, A., & Raytoch, S. 2014 Ciência Aberta: correntes interpretativas e tipos de ação. Liinc em Revista, Rio de Janeiro, v.10, n.2, p. 434‒450, novembro 2014.
• Cabezas Del Fierro, P. C. D., Meruane, O. S., Espinoza, G. V., & Herrera, V. G. (2018). Peering into peer review: Good quality reviews of research articles require neither writing too much nor taking too long. Transinformação, 30(2). https://doi.org/10.1590/2318-08892018000200006.
  » https://doi.org/10.1590/2318-08892018000200006
• Clinio, A., Albagli, S. Cadernos abertos de laboratório e publicações líquidas: novas tecnologias literárias para uma ciência aberta. Reciis - Rev Eletron Comun Inf Inov Saúde 2017nov.; 11 (sup) |[www.reciis.icict.fiocruz.br] e-ISSN 1981‒6278.
• Espinosa, Tobias. Reflexões sobre o engajamento de estudantes no Ensino Remoto Emergencial. Ensaio Pesquisa em Educação em Ciências (Belo Horizonte) [online]. 2021, v. 23. https://doi.org/10.1590/1983-21172021230122.
  » https://doi.org/10.1590/1983-21172021230122
• Hohendorff, J. V. (2021). Como Elaborar um Parecer de Artigo Científico? E porque Devemos Ser Revisores. Psicologia: Teoria E Pesquisa, 2021, v.37, e37. https://doi.org/10.1590/0102.3772e370001.
  » https://doi.org/10.1590/0102.3772e370001
• Longino, H. E. (2002). The fate of knowledge Princeton: Princeton University Press.
• Mendonça, P. C. C., Franco, L. G. A ciência aberta e a área de Educação em Ciências: perspectivas e diálogos. Ensaio: Pesquisa em Educação em Ciências (online). v. 23, p. 1‒5, 2021. | http://dx.doi.org/10.1590/1983-21172021230102.
  » https://doi.org/10.1590/1983-21172021230102
• Paula, H. F., Talim, S. L., Salema, C. S., & Lamilo, V. R. (2021). Engajamento de estudantes em um Ensino Remoto e Emergencial de Física. Ensaio Pesquisa em Educação em Ciências, 23. https://doi.org/10.1590/1983-21172021230117.
  » https://doi.org/10.1590/1983-21172021230117
• Pedri, P. & Araújo, R. F. 2021. Vantagens e desvantagens da revisão por pares aberta: consensos e dissensos na literatura. Revista eletrônica de biblioteconomia e ciência da informação, Florianópolis, v. 26, n. esp, p. 01‒18.
• Ross-Hellauer, T., Deppe, A., & Schmidt, B. (2017). Survey on open peer review: attitudes and experience amongst editors, authors and reviewers. PLoS One, 12(12), e0189311.
• Ross-Hellauer, T ., & Görögh, E. (2019). Guidelines for open peer review implementation. Research integrity and Peer Review, 4(4), 1‒12.
• SciELO DATA, Scientific Electronic Library Online (2018). Available online from: https://scielo.org/pt/sobre-o-scielo/scielo-data-pt/
  » https://scielo.org/pt/sobre-o-scielo/scielo-data-pt/
• Shintaku, M., Fagundes de Brito, R., Seabra Ferreira Jr., R., & Barraviera, B. (2020). Avaliação aberta pelos pares no âmbito da Ciência Aberta: revisão e reflexão. BIBLOS, 34(1), 161-175. https://doi.org/10.14295/biblos.v34i1.11189
  » https://doi.org/10.14295/biblos.v34i1.11189
• van Rooyen S., Black N., Godlee F. (1999). Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol Jul; 52(7):625‒9. http://dx.doi.org/10.1016/s0895-4356(99)00047-5.
  » https://doi.org/10.1016/s0895-4356(99)00047-5
• Varas Espinoza, G. (2015). El informe de arbitraje en el proceso de revisión por pares de artículos de investigación: Niveles de retroalimentación según el tipo de evaluador. 68 f. Tesis (Tesis de Magíster en Estudios Latinoamericanos con mención en Lingüística) - Universidad de La Serena, La Serena, Chile.

• 7
  1A discussion that has led to the Ensaio Pesquisa em Educação em Ciências editorial Shall we talk about authorship? by Bizerra and Sá (2022), which can be accessed at https://www.scielo.br/j/epec/a/YFzCCLQZhFp7yYWnkKQN3sq/?lang=pt.

• CECIMIG would like to thank National Council for Scientific and Technological Development (CNPq) and Research Support Foundation of the State of Minas Gerais (FAPEMIG) for the funding for editing this article.

Publication Dates

• Publication in this collection
  27 Mar 2023
• Date of issue
  2023