Research evaluation in national open science policies

ABSTRACT

Introduction:

Open science is increasingly prominent in international science policy agendas, converging towards the understanding that its consolidation as an academic practice depends on the reformulation of the current research evaluation system. Objective: This study aims to identify and analyze to what extent and in what way open science policies incorporate and address research evaluation. Methodology: A bibliographic and documentary research was carried out, enabling the selection of open science policies from five countries - Finland, Slovenia, the Netherlands, France, and Colombia. Results: It was observed that countries with more comprehensive policies recognize the need to reformulate recognition and reward mechanisms to include open science components. Only the French policy presents more concrete indications of changes. Colombia's National Open Science Policy 2022-2031 sets goals and execution deadlines for the adoption of open science in research evaluation and classification of groups and researchers, based on the proposal to review its accreditation systems for programs and institutions. The plans of the Netherlands, Slovenia, and Finland also anticipate changes after examining the current instruments and criteria adopted by them, whether by verifying the feasibility of using altmetric indicators (Netherlands), including new methods (Slovenia), or considering new and variable formats of publication (Finland). Conclusion: There is some limitation in moving from a supportive discursive policy towards open science to a policy with devices that imply concrete changes in the criteria and procedures used in evaluations.

KEYWORDS:
Evaluation; Science policy; Scientific Research; Open Science; Open access.

RESUMO

Introdução:

A ciência aberta tem tido um progressivo destaque nas agendas das políticas científicas internacionais, convergindo-se para o entendimento de que a sua consolidação como prática acadêmica depende da reformulação do atual sistema de avaliação da pesquisa. Objetivo: Identificar e analisar em que medida e de que modo políticas de ciência aberta incorporam e abordam a avaliação da pesquisa Metodologia: Realizou-se pesquisa bibliográfica e documental, que possibilitou estabelecer a seleção de políticas públicas de ciência aberta de cinco países - Finlândia, Eslovênia, Holanda, França e Colômbia. Resultados: Observou-se que os países com políticas consideradas mais abrangentes reconhecem a necessidade de reformulação dos mecanismos de reconhecimento e recompensa para contemplar componentes da ciência aberta. Apenas a Política francesa apresenta indicativos mais concretos de mudanças. A Política Nacional de Ciência Abierta 2022-2031, da Colômbia, indica metas e prazos de execução para a adoção da ciência aberta em processos de avaliação da pesquisa e de classificação de grupos e pesquisadores, a partir da proposta de revisão de seus sistemas de acreditação de programas e instituições. Os planos da Holanda, Eslovênia e Finlândia também preveem mudanças após realizarem exames dos atuais instrumentos e critérios que adotam, seja averiguando a viabilidade do uso de indicadores altmétricos (Holanda), seja incluindo novos métodos (Eslovênia) ou considerando novos e variáveis formatos de publicação (Finlândia). Conclusão: Constatou-se certa limitação em passar de uma política discursiva de apoio à ciência aberta para uma política com dispositivos que impliquem em mudanças concretas nos critérios e procedimentos usados nas avaliações.

PALAVRAS-CHAVE:
Avaliação; Política científica; Pesquisa científica; Ciência aberta; Acesso aberto.

1 INTRODUCTION

Open science has increasingly grown in importance in the agendas of international scientific policies. In the past few years, the number of initiatives and institutional policies aimed at fostering open access to the results of research financed with public resources have increased¹ . In the Registry of Open Access Repositories Mandatory Archiving Policies - ROARMAP alone 1.116 open access policies can be found in connection with universities, research institutions and research funders². More recently, several government regulations and policies of national scope encompassing open science as a broader construct have been formulated, even though an emphasis on publications and research data still prevails.

The denomination open science encompasses distinct approaches (Fecher; Friesike, 2014) and refers to a plurality of practices and initiatives including open access to data and publications as well as the opening of infrastructures (software and hardware), research methodologies and evaluation processes, together with the collaborative engagement of nonscientist individuals and communities, among other components (Albagli, Clinio, Raychtock, 2014).

At the same time, there is a convergence in the understanding that the dissemination and consolidation of open science as an academic practice substantially depends on the reformulation of the current system of research evaluation (O'Carroll et al., 2017; Boukacem- Zeghmouri, 2020; Beigel, 2021).

Criticism of scientific research evaluation systems is not new in Latin America - whether it be connected to recruiting or career advancement systems, to analyses of grant applications or financing projects or to processes of evaluation of departments and graduate programs. Among other aspects the prioritization of the article format as the sole or main vehicle of scientific communication as well as the growing employment of bibliometric indicators in the measurement of the quality of research are questioned, besides the widely spread premise that journals indexed in commercial international bibliographical data bases possess greater quality than those indexed in local-regional open data bases. Besides, the pressure to increase publication strengthens practices of secrecy and competition as well as favoring certain types of research and English as the reference language for publications. As a result, the hegemonic evaluation system is increasingly shaping a monochromatic picture of research work, which tends to deepen already existing asymmetries in what is considered mainstream or peripheric science (Guédon, 2011).

Open science agendas, particularly in their public and democratic perspectives (Albagli, 2015; Clinio, 2019; Chan; Okune; Sambuli, 2015, Albornoz et al., 2018; Albagli; Iwama, 2022), update and introduce new elements into the historic debates regarding the systems of evaluating, acknowledging, and rewarding scientific research. Beyond the new agendas and challenges related to the open access publication of articles and books (Appel; Albagli, 2019) and of the opening of research data (Jorge, 2022), questions arise that problematize criteria of excellence as well as dominant systems of evaluating research and researchers. Among other criteria, the acknowledgement and increased visibility of the contribution to science of other systems of knowledge; the collective and democratic governance of digital infrastructures that support scientific communication as well as the promotion of the multiplicity of languages are highlighted³ (Becerril-García; Aguado-López, 2019; Beigel, 2021; Vommaro; Rovelli, 2022). Considering that the evaluation of research constitutes an unavoidable aspect of science, this paper aims at identifying and analyzing to what extent and how public policies of open science in different countries encompass and approach the component of evaluation of scientific research in their normative frameworks.

2 METHODOLOGY

This is an exploratory, descriptive, and non-exhaustive study based on bibliographical and documental analysis, having as its main sources of information scientific articles, reports of multilateral organizations, documents, and information available on internet sites of government institutions.

The theoretical framework was composed by documents retrieved through searches without period limitation carried out on the phrase “open science policies” on the CAPES Journal Web Portal (Portal Periódicos CAPES) as well as on the data bases Scopus, Web of Science, Library, Information Science & Technology Abstracts (LISTA), Redalyc and SciELO Org, on January 13, 2022, as detailed in Table 1.

Table 1
Data bases, syntax, fields of search applied and recovered documents

Data bases, syntax, fields of search and recovered documents (January 13th 2022)
Data bases	Search syntax	Searched areas	Recovered documents	Types of documents	Language of documents
Capes Journal Web Portal (Portal Periódicos CAPES)	Topic "open science policies"	All fields	250	Articles (204); Reports 22); Groups of data (9); Video recordings (8); Textual resources (3), Images (1), Reviews (1), Conference reports (1), Other (1)	English (245); Portuguese (4), Spanish (4); Japanese (2); Croatian (2), Russian (1); Persian (1), Norwegian (1), Serbian (1)
Scopus	TITLE-ABS-KEY ("open science policies")	Title, summary, key word	32	Articles (20); Conference Paper (9); Reviews (2), Short interview (1)	English (27); Spanish (2); Japanese (1); Persian (1); Portuguese (1)
Web of Science	All Fields ("open science	All Fields	14	Articles (9); Proceedings	English (12); Japanese (1);
	policies")			Papers (4); Review articles (1)	Spanish (1)
Library, Information Science & Technology Abstracts (LIST)	All Fields ("open science policies")	All fields	4	Articles (4)	English (2); Japanese (1); Portuguese (1)
Redalyc	Key word, title, DOI, Whole text ("open science policies")	Article	3	Articles (3)	Portuguese (2)/ Spanish(1)
SciELO Org	All Indexes ("open science policies")	Article	1	Article (1)	English/Spanish (1)

Source: Elaborated by the authors based on data recovered from Capes Journals Web Portal, Scopus, Web of Science, Library, Information Science & Technology Abstracts (LISTA), Redalyc and SciELO Org.

Based on the reading of titles and summaries and after revision to exclude duplicate records, 7 articles, 3 reports, 2 proceedings and 1 book were selected to integrate the theoretical framework connected to the research objectives. Besides these, two other documents not recovered through the search of the data base were included in the body of reference: the UNESCO Recommendation on Open Science⁴ and the report Current tendencies in scientific policies of open science and open access in Ibero-America (Tendencias recientes en las políticas científicas de ciencia abierta y acceso abierto en Iberoamérica), produced by CLACSO in partnership with the Fundación Carolina.

The UNESCO Recommendation on Open Science was integrated into the framework because of its objective of presenting a concept of open science, simultaneously broad and unified, as well as of establishing commitments among signatory countries towards the development of open practices, encouraging them to revisit their research and academic career evaluation systems to align them with principles of open science. In turn, the report Current tendencies in scientific policies of open science and open access in Ibero-America (Tendencias recientes en las políticas científicas de ciencia abierta y acceso abierto en Iberoamérica) was also included because it presents a historical account as well as a quite thorough portrayal of the diversity of actions and experiences in open science in the Ibero-American expanse.

Based on the previous reference table which led to the selection of five countries (Finland, Slovenia, The Netherlands, France, and Colombia) a documentary survey of the regulatory frameworks of open science available on sites of governmental organizations of the respective countries was carried out⁵.

The preliminary survey of the first four countries had as a starting point the report An Analysis of Open Science Policies in Europe, volume 7, carried out by a partnership between the Scholarly Publishing and Academic Resources Coalition (SPARC Europe) and the Digital Curation Center (DCC). In its seventh edition, the study⁶ presents and analyzes fourteen documents of national policies of open data and open science, encompassing national legislation, plans, roadmaps, agreements and research codes of practice belonging to 12 member countries of the European Union (Belgium, Cyprus, the Czech Republic - with two analyzed documents -, Finland, France, Ireland, Lithuania, the Netherlands, Serbia, Slovakia, Slovenia and Spain), as well as 3 documents from countries associated to the EU: Norway, Switzerland and the United Kingdom.

Subsequently, the choice of these countries was corroborated by the study Policies of open science in Europe (Políticas de ciencia abierta en Europa), elaborated by Abadal and Anglada (2021), which considered European global policies presenting a wider perspective on open science and which carried out a concise analysis of the contents of documents of a national political nature. From this perspective, the Abadal and Anglada´s study (2021) considered the recommendations of the European Union, the state plans of Slovenia, Finland, the Netherlands and France, as well as some guidelines and roadmaps from university institutions and libraries such as the League of European Research Universities (LERU) and the Ligue des Bibliothèques Européennes de Recherche (LIBER), which have served as starting points for the elaboration of some initiatives in Latin America (Babini; Rovelli, 2020).

On the other hand, in the selection of the five countries, the report elaborated by Babini and Rovelli (2020) investigating initiatives connected to open science in nine countries - Recent Tendencies in scientific policies of open access and open access in Ibero-America (Tendencias recientes en las políticas científicas de ciencia abierta y acceso abierto en Iberoamérica) - was considered. Nevertheless, from this group, only Colombia was selected for this paper for the following reasons.

The Babini and Rovelli (2020) report brought together and analyzed guidelines, programs, normative apparatus and legal frameworks from 9 Ibero-American countries, chosen as a result of their active participation in the construction of initiatives regarding the opening of scientific knowledge: Argentina, Brazil, Chile, Colombia, Costa Rica, Cuba, Spain, Mexico and Peru, of which, at the time of the study, only Spain, Peru, Argentina and Mexico possessed any legislated regulation on open access policies for data and or publications. In Peru and in Argentina policies are compulsorily proposed through national laws, while in Mexico and Spain they are expressed as recommendations.

The analysis by Babini and Rovelli (2020), split into three dimensions - open access, open research data and open science - explored a number of regulations and initiatives at different levels that have been developed since 1990, seeking to demonstrate the achievements and limitations in each of the national scenarios, alerting to the needs of change in current policies of evaluation in order to highlight criticisms of traditional systems of evaluation and of impact indicators. At the end, the report presents the Proposal for a Declaration of Principles (Propuesta de Declaración de Principios), a document integrating the Series for changes in the evaluation of science in Latin America and the Caribbean (Serie para una transformación de la evaluación de la ciencia en América Latina y El Caribe)⁷, of FOLEC/CLACSO (Babini; Rovelli, 2020).

Despite the fact that Babini and Rovelli (2020) highlight the increase in manifestoes, recommendations and declarations that point towards the pressing need of revising systems and processes of evaluation as well as of bibliographic indicators currently in use - such as the San Francisco Declaration on Research Evaluation⁸ (DORA) and the Leiden Manifest on Research Metric⁹ - the national policies of the nine Ibero-American countries analyzed by them do not incorporate the evaluation component into their structures¹⁰, even though Spanish policies acknowledge the need to introduce improvements in technical-scientific evaluation in order to take into account open science and propose that the country should move forward in this direction¹¹. Above all, such actions deal with the development of repositories of data and publications, of products and services favoring open access to citizenship science, of technological infrastructure for the management, deposit, and visualization of scientific production, of actions aimed at the sensitization of researchers and of gathering around copyrights, among others (Babini; Rovelli, 2020).

Taking into account these aspects, after the identification of the group of countries with national policies-plans for open science identified in the studies surveyed, we opted for selecting for our corpus only those countries with broader and or better consolidated policies, possessing documents defining open science in a global way (and not focusing on open access to publications or on the implementation of repositories for data and or articles in isolation) and that capable of acting upon the CT&I systems and of leading public discourse and practices of open science at the national level.

Based on these principles, the selection followed that of European government policies¹² carried out by Abadal and Anglada (2021), choosing as a result the open science national plans of Finland, Slovenia, the Netherlands, and France. To this group, we added the National Policy for Open Science 2022 - 2031 (Política Nacional de Ciencia Abierta 2022-2031) for Colombia, published in August 2022, the first national policy for open Science in Latin America which displays the same characteristic of being a broad public policy, guiding actions and strategies in this area, and bringing together open access, open data and other opening practices in a thorough document.

Nevertheless, both the Abadal and Anglada (2021) study and the SPARC report (2021) do not consider the research evaluation dimension. Therefore, even though both considered the national policies of Finland, Slovenia, the Netherlands, and France, we propose a different framework.

The focus on the evaluation dimension is due to the fact that, even though it is considered crucial to the strengthening of open science, the mechanisms of acknowledgement and reward still adopt models and procedures of evaluating scientific activity that privilege the use of rankings and bibliometric indicators such as the Impact Factor (Clarivate Analytics) and the CiteScore, indexing on data bases such as the Web of Science (WoS) and Scopus (Elsevier), the journal article format (Salatino; López Ruiz, 2021) and English as the “language of science” (Sánchez-Tarragó, 2015). They do not take into account the variety of open practices, multilingualism, bibliodiversity, inclusive research, the different formats of generating knowledge such as the work in creating, cleaning, managing and curating data, protocols and open codes, citizen science, the dialogue with other knowledge systems. They do not consider plural characteristics and approaches as well as the varied experiences in cooperation, cocreation and sharing proposed by open science.

3 RESULTS

3.1 Review of the literature: the traditional model of evaluating research as the major obstacle for open science

Criticisms aimed at the use of quantitative indicators and at the employment of homogenizing criteria in science evaluation have been around for a long time. As described by Davyt and Velho (2000), research evaluation goes back to the 17th century, coinciding with the dawn of scientific knowledge in its modern format.

In the 20th century, at the end of World War II, governmental organizations were interested in measuring scientific activity, mainly because of changes in the way science was perceived (Velho, 2011). This new interest was reinforced by the birth of the Sociology of Science, formulated by Robert Merton, as well as by the development of indicator methodologies for science and technology (S &T). In particular, the 1960s were a turning point in the use of these instruments, with Derek de Solla Price´s conception of the “science of science”, which constituted a new area of investigation with an emphasis on Scientometrics (Davyt; Velho, 2000). The development of Scientometrics was driven forward by the creation in 1963 of the Science Citation Index¹³ (SCI), as well as of quantitative techniques and methods proposed by Eugène Garfield, also the founder of the Institute for Scientific Information (ISI) (Davyt; Velho, 2000).

From that moment on, despite frequent criticism and challenges (Davyt; Velho, 2000; Vessuri; Guédon; Cetto, 2013; Gingras, 2014; Aksnes; Langfeldt; Wouters, 2019), the emphasis in the use and production of new indicators to measure research activities has increased, motivated among others by: i) the computerization of data bases for the storage and retrieval of scientific information; ii) by the facilitation of evaluation processes by these instruments as they are seen to supposedly possess objective, neutral and transparent characteristics, plus the capacity of speeding up analyzes and of “bringing to light” the quality of the research, and, therefore, of subsidizing S&T policies¹⁴; and finally, iii) by developments in the managerial control and competitive logic arising from neo-liberal agendas that have spread out in public administration from the 1970s. (Mingers; Leydesdorff, 2015).

According to Beigel (2021), debates and critical reflections on the pathways of academic evaluation as well as studies and propositions of researchers and activists of open science movements followed different trajectories. However, in recent years, they have become increasingly intertwined. The author argues that the increased importance of indicators such as the Impact Factor¹⁵ in the accreditation of researchers and institutions has increasingly led to a scientific system that is “more hierarchical and commodified and which, far from opening up the world conversation on science, has contributed to its progressive shutting down”¹⁶ (Beigel, 2021, p. 229).

Paradoxically, at the same time that the right to science and the urgency of opening up scientific knowledge as well as of making it collaborative have been integrated into the priority agendas of different multilateral organisms - such as the Unesco, the World Trade Organization (OTW), the Organization for Economic Co-operation and Development (OECD) among others (Cueva; Méndez, 2022) -, the current research evaluation system has re-enforced values and behaviors in dissonance with principles of a more open, transparent, participative, plural, equitable and collaborative science.

In 2015, the OECD published the document Making Open Science a Reality, in which it analyzed the progress of member countries in opening results of research financed by public resources (OECD, 2015). The report called attention to the need of creating incentive mechanisms which should take into account not only the financial support for the so-called Article Processing Charges (APC) or to costs associated with the sharing of data but should also promote behaviors leading to the opening of science,

[...] including for example financial support to open science efforts, acknowledgment and reward of researchers undertaking open science actions, or the use of new and broader evaluation metrics that consider open science and its impact¹⁷ (OCDE, 2015, p. 89).

In Brazil, the report Green Book: Open Science and Open Data: mapping out of policies, infrastructures and strategies from a national and an international perspective (Livro Verde: Ciência Aberta e Dados Abertos: mapeamento de políticas, infraestruturas e estratégias em perspectiva nacional de internacional) published by the Oswaldo Cruz Foundation (Fundação Oswaldo Cruz - Fiocruz) in 2017, with a focus on the opening of research data, reported on the experience of the European Union and of other eight countries, establishing that the critical aspects for the implementation of open science policies and actions require a range of strategies, encompassing from the “need for new metrics in the evaluation of science, the development of technological infrastructure, [as well as] the definition of a legal framework, among others” (Santos; Almeida; Henning, 2017, p.19).

At the Latin America regional level, the Latin American Forum on Scientific Evaluation (Foro Latinoamericano sobre Evaluación Científica - FOLEC) was created in 2019 with the objective of debating and formulating proposals for the transformation of research evaluative practices in the region. From then on, Folec has elaborated a series of guidelines aimed at sensitizing researchers, institutions, and governments to the need of re-thinking current mechanisms of research evaluation taking into account approaches that are more plural, participative and local, as well as to aligning evaluation systems to open science practices (FOLEC, 2020).

In 2021, the UNESCO Recommendation for Open Science was published, to guide its 195 member States into adopting several strategies and measures with a view to fostering principles and practices of open science, stressing the need to promote a revision of systems of evaluating research and researchers in order to align them with principles of open science (Unesco, 2021).

Among the recommendations concerning research evaluation, Unesco states that member States must engage in the removal of barriers to open science, particularly those regarding career evaluation and reward systems. To this end, they must promote the development and implementation of evaluation systems that (Unesco, 2021, p.29):

• be based on existing efforts to improve ways of evaluating scientific results such as the San Francisco Declaration on Research Evaluation of 2012, with a greater focus on the quality of the results rather than on their quantity, and by the adequate use of diverse indicators and processes which might do away with metrics based on journals, such as the journal s impact factor.

• value all relevant scientific research activities and results, including high quality FAIR data and metadata, well documented and re-usable software, protocols and workflows, machine reading results summaries, as well as the teaching, reaching and engagement of social actors.

• consider evidence of the impact of the research and of the exchange of knowledge such as the broadening of the participation in the process of research, the influence on policies and practices, and the involvement in open innovation with non-academic partners.

• consider the fact that the diversity of subjects demands different approaches to open science.

• consider the fact that the evaluation of researchers in relation to the criteria of open must be adequate to the different stages in the career, with a special attention for those still at the beginning.

In July 2022 the Agreement on Reforming Research Assessment¹⁸ was launched by the European Commission, by the European Universities Association (EUA) and by Science Europe, signed by over 500 organizations - encompassing universities, research institutions, scientific societies as well as evaluation and funding agencies - which made the commitment of implementing a series of actions including, among others: giving up the inappropriate use of indicators such as the Impact Factor and the H-Index, the acknowledgment of the diversity of production in different languages, as well as abilities and practices of opening and collaboration (Agreement On Reforming Research Assessment, 2022).

In December 2022, the Agreement was institutionalized in the Coalition for Advancing Research Assessment - CoARA. It is a multiorganizational and multilateral enterprise with principles, targets, and an action schedule (a five-year deadline from the signing of the Agreement) to push forward changes in evaluation systems through the implementation of new criteria, tools and evaluation processes that acknowledge and reward the diversity of research results, activities and practices leading to an increase in quality and impact of science (Agreement on Reforming Research Assessment, 2022).

3.2 Documental Analysis: national policies

Based on the documental analysis of the set of open science national policies of Finland, Slovenia, the Netherlands, France, and Colombia, one can observe the still incipient design of new criteria and requirements of incentive and reward that seek to contemplate the open science agendas.

Finland

As early as 2014, Finland launched the Open Science and Research Initiative 2014 - 2017 coordinated by the Finnish Ministry of Education and Culture in cooperation with other ministries, universities, institutes, and research funders. The Initiative included the elaboration of a manual for open science, covering the mapping and evaluation of actions already developed by research institutions and financing organisms in the country with the aim of promoting open science and the adoption of relevant requirements and practices to this end (Finland, 2014).

Afterwards, Finland published its Declaration for Open Science and Research - 2020 - 2025) setting up other policies for open science in connection with these guidelines. These policies consider specific components such as: i) policy for open academia; ii) policy for open data and research methods; iii) policy for open access to academic publications; iv) policy for open education and educational resources. Its policy of open access to academic publications establishes that evaluation will consider new and changing publication formats; it also mentions that it will carry out regular revisions to discuss and indicate how new formats of publications will be considered in the evaluation systems of researchers and institutions.

The Declaration also determines that the work of managing and opening research data be acknowledged and considered in the evaluation of researchers. It recommends that monitoring of this work be carried out through a self-evaluation instrument to be developed by the Finish Coordination for Open Science (Finland, 2021).

Table 2
Summary of the key aspects of the evaluation dimension in the contexto of Finland s national policy for open science.

Survey of the dimension of research evaluation within the scope of Finish National Policy for Open Science
Country	Name of the Policy	Year of publication/ Related documents	Central idea of the document	Evaluation dimension (mechanisms of acknowledgement and reward)
Finland	Open Science and Research Initiative 2014-2017 Policies of Open Science and Research in Finland 2020- 2025	(2020-2025) Policy for Open Scholarships	Citizen Science and business cooperation	1) Refers to the document "Good practice in researcher evaluation. Recommendation for the responsible evaluation of a researcher in Finland", which states that the activities of researchers aimed at promoting open access will integrate the process of evaluation. (p. 7).
		(2020-2025) Policy for Open Access to Scholarly Publications	Open Access to articles in journals and conferences	1) Evaluation will consider the new and changing publication formats. It will carry out regular revisions as to how new formats of publication are considered in the evaluation systems of researchers and institutions. Funding agencies and other research funders must include open access as a criterion in their evaluation processes (p.9).
			Open Access of dissertations and academic papers.	At the planning stage* (access on December 6,2022)
		(2021-2025) Policy for Open Access to Research Data and Methods	Open Access for Research Data (FAIR)	1) The management and opening of research data will be considered in the work of the researcher. The monitoring will be carried out through a self-evaluation tool to be developed by Finish Coordination for Open Science as part of the national monitoring process (p. 11).
			Open Access to Research Methods	At the planning stage* (access on December 6, 2022)
		(2021-2025) Policy for the Open Education and Educational Resources	Open Access to Educational Resourcess	1) The work of elaborating REA will be evaluated according to the criteria of merit and in the planning of functions. The Open Science Coordination is planning to develop indicators and a knowledge base to subsidize merit evaluation for this item. (p.15).
			Open Educational Practices	At the planning stage* (access o December 6, 2022)

Source: Elaboration by the authors based on open science plans and policies - Finland.

Slovenia

Slovenia was one of the first European countries to develop a National Strategy of Open Access to Scientific Publications and Research Data in Slovenia 2015 - 2020 in line with the guidelines of the Horizon Europe 2020 project. The name of the policy does not make direct reference to the term open science which, at the time, was not widespread, but it refers to publications and research data within the same conceptual context (Abadal; Anglada, 2021).

The Slovenian strategy was preceded by an action plan and a pilot for the opening of research data. Its components encompass recommendations for: i) open access to scientific publications - with rules for deposit and accepted embargo periods and ii) open access to data - with the compulsory deposit of a set of metadata which describe the deposited research data. Besides articles and academic papers, all Slovenian journals that benefit from public resources must be in the open access format and be included in the Directory of Open Access Journals (DOAJ). Slovenian policies also determine that the evaluation of researchers, research organizations, programs and projects must encourage open access to scientific information in the shape of publications and research data, including within its scope new methods of evaluating science (Slovenia, 2015).

The Action Plan for the implementation of the National Strategy points out the need of setting up an evaluation system within which sets of research data be acknowledged as scientific publication based on achievement of quality criteria (Slovenije, 2017). Also, the Slovenian Research Agency integrates the cOAlition S and began to include the Plan S devices¹⁹ in all its new calls for research funding (Sparc, 2021). Nevertheless, there are no signs indicating changes in the system of incentives and rewards in the frameworks of the Slovenian policies analyzed.

Table 3
Synthesis of the key aspects concerning the evaluation dimension within the scope of the open science national policy of Slovenia.

Survey of the research evaluation dimension within the scope of the Slovenian Open Science National Policy
Country	Policy Denomination	Year of publication/ Related documents	Central ideas of the documents	Evaluation Dimension (acknowledgement and reward mechanisms)
Slovenia	National Strategy of Open Access to Scientific Publications and Research Data in Slovenia 2015 - 2020	(2015-2020) National Strategy of Open Access to Scientific Publications and Research Data	Open Access to Scientific Publications and Research Data (besides articles and academic papers, all Slovenian journals receiving public resources must be open access and included in DOAJ)	1) The evaluation of researchers, research organizations, research programs and projects must encourage open access to scientific information in the format of publications and research data. Criteria for the evaluation of science must also include new and relevant methods for the evaluation of science (p.11)
		(2015-2020) Akcijski načrt izvedbe nacionalne strategije odprtega dostopa do znanstvenih objav in raziskovalnih podatkov v sloveniji (Slovenian action plan towards implementing a national strategy of open access to scientific publications and research data)	1) Scientific publications and research data; 2) Scientific journals and academic papers by editors based in Slovenia; 3) National infrastructure for open access to scientific information; 4) Support for researchers; 5) Pilot program of open access to research data; 6) Open access in science evaluation	3) 1) The evaluation of researchers, research organizations, programs and projects must promote open access to scientific information in the shape of publications or research data. Criteria for evaluating Science must include new and appropriate forms of evaluating science. (p.8)2) Aims at establishing an evaluation system of research data. Stored research data that have been approved as relevant will be acknowledged as a scientific publication. Both intermediate and final reports of research programs and projects must include a self-evaluation dimension of the effects of open access publications of peer reviewed scientific articles related to the results of the research. The action plan will devise an analysis of the evaluation of science carried out by national funding agencies as well as by universities and research institutes, with the aim of elaborating a proposal for change towards open science (p.21)

Source: Elaboration by the authors based on Slovenian plans and policies for open science.

The Netherlands

The National Plan Open Science of the Netherlands was published in 2017, through the articulation of different organizations - the Ministry of Education, Culture and Science, national funding agencies (NWO and ZonMw), University Associations (VSNU), the National Library, among others - which, together, formed an effort coalition around the National Programme Open Science (NPOS). The aim was to accelerate the broadening of open access to scientific publications and research data, as well as to adapt evaluation and reward systems to open science objectives.

At this stage, the Netherlands had already progressed in compulsory policies and in initiatives such as GO FAIR²⁰, with demands of open access to publications and the presentation of plans for managing research data which were publicly funded. But the national policy points to the absence of explicit rewards for efforts and practices regarding the opening of science, stating that “in the present evaluation and reward systems the emphasis is often on the number of publications in prestigious journals with a high impact factor, often produced by wellestablished publishers and to which there is no open access, thus maintaining the culture of ‘publish or perish”²¹ (Netherlands, 2017, p. 10).

Besides, the Dutch plan expresses the need to analyze how the country´s instrument for evaluating research, known as the Standard Evaluation Protocol - SEP can foster the transition to open science so that it becomes integrated into the evaluation of researchers and of research projects. To this end, the NPOS Coalition is committed to initiating the changes recommended in the document, examining, to start with, alternative indicators (altmetrics) as a component of the evaluation of researchers. On the other hand, it emphasizes that the indicators to be adopted for these processes must undergo scrutiny by evaluation institutions (Netherlands, 2017, p. 25).

The 2018 The Netherlands’ Plan on Open Science: Open Science Monitor Case Study, a monitoring report ordered by the European Commission, listed the most important barriers to the Dutch ambition of having 100% of its publications in the open access format by the year 2020, among these the absence of incentives and rewards in the evaluation systems. The report indicates that the 2016 request for opening scientific publications and of setting up plans for data management contemplated only the perspective of funding organisms and the stages of communication of research results. To sum up, the monitoring report says that:

Dutch universities do little to none in terms of rewarding researchers for practicing open science. This can be observed from the appointment criteria of tenure track policy across all Dutch universities, which generally include research, teaching, valorisation, and management leadership. For example, Delft University of Technology has the following criteria in the research aspect of their tenure track policy: (1) conducting research, (2) supervising PhD students, (3) acquisition of indirect government funding and contract research and (4) publications. Open science was not included nor integrated into any of the four research elements. At the national level, the Standard Evaluation Protocol (SEP2015-2020) has included relevance to society to be assessed in research evaluation of disciplines. However, the document does not explicitly refer to open science practices. Hence, it is still unclear how individuals would be rewarded for practicing open science and how it could positively impact their career in a concrete manner at Dutch universities²² (Chan; Meijer, 2018, p. 10-11).

By consulting the site for the National Programme Open Science - NPOS, it was possible to observe the systematic publication of updates and of new strategies and initiatives for promoting open science, corroborating the acknowledgment by different studies of the active role played by the Netherlands in the adoption of open science (SPARC, 2021; Abadal; Anglada, 2021; Santos; Almeida; Henning, 2017). It was also possible to establish that the main organizations connected to research such as the Dutch Universities Association (VSNU), the Royal Academy of Arts and Sciences (KNAW) and the Research Council (NWO) produced a document in 2019, directing changes in evaluation processes in Dutch universities with the title Room for everyone’s talent: towards a new balance in the recognition and rewards for academics²³ (Rodrigues, 2022).

Regarding open science, the document states that the modernization of research evaluation systems requires assuring academics that these will acknowledge the adoption of practices of opening scientific work, of new types of production and co-creation that go beyond the traditional model of producing scientific research (Vsnu; Knaw; Nwo; Zonmw, 2019).

Table 4
Synthesis of the key aspects regarding the dimension of evaluation within the scope of the national policy for open science in the Netherlands

Survey of the research evaluation dimension within the scope of the Duthch National Policy for Open Science
Country	Name of the Policy	Year of publication/ Related documents	Central ideas in the documents	Scope of the evaluation (acknowledgement and reward mechanisms)
The Netherlands	National Plan Open Science (2017)	(2022) Open Science 2030 in the Netherlands (2021) Professionalizing Data Stewardship (2021) Collection National Open Science Festival on Zenodo (2020) Kennis en Krachten gebundeld - Citizen Science in NL (2020) NPOS Final report Exploring and optimizing the Dutch data landscape. (2019) Open Access policy for academic books in the Netherlands (2019) Report Transition Costs for Open Science in the Netherlands (2018) NPOS Recommendations Recognition and Rewards (2016) Call for Action on Open Science	Open access to Scientific Publications, Research Data and Adaptation of evaluation and reward systems to Open Science	1) The Funding Agency Netherlands Organization for Scientific Research (NWO) demands open access to publications as well as to plans of data management by researchers receiving its financial aid, and promises to launch studies focusing on Open Science in its evaluation procedures (p.26). The Plan indicates that there is a lack of explicit rewards for efforts and practices focusing on the opening. The Plan hopes to examine how the instrument Standard Evaluation Protocol (SEP) might facilitate the transition to Open Science, considering how to integrate Open Science into the evaluation of researchers and research projects. The NPOS Coalition promises to initiate changes by first considering alternative indicators (altimetric) when evaluating researchers. Indicators employed must be selected and used by evaluated institutes (p.25)

Source: Elaborated by the authors based on the Netherlands plans and policies for open science.

France

In 2018, France published its First Plan for Open Science (Premier Plan National pour la science ouverte). This first plan focused on three components: i) generalizing open access to publications; ii) structuring research data and making them available on open access; iii) integrating the country into a sustainable dynamic of European and international open science, ensuring the long-term sustainability of the ecosystem involving infrastructures and practices of sharing and opening. These components help to: shape the implementation of mechanisms of acknowledgement and reward fostering open practices; reduce the influence of quantitative and favor qualitative evaluation; encourage the adoption of open citation instead of citation in proprietary environments²⁴ that restrict their access; as well as the incentive to bibliodiversity and the development of abilities in open science, particularly in the field of intellectual property and peer review (France, 2018a).

In its Second National Plan for Open Science (Deuxième Plan National pour la science ouverte), launched in 2021, the country moved forwards to encompass the appreciation of the development of open codes in structures of evaluation and the creation of prizes for practices to promote the reuse of research data and the development of free software. The document also mentions the need to reduce the influence of the indicator Impact Factor through an effective measure: the exclusion of references to this indicator and to the H-index from calls for the presentation of projects and from forms for funding application. Finally, the document stresses the need of encouraging research organizations signatory of the DORA declaration into informing their evaluation committees and supporting them in the implementation of the adopted principles (France, 2021).

Table 5
Summary of the main aspects referring to the evaluation dimension within the scope of the national policy for open science in France

Survey of the research evaluation dimension within the scope of The French National Policy for Open Science
Country	Name of the Policy	Year of publication/ Related documents	Central ideas in the documents	Scope of the evaluation (acknowledgement and reward (mechanisms)
France	Premier Plan National pour la Science Ouverte (2018-2020)	(2022) Appel de Paris sur l’évaluation de la recherche (2021) Plan d’action logiciels libres et communs numériques (2021) Passeport pour la Science Ouverte / Guide pratique à l’usage des doctorantes et des doctorants, v.1 et 2 (2020) Déclaration conjointe du réseau des agences de financement françaises en faveur de la science ouverte (2020) Loi de programmation de la recherche (2017) Appel de Jussieu pour la science ouverte et la bibliodiversité (2016) Livre blanc: Une Science ouverte dans une République numérique (CNRS) (2016) Loi pour une République numérique	1) To generalize Open Access to Publications; To structure Research Data and make them available in open access format To integrate into a sustainable dynamic of European and international open science.	1) The evaluation of researchers will aim at reducing the quantitative dimension in favor of a more qualitative evaluation in accordance with the guidelines of the San Francisco Declaration on Research Evaluation (DORA), as well as the Leiden Manifesto, especially considering open citations, supporting the efforts of the Open Citations Initiative (I4OC). (p. 4)
		(2013) Politique science ouverte de l' Agence Nationale de la Recherche (ANR)
	Deuxième Plan National pour la Science Ouverte (2021-2024)		1) To generalize Open Access to Publications, with an emphasis in the Diamond model and on multilingualism. To Structure, Share and Open Research Data (FAIR); 3) To Open and promote source codes produced by research; 4) To transform practices in order to turn open science into the standard of research; 5) To integrate a sustainable dynamic of European and international open science.	1) To create a prize for practices promoting the reuse of research (p.13). To create a prize for teams developing free software (p.17). To increase the valuation of the production of software in the context of the careers of researcher, support personnel and in the evaluation of structures for research (p.18). To reduce the quantitative dimension in favor of a more qualitative approach, taking into account not only publications, but also the plurality of research results, using indicators in a rational way and supporting cooperation and opening over competition and secrecy (p. 21). To reduce the influence of the Impact Factor (IF), starting by the exclusion of any reference to this indicator as well as to the H-index in the texts of calls for projects and application forms. (p.22). To promote the use of narrative curricula to reduce the influence of quantitative in favor of qualitative evaluation (p.22). To engage research organizations signatory of DORA to inform their evaluation committees into supporting the effective implementation of the adopted principles (p.23).

Source: Elaboration by the authors based on plans and policies for open science - France.

Colombia

The national policy of Colombia, published in 2022, presents a thorough diagnostics of open science in the country and raises several issues concerning the dominant research evaluation systems affecting the recognition of the scientific production of the country. Among these issues, one can find the underrepresentation of Colombian journals indexed on data bases such as the Web of Science (WoS) and Scopus (Elsevier), as well as the use of the Impact Factor as a privileged criterium in research evaluation processes. It also mentions that, in the Colombian evaluation system, there are no incentives to promote open practices, as there are no open peer review systems.

With references to the UNESCO Recommendation on Open Science, to the Leiden Manifesto and to the San Francisco Declaration (DORA), Colombian policy encompasses an action plan with objectives, strategic actions and targets, among which a specific objective concerning evaluation: “To set up a system of metrics and incentives that promote, value and acknowledge practices, processes and results of open science in the Colombian scientific community and to integrate it into the models, systems of metrics and incentives of the CT&I activities existing in the country”²⁵ (Colombia, 2022, p. 51).

The policy goal foresees the formulation of a program of tax incentives for businesses, universities, the civil society, as well as the State carry out joint actions of research, innovation and development in line with practices of open science (Colombia, 2022).

Along the same lines, the policy suggests that, as from 2023, evaluation processes acknowledge open science within the scope of their systems of classification which leads to changes in the components integrating the Model for the Measurement of Groups and Researchers of Science (Modelo de Medición de Grupos e Investigadores de Minciencias), a Colombian tool containing the indicators used in the evaluation of science, technology and innovation results. Until 2023, the development and integration of accountable metrics and alternative indicators of open science are planned to value the diversity of products as well as the different processes and the impact generated by opening practices. For 2025, Colombia hopes to revise and to adjust the system of qualified registration as well as the accreditation of programs and institutions so that they start taking into account practices of open science as substantial indicators in self-evaluation processes and plans for self-development. Finally, for 2026, Colombian policy hopes to adopt a specific budget to prioritize the financing of research adopting practices of open science and aiming at promoting the dialogue among different areas of knowledge (Colombia, 2022).

Table 6
Summary of the main aspects referring to the evaluation dimension within the scope of Colombian national policy for open science

Survey of the research evaluation dimension within the scope of the Colombian National Policy for Open Science
Country	Name of the Policy	Year of publication/ Related documents	Central ideas in the document	Scope of the Evaluation (acknowledgment and reward mechanisms)
Colombia	Política Nacional de Ciencia Abierta 2022- 2031	(2022) Política Nacional de Ciencia Abierta 2022-2031	1) Open Science governance. opening and inclusion culture. metrics and incentives. Training and e expertise of participants. Open Science infrastructure.	1) To create a program of tax incentives so that businesses, teaching institutions and the civil society carry out joint actions promoting open science. To acknowledge practices of open science in processes of research evaluation and in the classification of individual researchers and groups by updating the Model for the Measurement of Groups and Researchers of Science (Modelo de Medição de Grupos e Pesquisadores do Minciência) in order to promote criteria of open science. To promote a specific budget for research contemplating practices of open science. To design and implement responsible metrics and alternative indicators that value the diversity of products, of processes as well as the different impacts generated by open
				science practices. 5) To adjust systems of accreditation to contemplate and value open science practices as indicators in processes of self-evaluation and self-development plans (p. 61-62).

Source: Elaboration by the author based on plans and policies for open science in Colombia.

4 DISCUSSION AND CONCLUSION

The review of the literature indicated a broad convergence among researchers of the topic in pointing out that the current system of research evaluation - based mostly on the employment of metrics associated with the number of publications and the number of citations (Impact Factor /CiteScore) - presents a major challenge to the promotion and consolidation of the principles and practices of open science as the dynamic standard for the academicscientific activity (Babini; Rovelli, 2020; Méndez, 2021; Rodrigues, 2022).

Disfunctions generated by this system, based on quantitative productivism, and synthetized by the phrase “publish or perish”, have given rise to several criticisms from part of the scientific community at a global level, particularly in Latin American countries, contributing to the launching of different proposals for reform along the last decade.

Based on the documents analyzed, it was possible to ascertain the growing importance of the issue of the evaluation of research and researchers in the context of open science policies, with more or less specific guidelines, indicating the need to reformulate these systems in order to strengthen open science actions, practices, and values.

These documents, as they have a guiding nature, with general guidelines, do not include details of the instruments and procedures to carry out the changes required in the apparatus and methods for evaluating research and professional progression in academic careers to encourage the adoption of principles and practices of open science. In effect, there is a certain limitation in moving from a discursive policy supporting open science to a policy with provisions that imply concrete changes in the criteria and procedures used in evaluating research, researchers, institutions, universities and projects.

The research showed that France is the country with an open science policy with clearer guidelines regarding the re-configuration of systems of recognition and reward of research and researchers as compared to the other countries studied, even though the topic is treated in a fragmented way in different parts and guidelines of its two open science plans. The Netherlands and Slovenia show progress towards carrying out reforms of their evaluation systems. In Finland, research institutions and funding organisms have been encouraged to consider the orientation towards open access to publications issuing from their results as part of their evaluation criteria. On the other hand, the Colombian open science policy is the only one among the countries in the study that contains a specific item dealing with the evaluation system, emphasizing the need to introduce changes in the current model of evaluation and incentives, adopting the concept of responsible metrics as one of its components.

The path followed by Unesco towards the broadening of the definition of open science - encompassing the commitment to multilingualism, bibliodiversity and different systems of knowledge, social sectors, cultures, diversity of formats, products and results - indicates the size of the challenge facing public policies of ST&I, as well as systems of research evaluation, in transforming their mechanisms, criteria and procedures mostly based on metrics. Aligning research evaluation systems to practices of open science requires a change in the culture, coordination, and articulation at the national and international levels among the different actors involved in the process. It also requires the replacement of a rationale that stimulates standardization, secrecy, and competition, by a rationale that contemplates different knowledge productions, as well as sharing, transparency, co-creation and collaboration in research.

Albornoz et al. (2018) call attention to power struggles underlying opening processes when they identify a major disconnection between the narratives on open science that promise equitable democratization, but that seem to re-enforce existing inequalities within the scientific system.

Bearing this in mind, to consider changes in the systems of evaluating research and the careers of researchers in accordance with the pillars and presuppositions of open science also requires reflection on what kind of opening one wants to promote, which set of values, behaviors, and practices one wants to stimulate.

In fact, the plans for open science analyzed point to possible changes in the systems of incentive and reward, especially when they question current models of evaluation that privilege the visibility of the research in high impact journals that are not in line with the assumptions of open science (the Netherlands, 2017). All plans foresee alternative forms of incentives. As an example, some propose rewards for data sharing, but they are yet to question productivism, standardization, evaluation centering on controls or on quantitative aspects. One exception is the French policy which is more explicit regarding this aspect and that indicates that it will reduce the influence of quantitative in favor of qualitative evaluation, promoting the use of narrative curricula of professional trajectories and excluding references to the Impact Factor and the H-index from public calls for projects and from application forms (France, 2021).

Thus, both national policies for open science as systems of incentive and reward originate in paradigms that represent an ensemble of ethical-political values encompassing a series of conflicts, tensions and negotiations among the different actors involved.

From these tensions, at least two perspectives of open science unfold schematically: a more pragmatic one (emphasizing values more closely connected to efficiency, productivity, acceleration in processes of data sharing and of the (re)production of research) and another one more democratic (focusing on values associated with inclusion, social and cognitive justice, bibliodiversity and multilingualism) (Albagli, 2015; Clinio, 2019).

Finally, it becomes clear that open science aims at a broader set of evaluation criteria than those currently employed (Babini; Rovelli, 2020), but whatever requirements come to replace them it is necessary to bear in mind that these are not neutral, impartial, or precise.

• 1
  Regarding the Brazilian perspective, a growing acknowledgment of the importance of open access as well as of the related set of practices and issues (open data, open licenses, open repositories), has resulted in the implementation of institutional policies connected to universities, research, and funding institutions over the past five years (Café et al., 2022).
• 2
  Available at: https://roarmap.eprints.org/. Access: April 7, 2023.
• 3
  The Helsinki Initiative on Multilingualism in Scientific Communication invites managers of public policies, universities, research institutions, funding agencies, libraries as well as individual researchers to promote access to knowledge in different languages and to ensure linguistic diversity in the evaluation and financing systems of research (Helsinki Initiative, 2019).
• 4
  The UNESCO Recommendation on Open Science was adopted by the Unesco General Conference in its 41st session, on November 23, 2021. Available at: https://l1nq.com/gL20b. Access on: December 12, 2022.
• 5
  All legal frameworks and documents analyzed are described by country in the Tables that make up the Results section of this paper.
• 6
  The SPARC report does not cover open access policies. On page 10, in explanatory note 53, only ten countries were listed. Not mentioning Belgium and Lithuania, countries included in the Summary and in the analysis, was probably a mistake. Regarding Italy (which is also included in the list of countries referred to in the Summary), the report indicates in its explanatory note 54 that the content of the policy was not analyzed because of it was available only in Italian. Besides these, the study systematized the regulatory frameworks of another fourteen European Union countries engaged with the open science movement and that have moved forward in the formulation of their policies: Austria, Bulgaria, Croatia, Denmark, Estonia, Germany, Greece, Hungary, Latvia, Luxembourg, Malta, Poland, Portugal, and Romania. (SPARC, 2021).
• 7
  FOLEC was established in 2019 by CLACSO with the aim of discussing, sharing knowledge and elaborating a collective proposal for reforming the research evaluation system in Latin America and the Caribbean. Available at: https://www.clacso.org/folec/. Accessed on: January 12, 2022.
• 8
  Elaborated in 2012 and published in 2013, the DORA Declaration is a global initiative that questions the increasing use of bibliometrics to measure the quality of scientific research. Available at: https://l1nq.com/pxVqL. Accessed on: April 10, 2023.
• 9
  Launched in 2015, the Leiden Manifesto points out that an incorrect use of quantitative indicators and metrics in processes of evaluating scientific performance and recommends ten principles for the adequate use of these instruments to subsidize process of evaluating science. Available at: http://www.leidenmanifesto.org/. Accessed on: April 10, 2023.
• 10
  It should be stressed that the National Policy of Open Science of Colombia integrated into our research was published in 2022, therefore after the Babini and Rovelli (2020).
• 11
  As an example, Spain is the country with the largest number of institutions and organisms signatory of the Coalition for Advancing Research Assessment (CoARA) and its Agencia Nacional de Evaluación de la Calidad y Acreditación (ANECA) is committed to promoting changes in the science evaluation system as from the approval of the new Royal Decree of Accreditation (Real Decreto de Acreditación) approved in July 2023. Available at: https://acesse.dev/uQL4D. Accessed on: October 05, 2023.
• 12
  Such national policies are also included in the report An Analysis of Open Science Policies in Europe, v7 (Sparc, 2021).
• 13
  Inspired by a reference system for legal cases created by Frank Shepard known as Shepard's legal citation system, Garfield observed that a Citation index could be useful in the scientific domain, both for the indexation of terms and for researching them, besides serving as a means of finding unknown authors (Mingers; Leydesdorff, 2015; Larivière; Sugimoto, 2018).
• 14
  Mugnaini, Jannuzzi and Quoniam (2004) added that science and technology indicators were employed from the post-war period onwards to “[...] to measure the effort in S&T through input indicators d) as volumes of investments in scientific and technological research. From the 1960s, output indicators (results) were used more often, given the need for measurements that allowed decision makers to evaluate returns on the investments made by them” (Mugnaini; Jannuzzi; Quoniam, 2004, p.125).
• 15
  As explained by Eugene Garfield (2006), the developer of this indicator (in collaboration with Irving H. Sher), the Impact factor (IF) was conceived in 1955 to “[...] help select sources of journals. To do that, we simply re-classified the index of author citations within the index of journal citation. Based on this simple exercise, we learned that at the start a central group of highly quoted journals had to be covered in the new Science Citation Index (SCI)” (Garfield, 2006, p. 90). Garfield explains that after the creation of the IF eight years were needed to formulate the theoretical and conceptual support of citation analysis based on approaches originating in works arising from Merton´s Sociology of Science and on Crawford, Griffith and Crane’s scientific communication (Araújo, 2006).
• 16
  In the original: “[...] más jerarquizado y mercantilizado que lejos de abrir la conversación mundial de la ciencia contribuyó a su progresiva cerrazón” (Beigel, 2021, p. 229).
• 17
  Not original: "including, for example, financial support for open science efforts, recognition and reward of researchers undertaking open science actions, or the use of new and broader evaluation metrics that take into account open science and its impact." (OECD, 2015, p. 89)
• 18
  Available at: https://acesse.dev/tkTqg Access on: January 15, 2023.
• 19
  The Plan S is a European Union project connected to the funding agencies of European countries (with the support of USA and China) which states that: “With effect from 2021, all scholarly publications on the results from research funded by public or private grants provided by national, regional and international research councils and funding bodies, must be published in Open Access Journals, on Open Access Platforms, or made immediately available through Open Access Repositories without embargo” (Plan S, 2020). Available at: https://www.coalition- s.org/about/. Access on: January 20, 2022.
• 20
  The GO FAIR network was founded in the Netherlands with strategies to promote the FAIR principles that state that data should be findable, accessible, interoperable, and reusable. Available at: https://www.go- fair.org/country/netherlands/
• 21
  From the original: “In the present evaluation and reward systems the emphasis is often on the number of publications in prestigious journals with a high impact factor, often produced by well-established publishers and to which there is no open access, thus maintaining the culture of ‘publish or perish’.” (Netherlands, 2017, p. 10).
• 22
  From the original: “Dutch universities do little to none in terms of rewarding researchers for practicing open science. This can be observed from the appointment criteria of tenure track policy across all Dutch universities, which generally include research, teaching, valorisation, and management leadership. For example, Delft University of Technology has the following criteria in the research aspect of their tenure track policy: (1) conducting research, (2) supervising PhD students, (3) acquisition of indirect government funding and contract research and (4) publications. Open science was not included nor integrated into any of the four research elements. At the national level, the Standard Evaluation Protocol (SEP2015-2020) has included relevance to society to be assessed in research evaluation of disciplines. However, the document does not explicitly refer to open science practices. Hence, it is still unclear how individuals would be rewarded for practicing open science and how it could positively impact their career in a concrete manner at Dutch universities (Chan; Meijer, 2018, p. 10-11).
• 23
  Available at: https://www.nwo.nl/en/position-paper-room-everyones-talent. Access on: January 20 2023.
• 24
  According to the site of the Initiative for Open Citations (I4OC), launched in 2018, data citation is not, in general, freely available for access; it is often subjected to inconsistent licenses that are difficult to analyze and is frequently unreadable by machines (I4OC, 2018). The open, machine-readable format of this data allows researchers as well as the public interested in any topic to keep up to date on the findings of published contents as well as allowing for its incorporation into systems such as Crossref as well as other computer systems. As explained by Redhead (2019), the Initiative for Open Science Open Citations (I4OC) cooperates with some academic publishers with the aim of promoting the unrestricted availability of citation data and metadata. Available at: https://i4oc.org/. Access on: April 5, 2023.
• 25
  From the original: “Instituir un sistema de métricas e incentivos que fomenten, valoren y reconozcan las prácticas, procesos y resultados de Ciencia Abierta de la comunidad científica colombiana e integrarlo a los modelos y sistemas de métricas e incentivos de las actividades de CTeI existentes en el país” (COLOMBIA, 2022, p. 51).

Acknowledgments:

Not applicable.

• Funding: This work was carried out with the support of the Coordination for the Improvement of Higher Education Personnel - Brazil (CAPES) - Funding Code 001.
• Ethical approval: Not applicable.
• Availability of data and material: Not applicable.
• JITA: IN. Open science.

REFERÊNCIAS

• ABADAL, E.; ANGLADA, L. Políticas de ciencia abierta en Europa. In: BORGES, M. M.; SANZ CASADO, E. (coord.). Sob a lente da Ciência Aberta: olhares de Portugal, Espanha e Brasil. Coimbra: Imprensa da Universidade, 2021. p. 45-66. Disponível em: https://doi.org/10.14195/978-989-26-2022-0 Acesso em: 22 jan. 2023.
  » https://doi.org/10.14195/978-989-26-2022-0
• AGREEMENT ON REFORMING RESEARCH ASSESSMENT. [S.l.:s.n], 2022. Disponível em: https://coara.eu/app/uploads/2022/09/2022_07_19_rra_agreement_final.pdf Acesso em: 22 jan. 2023.
  » https://coara.eu/app/uploads/2022/09/2022_07_19_rra_agreement_final.pdf
• AKSNES, D. W.; LANGFELDT, L.; WOUTERS, P. Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories. SAGE Open, [S.l], v. 9, n. 1, 2019. Disponível em: https://doi.org/10.1177/2158244019829575 Acesso em: 22 jan. 2023.
  » https://doi.org/10.1177/2158244019829575
• ALBAGLI, S. Ciência aberta em questão. In: ALBAGLI, S.; MACIEL, M. L.; ABDO, A. H. (Org.). Ciência aberta, questões abertas Brasília: Ibict; Rio de Janeiro: Unirio, 2015.
• ALBAGLI, S; CLINIO, A.; RAYCHTOCK, S. Ciência Aberta: correntes interpretativas e tipos de ação. Liinc em Revista, Rio de Janeiro, v.10, n.2, p. 434-450, nov. 2014. Disponível em: https://revista.ibict.br/liinc/article/view/3593 Acesso em: 14 abr. 2023.
  » https://revista.ibict.br/liinc/article/view/3593
• ALBAGLI, S.; IWAMA, A. Y. Citizen science and the right to research: building local knowledge of climate change impacts. Humanities and Social Sciences Communications, v. 9, n. 39, 2022. Disponível em: https://doi.org/10.1057/s41599-022-01040-8 Acesso em: 14 abr. 2023.
  » https://doi.org/10.1057/s41599-022-01040-8
• ALBORNOZ, D. et al Framing power: tracing key discourses in Open Science Policies. In: INTERNATIONAL CONFERENCE ON ELECTRONIC PUBLISHING, 22., Villeurbanne, 2018. [Proceeddings]… Villeurbanne, France: ELPUB, 15 jun. 2018.
• APPEL, A. L; ALBAGLI, S. Acesso Aberto em questão: novas agendas e desafios. Informação & Sociedade: Estudos, João Pessoa, v. 29, n. 4, p. 187-208, 28 dez. 2019. DOI: https://doi.org/10.22478/ufpb.1809-4783.2019v29n4.50113
  » https://doi.org/10.22478/ufpb.1809-4783.2019v29n4.50113
• ARAÚJO, C. A. Á. Bibliometria: evolução histórica e questões atuais. Em questão, Porto Alegre, v. 12, n. 1, p. 11-32, 2006.
• BABINI, D.; ROVELLI, L. Tendencias recientes en las políticas científicas de ciencia abierta y acceso abierto en Iberoamérica Ciudad Autónoma de Buenos Aires: CLACSO; Fundación Carolina, 2020.
• BECERRIL-GARCÍA, A.; AGUADO-LÓPEZ, E. the end of a centralized open access project and the beginning of a community-based sustainable infrastructure for Latin America: Redalyc.org after Fifteen Years. In: CONNECTING THE KNOWLEDGE COMMONS - From Projects to Sustainable Infrastructure: The 22nd International Conference on Electronic Publishing - Revised Selected Papers, 22., Marseille, 2019. Proceeddings… Marseille: OpenEdition Press, 2019 (généré le 18 janvier 2023). Disponível em: https://doi.org/10.4000/books.oep.9003 Acesso em: 14 abr. 2023.
  » https://doi.org/10.4000/books.oep.9003
• BEIGEL, F. La evaluación académica y el camino latinoamericano de la ciencia abierta.” In: BECERRIL-GARCÍA, A.; CÓRDOBA GONZÁLEZ, S. Conocimiento Abierto en América Latina: Trayectoria y Desafíos. [S.l.]: CLACSO, 2021. p. 229-54. Disponível em: https://doi.org/10.2307/j.ctv2v88f34.13 Acesso em: 22 jan. 2023.
  » https://doi.org/10.2307/j.ctv2v88f34.13
• BOUKACEM-ZEGHMOURI, C. Pourquoi les chercheurs ouvrent-ils leurs recherches? The Conversation, June 22, 2020. Disponível em: https://acesse.dev/Y20Hb Acesso em: 22 jan. 2023.
  » https://acesse.dev/Y20Hb
• CAFÉ, L. C. et al Análise de domínio de políticas institucionais de acesso aberto no Brasil. RDBCI: Revista Digital de Biblioteconomia e Ciência da Informação, Campinas, v. 20, 2022. DOI: https://doi.org/10.20396/rdbci.v20i00.8670092 Acesso em: 10 abr. 2023.
  » https://doi.org/10.20396/rdbci.v20i00.8670092
• CHAN, L.; OKUNE, A.; SAMBULI, N. O que é ciência aberta e colaborativa e que papeis ela poderia desempenhar no desenvolvimento?. In: ALBAGLI, S.; MACIEL, M. L.; ABDO, A. (org). Ciência aberta, questões abertas Brasília: IBICT, 2015. 312p. Disponível em: http:/livroaberto.ibict.br/handle/1/1060 Acesso em: 3 abr. 2022.
  » http:/livroaberto.ibict.br/handle/1/1060
• CHAN, T. T.; MEIJER, I. The Netherlands’ Plan on Open Science: open science monitor case study. Brussels: Office of the European Union, Oct. 2018. [Research and Innovation]. Disponível em: https://op.europa.eu/en/publication-detail/-/publication/20d4026e-4478-11e9-a8ed-01aa75ed71a1 Acesso em: 30 jan. 2022.
  » https://op.europa.eu/en/publication-detail/-/publication/20d4026e-4478-11e9-a8ed-01aa75ed71a1
• CLINIO, A. Ciência Aberta na América Latina: duas perspectivas em disputa. Transinformação, Campinas, v. 31, 2019. Disponível em: https://doi.org/10.1590/238180889201931e190028 Acesso em: 14 abr. 2023.
  » https://doi.org/10.1590/238180889201931e190028
• COLOMBIA. Ministerio de Ciencia Tecnología e Innovación - Minciencias. Politica Nacional de Ciencia Abierta 2022-2031 Bogotá D.C: [s.n.], jul. 2022. Disponível em: https://minciencias.gov.co/pdf/pdfreader?url=https://minciencias.gov.co/sites/default/files/politica_nacional_de_ciencia_abierta_-2022_-_version_aprobada.pdf Acesso em: 22 jan. 2023.
  » https://minciencias.gov.co/pdf/pdfreader?url=https://minciencias.gov.co/sites/default/files/politica_nacional_de_ciencia_abierta_-2022_-_version_aprobada.pdf
• CUEVA, J.; MÉNDEZ, E. Open Science and intellectual property rights. How can they better interact? State of the art and reflections. [Report of Study EUROPEAN COMMISSION]. [S.l.: s.n], 2022. Disponível em: https://op.europa.eu/en/publication-detail/-/publication/42d27e04-b715-11ec-b6f4-01aa75ed71a1/language-en Acesso em: 22 jan. 2023.
  » https://op.europa.eu/en/publication-detail/-/publication/42d27e04-b715-11ec-b6f4-01aa75ed71a1/language-en
• DAVYT, A.; VELHO, L. A avaliação da ciência e a revisão por pares: passado e presente: como será o futuro? História, ciências, saúde-Manguinhos, Rio de Janeiro, v. 7, n. 1, p. 93-116, mar./jun. 2000.
• FECHER, B., FRIESIKE, S. Open Science: one term, five schools of thought. In: BARTLING, S., FRIESIKE, S. (ed.). Opening Science Cham: Springer Open, 2014.
• FINLAND. Ministry of Education and Culture. Department for Higher Education and Science Policy. Open science and research leads to surprising discoveries and creative insights Open science and research roadmap 2014-2017. Helsinki: [s.n], 2014. [Reports of the Ministry of Education and Culture, Finland 2014:21]. Disponível em: https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/75210/okm21.pdf?sequence=1&isAllowed=y Acesso em: 30 jan. 2022.
  » https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/75210/okm21.pdf?sequence=1&isAllowed=y
• FINLAND. Open Science Coordination in Finland. Federation of Finnish Learned Societies. Declaration for Open Science and Research (Finland) 2020-2025 Helsinki: TJNK; TSV, 2020. Disponível em: https://avointiede.fi/sites/default/files/2020-02/declaration2020_0.pdf Acesso em: 30 jan. 2022.
  » https://avointiede.fi/sites/default/files/2020-02/declaration2020_0.pdf
• FINLAND. Open Science Coordination in Finland. Federation of Finnish Learned Societies. Open Research Data and Methods: National Policy and Executive Plan by the Higher Education and Research Community for 2021-2025 Helsinki: TJNK; TSV, 2021. [Responsible Research Series 7:2021. 3.v.] Disponível em: https://encr.pw/SIGUe Acesso em: 30 jan. 2022.
  » https://encr.pw/SIGUe
• FOLEC/CLACSO. Diagnóstico y propuestas para una iniciativa regional [S.l]: FOLEC/CLACSO, 2020. [Elaborado por Fernanda Beigel]. (Serie Para una transformación de la Evaluación de la Ciencia en América Latina y el Caribe - FOLEC, CLACSO). Disponível em: https://www.clacso.org/en/folec/ Acesso em: 15 abr. 2023.
  » https://www.clacso.org/en/folec/
• FRANCE. Ministère de L’Enseignement supérieur de la Recherche et de L’Innovation. Plan National pour la Science Ouverte Paris: Open Science, 2018a. Disponível em: https://encr.pw/lzQup Acesso em: 30 jan. 2022.
  » https://encr.pw/lzQup
• FRANCE. Pour une action publique transparente et collaborative: Plan d’action national pour la France: 2018-2020. Paris: Open Government Partnership, 2018b. Disponível em: https://www.etalab.gouv.fr/wp-content/uploads/2018/04/PlanOGP-FR-2018-2020-VF-FR.pdf Acesso em: 30 jan. 2022.
  » https://www.etalab.gouv.fr/wp-content/uploads/2018/04/PlanOGP-FR-2018-2020-VF-FR.pdf
• FRANCE. Ministère de l’Enseignement supérieur, de la Recherche et de l’Innovation. Deuxième Plan national pour la science ouverte: Généraliser la science ouverte en France 2021-2024 Paris: Open Science, 2021. Disponível em: https://encr.pw/PIVyH Acesso em: 30 jan. 2022.
  » https://encr.pw/PIVyH
• FRESSOLI, M.; ARZA, V. Los desafíos que enfrentan las prácticas de ciencia aberta. Teknokultura, Madrid, v. 15, n. 2, p. 429-448, 2018.
• GARFIELD, E. The history and meaning of the journal impact factor. Jama, [S.l.], v. 295, n. 1, p. 90-93, 2006. DOI: DOI: https://doi.org/10.1001/jama.295.1.90 Acesso em: 15 abr. 2023.
  » https://doi.org/DOI:» https://doi.org/10.1001/jama.295.1.90
• GINGRAS, Y. Les dérives de l’évaluation de la recherche: du bon usage de la bibliométrie. Paris: Raisons d’agir, 2014, 122 p.
• GUÉDON, J-C. El acceso abierto y la división entre ciencia “principal” y “periférica”. Crítica y Emancipación, Buenos Aires, ano 3, n. 6, p. 135-180, 2011. Disponível em: http://eprints.rclis.org/17570/1/CyE-6_Guedon-CLACSO.pdf Acesso em: 30 jan. 2022.
  » http://eprints.rclis.org/17570/1/CyE-6_Guedon-CLACSO.pdf
• HELSINKI INITIATIVE. Helsinki Initiative on Multilingualism in Scholarly Communication Helsinki: Federation of Finnish Learned Societies; Committee for Public Information, Finnish Association for Scholarly Publishing; Universities Norway & European Network for Research Evaluation in the Social Sciences and the Humanities, 2019. Disponível em: https://doi.org/10.6084/m9.figshare.7887059 Acesso em: 10 abr. 2023.
  » https://doi.org/10.6084/m9.figshare.7887059
• I40C. Initiative for Open Citations [S.l.: s.n]. 2018. [website]. Disponível em: https://i4oc.org/ Acesso em: 5 abr. 2023.
  » https://i4oc.org/
• JORGE, V. de A. Implementando um repositório de dados no contexto da ciência aberta: a experiência da Fiocruz. In: ENCONTRO DA REDE BRASILEIRA DE REPOSITÓRIOS DIGITAIS, 1., 2022. Anais.. [S. l]: Fiocruz/Icict; IBICT, 2022. 33 p.
• LARIVIÈRE, V.; SUGIMOTO, C. R. Mesurer la science Montréal: Les Presses de l'Université de Montréal, 2018.
• MÉNDEZ, E. Open science por defecto. La nueva normalidad para la investigación Arbor, Madrid, v.197, n.799, a587, 2021. Disponível em: https://doi.org/10.3989/arbor.2021.799002 Acesso em: 10 abr. 2023.
  » https://doi.org/10.3989/arbor.2021.799002
• MINGERS, J.; LEYDESDORFF, L. A review of theory and practice in scientometrics. European Journal of Operational Research, v. 246, n. 1, p. 1-19, 1 Oct. 2015. DOI: https://doi.org/10.1016/j.ejor.2015.04.002 Acesso em: 21 set. 2020.
  » https://doi.org/10.1016/j.ejor.2015.04.002
• MUGNAINI, R.; JANNUZZI, P. M.; QUONIAM, L. Indicadores bibliométricos da produção científica brasileira: uma análise a partir da base Pascal. Ciência da informação, Brasília, v. 33, n. 2, p. 123-131, 2004.
• NETHERLANDS. Dutch Ministry of Education, Culture and Science. National plan open science [S.l.:s.n], 9 febr. 2017. Disponível em: https://www.dtls.nl/2017/02/09/national-plan-open-science-presented/ . Acesso em: 30 jan. 2022.
  » https://www.dtls.nl/2017/02/09/national-plan-open-science-presented/
• O’CARROLL et al Evaluation of research careers fully acknowledging Open Science practices: Rewards, incentives and/or recognition for researchers practicing Open Science. Publications Office, Luxembourg, 2017. Disponível em: https://data.europa.eu/doi/10.2777/75255 Acesso em: 30 jan. 2022.
  » https://data.europa.eu/doi/10.2777/75255
• OCDE. Making Open Science a Reality. OECD Science, Technology and Industry Policy Papers, Paris, n. 25, 15 oct. 2015. 108 p. DOI: https://doi.org/10.1787/5jrs2f963zs1-en
  » https://doi.org/10.1787/5jrs2f963zs1-en
• RODRIGUES, E. A necessária e difícil reforma da avaliação da investigação. In: REFORMA da avaliação [Preprint de Capítulo do Livro] [S.l.: s.n.], 2022. Disponível em: https://encr.pw/0iX7Q Acesso em: 22 jan. 2023.
  » https://encr.pw/0iX7Q
• SALATINO, M.; LÓPEZ RUIZ, O. El fetichismo de la indexación: una crítica latinoamericana a los regímenes de evaluación de la ciencia mundial. Revista Iberoamericana de Ciencia, Tecnología y Sociedad -CTS, [S.l.], v. 16, n. 46, p. 73-100, 2021. Disponível em: https://l1nq.com/1fuIR Acesso em: 30 jan. 2022.
  » https://l1nq.com/1fuIR
• SÁNCHEZ-TARRAGÓ, N. Acceso abierto en Latinoamérica: desafiando la geopolítica del conocimiento. Revista Cubana de Información en Ciencias de la Salud, Habana, Habana, v. 26, n. 4, ago. 2015. Disponível em: https://acesse.dev/VITBU Acesso em: 22 jan. 2023.
  » https://acesse.dev/VITBU
• SANTOS, P. X.; ALMEIDA, B. A.; HENNING, P. (org.). Livro Verde - Ciência aberta e dados abertos: mapeamento e análise de políticas, infraestruturas e estratégias em perspectiva nacional e internacional. Rio de Janeiro: Fiocruz, 2017. 140 p.
• SLOVENIA. Government of the Republic of Slovenia. National Strategy of Open Access to Scientific Publications and Research Data in Slovenia 2015-2020 Liubliana: [S.n], 3 Sept. 2015. Number: 60300-5/2015/5. Disponível em: https://l1nq.com/3AHdC Acesso em: 22 jan. 2023.
  » https://l1nq.com/3AHdC
• SLOVENIJE. Vlada Republike Slovenije. Akcijski Načrt Izvedbe Nacionalne Strategije Odprtega Dostopa do Znanstvenih Objav in Raziskovalnih Podatkov v Sloveniji 2015‒ 2020 Liubliana: [S.l], 24 maj. 2017. Številka [número]: 60300-5/2015/10. Disponível em: https://l1nq.com/dscl0 Acesso em: 30 jan. 2022.
  » https://l1nq.com/dscl0
• SPARC. An Analysis of Open Science Policies in Europe, v.7. [Report]. [S.l]: SPARC EUROPE; DCC, Apr. 2021. 43 p. Disponível em: https://zenodo.org/record/4725817#.YfaYxurMIdU Acesso em: 30 jan. 2022.
  » https://zenodo.org/record/4725817#.YfaYxurMIdU
• UNESCO. Recommandation de l’UNESCO sur une science ouverte Paris: UNESCO, 2021. Disponível em: https://unesdoc.unesco.org/ark:/48223/pf0000379949_fre
  » https://unesdoc.unesco.org/ark:/48223/pf0000379949_fre
• VELHO, L. Conceitos de ciência e a política científica, tecnológica e de inovação. Sociologias, Porto Alegre, v. 13, n. 26, p. 128-153, jan./abr. 2011.
• VESSURI, H.; GUÉDON, J.-C.; CETTO, A. M. M. Excellence or quality? Impact of the current competition regime on science and scientific publishing in Latin America and its implications for development. Current Sociology, London, v. 62, n. 5, p. 647-665, 2014. DOI:10.1177/0011392113512839
  » https://doi.org/10.1177/0011392113512839
• VOMMARO, P.; ROVELLI, L. Desafíos a la evaluación de la investigación orientada a la movilización del conocimiento en transición hacia la ciencia abierta: un análisis a partir del caso de los Grupos de Trabajo del Consejo Latinoamericano de Ciencias Sociales. Analecta Política, Medellín, v. 12, n. 23, p. 1-18, 2022, Disponível em: http://dx.doi.org/10.18566/apolit.v12n23.a02 . Acesso em: 10 abr. 2023.
  » http://dx.doi.org/10.18566/apolit.v12n23.a02
• VSNU; KNAW; NWO; ZONMW. Room for everyone’s talent: towards a new balance in the recognition and rewards for academics. [S.l.: s.n], 2019. Disponível em: https://acesse.dev/prlLe Acesso em: 10 abr. 2023.
  » https://acesse.dev/prlLe
• PLAN S. Why Plan S Strasburg: Science Connect, 4 set. 2018. Disponível em: https://www.coalition-s.org/why-plan-s/ Acesso em: 22 jan. 2023.
  » https://www.coalition-s.org/why-plan-s/
• REDHEAD, C. OASPA e a Iniciativa para Citações Abertas Blog PPEC, [S.l.], v. 9, n. 1, set. 2019. Disponível em: https://periodicos.sbu.unicamp.br/blog/index.php/2019/09/19/oaspa/ Acesso em: 22 jan. 2023.
  » https://periodicos.sbu.unicamp.br/blog/index.php/2019/09/19/oaspa/
• WOUTERS, P. The citation: from culture to infrastructure. In: CRONIN, B.; SUGIMOTO, C. R. (ed.). Beyond bibliometrics: harnessing multidimensional indicators of scholarly impact. Cambridge: MIT Press, 2014. p. 47-66. Cap. 3.

Edited by

Editor: Gildenir Carolino Santos

Translation: Maria Cristina Matos

Data availability

Availability of data and material: Not applicable.

Publication Dates

• Publication in this collection
  27 Nov 2023
• Date of issue
  2023

History

• Received
  16 Apr 2023
• Accepted
  04 Oct 2023
• Published
  20 Oct 2023