The challenge of assessing the impact of science beyond bibliometrics

Abstracts

It is hoped that the knowledge produced by scientific research turns into concrete benefits (material or non-material). Bibliometrics, with its various indexes, became the reference of scientific evaluation as an “objective” measure of scientific impact. In Brazil, the evaluation system focused on CAPES has been of great importance for the development of postgraduate studies and research; however changes are necessary, among others, to its bibliometric approach. If we consider the area of health, the challenge of the Unified Health System (SUS) implies the search for scientifically sound alternatives, regarding questions ranging from diagnosis, cure and prevention of a variety of problems, to the organization of a macro-structure capable of giving broad and equal access to the resources required for improving the health of the population. The solutions require expertise and creativity on the part of the researchers and the expected products must include, but are not limited to scientific publications.

Researcher Performance Evaluation Systems; Scientific Publication Indicators; Periodicals as Topic; Science, Technology and Innovation Indicators

Espera-se que os conhecimentos produzidos pela investigação cientifica transformem-se em benefícios concretos (materiais ou não materiais). A bibliometria, com seus vários índices, tornou-se a referência da avaliação científica e a principal estratégia de medida objetiva do impacto. No Brasil, o sistema de avaliação da Coordenação de Aperfeiçoamento de Pessoal de Nível Superior tem sido de grande importância para o desenvolvimento da pós-graduação e da pesquisa; entretanto, identificam-se necessidades de mudanças, sendo uma delas o enfoque bibliométrico. Se considerarmos a área da saúde, o desafio do Sistema Único de Saúde implica na busca de alternativas cientificamente embasadas para um complexo de questões que vão do diagnóstico, cura e prevenção de uma variedade de problemas até a organização de uma macroestrutura capaz de dar amplo e igual acesso aos recursos demandados pela saúde da população. As soluções exigirão dos pesquisadores competência e criatividade e destes esperam-se produtos que incluam, mas que certamente não poderão se restringir às publicações científicas.

Sistemas de Créditos e Avaliação de Pesquisadores; Indicadores de Produção Científica; Publicações Periódicas como Assunto; Indicadores de Ciência, Tecnologia e Inovação

Se espera que los conocimientos producidos por la investigación científica se transformen en beneficios concretos (materiales y no materiales). La bibliometría, con sus diferentes índices, se convirtió en referencia de evaluación científica y en la principal estrategia de medida objetiva de impacto. En Brasil, el sistema de evaluación de la Coordinación de Mejoramiento de Personal de Nivel Superior ha sido de gran importancia para el desarrollo de la post-graduación y de la investigación, sin embargo, se identifican necesidades de cambios, siendo una de ellas el enfoque bibliométrico. Se consideramos el área de la salud, el desafío del Sistema Único de Salud implica la búsqueda de alternativas científicamente fundamentadas en una complejidad de aspectos que van del diagnóstico, cura y prevención de una variedad de problemas hasta la organización de una macro estructura capaz de dar amplio e igual acceso a los recursos demandados por la salud de la población. Las soluciones exigirán de los investigadores competencia y creatividad esperándose de estos productos que incluyan, pero que no podrán sólo restringirse a las publicaciones científicas.

Sistemas de Créditos y Evaluación de Investigadores; Indicadores de Producción Científica; Publicaciones Periódicas como Asunto; Indicadores de Ciencia, Tecnología e Innovación

INTRODUCTION

Given the immense effort dedicated to scientific investigation and to ensure that its results are not limited to “advances in knowledge”, it is expected that at some point this knowledge is transformed into action, yielding tangible benefits (material or non-material) for society. This long, and far from linear process between knowledge and action, although acknowledged and desirable, has not proved to be very practicable and there are difficulties with regards perception, consensus, measurement and assessment. This process, generically known as transfer, translation or, more recently, knowledge translation, ⁷ has been understood and interpreted in various ways. For some, it has always been the tempting idea that knowledge produced can be immediately translated into a tangible action, it being enough that we desire it to be so. This version leaves out the large cognitive, epistemological and political distances between producing the knowledge and producing the action.

Another issue which has often been a source of confusion concerns the differences between knowledge and information. Scientific knowledge involves a production process with various stages and ultimately aims to change the state of knowledge that was previously available. With regards information, its objective is to present a fact and how that fact is configured at a given moment and in a given context. Information may be knowledge, however, it is not scientific knowledge.

It is in this context, in which information, knowledge and action are mixed, that programs for evaluating academic performance, and specifically the production of scientific knowledge, are situated. The configuration of the assessment of scientific programs should, obviously, be linked to the configuration of the scientific programs. However, on reviewing current scientific programs it is notable that only some characteristics which are central to the scientific activity are included in the evaluation processes, whereas others, no less important, are either not included or are secondary.

Science, in whatever area, is disseminated in a written form; any new piece of scientific knowledge needs to be written, revised, reviewed and verified in order to be accepted by peers. Add to this the fact that the evolution of science contains cumulative elements: each new project needs to record and cite the prior knowledge needed to substantiate and justify new proposals. These characteristics formed the basis for developing the systems of bibliographic cataloging and, derived from these, the systems of counting citations, which made it possible for various current bibliometric indexes (e.g., number of citations, impact factor and h-index) to be developed and universally used in systems of scientific evaluation. With the exception of the systems of registering patents, which are limited to those areas of applied science that produce patentable products and which are used in scientific evaluation, nothing else has been developed to systematically record the potential effects of activity in the different scientific fields, apart from bibliographic effects. It is not by chance that bibliometrics, with its various indexes, became the reference for scientific evaluation and the principal strategy for “objective” measurement of the scientific impact of an individual, institution or journal, eventually becoming confused with scientometry. This occurred in spite of everyone being aware of its principal drawback: the fact that the indexes measured only the modus operandi part of the science (publishing and citations), leaving aside its various potential non-bibliographic effects on the world.

The quantity and the form of publishing and the intensity of citation varies between the scientific fields. However, it is important to recognize that there is no such thing as science without publishing or science without citation. Thus, bibliometric evaluation has taken on great relevance for scientific evaluation in general and is central to some scientific fields or ways of doing science. ⁶ However, one would expect this approach to have less significance in those fields of ways of doing science in which other products in addition to bibliographic production or citation are expected. This difference between ways of doing science was well established by Gibbons, ⁶ separating traditional academic science, based on disciplines and centered around a system of merit based on publication and on effects identified using bibliometry (named mode 1), from scientific practice aimed at problems, not restricted to specific disciplines and the products of which are not limited to publications, although of course not excluding them (named mode 2). However, in spite of proposals such as those of Gibbons having a strong influence on the way in which the organization of science is viewed, what is also notable is the tendency not to differentiate the evaluation processes, which continue to be strongly centered on bibliometry.

In 2000, in one of the first attempts to evaluate the excellence of research in area in which mode 2 predominates, the English Royal Academy of Engineering presented an innovative set of ideas about a way of perceiving and evaluating the impact of this way of doing science. ⁹ It is interesting that it reaffirms the centrality of bibliometric indexes in mode 1 evaluations and, where deemed necessary, withdraws its importance and centrality to mode 2 ways of doing science. In this mode, elements such as the practical results of the research are emphasized, these being considered essential in evaluating excellence.

Similarly, and using the earlier document as the main point of reference, in 2002 the Dutch Royal Academy of Arts and Sciences published a document ¹⁰ which clearly highlighted the importance of non-bibliometric impact in evaluating applied research in the area of health. Firstly, it emphasized broadening the coverage of bibliometric impact to cover more than just peer citations, in other words, exclusively among scientific manuscripts, so as to include other forms in which they are used, for example, in documents on which policies, norms and guidelines for professional practice are based. The need to focus the impact of applied research in health on health care providers and on health care organization processes and the process of designing, implementing and monitoring decisions was highlighted. In order to establish such advances more clearly, it was proposed that potential impact ( ex-ante ) – transforming social issues into research – be discriminated from effective impact ( ex-post ) – how scientific questions established previously were answered – and the subsequent scientific conclusions translated into practical solutions or political implications.

More than a decade after these seminal documents appeared, little advancement has been seen concerning bringing these proposals into being. With the exception of being able to use search engines (such as Google scholar), which recover bibliographic citations without limiting the search to scientific journals, ³ little has been produced in the way of new proposals for scientific evaluation, especially in the applied areas, or mode 2.

If the evaluation of scientific impact, internationally, has been focused on bibliometric indicators, in Brazil this process has taken on some very specific characteristics. There, academic research has been linked to the development of post-graduate courses. Post-graduate programs, enclosed in universities that, if not entirely rejecting them, regarded them as foreign bodies, have become epicenters of scientific investigation activities and sources of prestige in academic activity. Within the systems of scientific evaluation, bibliometric indicators (number of publications, number of citations, impact factor the scientific journals, h-index of the researchers or the journals, among others) have been avidly assimilated into the system of evaluating scientific activities and academics, as in the international scene, also known as “quantified control”. ⁴

The principal system of scientific evaluation in the country is the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – CAPES (Coordination of Improvement of Higher Education Personnel) and the unit of evaluation is post-graduate programs and scientific activities linked to them. Brazil is perhaps the only country in the world to use such an approach. Although this evaluation effort, which began over 30 years ago, has been central in kick starting post-graduate activities and scientific activities on which they are based in the country, the need for substantial changes has arisen, accompanying changes in society’s focus on scientific development, coming to include evaluative parameters such as applications and innovations produced by scientific activity. The current focus based on number of articles, impact factor or journals’ h-index has drawbacks and is unsatisfactory, as has been shown in terms including public demonstrations calling for its exclusion as a criterion for scientific evaluation. ^{1 , 5}

In Brazil, the need to develop new parameters for scientific evaluation which seek to recognize the contribution made by different modes of doing science, by different communities, institutions and scientific groups to material and non-material advances expected to occur in society has become evident, as has the need to transform this evaluative process into an activity which includes the perspective of the various direct sources of financing scientific investigation, of the institutions which carry it out (universities and research institutes) and of society’s expectations of scientific activities.

For a project of such change to be realized, it also becomes necessary to understand the base upon which the projects and relationships of power within and among the scientific communities in the country are configured. Thus, Barata et al ² have shown that, based on current evaluation criteria, the area of health care, despite its importance to society and its good performance when compared with other scientific areas using traditional parameters, has been, historically, one of the areas that had the lowest allocation of investment in research. The proposal of constructing a new Brazilian Unified Health System (SUS), in order to fulfill the principle established in the 1988 Constitution that very citizen has a right to health care and that this is a duty of the State, produced a huge political, ⁸ scientific and technological challenge. ¹¹ They include the search for scientifically sound alternatives for a diverse set of activities ranging from diagnosis, cure and prevention of a variety of health problems to the organization of a macrostructure capable of providing comprehensive and equal access to the recourses necessary for the health of the population. These are complex challenges for members of the scientific community in the country, demanding expertise and creativity and, also, production which includes, but is by no means limited to scientific publications.

REFERÊNCIAS

• ¹
  Alberts B. Impact factor distortions. Science . 2013;340(6134):787. DOI:10.1126/science.1240319
• ²
  Barata RB, Portela LE, Aragão E, Barreto ML. The configuration of the Brazilian scientific field. Ann Acad Bras Cienc. 2013. No prelo.
• ³
  Barreto ML, Aragão E, Souza LEPF, Santana TM, Barata RB. Diferenças entre as medidas do índice-h geradas em distintas fontes bibliográficas e engenhos de busca. Rev Saude Publica. 2013;47(2):231-8.
• ⁴
  Burrows R. Living with the h-index? Metric assemblages in the contemporary academy. Sociol Rev. 2012;60(2):355-72. DOI:10.1111/j.1467-954X.2012.02077.x.
  » https://doi.org/10.1111/j.1467-954X.2012.02077.x
• ⁵
  Ferreira RC, Antoneli F, Briones MRS. The hidden factors in impact factors:aperspective from Brazilian science. Frontiers in Genetics 2013; 4:130. DOI:10.3389/fgene.2013.00130
  » https://doi.org/10.3389/fgene.2013.00130
• ⁶
  Gibbons M, Limoges C, Nowotny H, Schwartzmann S, Scott P, Trow M. The new production of knowledge: the dynamics of science and research in contemporary societies. London: Sage; 1994.
• ⁷
  Guimarães R. Pesquisa Translacional: uma interpretação. Cienc Saude Coletiva. 2013;18(6):1731-44. DOI:10.1590/S1413-81232013000600024
  » https://doi.org/10.1590/S1413-81232013000600024
• ⁸
  Paim J, Travassos C, Almeida C, Bahia L, Macinko J. The Brazilian health system: history, advances, and challenges. Lancet . 2011;377(9779):1778-97. DOI:10.1016/S0140-6736(11)60054-8
  » https://doi.org/10.1016/S0140-6736(11)60054-8
• ⁹
  Royal Academy of Engineering. Measuring excellence in engineering research. London; 2000.
• ¹⁰
  Royal Netherlands Academy of Arts and Sciences, Council for Medical Sciences. The societal impact of applied health research: towards a quality assessment system. Amsterdam; 2002.
• ¹¹
  Victora CG, Barreto ML, Leal MC, Monteiro CA, Schmidt MI, Paim J, et al. Health conditions and health-policy innovations in Brazil: the way forward. Lancet . 2011;377(9782):2042-53. DOI:10.1016/S0140-6736(11)60055-X
  » https://doi.org/10.1016/S0140-6736(11)60055-X

Publication Dates

• Publication in this collection
  Aug 2013

History

• Received
  2 Aug 2013
• Accepted
  19 Aug 2013