Measures of experience and scientometrics to evaluate the impact of scientific production

Abstracts

“Experiencimetrics” is proposed as a measure of researchers’ academic experience, to complement the existing measure, scientometrics, when evaluating expertise in research. The article is partly based on the author’s scientific output, and explores the possibilities of analyzing impact from a qualitative perspective. A synthesis is produced, with possible implications for this analysis, highlighting some options for evaluating the impact of research and articles which are not limited to counting author’s citations in order to calculate the impact factor.

Research Personnel; Researcher Performance Evaluation Systems; Scientific Publication Indicators; Periodicals as Topic

Propõe-se a “experienciometria” como medida da experiência acadêmica de pesquisadores de modo a complementar a medida consagrada da cienciometria nas avaliações de competências em pesquisa. O artigo baseia-se em parte na produção cientifica do autor, utilizada sob uma perspectiva qualitativa para explorar possibilidades de análise de impactos. Ensaia-se uma síntese com possíveis implicações dessa análise, indicando algumas possibilidades para se avaliar impacto de pesquisas e artigos que não se restrinjam exclusivamente à contagem de números de citações dos autores para o cálculo do fator de impacto.

Pesquisadores; Sistemas de Créditos e Avaliação de Pesquisadores; Indicadores de Produção Científica; Publicações Periódicas como Assunto

Se propone la “experienciometría” como medida de la experiencia académica de investigadores con el objetivo de complementar la medida consagrada de la cienciometría en las evaluaciones de competencias en investigación. El artículo en parte, se basa en la producción científica del autor, utilizada bajo una perspectiva cualitativa para explorar posibilidades de análisis de impactos. Se ensaya una síntesis con posibles implicaciones de éste análisis, indicando algunas posibilidades para evaluar el impacto de investigaciones y artículos, que no se limiten exclusivamente al contaje de números de citaciones de los autores para el cálculo del factor de impacto.

Investigadores; Sistemas de Créditos y Evaluación de Investigadores; Indicadores de Producción Científica; Publicaciones Periódicas como Asunto

INTRODUCTION

When considering the ingenuity of the criteria of meritocracy engendered by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) to qualify post-graduate programs, it is necessary both to improve the quantitative mechanisms and to examine new ways of evaluating the impact of the researchers and programs.

On following the process of accreditation of a Masters as post-graduate coordinator, I published an opinion piece entitled: “There are many doctors, but few ‘social entrepreneurs’”. ¹

Entrepreneurship or taking a risk would be the expected result of the “research – teaching- extension” triad and combined mechanisms of evaluation would need to be practiced. As mentioned in Ribeiro, RJR “the university will truly become university when the prince (research) kisses the frog (extension)” (Almeida Filho, 2007). ⁸

And how is this to be done? Perhaps by identifying when research also “serves humanity and not just science”.

Borrowing Packer’s expression, ^a that “a doctor may have read one scientific article and have saved two lives”, by not reporting this in a systematic manner to the scientific community, “he helps humanity, but not science”. Thus, the author of said article read by the doctor will never know of the impact caused by article.

Moreover, Packer ^a (2012) states that the scientific community recognizes the need for this type of qualification of research or of its repercussions on the life of the subjects; but, for lack of a validated methodology, we continue to use the enshrined measure of scientometrics ³ to evaluate the impact of journals, articles and researchers.

Would this attempt to qualify the impact of research/article/researcher not be the subterfuge of someone who would never achieve the competitive quantitative levels, hiding behind qualitative raptures? We have not reached a conclusion on this insinuation. According to Packer ^a (2012), the “science of citations” is also susceptible to Pareto’s Law, known as the “80-20 principle”, in which 80% of citations would always be attributed to 20% of researchers. Coimbra Jr. ¹⁰ argues that the field of Public Health is not represented in the “lions share” of the most cited journals, not even in international journals, such as the American Journal of Public Health and the International Journal of Epidemiology.

So the “crumbs” of research promotion would be left for those 80% of researchers with an inferior number of citations? Of that contingent of research, what has the greatest potential to help humanity? Are there other forms of evaluating the impact of articles that does not involve simply counting citations?

The aim of this article was to suggest possibilities for the qualitative analysis of the impact of scientific production.

A POSSIBLE STARTING POINT

In 1993, a broad piece of research entitled “Health and Environment: a study of intra-urban differences in developing countries, in Acra, capital of Ghana and Sao Paulo, Brazil”, ¹⁶ demonstrated the existence of a social gradient with regards the risk of dying from various selected chronic diseases, external causes, respiratory disease and communicable diseases when comparing different geographical territories in the cities studied. ^{7 , 14 , 15}

More than an ecological epidemiological study of intra-urban differences, it practiced the exercise in negotiation of constructing composite indicators, ^{b , c} which influenced a set of research into violence, quality of life and social exclusion/inclusion. ^{4 - 6}

One of those pieces of research was the Violence Risk Map of Sao Paulo, SP, Southeastern Brazil. ⁶ This initiated maps of violence risk in the cities of Curitiba, PR, Rio de Janeiro, RJ, Southeastern Brazil and Salvador, BA, ³ Northeastern Brazil, forming part of Federal Government efforts to draw up a National Human Rights Plan, created in 1996 by the Ministry of Justice.

It was decided to report the results of these colored maps in the form of an easy-to-read bulletin, divulged as a tool for informing policies and raising society’s awareness of the magnitude of urban violence.

Following this logic, it was decided to report the results of mapping of violence in the city of Sao Paulo, exclusively, in that city’s press.

The material, carrying the headline “Jardim Ângela is the most dangerous place in the world”, ^d introduced the report with the results of the research based on a small chart which compared mortality rates from homicide in Cali and Medellin, Colombia, with neighborhoods in the city of Sao Paulo, where the rate in Jardim Ângela exceeded the rates of the Colombian cities, at the time considered to be extremely violent.

Following this, there was a steep decline in the violence rates in Jardim Ângela which persists to this day. There was a 73.3% drop in the homicide rate, according to the text below:

The district of Jardim Ângela, in the south of Sao Paulo, with a population of 300 thousand inhabitants was, in 1996, considered by the UN to be the most violent place on the planet. To give an idea of the gravity of the situation, in 2001, 277 murders were recorded. Since then the rate has dropped year on year and, in 2004 the district recorded 151 murders, a reduction of 45%.

But the best was yet to come, this year, at time of writing, 26 violent deaths have been recorded, compared with 65 in the same period last year, and 120 in 2001. This shows that in fewer than four years the homicide rate in Jardim Ângela has fallen by more than 73%. Instituto Sou da Paz. ^e

It is not intended that this narrative establishes a cause-effect relation between the headline in 1996 and the significant decrease in homicide rates in 2005. However, this headline has repercussions up to the present day in newspapers and seems to play a social marketing role, influencing the mobilization and rolling out of actions aimed at this area, which was pushed into the spotlight by the headline, which did nothing more than illuminate a situation that had existed long before the headline.

It seems that a newspaper headline, which does not come within the scope of science, indicated that the research may have had a “great impact on humanity”.

OTHER TYPES OF IMPACT

Who has not heard the expression “This had a huge impact on me”? With this expression, the subject is talking of something that affected them, something they experienced.

According to Bondía: ⁹

“Experience, the possibility that something might happen to us or touche us, requires a gesture of interruption, a gesture which is almost impossible nowadays because the lack of time: it requires one to stop and think, stop to look, stop to listen… suspend automated actions, cultivate attention and gentleness, open the eyes… cultivate the art of finding, being very quiet, being patient and giving oneself time and space”. ⁹

Thus, Bondía ⁹ alerts us that a society “constituted under the sign of information” makes experiencing difficult.

There is no room here for the conflict between knowledge/information and knowing from experience/feeling, each pair with their purpose, their ethics and aesthetics, but we must advocate for adequate tension between the two pairs.

However, Bondía ¹⁶ comments that modern science distrusts experience and creates a safe path translated by the method which reduces experience to experimentation.

If the logic of experimentation is agreement, consensus and homogeneity, in knowing from experience, difference, heterogeneity and singularity are important. ⁹

If the scientometric base ¹³ is the impact factor and other bibliometric indicators producing objective, regular, repeatable, comparable, impersonal, infinite, universal results, it would be very daring of Brazilian researchers to be able to explore the development of “experiencimetrics”, which would produce personal and comparable results and which would also contribute to better understanding of the scientific research mechanism as a social activity.

RESISTENCE TO A DECLARED ELITISM?

In the “preamble” of the “rodízio” traffic law in Sao Paulo – limiting which days cars with particular number plates can be used – it states: “according to research by Prof. Pedro Jacobi, who pointed out the perceived increased pollution in the city by housewives in a neighborhood by the quantity of soot forming on the sheets hanging out on the clothes lines”. ^f

On seeing this described in the law, it is interpreted immediately as being a genuine impact of scientific research.

Universities in Europe are in the process of creating ways of evaluating impact that are not limited to citation indexes, which frequently lead to opinions that the academic world is split between those who do “little science” and those who do “real” science when comparing the citation indexes for different researchers.

What needs to be done is to find forms of inducing/designing a wider repertoire of indicators for evaluating the career of a university professor, considering other fields of activity, measured with the same value as the capacity for disseminating scientific articles, so that the quantitative criterion of citations is not solely predominant.

FINAL CONSIDERATIONS

First, it is necessary to recognize the provisory character of this work – tentative and precarious, in the words of Deleuse & Guattarri: ¹¹

“ Also, above all in the theoretical domain, any pragmatic rough sketch is better than a copy of concepts with cuts and progresses which never change. The imperceptible rupture instead of a significant cut. ”

Second, it is necessary to recover the context of academic life so as to “tighten” meritocracy in research as a determining principle in evaluating academic performance and advocate that other areas of academic performance also acquire the same degree of evaluative relevance as research and publishing.

The third point that needs to be highlighted is: what is the purpose of science? To help humanity or to self- reproduce among peers? How is it possible to know if it is really helping humanity? At the moment we only have the quantitative methods already validated, derived from scientometrics, but we lack methods which could also qualify the impact of science/research in the sense of its “contribution to humanity”. Moreover, the most enshrined method – the impact factor measured by number of citations – has its limitations and critics such as, for example, that attributed by Pinto & Andrade: ¹³

“ Science could benefit from the application of these scientometric parameters, but its growth and quality demand work in all directions so that important areas of knowledge do not atrophy for not being in fashion, like those which gather the highest number of scientists and only appear in high impact factor journals... ” ¹³

In order to achieve the aspiration that research’s main purpose is to “help humanity”, indirect criteria could be established, for example, potential or realized capacity of research to influence public policies: (1) level of participation of social players in the process of drawing up the research; (2) communication strategies between the social players and the researchers involved; (3) clarity in defining objectives; (4) visibility of the research process and dissemination of the results; (5) meeting deadlines; and (6) achieving previously established objectives. ²

These criteria are only a first approximation for evaluating the elements research that aim to influence policies. Other elements may not have been identified, and other studies exploring these issues are welcome, as would a clearer definition of indicators which allow the association between these elements and influence on public policies to be certified. In this discussion, the emphasis is on the fact that, in the case of there being genuine “scientific willingness” on the part of the researchers in influencing policy through their work, the abovementioned criteria need to be directly and explicitly included in the design of the research as elements that could potentially influence the formulation of public policies. ²

Thus, these elements indicate the possibility of searching for a mode of analysis to highlight “knowledge from experience” of the research in influencing public policies, blazing trails to constitute evaluative “experiencometrics”.

A fourth thread of discussion, connected to the previous one, proposes inserting perceived and documented “narratives” (knowledge from experience) of the impacts of the research, for example, in subjects, population groups, territories or laws such as those exemplified here: (1) in the case of the “Rodízio” law and the influence of research by Prof. Pedro Jacobi and (2) the case of the Violence Risk Map of Sao Paulo City and its connection with “Jardim Ângela”.

“ Narrative is a linguistic form characterized by: presenting finite and longitudinal time sequence; presupposing the existence of a narrator and an audience, whose visions of the world are embedded in how the stories are told;... engaging the audience and inviting them to interpret. ” ¹²

Therefore, it would become the role of researchers to position themselves as a collector of experiences, inspired by the willingness to share the perceived and observed impact of their research. Such an exercise may even be encouraged by preconceived scripts during the evaluation processes, and incorporated into CAPES or into the Currículo Lattes network, e.g., and they would have some weight in evaluating research, post-graduate programs and requests for subsidies to development agencies.

Grossman & Cardoso ¹² give some indications that may serve for proposing this script when they affirm that “narratives are structured on five elements, namely: the facts, the characters, time, space and the narrator”. Moreover, it is possible to suggest creating a script which would come with documental material in order to support and state in detail the facts.

REFERÊNCIAS

• ¹
  Akerman M. Há muitos doutores, mas faltam empreendedores sociais. Diario Grande ABC. 2005 ago 7:13.
• ²
  Akerman M. Examinado elementos que possam influenciar a formulação de políticas em estudos que utilizam indicadores compostos: o chão contra o cifrão. Cienc Saude Coletiva. 2000;5(1):115-24. DOI: 10.1590/S1413-81232000000100010
  » https://doi.org/ 10.1590/S1413-81232000000100010
• ³
  Akerman M, Bousquat A. Mapa de risco da violência das cidades de São Paulo, Curitiba, Salvador e Rio de Janeiro. Sao Paulo Perspec.2000;13(4):112-20. DOI: 10.1590/S0102-88391999000400012
  » https://doi.org/ 10.1590/S0102-88391999000400012
• ⁴
  Akerman M, Massei W, Cabral S, Broch A, Cremonese A, Alves TS, et al. A concepção de um projeto de observatório de qualidade de vida: relato de uma experiência realizada em Campinas. Saude Soc. 1997;6(2):83-100. DOI: 10.1590/S0104-12901997000200010
  » https://doi.org/ 10.1590/S0104-12901997000200010
• ⁵
  Akerman M. Qualidade de vida e exclusão: quanto custa incluir? Debates Socioambient. 1996;2(4):8-9.
• ⁶
  Akerman M, Bousquat A Mapa de risco da violência da cidade de São Paulo. São Paulo: Centro de Estudos de Cultura Contemporânea; 1996.
• ⁷
  Akerman M, Stephens C, Campanario P, Maia PB. Saúde e meio ambiente: uma análise de diferenciais intra-urbanos enfocando o Município de São Paulo, Brasil. Rev Saude Publica. 1994;28(4):320-5. DOI: 10.1590/S0034-89101994000400013
  » https://doi.org/ 10.1590/S0034-89101994000400013
• ⁸
  Almeida Filho N. Universidade nova: textos críticos e esperançosos. Brasília/Salvador: Editora UnB/EDUFBA; 2007.
• ⁹
  Bondia JL. Notas sobre a experiência e o saber da experiência. Rev Bras Educ. 2002;19(1):20-9. DOI: 10.1590/S1413-24782002000100003
  » https://doi.org/ 10.1590/S1413-24782002000100003
• ¹⁰
  Coimbra Jr CEA. Produção científica em saúde pública e as bases bibliográficas internacionais. Cad Saude Publica. 1999;15(4):883-8. DOI: 10.1590/S0102-311X1999000400022
  » https://doi.org/ 10.1590/S0102-311X1999000400022
• ¹¹
  Deleuze G, Guatarri F. Mil Platôs. Rio de Janeiro: Editora 34; 1995.
• ¹²
  Grossman E, Cardoso MHCA. As narrativas em medicina: contribuições à prática clínica e ao ensino médico. Rev Bras Educ Med. 2006;30(1):6-14. DOI: 10.1590/S0100-55022006000100002
  » https://doi.org/ 10.1590/S0100-55022006000100002
• ¹³
  Pinto AC, de Andrade, JB. Fator de impacto de revistas científicas: qual o significado deste parâmetro? Quim Nova. 1999;22(3):448-53. DOI: 10.1590/S0100-40421999000300026
  » https://doi.org/ 10.1590/S0100-40421999000300026
• ¹⁴
  Stephens C, Akerman M, Maia PB. Health and environment in São Paulo, Brazil: methods of data linkage and questions for policy. World Health Stat Q. 1995;48(2):95-107.
• ¹⁵
  Stephens C, Akerman M., Avle S, Maia PB, Campanário P, Doe B, Tetteh D. Urban equity and urban health: using existing data to understand inequalities in health and environment in Accra, Ghana and Sao Paulo Brazil. Environ Urban.1997;9(1):181-202.
• ¹⁶
  Stephens C, Timaeus I, Akerman M, Avle S, Maia PB, Campanário P, Harpham T. Field study: São Paulo, Brazil. In: Stephens C, organizador. Environment and health in developing countries: an analysis of intra-urban differentials using existing data. London: London School of Hygiene and Tropical Medicine; 1994. p. 49-80.

• a
  Packer A. Comunicação pessoal; 2012.
• b
  Akerman M. Diferenciais intra-urbanos de saúde: estudo de caso de macrolocalização de problemas como estratégia de influenciar políticas públicas In: Equidade e saúde: contribuições da epidemiologia. Rio de Janeiro: ABRASCO; 1997. v.1, p. 177-86.
• c
  Akerman M. Metodologia de construção de indicadores compostos: um exercício de negociação intersetorial In: Condições de vida e saúde. Rio de Janeiro: ABRASCO; 1997. v.1, p. 95-113.
• d
  Folha de São Paulo. Jardim Ângela é o lugar mais perigoso do mundo, 1996, caderno cotidiano, página 3.
• e
  Instituto Sou da Paz. Homicídios caem 73.3% no Jardim Ângela. São Paulo; 2005 [cited 2012 Jan 28]. Available from: http://www.soudapaz.org/Home/tabid/630/EntryID/740/language/pt-BR/Default.aspx
• f
  Centro de Estudos de Cultura Contemporânea. Coordenadoria de Educação Ambiental. Respira São Paulo: resultado da pesquisa sobre a Operação Rodízio. São Paulo; 1997.
• Article based on the Doctoral thesis of Akerman M., entitled: “Impacto. Que impactos? Uma tentativa de diálogo com meu leitor”, submitted to the Faculdade de Saúde Pública, Universidade de São Paulo , 2012.

Publication Dates

• Publication in this collection
  Aug 2013

History

• Received
  3 Feb 2013
• Accepted
  11 June 2013