Measures of Experience and Scientometrics to Evaluate the Impact of Scientific Production

" Experiencimetrics " is proposed as a measure of researchers' academic experience, to complement the existing measure, scientometrics, when evaluating expertise in research. The article is partly based on the author's scientific output, and explores the possibilities of analyzing impact from a qualitative perspective. A synthesis is produced, with possible implications for this analysis, highlighting some options for evaluating the impact of research and articles which are not limited to counting author's citations in order to calculate the impact factor. When considering the ingenuity of the criteria of meritocracy engendered by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) to qualify postgraduate programs, it is necessary both to improve the quantitative mechanisms and to examine new ways of evaluating the impact of the researchers and programs. On following the process of accreditation of a Masters as postgraduate coordinator, I published an opinion piece entitled: " There are many doctors, but few 'social entrepreneurs' ". 1 Entrepreneurship or taking a risk would be the expected result of the " research – teaching-extension " triad and combined mechanisms of evaluation would need to be practiced. As mentioned in Ribeiro, RJR " the university will truly become university when the prince (research) kisses the frog (extension) " (Almeida Filho, 2007). 8 And how is this to be done? Perhaps by identifying when research also " serves humanity and not just science ". Borrowing Packer's expression, a that " a doctor may have read one scientific article and have saved two lives " , by not reporting this in a systematic manner to the scientific community, " he helps humanity, but not science ". Thus, the author of said article read by the doctor will never know of the impact caused by article. Moreover, Packer a (2012) states that the scientific community recognizes the need for this type of qualification of research or of its repercussions on the life of the subjects; but, for lack of a validated methodology, we continue to use the enshrined measure of scien-tometrics 3 to evaluate the impact of journals, articles and researchers. Would this attempt to qualify the impact of research/ article/researcher not be the subterfuge of someone who would never achieve the competitive quantitative levels, hiding behind qualitative raptures? We have not reached a conclusion on this insinuation. According to Packer a (2012), the " science of citations " is also susceptible to Pareto's Law, …

When considering the ingenuity of the criteria of meritocracy engendered by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) to qualify post-graduate programs, it is necessary both to improve the quantitative mechanisms and to examine new ways of evaluating the impact of the researchers and programs.
On following the process of accreditation of a Masters as post-graduate coordinator, I published an opinion piece entitled: "There are many doctors, but few 'social entrepreneurs'". 1 Entrepreneurship or taking a risk would be the expected result of the "research -teaching-extension" triad and combined mechanisms of evaluation would need to be practiced.As mentioned in Ribeiro, RJR "the university will truly become university when the prince (research) kisses the frog (extension)" (Almeida Filho, 2007). 8d how is this to be done?Perhaps by identifying when research also "serves humanity and not just science".Borrowing Packer's expression, a that "a doctor may have read one scientific article and have saved two lives", by not reporting this in a systematic manner to the scientific community, "he helps humanity, but not science".Thus, the author of said article read by the doctor will never know of the impact caused by article.
Moreover, Packer a (2012) states that the scientific community recognizes the need for this type of qualification of research or of its repercussions on the life of the subjects; but, for lack of a validated methodology, we continue to use the enshrined measure of scientometrics 3 to evaluate the impact of journals, articles and researchers.
Would this attempt to qualify the impact of research/ article/researcher not be the subterfuge of someone who would never achieve the competitive quantitative levels, hiding behind qualitative raptures?We have not reached a conclusion on this insinuation.According to Packer a (2012), the "science of citations" is also susceptible to Pareto's Law, known as the "80-20 principle", in which 80% of citations would always be attributed to 20% of researchers.Coimbra Jr. 10 argues that the field of Public Health is not represented in the "lions share" of the most cited journals, not even in international journals, such as the American Journal of Public Health and the International Journal of Epidemiology.iNTRoDUCTioN So the "crumbs" of research promotion would be left for those 80% of researchers with an inferior number of citations?Of that contingent of research, what has the greatest potential to help humanity?Are there other forms of evaluating the impact of articles that does not involve simply counting citations?
The aim of this article was to suggest possibilities for the qualitative analysis of the impact of scientific production.

A PoSSiBLe STARTiNG PoiNT
In 1993, a broad piece of research entitled "Health and Environment: a study of intra-urban differences in developing countries, in Acra, capital of Ghana and Sao Paulo, Brazil", 16 demonstrated the existence of a social gradient with regards the risk of dying from various selected chronic diseases, external causes, respiratory disease and communicable diseases when comparing different geographical territories in the cities studied. 7,14,15re than an ecological epidemiological study of intra-urban differences, it practiced the exercise in negotiation of constructing composite indicators, b,c which influenced a set of research into violence, quality of life and social exclusion/inclusion. [4][5][6] One of those pieces of research was the Violence Risk Map of Sao Paulo, SP, Southeastern Brazil. 6This initiated maps of violence risk in the cities of Curitiba, PR, Rio de Janeiro, RJ, Southeastern Brazil and Salvador, BA, 3 Northeastern Brazil, forming part of Federal Government efforts to draw up a National Human Rights Plan, created in 1996 by the Ministry of Justice.
It was decided to report the results of these colored maps in the form of an easy-to-read bulletin, divulged as a tool for informing policies and raising society's awareness of the magnitude of urban violence.
Following this logic, it was decided to report the results of mapping of violence in the city of Sao Paulo, exclusively, in that city's press.
exceeded the rates of the Colombian cities, at the time considered to be extremely violent.Following this, there was a steep decline in the violence rates in Jardim Ângela which persists to this day.There was a 73.3% drop in the homicide rate, according to the text below: The district of Jardim Ângela, in the south of Sao Paulo, with a population of 300 thousand inhabitants was, in 1996, considered by the UN to be the most violent place on the planet.To give an idea of the gravity of the situation, in 2001, 277 murders were recorded.Since then the rate has dropped year on year and, in 2004 the district recorded 151 murders, a reduction of 45%.
But the best was yet to come, this year, at time of writing, 26 violent deaths have been recorded, compared with 65 in the same period last year, and 120 in 2001.This shows that in fewer than four years the homicide rate in Jardim Ângela has fallen by more than 73%.Instituto Sou da Paz. e It is not intended that this narrative establishes a cause-effect relation between the headline in 1996 and the significant decrease in homicide rates in 2005.However, this headline has repercussions up to the present day in newspapers and seems to play a social marketing role, influencing the mobilization and rolling out of actions aimed at this area, which was pushed into the spotlight by the headline, which did nothing more than illuminate a situation that had existed long before the headline.
It seems that a newspaper headline, which does not come within the scope of science, indicated that the research may have had a "great impact on humanity".

oTHeR TYPeS oF iMPACT
Who has not heard the expression "This had a huge impact on me"?With this expression, the subject is talking of something that affected them, something they experienced.
According to Bondía:  finding, being very quiet, being patient and giving oneself time and space". 9us, Bondía 9 alerts us that a society "constituted under the sign of information" makes experiencing difficult.
There is no room here for the conflict between knowledge/information and knowing from experience/ feeling, each pair with their purpose, their ethics and aesthetics, but we must advocate for adequate tension between the two pairs.However, Bondía 16 comments that modern science distrusts experience and creates a safe path translated by the method which reduces experience to experimentation.
If the logic of experimentation is agreement, consensus and homogeneity, in knowing from experience, difference, heterogeneity and singularity are important. 9 the scientometric base 13 is the impact factor and other bibliometric indicators producing objective, regular, repeatable, comparable, impersonal, infinite, universal results, it would be very daring of Brazilian researchers to be able to explore the development of "experiencimetrics", which would produce personal and comparable results and which would also contribute to better understanding of the scientific research mechanism as a social activity.

ReSiSTeNCe To A DeCLAReD eLiTiSM?
In the "preamble" of the "rodízio" traffic law in Sao Paulo -limiting which days cars with particular number plates can be used -it states: "according to research by Prof. Pedro Jacobi, who pointed out the perceived increased pollution in the city by housewives in a neighborhood by the quantity of soot forming on the sheets hanging out on the clothes lines".f On seeing this described in the law, it is interpreted immediately as being a genuine impact of scientific research.
Universities in Europe are in the process of creating ways of evaluating impact that are not limited to citation indexes, which frequently lead to opinions that the academic world is split between those who do "little science" and those who do "real" science when comparing the citation indexes for different researchers.
What needs to be done is to find forms of inducing/ designing a wider repertoire of indicators for evaluating the career of a university professor, considering other fields of activity, measured with the same value as the capacity for disseminating scientific articles, so that the quantitative criterion of citations is not solely predominant.

FiNAL CoNSiDeRATioNS
First, it is necessary to recognize the provisory character of this work -tentative and precarious, in the words of Deleuse & Guattarri: 11 "Also, above all in the theoretical domain, any pragmatic rough sketch is better than a copy of concepts with cuts and progresses which never change.The imperceptible rupture instead of a significant cut." Second, it is necessary to recover the context of academic life so as to "tighten" meritocracy in research as a determining principle in evaluating academic performance and advocate that other areas of academic performance also acquire the same degree of evaluative relevance as research and publishing.
The third point that needs to be highlighted is: what is the purpose of science?To help humanity or to selfreproduce among peers?How is it possible to know if it is really helping humanity?At the moment we only have the quantitative methods already validated, derived from scientometrics, but we lack methods which could also qualify the impact of science/research in the sense of its "contribution to humanity".Moreover, the most enshrined method -the impact factor measured by number of citations -has its limitations and critics such as, for example, that attributed by Pinto & Andrade: 13 "Science could benefit from the application of these scientometric parameters, but its growth and quality demand work in all directions so that important areas of knowledge do not atrophy for not being in fashion, like those which gather the highest number of scientists and only appear in high impact factor journals..." 13 In order to achieve the aspiration that research's main purpose is to "help humanity", indirect criteria could be established, for example, potential or realized capacity of research to influence public policies: (1) level of participation of social players in the process of drawing up the research; (2) communication strategies between the social players and the researchers involved; (3) clarity in defining objectives; (4) visibility of the research process and dissemination of the results; (5) meeting deadlines; and (6) achieving previously established objectives. 2 These criteria are only a first approximation for evaluating the elements research that aim to influence policies.Other elements may not have been identified, and other studies exploring these issues are welcome, as would a clearer definition of indicators which allow the association between these elements and influence on public policies to be certified.In this discussion, the emphasis is on the fact that, in the case of there being genuine "scientific willingness" on the part of the researchers in influencing policy through their work, the abovementioned criteria need to be directly and explicitly included in the design of the research as elements that could potentially influence the formulation of public policies. 2us, these elements indicate the possibility of searching for a mode of analysis to highlight "knowledge from experience" of the research in influencing public policies, blazing trails to constitute evaluative "experiencometrics".
A fourth thread of discussion, connected to the previous one, proposes inserting perceived and documented "narratives" (knowledge from experience) of the impacts of the research, for example, in subjects, population groups, territories or laws such as those exemplified here: (1) in the case of the "Rodízio" law and the influence of research by Prof. Pedro Jacobi and (2) the case of the Violence Risk Map of Sao Paulo City and its connection with "Jardim Ângela"."Narrative is a linguistic form characterized by: presenting finite and longitudinal time sequence; presupposing the existence of a narrator and an audience, whose visions of the world are embedded in how the stories are told;... engaging the audience and inviting them to interpret." 12erefore, it would become the role of researchers to position themselves as a collector of experiences, inspired by the willingness to share the perceived and observed impact of their research.Such an exercise may even be encouraged by preconceived scripts during the evaluation processes, and incorporated into CAPES or into the Currículo Lattes network, e.g., and they would have some weight in evaluating research, post-graduate programs and requests for subsidies to development agencies.
Grossman & Cardoso 12 give some indications that may serve for proposing this script when they affirm that "narratives are structured on five elements, namely: the facts, the characters, time, space and the narrator".Moreover, it is possible to suggest creating a script which would come with documental material in order to support and state in detail the facts. 9 Instituto Sou da Paz.Homicídios caem 73.3% no Jardim Ângela.São Paulo; 2005 [cited 2012 Jan 28].Available from: http://www.soudapaz.