Scientific Performance of Brazilian Researchers in Pharmacology with grants from CNPq: A comparative study within the Brazilian categories

ABSTRACT

In Brazil, scientific performance of researchers is one important criteria for decision-making in grant allocation. In this context, this study aimed to evaluate and compare the profile of 82 seniors' investigators (graded as level 1A-D) which were receiving CNPq (National Council for Scientific and Technological Development) productivity grant in Pharmacology, by analyzing the pattern of citation of their papers and h-index. Total documents, citations (with and without self-citations) and h-index (with and without self-citations) were retrieved from the Scopus database. The results indicated a clear difference among researchers from the higher categories (1A and 1B) in most of the parameters analyzed. However, no noticeable differentiation was found between researchers from grant category 1C and 1D. The results presented here may inform the scientific community and the grant agencies on the profile of PQ 1(A-D) fellows of Pharmacology, and may help to define new differences within CNPq grant categories, and consequently, a better allocation of grants.

Key words:
bibliometric analyses; Brazilian scientific performance; citation analysis; CNPq; pharmacology; scientometrics

RESUMO

No Brasil, o desempenho científico dos pesquisadores é um dos critérios importantes para a tomada de decisões na concessão de bolsas. Neste contexto, este trabalho teve como objetivo avaliar e comparar o perfil de 82 pesquisadores sêniors (nível 1A-D), os quais recebem bolsa de produtividade CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) em Farmacologia, analisando o padrão de citações de seus artigos e o índice h. O número total de documentos, citações (com e sem autocitação) e o índice h (com e sem autocitação) foram extraídos do banco de dados Scopus. Os resultados obtidos indicaram uma diferença nítida entre os pesquisadores de alta categoria (1A e 1B) na maioria dos parâmetros analisados. No entanto, não houve uma diferença notável entre os pesquisadores das categorias 1C e 1D. Os resultados obtidos podem informar a comunidade científica e as agências de fomento sobre o perfil dos bolsistas PQ 1(A-D) em Farmacologia. E podem ajudar a definir novos critérios de diferenciação entre as categorias de bolsas de produtividade, e consequentemente, uma melhor alocação de bolsas.

Palavras-chave:
análises bibliométricas; perfil científico brasileiro; análise de citação; CNPq; farmacologia; cientométria

INTRODUCTION

Scientific performance of an individual researcher can be evaluated at different periods of the career (i.e., by the academic position, grants awards and scientific prizes) or more objectively by using bibliometric parameters. Although bibliometric analyses have been criticized due to its limitations in assessing scientific output of an individual scientist (Campanario and Molina 2009), it is generally accepted by the scientific community as an efficient tool for assessing, measuring and comparing the performance/productivity of scientists, entire institutions, departments, countries, etc (Harnad 2008, Haeffner-Cavaillon and Graillot-Gak 2009, Inglesi-Lotz and Pouris 2011, Jia et al. 2014, Díaz-Faes et al. 2015). In addition, bibliometric analyses play a major role in the critical decision of allocating funding resources and in determining policies of research (Katz 2000, Tian et al. 2008, Li et al. 2010, Pinto and Matias 2011, Besselaar and Sandstrom 2015).

In the evaluation processes of scientific performance, bibliometric parameters such as the total number of published papers and their subsequent citations are of utmost importance (Inglesi-Lotz and Pouris 2011, Peng 2011, Rodríguez-Navarro 2012, Roos et al. 2014, Zhu et al. 2014). In Brazil, the evaluation system from the Coordination for the Improvement of Higher Education Personnel (CAPES) requires high performance from scholars in terms of research, publications and, more recently, the citations has been cogitated as potential parameter for individual and institutional scientific evaluation (Meneghini 2011, Baccini et al. 2014).

Few studies have analyzed the profile and scientific output of Brazilian researchers receiving the CNPq (National Council for Scientific and Technological Development) scientific productivity grants in different fields or subfields of knowledge (Santos et al. 2010, Oliveira et al. 2011, 2013, Leite et al. 2012, Martelli-Junior et al. 2010). Those cited studies were more descriptive in nature and, as role, the citation profile of the researchers has not been evaluated in deep. We argue that citation pattern of a given author can give an indication of his/her influence as generator of new knowledge (by analyzing citations received by original research articles, ORAs) in a field or subfield.

Here, we decided to analyze the citation pattern of researchers currently receiving the productivity fellowship from CNPq in Pharmacology subarea or subfield of biomedical sciences. The selection of this field or subfield in the BF committee (BF is the abbreviation used by the CNPq to describe the field including biochemistry, pharmacology, neuroscience, physiology and biophysics) was based on the fact that authors of the present study have a good knowledge of the area (and subarea) in Brazil, and the choice of parameters was based on the assumed assumption that the citation profile of a researcher may give a better idea about his/her performance than the place where papers were published (i.e., the journal impact factor). Furthermore, since the distribution of CNPq researchers in the levels from D (lowest grade) to A (for consolidate and highly productive investigators) in the category 1 is expected to reflect the degree of independence and the time of experience of the researchers, the analysis of the entire production can give a better idea of the importance of a given researcher in relation to these two aspects. However, to the best of our knowledge, there is no study investigating the profile of CNPq researchers from Pharmacology.

Considering that there is no information regarding the performance of Brazilian researchers in the area of Pharmacology, and that the category 1 of CNPq researchers is attributed only to researchers with scientific productivity (Arruda et al. 2009) and those considered of good quality in terms of comparisons within the Brazilian pairs, the present study was designed to analyze and compare the performance of Brazilian researchers receiving CNPq grants category 1 (1A-D) in Pharmacology. Our research was formulated as follow: (1) What is the scientific profile of those researchers regarding data sources indexed in Scopus? (2) Is there a clear difference within the four levels of CNPq category 1? To answer these questions, we analyzed and compared the total documents; total publications (research articles and review papers) of each researcher by grant category. The bibliometric indicators used here were: total documents, number of original research article (ORA) and review paper (RP), total citation, citation without self-citation, total h-index and h-index without self-citations.

MATERIALS AND METHODS

Participants

A list of eighty two (82) researchers registered as recipients of scientific productivity grant from the National Council for Scientific and Technological Development (CNPq) in Pharmacology was downloaded from CNPq website (www.cnpq.br) in November 2013. They were males and females (Table I) having permanent positions in their institution, distributed in different regions of Brazil (Table II). According to the CNPq classification, these PQ researchers were categorized as 1A, 1B, 1C and 1D. The estimated time of career varied among categories: from 32 to 58 years (grant 1A researchers), from 31 to 43 years (grant 1B), from 21 to 28 years (researchers receiving grant 1C), and from 18 to 27 years (grant 1D). The approximate time of career was calculated by the difference between the last and the first publication (2013 or 2014) indexed in Scopus database).

TABLE I
Distribution of research grant in Pharmacology by gender and CNPq grant category (N = 82) (obtained on December 2013).

CNPq grant category	Male	Female	Total
1A	18 (21.95)	2 (2.44)	20 (24.39)
1B	7 (8.54)	13 (15.85)	20 (24.39)
1C	11 (13.41)	13 (15.85)	24 (29.27)
1D	9 (10.98)	09 (10.98)	18 (21.95)
Total	45 (54.88)	37 (45.12)	82 (100)

• The number in parenthesis indicates the percentage in relation to the total number of fellows (N = 82).

TABLE II
Distribution by institution, region and state of CNPq research grant (PQ 1A-D) in Pharmacology.

Institution	Region	State	Number of grant	%
USP-Universidade de São Paulo	SE	SP	21	25.61
UFSC-Universidade Federal de Santa Catarina	S	SC	09	10.97
FIOCRUZ-Fundação Oswaldo Cruz	SE	RJ	07	08.54
UFC-Universidade Federal do Ceará	NE	CE	07	08.54
UFMG-Universidade Federal de Minas Gerais	SE	MG	07	08.54
UNICAMP-Universidade Estadual de Campinas	SE	SP	06	07.32
IBU-Instituto Butantan	SE	SP	04	04.88
PUCRS-Pontifícia Universidade Católica do Rio Grande do Sul	S	RS	04	04.88
UFSM-Universidade Federal de Santa Maria	S	RS	04	04.88
UNIFESP-Universidade Federal de São Paulo	SE	SP	03	03.66
INCA-Instituto Nacional do Câncer	SE	RJ	02	02.44
UFRGS-Universidade Federal do Rio Grande do Sul	S	RS	02	02.44
UFRJ-Universidade Federal de Rio de Janeiro	SE	RJ	02	02.44
UCS-Universidade de Caxias do Sul	S	RS	01	01.22
UERJ-Universidade do Estado do Rio de Janeiro	SE	RJ	01	01.22
UFCSPA-Universidade Federal de Ciências da Saúde de Porto Alegre	S	RS	01	01.22
UFPB-Universidade Federal da Paraíba	NE	PB	01	01.22
Total	-	-	82	100

• SP, São Paulo; RJ, Rio de Janeiro; RS, Rio Grande do Sul; PB, Paraíba; CE, Ceará; MG, Minais Gerais; SC, Santa Catarina. Dezember, 2013. Among the 26 states and the 5 regions of Brazil, the grants were distributed only in 7 states and 3 regions.

Data Collection

The scientific performance of each researcher was evaluated using the Scival-Scopus database (www.scopus.com). The data were collected from December 2013 to June 2014. Each researcher was evaluated for his/her total production (entire career).

The parameters used to evaluate the entire career of the researchers were: total documents indexed in Scopus database, number of articles (original research and reviews), total citations (with and without self-citations) and h-index (with and without self-citations).

Statistical Analysis

The results are expressed as mean by researcher category ± S.E.M (standard error of mean).

RESULTS AND DISCUSSION

Scientific Productivity and Total Citations

The distribution of total documents published during the entire career of the researchers as retrieved from the Scopus database in April 2014, is shown in Fig. 1a. The total documents included: conference paper, erratum, article, letter, editorial, notes, review, short survey and book chapter. Researchers from grant category 1A published about 250 documents, which is about two times the number published by researchers in the category 1B. A similar number of documents was found for researchers in category 1C and 1D, with an average of 102.83 and 91.39 respectively (Fig. 1a).

Figure 1
Distribution of total documents (a), original research articles (ORA) (b) and review papers (RP) (c) published by Brazilian researchers from CNPq grant category 1A-D in Pharmacology. The numbers above indicate the mean of total documents, ORA or RP per researcher.

From the total documents indexed in Scopus database, we determined the number of original research articles (ORA) and review papers (RP) (Fig. 1b, c). The results demonstrate that the average of ORA published by researchers from grant category 1A (220.4) is almost two times higher than those published by researchers from grant category 1B (111.55), and that researchers from category 1C (88.29) and 1D (83.33) have similar average of published ORA (Fig. 1b). The range of distribution of total ORA by grant category was as follow: 1A (62-437), 1B (49-277), 1C (30-226) and 1D (45-138).

Similar to ORA, researchers from 1A category had the highest number of RP with an average of 9.85 (1-43), followed by researchers from grant category 1B (0-28), with a mean of 7.05 RP per researcher (Fig. 1c). The average number of RP per researcher from grant category 1C and 1D was similar (3.33) with a distribution range of 0-18 and 0-19, respectively (Fig. 1c).

Citations are generally viewed as an important indicator of the impact of the researcher in science, and it therefore serves as an objective and quantitative indicator for evaluating research performance. The h-index measures the cumulative impact of publications of a scientist, considering that the number of citations is a useful variable to evaluate the relevance and/or recognition of a paper. In the present study, the number of citations and the h-index were analyzed as quantitative (and qualitative) indicators of a scientist receiving CNPq fellowships. The results presented in Fig. 2a indicate that the number of total citations received by the researchers during the entire scientific career was related with the category of the researchers. Researchers from 1A category received 2.5 times more citations than researchers of 1B category. The mean of total citations was 5529.5 for 1A and 2207.6 for 1B category, with a distribution range of 1755 to 14349 (1A) and 842 to 6974 (1B). The total citations of researchers in category 1C and 1D presented almost the same pattern of citations, with 1622.08 (226-5865) and 1527.11 (467-3037) for 1C and 1D, respectively (Fig. 2a).

Figure 2
Distribution of the total citations with (a) and without self-citations (b) of Brazilian researchers from CNPq grant category 1A-D in Pharmacology. The numbers above indicate the mean of citations per researcher.

Self-citations of the authors have been reported to inflate their h-index (Bartneck and Kokkelmans 2011). Therefore, when the self-citations of all authors was excluded from the total citations, the mean of citations decreased from 5529.5 to 3911.35 (1044-10800) for grant category 1A, from 2207.6 to 1673.7 (524-5788) for 1B, from 1622.1 to 1147.42 (150-4261) for 1C and from 1527.1 to 1070.39 (323-1911) for 1D (Fig. 2b).

H -Index

The h-index (with and without self-citations) associated with the total citations of the researchers is illustrated in Fig. 3a, b. The results show that the mean of h-index of researchers from grant category 1A was higher than that of category 1B, while that of researchers from grant category 1C and 1D was almost equal (Fig. 3a). The same tendency was observed when analyzing the h-index without self-citations (Fig. 3b). In fact, the h-index reflects both the number of publications and the number of citations per publication. Most of researchers in 1A and 1B categories are normally older than that of low categories; however, their "academic age" was not taken into account in this study. Furthermore, the scopus database considers only the citations received from 1996 to the present (in the present study). The h-index grows as citations accumulate and thus it depends on the "academic age" of the researcher. Based on the fact that only the most highly cited articles contribute to the h-index, we can presume that the higher level of the mean of h-index of grant category 1A and 1B, in comparison to grant category 1C and 1D may be due not only on their highly cited papers, but also to differences in the professional age.

Figure 3
Total h-index with (a) and without self-citations (b) of Brazilian researchers from CNPq grant category 1A-D in Pharmacology.

FINAL CONSIDERATIONS

In summary, the present study evaluated and compared the profile of CNPq researchers receiving grant productivity 1A, 1B, 1C and 1D, in the area of Pharmacology, based on their total production (i.e., throughout their scientific career) as indexed in the Scopus database. In general, the results showed in this study indicated differences between researchers from grant category 1A when compared to 1B in terms of number of documents indexed, number of research articles and review papers, citations, h-index. On the other hand, there were no noticeable differences between grant category 1C and 1D in most of the parameters analyzed. Based on the bibliometric aspect investigated, it is possible to conclude that researchers from grant category 1A are well ahead of researchers of inferior categories. Consequently, the overall assessment seems to be on the expected path. However, there are some distortions in performance that are difficult to conciliate with relativistic performance analysis. Distortions occurred in all levels of fellowship. In the case of 1A and 1B researchers, it can represent the presence of some old researchers that are now no more active in terms of research, but that had done important contributions to Brazilian science in the past. The low bibliometric performance of those researchers was worsened by the fact that Scopus database only consider the citations received after 1996. Consequently, to some of those old researchers, important contribution to the field in the past might be neglected. About 10 years ago, CNPq created the senior category to encompass the researchers that had contributed in the past and now are no more as active scientist as they were in the past. The existence of such type of distortions in relation to those researchers that have contributed in the past is acceptable and may reflect the relativistic weight in the analysis and distribution of resources in Brazilian sciences. Optimistically, the presence of low performing subjects in all the categories have to be better scrutinized to define distortions that cannot be explained by relativistic performance. Indeed, the presence of researchers with low number of papers, low levels of citations suggests that his or her classification as productive scientist is biased by faults in the judgment system or by corporative interference. Overall, the results presented here may inform the scientific community and the grant agencies on the profile of PQ 1(A-D) fellows of Pharmacology, and may help to define new differences within CNPq grant categories, and consequently, a better allocation of grants.

LIMITATIONS OF THE STUDY

As our searches were performed until June 2014, citation counts of the latest publications may not reflect the actual values of these researchers. Another limitation can be the source of data, because, not all scientific literature are published in sources indexed by Scopus. In addition, Scopus presents some problems in indexing periodicals; however, the problems found in Scopus are similar to those found in other important sites such as Web of Science and Google Scholar.

It should be stressed that there were some difficulties in this work. For instance, one review paper published in Chemical Reviews was found in articles section. In such case, we considered the articles as review paper, because of the knowledge that this journal only publish review papers.

ACKNOWLEDGMENTS

This work was supported by grants from Universidade Federal do Rio Grande do Sul (UFRGS), Universidade Federal de Santa Maria (UFSM), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), CNPq - The World Academy of Sciences (TWAS), Fundação de Aparo À Pesquisa do Estado do Rio Grande do Sul (FAPERGS), Financiadora de Estudos e Projetos (FINEP), Rede Instituto Brasileiro de Neurociência (IBN Net) and Instituto Nacional de Ciência e Tecnologia em Excitotoxicidade e Neuroproteção (INCT-EN).

REFERENCES

• Arruda D, Bezerra F, Neris VA, Toro PR and Wainer J. 2009. Brazilian computer Science research: Gender and regional distributions. Scientometrics 79: 651-665.
• Baccini A, Barabesi L, Cioni M and Pisani C. 2014. Crossing the hurdle: the determinants of individual scientific performance. Scientometrics 101: 2035-2062.
• Bartneck C and Kokkelmans S. 2011. Detecting h-index manipulation through self-citation analysis. Scientometrics 87: 85-98.
• Besselaar P and Sandstrom U. 2015. Early career grants, performance, and careers: A study on predictive validity of grant decisions. J Informetr 9: 826-838.
• Campanario JM and Molina A. 2009. Surviving bad times: the role of citations, self-citations and numbers of citable items in recovery of the journal impact factor after at least 4 years of continuous decreases. Scientometrics 81: 859-864.
• Díaz-Faes AA, Costas R, Galindo MP and Bordons M. 2015. Unravelling the performance of individual scholars: Use of canonial biplot analysis to explore the performance of scientists by academic rank and scientific field. J Informetr 9: 722-733.
• Haeffner-Cavaillon N and Graillot-Gak C. 2009. The use of bibliometric indicators to help peer-review assessment. Arch Immunol Ther Exp (Warsz) 57: 33-38.
• Harnad E. 2008. Validating research performance metrics against peer rankings. Ethics Sci Environ Polit 8: 103-107.
• Inglesi-Lotz R and Pouris A. 2011. Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa. Scientometrics 88: 747-760.
• Jia ZJ, Hong B, Chen DM, Huang QH, Yang ZG, Yin C, Deng XQ and Liu JM. 2014. China's growing contribution to global Intracranial aneurysm research (1991-2012): A bibliometric study. PloS ONE 9: e91594.
• Katz JS. 2000. Scale-independent indicators and research evaluation. Sci Public Policy 27: 23-36.
• Leite BD, Oliveira EA, Queiroz IN, Martelli DR, Oliveira MC and Martelli Junior H. 2012. Profile of the researchers with productivity grants in the Brazilian National Research Council (CNPq) of the physical education area. Motricidade 8: 90-98.
• LI J, Sanderson M, Willett P, Norris M and Oppenheim C. 2010. Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments. J Informetr 4: 554-563.
• Martelli-Junior H, Martelli DR, Quirino IG, Oliveira MC, Lima LS and Oliveira EA. 2010. CNPq-supported medical researchers: a comparative study of research areas. Rev Assoc Med Bras 56: 478-483.
• Meneghini R. 2011. Citations to papers from Brazilian institutions: a more effective indicator to assess productivity and the impact of research in graduate programs. Braz J Med Biol Res 44: 738-747.
• Oliveira EA, Pécoits-Filho R, Quirino IG, Oliveira MC, Martelli DR, Lima LS and Martelli-Júnior H. 2011. Profile and scientific production of CNPq researchers in Nephrology and Urology. J Bras Nefrol 33: 31-37.
• Oliveira MC, Martelli DR, Pinheiro SV, Miranda DM, Quirino IG, Leite BG, Colosimo EA, Silva AC, Martelli-Júnior H and Oliveira EA. 2013. Profile and scientific production of Brazilian National Council of Technological and Scientific Development researchers in Pediatrics. Rev Paul Pediatr 31: 278-284.
• Peng C. 2011. Focus on quality, not just quantity. Nature 475: 267.
• Pinto A and Matias M. 2011. Indicadores Científicos e as Universidades Brasileiras. Informação 16: 1-18.
• Rodríguez-Navarro A. 2012. Counting highly cited papers for University research assessment: conceptual and technical issues. PLos ONE 7: e47210.
• Roos DH, Calabró L, De Jesus SL, Souza DO, Barbosa NV and Rocha Jbt. 2014. Brazilian scientific production in areas of biological sciences: a comparative study on the modalities of full doctorate in Brazil or abroad. Scientometrics 98: 415-427.
• Santos Ncf, Cândido Lfo and Kuppens CL. 2010. Produtividade em pesquisa do CNPq: análise do perfil dos pesquisadores da química. Quím Nova 33: 489-495.
• Tian Y, Wen C and Hong S. 2008. Global scientific production on GIS research by bibliometric analysis from 1997 to 2006. J Informetr 2: 65-74.
• Zhu J, Hassan S, Mirza HT and Xie Q. 2014. Measuring recent research performance for Chinese universities using bibliometrics methods. Scientometrics 101: 429-443.

Publication Dates

• Publication in this collection
  15 Aug 2016
• Date of issue
  2016

History

• Received
  17 July 2015
• Accepted
  07 Dec 2015