The fellowships for research productivity (PQ) distributed annually by the National Council of Scientific and Technological Development (Conselho Nacional de Desenvolvimento Científico e Tecnológico - CNPq) generate, historically, a series of concerns in the scientific community. With the objective of enlightening this community in relation to the criteria adopted by the Multidisciplinary Advisory Committee (AC) in Health for the granting of PQ funds, we decided to produce this text.
An essential point is to emphasize that the PQ fellowship of these AC are primarily distributed for researchers with initial training in the areas of the committee and institutional bounds to the units, departments and graduate programs in the areas. Guidance activities in programs not yet certified by the Coordination for the Improvement of Higher Education Personnel (Coordenação de Aperfeiçoamento de Pessoal de Ensino Superior - CAPES) are disregarded, as well as the scientific productions not related to the areas of the committee.
It should be noted that there are minimum criteria to join each level, publicly available in the CNPq website. It is noteworthy that we keep receiving numerous requests which do not meet these minimum criteria, including from undergraduate and graduate students.
A diagnosis of the AC situation is necessary. Physical Education currently has 85 existing fellowships, Physical Therapy and Occupational Therapy have 65 fellowships and Speech Language Pathology and Audiology have 51 PQ fellowships. The fellowships for Physical Education are distributed into eight states of the Federation, considering 40% are allocated to teachers in the state of São Paulo. The fellowships for Physical and Occupational Therapy are allocated to researchers from seven states, considering that 63% of scholars work in the state of São Paulo. In Speech Language Pathology and Audiology, the fellowships are concentrated in six states, considering that 71% of them are allocated to researchers affiliated to São Paulo institutions.
From the total of AC fellowships, 62.5% are level 2, 18.5% of them are 1D, 5.5% are 1C, 8.5% are level 1B and 5.0% of them are level 1A. These percentages are in disagreement with the CNPq regulations, which suggest 10% of fellowships in levels 1A and 1B and 50% in level 2. The AC drafted documents to the board of CNPq requesting the review of this scenario, with an increase in quotas 1, especially in levels 1A and 1C. It is also noteworthy that in the last three trials, there was no allocation of new quotas for our AC. The matter was approached in a recent document sent to the board of CNPq.
From now on, we will present some information on the trial for PQ fellowships in 2015. The most complex evaluation task was to define the indicators to be included in the calculation algorithm, with their respective weights. The AC chose to use five indicators, with the following weights: scientific production in the evaluated period (35%), guidance (25%), the H index of the Institute for Scientic Information - ISI (20%), average citations per article, calculated based on the Scopus (15%) and research project submitted (5%). It should be noted that such criteria were only applied to the applicants who met the minimum criteria published in the CNPq website. For example, researchers with predominant activity in other areas or who did not meet the minimum criteria for guidance and scientific production were eliminated in this first evaluation stage.
Each one of these indicators has a specific algorithm of standardized calculation, considering that the final score ranges from 0 to 100. In the case of the project, they are evaluated by two ad-hoc reviewers. The concept "excellent" is equivalent to the score 5, the concept of "good" is equivalent to 3, the concept "regular" is equivalent to 2 and the concept "weak" is equivalent to 1. The mean of both evaluations is used in the final score. In the trial of 2015, only 16% of the projects were classified as "excellent" by the two evaluators.
As for the H index, initially, a calculation of the distribution percentage 95 is made, considering that this values is equivalent to the maximum score (20). The score for each researcher is calculated by the rule of three in comparison to the percentage 95. In gross terms, the mean H index of applicants in 2015 was 5.2, ranging from 0 to 19. The percentage 95 corresponded to the value 13. The mean H index of Physical Education applicants was 5.4, compared to 6.0 in Physical Therapy and Occupational Therapy and 3.0 in Speech Language Pathology and Audiology.
In relation to the citations, the calculation is made identically as in the H index. In gross terms, the mean of citations per article was 5.6, mean score 5.2 citation/articles in Physical Education, 6.6 in Physical Therapy and Occupational Therapy and 4.0 in Speech Language Pathology and Audiology.
The score of scientific production is calculated in a complex way. Each article published in journals with impact factor over four is equivalent to 100 points. The weight of the publications in journals of lesser impact decreases, reaching 50 publications in journals with impact factors between 0 and 0.5 and 10 for productions in journals without impact factor. Articles published in journals without peer review are disregarded. Each book is equivalent to 80 points and each book chapter is equivalent to 40 points. A bonus is given for articles published as first or last author, and then as second or before last author. This algorithm generates a continuous score, which is then processed in the same way as it was described for the H index and the citations index.
In the guidelines, three points are awarded for the guidance of complete PhD, two points for post-PhD complete supervisions and one point for the guidance of complete Masters degree. Half these values is attributed to ongoing guidance. The final score with weight 25 is calculated in a similar way to the one of the other indicators. However, this indicator has maximum value, equivalent to the titration of 0.5 PhDs and 1 Master per year. All teachers who achieve these values receive the maximum score in the guidance category.
The continuous score, weighting 100, correlates to all indicators. The strongest correlation was with the H index (r=0.83), followed by the scientific production (r=0.73), mean citations (r=0.66), guidance (r=0.58) and research project (r=0.42). The mean final score was 47 points among the applicants of Physical Education, 48 among applicants of Physical Therapy and Occupational Therapy and 36 among the applicants of Speech-Language Pathology and Audiology.
Considering the availability of quotas renovation in 2015, it was possible to fulfill 30% of the demand in Physical Education, 31% of the demand in Physical Therapy and Occupational Therapy and 49% of the demand in Speech-Language Pathology and Audiology. Such numbers prove the competitiveness of the system and stimulate this AC to search for continuous improvement of the evaluation criteria.
The AC conducted then a series of analysis comparing the contemplated researchers and the non-contemplated ones with fellowship quotas in 2015. In Physical Education, the mean H index of the contemplated was 8.3, against 2.0 among the non-contemplated ones. The mean of citations was 8.2 among the contemplated ones and 2.7 among the non-contemplated ones. In Physical Therapy and Occupational Therapy, the differences were smaller. The mean H index of the contemplated was 7.6, compared to 4.8 among the non-contemplated ones. The means of citations were from 11.1 and 4.1, respectively. In the Speech-Language Pathology and Audiology, the mean H index of the contemplated was 3.6, against 1.8 among the non-contemplated ones. The mean citations were from 4.6 among the contemplated and 2.0 among the non-contemplated ones.
The AC would like to use this space to provide some signals for the area. The filling out of the Lattes curricula should be improved; we detected many cases of event summaries inserted as "Complete articles published in journals" in the curricula of the applicants. The same way, editorial and letters to the editor have not always been clearly identified in this way by the applicants. There is a trend in the committee to appreciate more and more the quality of the scientific production, rather than an evaluation based only in quantity. In the trial of 2014, the production scores had relevance of 50 points, being reduced to 35 points in 2015. The H index doubled its values, as well as the weights of citation index and historical guidance increased.
We could not fail to use this space to thank the ad-hoc reviewers of the AC. However, some considerations should be made. In many cases, the opinions are overly succinct, preventing a more adequate understanding of the projects submitted. Besides that, it should be noted that the overall evaluation of the ad-hoc endorsement for the PQ fellowships should focus on the project submitted, rather than on the applicant's curriculum, which is evaluated by the remaining indicators described here. Finally, it is essential that the reviewers judge the proposals based on the criteria established by the AC.
Thus, the AC is committed to publish the detailed results of all trials made, with the only limit of legal restrictions imposed by the CNPq and by the Federal Constitution. Applicants should feel free to send electronic messages to members of the AC in case they have any doubts about the trial or their overall and specific scoring for each indicator.
This editorial is being published simultaneously in the Brazilian Journal of Physical Therapy, Occupational Therapy Journals of UFSCar (Cadernos de Terapia Ocupacional da UFSCar ), CoDAS and Movement (Movimento), and also published in the CNPq website. The objective of this joint publication is to reach out for researchers of different areas which make up the AC. It is expected that this publication stimulates a broad scientific debate of constant improvement of the evaluation criteria of the AC.
Publication Dates
-
Publication in this collection
Dec 2015
