Scientific output in the collective health area: journal profile and evaluation by Capes

Abstracts

A saúde coletiva se caracteriza como campo do conhecimento abrangente e complexo, apresentando-se como desafio para o processo de avaliação da pós-graduação realizado pela Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes). O objetivo do trabalho foi descrever o panorama da pós-graduação em saúde coletiva e discutir aspectos dos critérios de sua avaliação. A avaliação da pós-graduação é realizada anualmente, levando em conta a estrutura do programa, corpo docente e discente, atividades de pesquisa e formação, teses e dissertações e produção intelectual. A avaliação deste último item tem sido criticada por privilegiar a publicação de artigos e ancorar a valoração da produção nas bases de indexação dos periódicos e nos seus índices de impacto. Apesar das críticas e restrições ao processo de avaliação, a análise da produção científica brasileira, no geral e no caso específico da saúde coletiva, mostra um quadro positivo, com aumento expressivo do número de cursos de pós-graduação, docentes e alunos. Assim, o panorama da pós-graduação se apresenta favorável, mas aperfeiçoar a avaliação continuamente é essencial para que ela cumpra o seu papel de fomento e estímulo.

Avaliação; Educação de pós-graduação; Saúde pública; Publicações; Produção científica

Scientific output in the Collective Health area: journal profile and evaluation by Capes

Aluísio J D Barros

Programa de Pós-graduação em Epidemiologia. Universidade Federal de Pelotas. Pelotas, RS, Brasil

^{Correspondence}

ABSTRACT

Collective Health is characteristically a broad and complex field of study, presenting itself as a major challenge for the evaluation process carried out by Capes. The aim of the present work was to provide a panorama of postgraduate education in the field of collective health and to discuss aspects of the criteria adopted for its evaluation. The evaluation of postgraduate programs is carried out on a yearly basis. For this, program structure, its academic staff and students, research and teaching activities, theses and dissertations, and intellectual production are taken into account. The evaluation of the latter has been subject of criticism for favoring the publication of papers and for basing ranking of production on journal indexing and impact factors. Despite criticisms and reservations towards the evaluation process, the analysis of the Brazilian scientific production, in general and in the case of Collective Health, shows a very positive scenario, with an important increase in the number of postgraduate courses, teachers, and students. Thus, the panorama of postgraduate programs is favorable, but continuously rethinking and perfecting the evaluation process is essential to guarantee its contribution to the growth and strengthening of the system.

Keywords: Evaluation. Education, graduate. Public health. Publications.

INTRODUCTION

The history of collective health in Brazil, as is often the case in this country, is the result of a long process, begun in the late nineteenth century, the time of Oswaldo Cruz' struggle against yellow fever and other contagious diseases. This history is outlined by Nunes¹¹ in a recent article, where several other references may be found. Throughout this process, the collective health area has incorporated ideas of political character, derived from the social sciences and linked to public health issues, but in which public health remained as the major objective. From this perspective, the biological sciences in general, and medicine in particular, have always had a dedicated space in the collective health area, as has the quantitative approach characteristic of epidemiology. In addition to this diversity, the ever faster progress in technology and knowledge have brought collective health into closer contact with other fields of knowledge, such as engineering, genetics, pharmacology, chemistry, architecture, and economics.

In this context, collective health places in intense – and often uncomfortable – proximity professionals from various areas, with different work tools, and frequently with very distinct immediate goals. Such divergence has been the subject of countless reflections on the definition of collective health as a field of knowledge and of its object, as discussed by Nunes.¹¹ The integration of different areas or disciplines that coexist within the domain of collective health is often insufficient.¹⁰ Nevertheless, there are interesting examples of successful associations, such as the work of Behague et al² on caesarian sections in Brazil, which associates an epidemiological survey with ethnographical research, to cite just one example. Ultimately, such diversity provides a range alternatives to approach the various aspects of that which is the point of convergence of the different interests: population health.⁵

Epistemological reflections define collective health as a "scientific field",¹² or as a "complex disciplinary domain",⁵ rather than as a discipline or specialty of medicine. The scope and complexity of the area has been the subject of a large body of literature. A topic of intense debate within this literature is the evaluation of postgraduate programs in the field of collective health,^5,9 which is the object of the present article.

Although scarce when compared to other areas, postgraduate programs in collective health show an impressive diversity. Subjects covered include environmental and occupational health; epidemiological surveillance, from endemic diseases to medication use; sociological, anthropological, and historical aspects of health, disease, and health care, including the evaluation of health care systems, programs, and services; and epidemiology applied to prevention, from primary prevention, at the policy level, to tertiary prevention, closely related to clinical medicine and surgery. Its intimate association with different levels of management in the Sistema Único de Saúde (SUS - Brazilian Health System) is another important characteristic of collective health in Brazil, the area's scientific capital being fundamental for the definition of policies in a system that is intended to be universal, integral, and equitable.

Thus, the evaluation process currently conducted by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes - Coordination for the Improvement of Higher Education Personnel) faces the challenge of understanding and valorizing a diverse and complex area, which is expected to increase in quantity and quality due to its strategic importance to the country. Thus, the aim of the present article was to describe the process of evaluation of Brazilian postgraduate programs by Capes, and to discuss certain aspects relevant to public health, especially regarding the value attributed to of scientific output.

EVALUATION: OBJECTIVES, STRATEGIES, AND CRITERIA

The primary aim of the Capes evaluation process is to "promote the evolution of the entire Sistema Nacional de Pós-Graduação (SNPG - National Postgraduate System), and of each individual program, by setting goals and challenges that express current advancements in science and technology and the increase of national competence in this field",¹ In addition, other goals include "contributing to the improvement of each individual program", "contributing to the increase in program efficiency", and "providing subsidy for the definition of postgraduate development policies and bases for decision making by governmental agencies regarding investments". In parallel to goals related to the progress of and the investment of public resources in postgraduate education, a further goal, dealing with the laws regulating postgraduate education, is to "provide the basis, in terms of current regulations, for the National Education Council's appraisal regarding the authorization, recognition, and renewal of recognition of Brazilian Master's and Doctoral programs – a legal requirement for such programs to emit diplomas with national validity, recognized by the Ministry of Education".

This last goal is especially important in a country where the demand for postgraduate training is large, and poorly met by the public higher education network. Without proper regulations and accreditation, the postgraduate system in Brazil would be easy prey for institutions more concerned with profit than with providing high-quality training. This could reach enormous proportions in an academic milieu in which the evaluation of merit is strongly associated with the degrees one has obtained. Therefore, requiring a minimal degree of quality from postgraduate programs is an important role for this process, although it is not the main focus of evaluation.

The evaluation of established programs (recommended by Capes) is currently coordinated by a representative and an associate coming from the field being evaluated, who receive direct aid from a team indicated by the representative and sanctioned by Capes. The task of this Evaluation Committee is to evaluate programs on a three-year basis and to revise and update evaluation criteria in consonance with researchers in the field, represented by the Forum of Postgraduate Program Coordinators, and, in the case of collective health, by the Associação Brasileira de Pós-graduação em Saúde Coletiva (Abrasco - Brazilian Association of Collective Health Postgraduate Programs). Evaluation is divided into two stages: the continuous evaluation of program evolution (or lack thereof), which takes place during the first two years and does not alter the grade received by the course, and the triennial evaluation, which occurs during the last year, and which, based on the information accumulated throughout the entire three-year period, leads to the revision of the program's current grade.

Programs are graded as follows: 2 (insufficient, leading to the interruption of the program's activities), 3 (regular), 4 (good), and 5 (very good). Programs considered as "outstanding" and which include doctoral training may receive grades 6 and 7. These grades require clear international insertion, in addition to a strong academic output. Programs are evaluated based on seven items: program purpose, faculty, research activities, training activities, students, theses and dissertations, and intellectual output. The evaluation form currently in use was introduced in 1997, when Capes promoted an extensive revision of the evaluation process. On this occasion, each area was required to define, within certain limits, the weight of the criteria that were part of each item. This contribution from the evaluated field itself was part of a movement towards peer-reviewed evaluation. Despite the participation of all areas in the revision of the evaluation form, an overall change in philosophy was obvious, and Fonseca⁶ (2001) considered that the room for maneuver within each area was limited. A new five-section form that includes socially relevant indicators was recently approved by Capes (June 2006) and will undergo a similar weighting process before it is used in the next evaluation. Current forms and criteria may be obtained from the Capes website.²

The most controversial and widely debated criteria are, without doubt, those related to intellectual output. The revision of the evaluation process prioritized the divulgation of intellectual output in wide-circulation media, essentially journals. In many areas, intellectual output began to be classified according to the characteristics of journals. Journals were classified using a system named "Qualis",³ which considers both circulation and impact factor. This system predicts three levels of quality (A, B, and C) and three levels of circulation (international, national, and local). In the great area of health sciences (medicine i, ii, and iii, nursing, dentistry, pharmaceutics and physical therapy/physical education, and collective health), the following criteria were defined:

• International A and B: journals indexed in the Science Citation Index and Social Sciences Citation Index of the Institute for Scientific Information (ISI), currently named Thomson Scientific.

  ³ Journals with impact factors above the median of the evaluation area are considered as level A, and all others as level B.

• International C: journals indexed in the MEDLINE, International Pharmaceutical Abstracts (IPA), International Nursing Index (INI); Cumulative Index to Nursing & Allied Health Literature (CINAHL), SportDiscus, ERIC, Tropical Diseases Bulletin, Sociological Abstracts, and Planning/Policy & Development databases.

• National A: journals indexed in the SciELO (Scientific Eletronic Library Online) database.

• National B: journals indexed in the LILACS (

  Literatura Latino-Americana de Ciências da Saúde), EMBASE Excerpta Medica, and Psyclit/PsycInfo, databases, or by national scientific societies representative of the area of evaluation.

• National C: other journals. This category does not apply to health sciences.

In a decision supported by intense debate at the Forum of Postgraduate Program Coordinators and Abrasco, three journals considered as central to the divulgation of knowledge and as essential for dialogue between researchers in collective health were classified as Qualis International A: Revista de Saúde Pública (FSP/USP), Cadernos de Saúde Pública (ENSP/Fiocruz), and Revista Panamericana de Salud Pública (OPAS).

FAR FROM A CONSENSUS

Criticisms to the postgraduate evaluation process, in the form of articles, letters, and e-mails, have been frequent. Despite criticisms, the evaluation process only makes sense if thought of as an instrument for fostering the development of postgraduate programs, based on the views of the members of the area itself, and which indicates the best course to be followed.

The great majority of criticisms to the evaluation of collective health are not against evaluation per se, that is, are not against the concept of evaluating, but are mostly constructive in character, aiming to improve the evaluation process. Restrictions have been made to certain aspects of the evaluation process, or of the criteria employed. One of these is related to the inadequacy of current criteria to the typical output of social sciences, traditionally based on books and book chapters. Such output would be less valued than journal articles by current criteria, placing social science researchers in disadvantage. Another issue concerns authorship. Publications with multiple authors are frequent in clinical and epidemiological studies, whereas a book with a single author contributes as a single publication to the author and to the program to which he or she belongs (issues concerning the time necessary for the maturation and production of a book, as well as its "shelf life" are beyond the scope of the present article).

From this perspective, it is important that the diversity of production and the characteristics of each field be acknowledged and respected, both internally and by other areas, and that they be contemplated equally by evaluation criteria. Although this is an easy enough statement to make, the practical implementation of such criteria is complex, due to the need, at some point, to establish an equivalency between articles and books or chapters.

Another frequent point of criticism is the exclusive use of the ISI/Thomson Scientific impact factor for journal classification. The way in which journal impact is calculated in this system does not allow for direct comparison between journals from different areas, or even different sub-areas.¹³ Major factors contributing to this are journal density (i.e., the mean number of citations in articles of a given journal) and rate of obsolescence. The impact factor is directly proportional to both these variables, so that areas with many citations per article, and in which most citations of an article are made soon after its publication have the greatest impact factors. Thus, mean impact values in different fields of knowledge show wide variation – for instance, 0.6 for social sciences, 1.7 for clinical medicine, and 3.1 for biological sciences.¹

Other problems related to the use of a single impact factor as an evaluation criterion include the lapse of time necessary for a group of authors to produce an article and cite an article that has been read. If this lapse is long, the citation will not revert into impact, given that it will occur outside the evaluation period of ISI/Thomson Scientific. Finally, the impact factor of a journal is strongly determined by the presence, in the system, of other authors that regularly cite its articles. In collective health, the eventual entry of the Cadernos de Saúde Pública into citation index calculations will increase the impact factor of the Revista de Saúde Pública, given that the denominator (number of articles published) will remain the same, but the numerator (number of citations received computed by the system) will increase due to the strong dialogue between these two journals.

The SciELO system also publishes bibliometric indices similar to those calculated by ISI/Thomson Scientific, including two and three-year impact factors, immediacy index, and half-life. Of the 140 Brazilian journals indexed, Cadernos de Saúde Pública, Ciência & Saúde Coletiva, Revista de Saúde Pública, and Revista Latino-americana de Enfermagem occupy the second to fifth positions in the impact factor ranking (two years) for 2005, with impact values ranging from 0.59 to 0.47. On the other hand, Brazilian Journal of Medical and Biological Research and Memórias do Instituto Oswaldo Cruz, the Brazilian journals with greatest impact in the ISI/Thomson Scientific index (0.82 and 0.74, respectively), are given lower scores by the SciELO system: 0.12 and 0.29, respectively. Such discrepancies underscore the relativity with which impact factors must be considered for the evaluation of quality.

The considerations presented above do not intend to invalidate impact factor as a measure of relevance of a journal or publication. What should be kept in mind are the limitations and actual meaning of such indices. According to Amin,¹ the impact factor is a very useful measure of citation, but is not a direct measure of quality, and should thus be used with caution.

Finally, other points of debate include the "publication-centered" view of postgraduate education, without considering the quality of the thesis/dissertation "product", its social relevance, and other technical output, including support to SUS (latu sensu). However, the evaluation of some of these other forms of output is set back by difficulties related to the operationalization of evaluation criteria, as is the case with social relevance, which is also sought in the field of education.⁸ The new evaluation form recently approved by Capes shows an attempt to advance in this subject, including a section on "social insertion", with indices of educational, social, technological, and economic impact, as well as indicators of solidarity between postgraduate programs and of their visibility and transparency.

EVALUATING THE EVALUATION

Given this, it is pertinent to ask whether evaluation has been beneficial to Brazilian postgraduate education in general, and to the collective health area in particular. As is the case with the evaluation of a health intervention, a more specific answer to this question would require a controlled study, in which a "treated" group would be compared to a "control" group. But such technological innovation is dispensable if the evaluator is satisfied with results that indicate that the desired direction is being taken (adequacy evaluation).⁷ Thus, we present a panorama of the evolution of postgraduate education in recent years as an indication of the adequacy of the evaluation process.

In 2004, Capes elaborated the Plano Nacional de Pós-graduação (PNPG - National Postgraduate Plan) for the 2005-2010 period.⁴ The PNPG presents a retrospective of the evaluation process since its beginning, in 1976, showing some impressive results. In the 14 years between 1976 and 1990, the number of postgraduate programs more than doubled, from 673 to 1,485. This number again almost doubled after another 14 years, reaching 2,993 in 2004. The number of teachers in these programs showed a similar trend, increasing from 17,542 (tenured teachers) in 1990 to 32,354 (total teachers) in 2003.⁴ The increase in number of degrees conferred was even greater – a 4.8-fold increase, from 5,737 in 1990 to 27,630 in 2003. In other words, the degree/teacher ratio more than doubled in the period.

This same report⁴ showed that growth of scientific output indexed by ISI/Thomson Scientific was also much greater than the increase in the number of teachers. ISI-indexed publications increased from 3,566 in 1990 to 12,596 in 2003. In other words, the mean number of ISI publications per teacher practically doubled in the period. Curiously, the mean number of citations received by Brazilian publications remained stable (at around nine) until 1996, when a decreasing trend started (Figure 1).

The collective health area showed an increase in the number of programs and in scientific output in the form of articles between 2001 and 2004.⁵

In 2001, 952 articles were published, versus 1,360 in 2004 (Table 1). The increase in output published in Qualis International A/B journals was even greater – 51%. However, the proportion of articles published in such journals over the total number of articles published increased only slightly, from 55% in 2001 to 60% in 2004. Table 1 suggests an increase in the publication of articles in journals of all categories, thus maintaining proportionality between Qualis categories.

There are striking differences between programs in terms of the percentage of total output published in Qualis International A/B journals, which ranged from 9% to 72%. Most programs published 20-30% of their output in International A/B journals, with a median value of 34%. Of the 32 programs, five had more than 50% of their output published in International A/B journals. As to the grades attributed to programs by Capes, there was a consistent difference in this regard (p=0.015), courses with grades 3, 4, 5, and 6, showing mean values of 29, 33, 45, and 54%, respectively.

In the 2001-2004 period, over 4,500 papers in the collective health area were published in 1,354 journals. The 12 journals most used accounted for one third of this output (Table 2). The first four journals account for 25% of published articles, with the first three of these corresponding to the greatest impact factors in the SciELO system, with the exception of one journal from the agrarian sciences area (with an impact factor of 0.82). One-half of the output in the field was published in the 60 most used journals, of which the least used had published nine articles. This group of 60 included eight international journals, of which Lancet had the greatest number of publications (19), followed by American Journal of Public Health, with 14 publications; International Journal of Epidemiology and Diabetes Care, each with 13 published articles; Bulletin of the World Health Organization and Social Science and Medicine, each with 12 articles; and, finally, Environmental Health Perspectives and Journal of Clinical Microbiology, with nine articles each.

CONCLUSIONS

The above presented data show a clear-cut increase in Brazilian postgraduate education in general and in the collective health area specifically. From the perspective of adequacy evaluation, changes have taken place exactly in the direction desired, from the program perspective. Thus, although specific aspects of evaluation may still be questioned, it is impossible attribute a deleterious effect to this process. In a scenario of low competitiveness, as are Brazilian universities, the evaluation process seems to work as a driving force. In countries as the United States or England this role is played by the dispute for and subsequent maintenance of positions in universities, given that most teachers are hired for predetermined periods, using funds from grants and research projects (soft money).

The progress seen in collective health is evidence of the area's potential for growth. This growth is necessary to fulfill the increasing demand from universities and, especially, from SUS. With this in mind, it is essential to improve evaluation, so that it may stimulate the growth of segments covered by this field of evaluation, and, more importantly, so that it may promote greater and more meaningful integration between these segments, producing more creative and efficacious responses to the numerous health problems in Brazil. It is therefore a priority to discuss issues related to output in the form of books and book chapters, and to study in depth the different indexing databases and sources of bibliometric indicators, so as to optimize the Qualis classification for fulfilling its major objective, which is to stimulate the publication of articles in the journals of greatest relevance to the field. From this standpoint, reaching the appropriate target-audience and providing open access are aspects that should be valued in addition to the journal's citation potential.

REFERENCES

Received: 6/5/2006

• 1. Amin M, Mabe M. Impact factors: use and abuse. Oxford, UK: Elsevier Science; 2000. Report n. 1. Disponível em http://www.elsevier.com/framework_editors/pdfs/Perspectives1.pdf [Acesso em 12 jul 2006]
• 2. Behague DP, Victora CG, Barros FC. Consumer demand for caesarean sections in Brazil: informed decision making, patient choice, or social inequality? A population based birth cohort study linking ethnographic and epidemiological methods. BMJ. 2002;324:942-5.
• 3. Capes - Diretoria de Avaliação. QUALIS: concepção e diretrizes básicas. Rev Bras Pós-Graduação. 2004;1:149-151.
• ⁴
  CAPES/MEC. Plano Nacional de Pós-graduação - PNPG 2005-2010. Brasília (DF): Coordenação de Aperfeiçoamento de Pessoal de Nível Superior; 2004. Disponível em http://www.capes.gov.br/capes/portal/conteudo/PNPG_2005_2010.pdf [Acesso em 12 jul 2006]
• 5. Caponi S, Rebelo F. Sobre juízes e profissões: a avaliação de um campo disciplinar complexo. PHYSIS Rev Saúde Coletiva. 2005;15:59-82.
• 6. Fonseca C. Avaliação dos programas de pós-graduação: do ponto de vista de um nativo. Horiz Antropol. 2001;7:261-75.
• 7. Habicht JP, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol. 1999;28:10-8.
• 8. Horta JSB, Moraes MCM. O sistema CAPES de avaliação da pós-graduação: da área de educação à grande área de ciências humanas. Rev Bras Educ. 2005;30:95-116.
• 9. Luz MT. Prometeu acorrentado: análise sociológica da categoria produtividade e as condições atuais da vida acadêmica. PHYSIS Rev Saúde Coletiva. 2005;15:39-57.
• 10. Minayo MCS, Assis SG, Deslandes SF, Souza ER. Possibilidades e dificuldades nas relações entre ciências sociais e epidemiologia. Ciênc Saúde Coletiva. 2003;8:97-107.
• 11. Nunes ED. Pós-graduação em saúde coletiva no Brasil: histórico e perspectivas. PHYSIS Rev Saúde Coletiva. 2005;15:13-38.
• 12. Paim JS, Almeida-Filho N. Saúde coletiva: uma "nova saúde pública" ou campo aberto a novos paradigmas? Rev Saúde Pública. 1998;32:299-316.
• 13. Strehl L. O fator de impacto do ISI e a avaliação da produção científica: aspectos conceituais e metodológicos. Ci Inf. 2005;34:19-27.

1

Evaluation aims and components, Capes site. Available from

http://www

.capes.gov.br/capes/portal/conteudo/10/Objetivos_ Componentes_Avaliacao.htm [access in 2006 May 25]

2

http://www.capes.gov.br

3

http://scientific.thomson.com

4

Growth may be overestimated due to the difference in the definition of teacher.

5

Unpublished data provided by Capes.

Publication Dates

• Publication in this collection
  01 Dec 2006
• Date of issue
  Aug 2006

History

• Received
  05 June 2006