Pitfalls of artificial grouping and stratification of scientific journals based on their Impact Factor: a case study in Brazilian Zoology

OPINION

Pitfalls of artificial grouping and stratification of scientific journals based on their Impact Factor: a case study in Brazilian Zoology

Fábio A. Machado; Hussam Zaher^*

Museu de Zoologia, Universidade de São Paulo. Avenida Nazaré 481, Ipiranga, 04263-000 São Paulo, SP, Brazil

ABSTRACT

The present contribution explores the impact of the QUALIS metric system for academic evaluation implemented by CAPES (Coordination for the Development of Personnel in Higher Education) upon Brazilian Zoological research. The QUALIS system is based on the grouping and ranking of scientific journals according to their Impact Factor (IF). We examined two main points implied by this system, namely: 1) its reliability as a guideline for authors; 2) if Zoology possesses the same publication profile as Botany and Oceanography, three fields of knowledge grouped by CAPES under the subarea "BOZ" for purposes of evaluation. Additionally, we tested CAPES' recent suggestion that the area of Ecology would represent a fourth field of research compatible with the former three. Our results indicate that this system of classification is inappropriate as a guideline for publication improvement, with approximately one third of the journals changing their strata between years. We also demonstrate that the citation profile of Zoology is distinct from those of Botany and Oceanography. Finally, we show that Ecology shows an IF that is significantly different from those of Botany, Oceanography, and Zoology, and that grouping these fields together would be particularly detrimental to Zoology. We conclude that the use of only one parameter of analysis for the stratification of journals, i.e., the Impact Factor calculated for a comparatively small number of journals, fails to evaluate with accuracy the pattern of publication present in Zoology, Botany, and Oceanography. While such simplified procedure might appeals to our sense of objectivity, it dismisses any real attempt to evaluate with clarity the merit embedded in at least three very distinct aspects of scientific practice, namely: productivity, quality, and specificity.

Key words: CAPES; IF; JIF; QUALIS; scientometry.

The quantification of academic merit is not an easy task. Although it plays a central role in the academic environment, it is largely undefined in objective terms (LEWIS & DOYNO 1983) and there are increasing doubts that it can be approached in a simple or operational way (BOLLEN et al. 2009). In spite of these problems, in the past few years the scientific community witnessed an increasing concern from governmental agencies around the world to determine metrical systems capable of evaluating, for funding purposes, the productivity of scientists, institutions and graduate programs through the impact of published research.

One of the first initiatives in Brazil intended to formalize a metric system for academic evaluation took place in 1996 and 1997, when the Ministry of Education, represented by the Coordination for the Development of Personnel in Higher Education (CAPES), established a format that sought to evaluate the Brazilian Graduate Programs in all available fields of knowledge (SOUZA & PAULA 2002). For that purpose, CAPES developed the QUALIS database, which provides rankings of journals and books in each of the 47 main areas of knowledge defined by CAPES (see: http://qualis.capes.gov.br/webqualis/EmailRepresentantesAreas.faces). Each main area is regulated by a specific committee, which is in charge of defining rules for ranking all journals and books related to the area in question. Journals related to natural sciences, such as Zoology, Botany, and Ecology, have been traditionally ranked only through their impact factor (IF), as calculated by the Institute of Scientific Information (ISI/Web of Knowledge). The IF is defined as the ratio between the citations that a journal receives in a period of two years by the number of articles published by that journal in the same period, and was originally proposed as a way to objectively measure and compare the relevance of journals to the scientific community (GARFIELD 2006). Thus CAPES evaluates the quality of each Graduate Program through the IF of the journals in which the articles of the researchers associated to that specific Program are published. As explicitly stated by CAPES (see additional Information in ^{Appendix 11}: "Reestruturação do QUALIS 2009"), one of the main intents of the QUALIS database is to redirect the Brazilian scientific production towards journals with higher IFs, ultimately discouraging Brazilian scientists to publish in journals with low IFs.

Although the system has its own merits, it fails to take into account the specificities of many areas of research, forcing an unwarranted competition between distinct areas of knowledge, and even between different disciplines within the same field of knowledge. Strong criticisms have been recently raised against the QUALIS system in such distinct areas as Medicine (e.g. ROCHA-E-SILVA 2009), Chemistry (e.g. ANDRADE & GALEMBECK 2009), and Zoology (PENTEADO FILHO 2009, LANGGUTH 2009). Although we found common grounds in most criticisms raised within these areas, we will focus our analysis in Zoology as a case study that can be extended to other fields.

Brazil is the most biodiverse country in the world, encompassing approximately 13% (c.a. 1.8 million species) of the global biodiversity (LEWINSOHN & PRADO 2005). Such unique natural resources stimulated the establishment of a strong tradition in Zoological research (ZARUR 1994), with a large and dynamic academic community, several long-standing societies, three major zoological museums, and an unprecedented contribution in the exploration and study of South American fauna (MICHÁN & LLOENTES-BOUSQUET 2010). This tradition rests heavily on an anatomical and descriptive biology, with a major part of the Brazilian zoologists working in the field of systematics and taxonomy (MARQUES & LAMAS 2006).

The use of the IF in the evaluation of taxonomy has long been thought as extremely problematic and incapable of expressing true academic merit for this area (KRELL 2000, 2001, MINELLI 2003). It is probably for that reason that the release of the 2009 QUALIS report generated a strong negative reaction from the zoological community. Criticisms toward the new QUALIS culminated in an open letter from the Brazilian Society of Zoology, signed by 130 zoologists, botanists, editors, and Graduate Program coordinators, questioning the methods of evaluation of the scientific production implemented by the agency and, more specifically, the use of the journals' IF as the only criterion of evaluation (a copy of the open letter can be found at http://www.sbzoologia.org.br/noticia_interna.php?idnoticia = 75; see also PENTEADO FILHO 2009).

Even though the validity of the IF have been strongly questioned as an indexing criterion (e.g. SEGLEN 1997, GLÄNZEL & MOED 2002, GARFIELD 2006, EWING 2006, ADLER et al. 2008, and references therein), the real effects of its use as the only criterion for academic merit in Brazilian Zoology are still not clear. In the present contribution, we explore the use of the IF by CAPES, and its impacts in the evaluation of Brazilian Zoological research. Two main points implied by the QUALIS classification were evaluated, namely: if the QUALIS database can be reliable as a normative factor to guide researchers towards more appropriate choices of journals for publishing; if Botany, Oceanography, and Zoology possess the same publication profiles and can thus be grouped together in a cohesive area of research, as defined by QUALIS. Additionally, we tested CAPES' recent suggestion that the area of Ecology would represent a fourth field of research with a similar publication profile as the one determined by Botany, Oceanography, and Zoology.

MATERIAL AND METHODS

QUALIS database

The QUALIS database consists of a list of indexed journals belonging to a specific area of knowledge that are ranked according to a given criterion defined by a committee of specialists. The list of criteria used may be significantly distinct between the areas of knowledge. An area of knowledge might correspond to a field of research, such as Biochemistry, or may include several distinct fields of research such as Zoology and Genetics. In the context of the QUALIS database, Zoology finds itself within the area called "Biological Sciences I" ("Ciências Biológicas I"; called here "CBI"), which includes also the fields of Genetics, General Biology, Botany, and Oceanography.

Recently, the CBI committee decided to subdivide the area in two subareas, "GBG" and "BOZ", with the former encompassing the fields of Genetics and "General Biology" and the latter including Botany, Oceanography and Zoology. Such subdivision was justified by the committee as a necessary step to remove discrepancies due to a "bimodal heterogeneity" present in the IFs of the two subareas. The new subdivision was implemented for the evaluation triennium of 2007-2009, and allowed the CBI committee to calculate median scores and strata thresholds for BOZ and GBG separately. Table I and figure 1 represent, respectively, a section of the list of journals used by the CBI committee to calculate the median of BOZ, and the threshold values of the six strata depicted as a scale for the stratification of BOZ journals.

The subarea BOZ is composed by 719 scientific journals. From these, 441 were indexed in JCR and had IF calculated for at least the last 10 years. Journals directly related to the field of Zoology were identified based on the scope of the journal and the topic of the papers published. We also included several "multidisciplinary" journals that were non-taxon-specific, but were of main interest for zoologists in general (e.g. American Naturalist [ISSN: 0003-0147], Cladistics [ISSN: 0748-3007], Systematic Biology [ISSN: 1063-5157]). The list resulted in a total of 274 journals related to the field of Zoology, from which 147 presented indexed IF.

Data acquisition and statistical analyses

We obtained a list of journals from the CBI area of the QUALIS database for the year of 2009 in the WEBQUALIS webpage (http://qualis.capes.gov.br/webqualis/). CBI journals were subsequently organized in separate GBG and BOZ groups, based on the documentation provided by the CBI committee (see ^{Appendix 22}: "Lista de revistas e cálculo do IF mediano da BOZ"). The present analysis will focus on the BOZ group.

According to CAPES, the QUALIS database is reliable as a normative factor. In order to evaluate this assertion, we computed the IFs of all journals classified as BOZ for the years of 2006 and 2008, using the strata defined by QUALIS for the triennium of 2007-2009, and quantified the number of journals that changed their category between 2006 and 2008. This was calculated for Zoology alone and for the whole group of journals pertaining to BOZ (Tab. II).

According to the committee, the subdivision of CBI in the subareas of BOZ and GBG was stimulated by the argument that both subareas have distinct publication profiles. To test this hypothesis, we computed the difference between the medians shown by BOZ and Zoology. We then carried out a resampling procedure (MANLY 1997) intended to identify if such claimed differences are significant: 10,000 random samples of N journals were drawn from BOZ without replacement, with N being the number of journals that were assigned to the Zoology subarea. We then estimated the median of the IF for each sample and calculated the difference between the obtained value and the observed median of BOZ. This procedure resulted in an empirical distribution of the differences of medians between BOZ and the subsamples with size N. The empirical P-value was then calculated as the proportion of permutations that produced absolute differences superior to that found between the observed values. In other words, if Zoology were actually a subarea of BOZ, we would expect the observed differences between medians to fall within the range of differences estimated from random samples of BOZ with the same size. Finally, we repeated this procedure for the years of 1998 to 2004, 2006, and 2008, with the intent of evaluating the historical trend of these areas. The years of 2005 and 2007 were not available for the present study.

The recent suggestion made by the CBI committee that the area of Ecology would represent a fourth field of research with a similar publication profile exhibited by Botany, Oceanography, and Zoology was tested with the same resampling procedure described above. We randomly sampled N journals from a "Biodiversity" group composed of journals from Ecology (drawn from the Ecology QUALIS area), Botany, Oceanography and Zoology, with N being the number of BOZ journals. This procedure can be understood as a test to investigate if Ecology and BOZ do show a similar citation pattern. The same test was extended for all available years.

RESULTS AND DISCUSSION

The QUALIS stratification

The CBI committee performed a stratification of the journals based on the scientific production of all officially recognized graduate programs associated to the fields of Zoology, Botany, and Oceanography. The procedure consisted in the elaboration of a list of all papers published in the last three years by researchers belonging to the graduate programs of Zoology, Botany, and Oceanography. CAPES retrieved this information from the annual reports sent every year by each graduate program, and in which a detailed list of the scientific production is provided. The resulting lists of publications are summed together, removing repeated entries of the same article. Each paper receives a score equivalent to the IF of the journal in which the article was published. Journals that lack a calculated IF in ISI were not taken into account, except for a rooster of 10 indicated journals, chosen by the committee and added to the QUALIS calculation with an IF equal to zero (Tab. I). To establish a basis for the stratification, the committee calculated the median of these scores and the thresholds of the strata were then defined over this value, multiplied by different factors for each threshold (Fig. 1). As a result of this procedure, journals belonging to very distinct fields of knowledge were included in the list of journals used for the calculation of the median for the CBI area (Fig. 2).

This procedure could be regarded, at first glance, to represent a fair methodology of evaluation. However, it does not appear to be the case for at least the BOZ subarea evaluated in the present study. Publication listings obtained from the graduate programs belonging to BOZ were not filtered in order to retain only the journals related to the fields of Zoology, Botany, and Oceanography. Indeed, all journals were used in the calculations, including those from fields of knowledge that are only distantly or not at all related to BOZ (e.g., Engineering, Medicine; Fig. 2), and which IFs are incompatible with the latter (present study; AMIN & MABE 2000). However, the implications of such a heterogeneous list of journals composing the subarea BOZ is still poorly understood. That said, the procedure proposed by CAPES seems to be unnecessarily convoluted: it only serves to produce an arbitrary parameter of stratification, a goal that could be achieved through the use of any value of multiplication based on the median.

Stability of the strata in the QUALIS system

In order to serve as a normative factor, the strata proposed in QUALIS should be stable along the years. In other words, if one intends to publish in the upper strata of QUALIS (i.e., A1 and A2), journals in these given strata would preferably stay within the same stratum from one triennium to the other.

Journals from BOZ and Zoology, however, presented highly unstable IF values between the years of 2006 and 2008 (Figs 3-6), with approximately one third of the 440 journals changing from a stratum to another (Tab. II). The majority of the journals that changed their status had a decrease in their IF, thus being classified in a lower stratum. A significant number of journals belonging to stratum A1 showed a decrease in IF, with 19 and 26% of the journals being classified in lower strata for BOZ and Zoology, respectively. Strata B2 and B3 were particularly unstable, with more than half of their journals being classified in other strata, resulting in an almost complete overlap between them. B4 surprisingly presented no misclassifications due to the fact that most journals classified in this stratum lack an IF score for 2006 and 2008, indicating that this stratum is mainly comprised of newly indexed journals.

Our analysis shows that the QUALIS method of stratification proposed by the committee is inappropriate as a guideline for publication improvement, with approximately one third of the journals (for both BOZ and Zoology) changing their strata between years. This result is not surprising: even though the IF can be considered generally stable along the years (GLÄNZEL & MOED 2002), stochastic factors can have a great deal of influence in changes between the years (AMIN & MABE 2000, OGDEN & BARTLEY 2008). These factors are more relevant for journals that publish fewer articles due to the increased influence of individual articles (ADLER et al. 2008). This pattern can be particularly damaging when coupled with the instability of QUALIS stratification and its normative aspects. Publications with lower IF and fewer articles published would be more subject to larger proportional random fluctuations in their IF. If the IF is the sole basis for QUALIS, then these journals would likely be dismissed as unreliable, a process that could happen within BOZ and Zoology (Tab. II), with journals from these fields being effectively classified two ranks away from their original stratum (Figs 3-6). The small absolute differences between the strata thresholds (Fig. 1) probably contribute to this problem, rendering the QUALIS strata inadequate to guide zoologists in their decisions.

The BOZ grouping

Historically, Zoology is much more stable than BOZ, presenting a relatively constant median IF between 1998 and 2004, with a moderate increase from 2004 to 2008. BOZ, however, shows a bimodal pattern of increase, with a moderate slope between 1998 and 2003 and a more angled one from 2003 to 2008 (Fig. 7). The permutation analysis indicates that the median IF for Zoology cannot be found through random samples within BOZ journals for all years, except for the year of 1998 (Tab. III). It is clear that from 2002 to 2008, Zoology completely diverged from the expected trajectory of median IFs for BOZ (Fig. 7).

This divergence seems to impact negatively the classification of Zoology journals (Fig. 8). While Botany and Oceanography show almost half of their journals in the two uppermost strata, Zoology journals tend to be classified in lower strata, with only one third of them in the upper ones (A1 and A2). It is worth noticing that journals from BOZ that were clearly related to the area of Ecology, such as Chemoecology [ISSN: 0937-7409], Conservation Biology [ISSN: 0888-8892] and Journal of Tropical Ecology [ISSN: 0266-4674], presented an evident bias towards the upper strata. This is expected since ecological journals do tend to show higher IFs (Fig. 7) and are thus "better" classified when grouped with journals that tend to show lower IFs.

CBI subareas were intended to deal with the obvious incompatibilities between distinct fields with conspicuously disparate IF values (see QUALIS documentation in http://qualis.capes.gov.br/webqualis/). Although the segregation between BOZ and GBG correctly addressed the main dichotomy found in previous versions of QUALIS, it does not address the individual characteristics of Botany, Oceanography and Zoology. Our analysis shows that the insertion of Zoology within BOZ is artificial, and that it cannot be considered a valid sub-sample of this subarea of CBI. This grouping is actually detrimental for Zoology, as it tends to present lower IFs, reinforcing the tendency shown by Zoology journals to score poorly under the QUALIS stratification scheme (Fig. 8).

It could be argued that this is a legitimate trend for the field of Zoology, and that the lower IFs and, consequently, lower QUALIS classification actually reflects a poor academic performance. It is known, however, that comparisons between distinct areas are not straightforward: each discipline (and even subdisciplines) has its own tradition and pattern of citations (EWING 2006), affecting the meaning and validity of IF for these different areas of knowledge. In Zoology, and more specifically in taxonomy, the meaning of the IF is patently decoupled from the principle of academic merit (KRELL 2000, 2001, MINELLI 2003). Taxonomic papers do not tend to be cited within the "window" of two years of IF calculations, rendering this measure of impact almost meaningless for this field (KRELL 2000, 2001, MINELLI 2003). Under these conclusions, the clear bias shown by the sole use of IF values for evaluation of scientific journals reinforces the impression that the QUALIS system might threaten in the near future the growing effort by zoologists to describe the Brazilian biodiversity (MARQUES & LAMAS 2006, MICHÁN & LLORENTES-BOUSQUET 2010).

The "Biodiversity" group

A recent presentation given by CAPES officials highlighted their intention, for the next triennium (2010-2012), to merge BOZ journals with journals from the field of Ecology, forming a new QUALIS area called "Biodiversity". This new grouping was justified through a theoretical perspective, instead of a purely scientometric one. Indeed, according to the document from CAPES, the areas involved show "thematic affinities and interfaces" that justify a "Biodiversity area"³. However, although the involved fields of knowledge likely demonstrate a common theoretical background, it is still arguable that they share a common scientometric profile.

Journals belonging to the Ecology area show a moderate initial increase of their IF from 1998 to 2003, with a more angled slope appearing from 2003 to 2006 that decreases slightly from 2006 to 2008 (Fig. 7). The area of Ecology in QUALIS not only shows a more accelerated increase of its historical IF, but also presents a higher initial IF than that of Zoology (Fig. 7).

Our preliminary evaluation of the Biodiversity area reveals that BOZ could not be interpreted as a sub-sample of a greater area that combines both Ecology and BOZ journals. The differences were significant for all years, except for 1998 and 2001 (Tab. III). Despite these exceptions, the pattern is very clear and shows that the creation of a Biodiversity area would underestimate all fields represented in BOZ.

Our results indicate that the creation of a Biodiversity area would result in a highly heterogeneous field in terms of citation profiles, a problem that was supposedly addressed through the subdivision of CBI. The grouping of Ecology with BOZ would lead to an unfair competition between different areas. The higher IF of Ecology would guarantee a higher classification in QUALIS for the journals of this area, downplaying the importance of basic science, such as Zoology and Botany. The drawbacks of this practice could be two-fold: 1) mainly descriptive sciences, such as Zoology, would be forced to compete in unreasonable terms, and 2) mainly experimental sciences, such as Ecology, would see a significant number of journals migrate to higher strata of the Qualis system without any real improvement of their IF.

Final remarks

The Brazilian scientific community reacted to the newest version of the QUALIS with perplexity. Many editorials were quick to point out that the new methodology did not reflect academic merit and was downplaying the role of classical journals and probably of whole areas of research (ROCHA-E-SILVA 2009, ANDRADE & GALEMBECK 2009). The most unified response came from the societies of Zoology and Botany in the form of an open letter expressing deep concern for the use of the IF as the only index of evaluation of academic merit (see also PENTEADO FILHO 2009, LANGGUTH 2009). We agree with these criticisms and have presented here a comprehensive evaluation of the potential pitfalls of the procedure adopted by the CBI committee for the BOZ group and, more specifically, for the field of Zoology.

The meaning of the IF in distinct fields of scientific knowledge has been discussed at length, and the validity of its use as an absolute measure of merit have been barely supported by researchers in scientometry (e.g. GLÄNZEL & MOED 2002, EWING 2006, ADLER et al. 2008). Nevertheless, the growing culture in favor of "objectivity" has lead administrators and policymakers to embrace the IF as a quick tool for decision-making, ignoring concomitantly the basic question of what is being actually measured (EWING 2006).

The new QUALIS definitely joined this global trend, ignoring any thoughtful evaluation of the meaning of "merit." The reinforcement of the IF as the only parameter used by governmental agencies, such as CAPES and CNPq, to evaluate scientific merit could have more profound consequences on Brazilian research cycles. A continuous pressure to publish high impact articles might indeed alter scientific practices by stimulating data falsification (FANELLI 2010a) and selective publishing of positive results (FANELLI 2010b). Furthermore, the focus on IF could lean unjustified pressure on national journals that do not conform to the required pattern of citation, possibly leading to less governmental funding and incentive. More troublesome, however, are the already visible collateral effects on descriptive sciences, such as taxonomy, as opposed to experimental sciences. While Zoology and Botany definitely fall in the first group, Ecology tends to be included in the second one. This dichotomy seems to be illustrated in our analysis by a discrepant gap between these fields (Fig. 7). The decision to unite them in a single "Biodiversity" area will increase the already existing gap between Zoology and other fields included in BOZ. Brazilian Zoology is a very competitive field when evaluated independently (ADAMS & KING 2010, MICHÁN & LLORENTES-BOUSQUET 2010), and its insertion in artificial groups could intensify the already troubling global trend of marginalization of Organismal Biology, undermining the long-term achievements of the Brazilian "National Zoology Program" (CARVALHO et al. 2007).

One could argue that the only way to evaluate the merit of researchers from a specific field of knowledge would be to analyze the quality of their publication (EWING 2006). Although we do agree with this argument, we believe that the new QUALIS method of stratification does not represent an appropriate solution for this endeavor. The procedure used for the stratification of the journals reduces all aspects related to the pattern of publications present in Zoology, Botany, and Oceanography to only one parameter of analysis, i.e., the impact factor calculated for a comparatively small number of journals. While such simplification might appeal to our sense of objectivity, it dismisses any real attempt to evaluate with clarity at least three very distinct aspects of scientific practice that seem to represent "academic merit," namely: productivity, quality, and specificity. For QUALIS to become an efficient and uncontroverted evaluation tool of academic merit, it must incorporate more than one criterion of evaluation (not only the IF of journals), in order to take into account the specificity of the area being analyzed, for a better and fair judgment of the productivity of the scientist and quality of his research.

Any attempt of evaluation of academic merit should rests upon procedures that clearly take into account these three factors. A clear delimitation of the specificity of the area pool of journals or researchers to be evaluated is crucial for the outcome of a fair result.

ACKNOWLEDGEMENTS

We are deeply indebted to Abel Packer, Rogerio Meneghini, Rogerio Mugnaini, Walter A. Boeger, Marcelo Carvalho, Luis F. Silveira, Miguel T. Rodrigues, Celio Haddad, Augusto Abe, Carlos A. N. Iannini, Ana Lúcia da C. Prudente, Rodney R. Cavichioli, Sionei R. Bonatto, Luciane Marinoni, Carlos Lamas, Marcos Tavares, Carlos R. F. Brandão, Mario de Vivo, Mario de Pinna, Alfredo Langguth for helpful and enlightening discussions about scientific production and impact factors. We are also grateful to Pedro Gnaspini and Marcio de C. Silva Filho for providing the BOZ list of journals used by CBI to calculate the QUALIS stratification for the subarea. This research was supported by the Fundação de Amparo à Pesquisa do Estado de São Paulo BIOTA/ FAPESP, Grant number 02/13602-4 (to H.Z.).

LITERATURE CITED

Submitted: 28.VI.2010; Accepted: 29.VII.2010.

Editorial responsibility: Walter A.P. Boeger

Appendix

Appendex 1 - Click to extend

Appendix 1

Appendex 2 - Click to extend

Appendix 2

• ADAMS, J. & C. KING. 2010. Global Research Report: Brazil. Research and collaboration in the new geography of science. UK, Thomson Reuters, Evidence Ltd. Available online at: http://researchanalytics.thomsonreuters.com/m/pdfs/GRR-Brazil-Jun09.pdf [Accessed: 9.VI.2010]
• ADLER, R.; J. EWING & P. TAYLOR. 2008. Citation Statistics, a Report from the Joint Committee on Quantitative Assessment of Research (IMU, ICIAM, IMS). Available online at: http://www.ams.org/ewing/Documents/CitationStatistics-FINAL1.pdf [Accessed: 9.VI.2010]
• AMIN M. & M. MABE. 2000. Impact factors: use and abuse. Perspectives in Publishing. Oxford, Elsevier Science, Newsletter no. 1. Available online at: http://www.igh.org/publicaccess/Perspectives1.pdf [Accessed: 11.VI.2010]
• ANDRADE J. B. & F. GALEMBECK. 2009. QUALIS: Quo Vadis?. Química Nova 32: 5. doi: 10.1590/S0100-40422009000100001.
• BOLLEN J.; H.V. SOMPEL; A. HAGBERG & R. CHUTE. 2009. A principal component analysis of 39 scientific impact measures. PLoS ONE 4 (6): e6022. doi:10.1371/journal.pone.0006022.
• CARVALHO, M.R.; F.A. BOCKMANN; D.S. AMORIM; C.R.F. BRANDÃO; M. VIVO; J.L. FIGUEIREDO; H.A. BRITSKI; M.C.C. PINNA; N.A. MENEZES & F.P.L. MARQUES. 2007. Taxonomic impediment or impediment to taxonomy? A commentary on systematics and the cybertaxonomic-automation paradigm. Evolutionary Biology 34 (3): 140-143. doi: 10.1007/s11692-007-9011-6.
• EWING, J. 2006. Measuring journals. Notices of the American Mathematical Society 53 (9): 1049-1053. Available online at: www.ams.org/notices/200609/comm-ewing.pdf [Accessed: 9.VI.2010]
• FANELLI, D. 2010a. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE 4 (5): e5738. doi:10.1371/journal.pone.0005738.
• FANELLI, D. 2010b. Do pressures to publish increase scientists' bias? An empirical support from US States data. PLoS ONE 5 (4): e10271. doi:10.1371/journal.pone.0010271.
• GARFIELD, E. 2006. The history and meaning of the Journal Impact Factor. Journal of the American Medical Association 295 (1): 90-93.
• GLÄNZEL, W. & H.F. MOED. 2002. Journal impact measures in bibliometric research. Scientometrics 53 (2): 171-193. doi: 10.1023/A:1014848323806.
• KRELL, F.T. 2000. Impact factors aren't relevant to taxonomy. Nature 405: 507-508. doi:10.1038/35014664.
• KRELL, F.T. 2001. Why impact factors don't work for taxonomy. Nature 415: 957. doi:10.1038/415957a.
• LANGGUTH, A. 2009. Ditadura da CAPES. Boletim da Sociedade Brasileira de Mastozoologia 54: 11-12.
• LEWINSOHN, T.M. & P.I. PRADO. 2005. How many species are there in Brazil? Conservation Biology 19 (3): 619-624. doi:10.1111/j.1523-1739.2005.00680.x.
• LEWIS, L.S. & V. DOYNO. 1983. The definition of academic merit. Higher Education, 12: 707-719. doi: 10.1007/BF00132426.
• MANLY, B.F.J. 1997. Randomization, Bootstrap and Monte Carlo Methods in Biology. London, Chapman and Hall, 461p.
• MARQUES, A.C. & C.J.E. LAMAS. 2006. Taxonomia zoológica no Brasil: estado da arte, expectativas e sugestões de ações futuras. Papéis Avulsos de Zoologia 46: 139-174. doi: 10.1590/S0031-10492006001300001.
• MICHÁN L. & J. LLORENTE-BOUSQUETS. 2010. Bibliometria de la sistematica biologica sobre America Latina durante el siglo XX en tres bases de datos mundiales. Revista de Biología Tropical/International Journal of Tropical Biology and Conservation 58 (2): 531-554.
• MINELLI, A. 2003. The status of taxonomic literature. Trends in Ecology & Evolution 18 (2): 75-76. doi:10.1016/S01695347(02)00051-4.
• OGDEN, T.L. & D.L. BARTLEY. 2008. The ups and downs of journal impact factors. Annals of Occupational Hygiene 52 (2): 73-82. doi:10.1093/annhyg/men002.
• PENTEADO FILHO, R.C. 2009. O problema não é o mérito dos periódicos. Boletim da Sociedade Brasileira de Mastozoologia 54: 10-11.
• ROCHA-E-SILVA, M. 2009. O novo Qualis, que não tem nada a ver com a ciência do Brasil. Carta aberta ao presidente da CAPES. Clinics 64 (8): 721-724. doi: 10.1590/S180759322009000800002.
• SEGLEN, P.O. 1997. Citations and journal impact factors - questionable indicators of research quality. Allergy 11: 1050-1056.
• SOUZA, E.P. & M.C.S. PAULA. 2002. QUALIS: a base de qualificação dos periódicos científicos utilizada na avaliação CAPES. INFOCAPES - Boletim Informativo da CAPES 2002 abril-junho 10 (2): 7-25.
• ZARUR, G.C.L. 1994. Schools and paradigms in Brazilian Zoology. Interciencia 19 (4): 183-190.

Appendix 1

Appendix 2

1

Available as Online Supplementary Material accessed with the online version of the manuscript at

http://www.scielo.br/zool

2

Available as Online Supplementary Material accessed with the online version of the manuscript at

http://www.scielo.br/zool

3

Diretrizes da CBI-CAPES para os Programas de Pós-Graduação, Brasília, August, 2009.

Publication Dates

• Publication in this collection
  27 Sept 2010
• Date of issue
  Aug 2010

History

• Received
  28 June 2010
• Accepted
  29 July 2010