Assessing the quality of a scientific journal: the case of Revista Brasileira de Psiquiatria

Abstracts

A produção científica no campo da Psiquiatria vem crescendo de modo exponencial, como demonstrado pelo atual número de revistas científicas na área. As estratégias usadas para avaliar objetivamente a qualidade de uma determinada publicação estão hoje baseadas fundamentalmente em critérios de indexação e citação. O conceito de fator de impacto, que é uma estimativa da freqüência com que um determinado artigo foi citado em um dado período, é fundamental para compreender o modo como os periódicos científicos são avaliados. A recente indexação da Revista Brasileira de Psiquiatria nos dois principais bancos de dados da literatura médica proporcionou à revista uma maior visibilidade, como sugerido pelo número crescente de citações recebidas ao longo dos últimos anos. Um correto entendimento do conceito de fator de impacto (uma estimativa da freqüência com que um determinado artigo foi citado em um dado período) é fundamental para compreender o modo como as revistas científicas são avaliadas.

Estudos de avaliação; Bases de dados bibliográficas; Bibliometria; Índices; Fator de impacto

SPECIAL ARTICLE

Assessing the quality of a scientific journal: the case of Revista Brasileira de Psiquiatria

Christian Kieling^I; Renata Rocha Fernandes Gonçalves^II

^IMedical School, Universidade Federal do Rio Grande do Sul (UFRGS), Porto Alegre (RS), Brazil

^IIFundação Faculdade Federal de Ciências Médicas de Porto Alegre (FFFCMPA), Porto Alegre (RS), Brazil

^{Correspondence}

ABSTRACT

Scientific production in the field of Psychiatry is growing exponentially, as reflected by the number of scientific journals now available. Current strategies to objectively estimate the quality of a specific publication are mostly based on indexation and citation criteria. The concept of impact factor, a measure of the frequency with which an article has been cited in a particular period, is key to understanding the way journals are evaluated. The recent indexation of Revista Brasileira de Psiquiatria in the two major medical databases has increased the journal's visibility, as indicated by the rising number of citations received over the last years. As researchers' scientific productivity measures are largely associated with citation indices of journals in which manuscripts are published, it is essential that Revista Brasileira de Psiquiatria continues to attract high quality, innovative articles, with the ultimate goal of providing state-of-the-art continuing medical education for mental health professionals.

Descriptors: Evaluation studies [publication type]; Databases, bibliographic; Bibliometrics; Indexes; Impact factor

Introduction

The development of accurate and objective strategies to identify scientific journals that publish high-quality research can help clinicians select the most valuable medical literature to review. This may be especially important in face of the substantial number of publications currently available in the field of psychiatry and mental health. Although it is difficult to estimate the actual number of current psychiatry journals (a 1999 study identified a total of 977),¹ the number of publications indexed in two of the major medical databases – ISI and Medline – suffices to demonstrate the magnitude of the scientific production in the field. As of March 2007, for the category Psychiatry, 94 journals were indexed in ISI and 141 in Medline.

The Institute of Scientific Information (ISI), now part of Thomson Scientific, is a database that provides information in the fields of science, social science, technology, arts and humanities (the database is usually accessed through the web under the name Web of Science). Medline is a bibliographic reference database that focuses on health sciences; it was developed by the US National Library of Medicine and includes approximately 5,000 international titles in 30 languages. Evidently, there is some degree of overlap between these two databases.

Annually, ISI publishes the Journal Citation Report (JCR), which offers a set of resources for the evaluation of scientific journals. The JCR informs the total number of citations received by a journal or a subject category (e.g., Psychiatry); calculates the journal's impact factor; provides a list of journals that have cited a specific journal (citing list) or the journals cited by a certain title (cited journals); and shows the most frequently cited journals in a field.

According to the latest JCR, the number of articles published in the 94 ISI-indexed Psychiatry journals has been gradually increasing: in 2003, they were 8,733; in 2004, 9,137; and in 2005, 9,492. JCR can also be used to estimate the turnover rate of the body of work on a subject. In the case of Psychiatry, in 2005, the median half-life of articles was about seven years. This number is calculated based on the fact that 50% of all citations received (aggregate cited half-life) and made (aggregate citing half-life) by all Psychiatry journals in that year refer to articles from the 1999-2005 period – more precisely, the cited and citing half-lives were 6.9 and 7.3 years, respectively. Through the immediacy index, JCR also indicates the speed with which citations to a specific journal appear in the literature, by calculating the average number of times that an article is cited over the course of the year in which it was published.

Citation data also provide the elements for the calculation of a journal's impact factor (IF). According to Thompson Scientific,² impact factors are computed by dividing the number of current year citations to articles published in the two previous years by the total number of articles published in the two previous years (Figure 1). The IF is, thus, a measure of the frequency with which the average article in a journal has been cited in a particular year. It was proposed originally by Eugene Garfield in 1955,³ with the goal of eliminating some of the biases of citation counts which favor large journals over small ones, or frequently issued journals over less frequently issued ones, and of older journals over newer ones.⁴ Nevertheless, publication of review articles (which usually receive more citations than original research articles) or a small group of very highly cited research papers can significantly inflate a journal's IF.

Moreover, the use of citation-based criteria for measuring the quality of research is biased. Authors' nationality has been shown to influence the number of citations acquired by an article. In a clear example of geographical bias, country of origin – whether a developed or developing nation – showed to be a better predictor of number of citations than quality ratings.⁵ Developing countries are underrepresented in the editorial and advisory boards of ten leading Psychiatry journals.⁶ In six major European and American psychiatric journals, only 6% of the articles published between 1996 and 1998 were written by authors from countries outside Western Europe, North America, and Australia/New Zealand. In addition, of the three journals for which submission records were obtained, all had significantly decreased acceptance rates for submitted manuscripts from developing countries.⁷ In a retrospective study of publications in four leading generalist Psychiatry journals from various parts of the world, a slight decrease in frequency of publications from local authors when comparing two 2-year periods, 1991-1992 and 2001-2002, was observed. Nevertheless, the number of contributions from South American authors in the journals investigated was still piteous.⁸

Much debate is currently taking place regarding other methods intended to replace or supplement the IF. There are, of course, other indexes that can be used to assess a journal's quality, such as data regarding acceptance rate, time from submission to acceptance/publication, as well as other measures of a journal's penetration in the international literature.

The case of RBP

The Revista Brasileira de Psiquiatria (RBP) has been indexed in Medline since 2003,⁹ and in ISI since 2005.¹⁰ As one of the major problems affecting scientific journals from developing countries is their limited accessibility and visibility, being listed in these two virtual databases is certainly a major achievement, which also confirms the status that the journal has now attained in the field. More than changing the status of the journal according to Brazilian government funding agencies, the inclusion of RBP in international databases has widened its potential audience and resulted in an increase in the number of submissions.

However, as IFs are calculated with data from the two previous years, the first official index for RBP will not be released until mid 2008 (corresponding to the 2007's IF). Nevertheless, since the information required for calculating the IF is already available online, an unofficial estimate for RBP's IF in the last five years can be calculated (Figure 2). A trend of geometrical growth was observed in the last years, an increase also reflected in the total number of citations received in the period: 22 in 2004, 66 in 2005, and 133 in 2006.

The analysis of all citations received by RBP reveals that the majority of authors citing the journal (66%) is from Brazil, while 23% are from the United States, 6% from England, and 5% from Canada. However, English is by far the language more frequently used by articles citing RBP (79% of the total), with Portuguese and Spanish corresponding to 18% and 2%, respectively. This finding probably reflects the established use of English as the official language of scientific publications.¹¹

RBP has been cited by other Brazilian publications (Arquivos de Neuropsiquiatria and Revista de Saúde Pública, among those with the highest number of citations to RBP), as well as by leading journals in the field of Psychiatry (Archives of General Psychiatry, Molecular Psychiatry, Biological Psychiatry, among others). Of all citations received by RBP in all times, 18% were self-citations (cites from RBP papers to RBP papers).

Operational indexes can also be helpful when assessing a journal's quality. The acceptance rate at RBP is now at 40%, including requested articles. In 2005, manuscripts sent to RBP had a mean time from submission to approval of 130 days and from submission to publication of 348 days. These figures improved to 99 days and 230 days, respectively, in 2006. To draw a parallel between these numbers and those attained by leading medical journals, JAMA recently informed that its acceptance rate is about 8%, with an acceptance time of 73 days and a publication time of 115 days in 2005 (in 1999, they were 123 and 183, respectively).¹²

Discussion

For a concept that is so widely used and so highly respected, it is surprising how poorly understood the definition of impact factor is. By ignoring the way it is calculated, people may be unaware of just how limited it is as a means of assessing the actual impact of a particular article in a specific journal. A journal's IF can be a reliable indicator of the short-term visibility and widespread dissemination of a publication; however, IFs cannot (and are not meant to) inform just how thoroughly read and discussed the journal is outside the academy, including its potential readership among health policy makers, non-governmental agencies, medical students and educators.

Thus, if may sometimes be a poor estimate of the overall impact of a publication. Considering that most journals in Portuguese are not indexed in ISI and that RBP receives a large proportion of its citations from Brazilian authors and from Portuguese-derived articles, it is reasonable to assume that the journal's if would be higher if this evaluation took into account, for instance, citations from the three other most important Brazilian psychiatric journals (Jornal Brasileiro de Psiquiatria, Revista de Psiquiatria Clínica, and Revista de Psiquiatria do Rio Grande do Sul), which are currently not indexed in ISI. The latter two, however, are part of the Scientific Electronic Library Online (SciELO). SciELO is home to an increasing number of open access journals from Latin America and, recently, other countries. All publications available in the SciELO collection follow internationally accepted quality standards.¹³

Only citations made over two years after the publication of an article are included in the calculation of IF. This has an important effect on results, as journals in rapidly-developing research areas, such as bioinformatics, are more likely to publish papers at a fast pace from submission to acceptance, and these papers are more often cited within two years of their publication, leading to a high IF. On the other hand, the citation half-life in a field as Psychiatry is approximately seven years, and thus many papers are still recurrently cited much longer than two years. Therefore, combined indices including IF and citation half-life would better translate the citation report of journals in different fields.

Although, typically, an article in a high-IF journal tends to be more frequently cited than an average article in a relatively low-IF journal, IF alone is not a reliable indicator of the quality of an individual article or author. Studies investigating the relationship between citation factors and trial's methodological quality showed that journals with higher citation factors did not publish higher quality clinical trials¹⁴ and that invalid articles continued to be cited after their retraction.¹⁵

The lack of correlation between IF and journal quality also stems from weaknesses of the algorithm itself. A reduced proportion of articles counts for a large percentage of citations, as the ISI algorithm constitute a measure of mean citation frequency for a publication. Different journals reported that only a small percentage of their items received most of the citations that generated their IFs: in Nature, during 2004, 25% of the papers were responsible for 89% of citations.¹⁶ Journals publishing a large number of "non-citable items" – such as letters, editorial notes, and commentaries – can achieve higher IF, as the algorithm includes these items only in the numerator (and not in the denominator). Additionally, a high frequency of self-citations could theoretically increase a journal's IF (although a 2002 study evidenced that this was not the case for leading journals, which usually have self-citation rates under 20% – a proportion thus defined as acceptable by ISI).¹⁷

While citation indexes were originally designed for information retrieval purposes, they are increasingly used for bibliometrics and other studies involving scientific quality evaluation. An increasing number of funding agencies is using IF as a criterion in grant decisions. In Brazil, for instance, journals' IFs have been important indices employed for the evaluation of post-graduation programs. Among several indices used by the governmental agency Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), programs are also assessed by the publications derived from students and researchers. In the field of Psychiatry, if a paper is published in a journal with an IF higher than 1 it is classified as International A, if the paper is in the JCR and IF is lower than 1 it is classified as International B. Journals listed in Medline and not in the JCR are classified as International C. The journals in the SciELO are classified as National A (for a detailed description of this rating system, see Qualis website).¹⁸

One issue that arises from this practice is to what extent IF is a valid and reliable measure of a scientist's production. In parallel to indexes devised to assess journals, new indexes have been proposed to estimate the quality of the scientific production of a researcher. The H-index, also known as the Hirsch number, quantifies the scientific output of an individual author based on the citations each article receives. A scholar with an index of h has published at least h papers with at least h citations each (e.g., an h index of 25 means that a scientist has published at least 25 papers, each of them with at least 25 citations).¹⁹ While indices devised to evaluate a journal's impact in the scientific literature incorporate only current citations to recently published papers, those developed to estimate the quality of an individual researcher end up encompassing a scientist's whole career. This has a significant impact if one intends to compare scientists at different points in their careers (as it may be the case in grant decisions, for instance). Another limitation of lifetime measures is that a scientist with a large number of citations in the past can profit from his/her scientific capital even if he/she has not been productive in recent years. Options to avoid such biases include adjusting indices for age or time since graduation, as well as restricting analyses to a recent time period (e.g., last five or ten years).

An alternative to augment the sensitivity of IF in evaluating journals is to weight each received citation according to how many times the citing journal is itself cited. This has been called the PageRank (PR) rationale, in which receiving a citation from a highly-cited journal counts differently from being cited by a low-cited publication. A combination of PR and IF generates the newly proposed Y-index. For the year 2003, the different methodologies employed by IF and Y-index yielded distinct journal ranking lists: while the IF top three publications included only review journals (Annual Review of Immunology, Annual Review of Biochemistry, and Physiological Reviews), the Y-index put high prestige journals – Nature, Science and New England Journal of Medicine – in the first three positions.^20-21

As academic institutions and funding agencies increasingly adopt indices to evaluate scientific productivity, further research on the appropriateness of such measures remains highly opportune. This debate should include issues regarding the biases in citation records, the linguistic predominance of English, the open access policy, and the analysis of not citation-based indices.

In the case of RBP, the implementation of the ahead of print system, by which manuscripts are published online (with abstracts available in Medline) prior to the print version, has increased the exposure of RBP articles to the scientific community. The adoption of an online manuscript submission system in the near future will globally contribute to the peer-review process speed and accuracy. Visibility is also enhanced by the free availability of the journal's content and the publication of articles in English. RBP now faces the challenge of continuous improvement of its quality indices, such as sustaining the rapidly growing citation rate in order to achieve an initial IF above 1 in the year 2007. All these features enable the attraction of high-quality original and review articles to RBP, with the ultimate goal of publishing innovative research and providing outstanding continuing medical education for mental health professionals.

Acknowledgements

We thank Fernanda Vergueiro for providing data on RBP editorial processes. We are also indebted to Euripedes C. Miguel, Jair Mari, and Luis Augusto Rohde for their thoughtful review of the manuscript.

Financing: None

Conflict of interests: Christian Kieling is currently Junior Editor of Revista Brasileira de Psiquiatria

Submitted: April 16, 2007

Accepted: April 16, 2007

• 1 McDonald L, Taylor L, Adams C. Searching the right database. A comparison of four databases for psychiatry journals. Health Libr Rev. 1999;16(3):151-6.
• 2 Thomson Scientific. ISI JCR Glossary. [cited Jan 15, 2006] Available from: www.thomsonisi.com/glossary
• 3 Garfield E. Citation indexes to science: a new dimension in documentation through association of ideas. Science. 1955;122(3159):108-11.
• ⁴
  4 The ISI impact factor. Current Contents. [cited Jun 20, 1994]. Available from: http://scientific.thomson.com/free/essays/journalcitationreports/impactfactor
• 5 West R, McIlwaine A. What do citation counts count for in the field of addiction? An empirical evaluation of citation counts and their link with peer ratings of quality. Addiction. 2002;97(5):501-4.
• 6 Saxena S, Levav I, Maulik P, Saraceno B. How international are the editorial boards of leading psychiatry journals? Lancet. 2003;361(9357):609.
• 7 Patel V, Sumathipala A. International representation in psychiatric literature: survey of six leading journals. Br J Psychiatry. 2001;178:406-9.
• 8 Catapano LA, Castle DJ. How international are psychiatry journals? Lancet. 2003;361(9374):2087.
• 9 Bressan RA, Miguel EC, Mari JJ, Rohde LA, Mercadante MT. We have arrived at Medline, whither now? Rev Bras Psiquiatr. 2004;26(1):1.
• 10 Bressan RA, Miguel EC, Mari JJ, Rohde LA, Mercadante MT. We have reached the ISI! Rev Bras Psiquiatr. 2005;27(3):170-1.
• 11 Meneghini R, Packer AL. Is there science beyond English? Initiatives to increase the quality and visibility of non-English publications might help to break down language barriers in scientific communication. EMBO Rep. 2007;8(2):112-6.
• 12 DeAngelis CD, Fontanarosa PB. Thank You, JAMA Peer Reviewers and Authors. JAMA. 2006;295(10):1171-2.
• 13 Meneghini R, Mugnaini R, Packer AL. International versus national oriented Brazilian scientific journals. A scientometric analysis based on SciELO and JCR-ISI databases. Scientometrics. 2006;69(3):529-38.
• 14 Berghmans T, Meert AP, Mascaux C, Paesmans M, Lafitte JJ, Sculier JP. Citation indexes do not reflect methodological quality in lung cancer randomized trials. Ann Oncol. 2003;14:715-21.
• 15 Pfeifer MP, Snodgrass GL. The continued use of retracted, invalid scientific literature. JAMA. 1990;263:1420-3.
• 16 Editorial. Not so deep impact. Nature. 2005;435:1003-4.
• 17 McVeigh ME. Journal self-citation in the journal citation reports - science edition (2002). [cited Jan 15, 2002] Available from: http://scientific.thomson.com/free/essays/journalcitationreports /selfcitation2002
• ¹⁸
  18 CAPES. Critério para classificação de periódicos no sistema Qualis: Grande Área de Saúde – triênio 2004-2006. [cited 15 Jan 2007] Available from: http://servicos.capes.gov.br/arquivos/avaliacao/webqualis/criterios2004_2006/Criterios_Qualis_2005_16.pdf
• 19 Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci USA 2005;102(46):16569-72.
• 20 Bollen J, Rodriguez MA, Van de Sompe H. Journal status. Scientometrics. 2006;69(3):1-16. [cited 20 Jan 2007] Available from: http://www.soe.ucsc.edu/~okram/papers/journal-status.pdf
• 21 Ball P. Prestige is factored into journal ratings. Nature. 2006;439(7078):770-1.

Publication Dates

• Publication in this collection
  06 July 2007
• Date of issue
  June 2007

History

• Accepted
  16 Apr 2007
• Received
  16 Apr 2007