Additional comments on the paper by Thomas et al: how to evaluate "quality of publication"

LETTER TO THE EDITOR

Further comments on the paper by Thomas et al: how to evaluate "quality of publication"

Paulo Roberto Benchimol-Barbosa; Ricardo Luiz Ribeiro; Eduardo Correa Barbosa

Hospital Universitário Pedro Ernesto - Universidade do Estado do Rio de Janeiro, Rio de Janeiro, RJ - Brazil

^{Mailing address}

Keywords: Bibliometrics; journal impact factor; scientific and technical publications.

In contribution to the comments previously addressed on the paper by Thomas et al we would like to present the following insights¹. We partly agree with the view expressed by the authors in the Conclusion, since the evaluation of the quality of publications by peers can be decisive in situations in which merely quantitative analysis may be misleading. Indeed, issuing of an unbiased opinion on a paper is a prerequisite for publication. We would like to quote a famous research to highlight the importance of criterious peer's evaluation.

Chemists Stanley Pons, a doctoral student from the University of Utah and Martin Fleischmann, his advisor, announced in 1989, which would be the most revolutionary discovery of all times: cold fusion, an inexhaustible source of clean energy. The paper was published as a preliminary communication in that year and was followed by a number of attempts to replicate the experiment². Altogether, the article received 629 citations²!

A true scientific best-seller? Not at all. In fact, the alleged discovery has never been replicated. The quotations came from articles that strongly disputed the validity of both the experiment and the results published. Later that same year, the research a received formal communication from the U.S. government, revised and updated in 1999, putting down the way the research had been conducted and published: cold fusion has never existed³!

Thus, measuring the relevance of a scientific article solely based on the impact factor can lead to inappropriate conclusions about the merit of a particular publication. Thus, in 2010, the National Council of Medical Research of the Australian government decided to abandon the use of impact factor in reviewing requests for grants to aid research and post-graduate studies⁴. Careful peer's evaluation is always needed to resolve conflict situations.

Reply

We agree that the validation of the scientific quality of paper, using only the current bibliometric indexes, is not exempt of controversy. In 2005, Andrew Coats made a profound reflection on the difficulty of finding a bibliometric index able to assess accurately the impact of academic production at a time when the computer systems were already a great aid of the dissemination of scientific knowledge¹. The author compared the ten most quoted articles and the most downloaded over the Internet, among the total number of manuscripts published in International Journal of Cardiology within 12 months. His conclusion was that there was a parallelism between these two groups of papers. In general, the classical quotations favor the original papers, while the most accessed on the Internet tend to be those papers that include reviews, new themes and highlights, such as research on stem cells or treatment of rare diseases.

Assuming that, a question arose of what is most important, the higher number of quotations or Internet access. Unfortunately, this response has not yet been obtained. The editors, aware of the modernism of publications on the Internet, partly solved the problem by posting on the websites of their journals the most accessed papers. In turn, the Journal of Citations, from Thompson Reuters, continues to publish annually the impact factor of journals indexed in its base, taking into account the number of citations of papers.

The number of accesses to a particular paper, however, is also subject to biases. Thus, it is worth making a few questions: Do all accesses correspond to the full reading of the manuscript? If read, is its content truly relevant? How many of these "papers read" deserve to be quoted? This shows the difficulty of assessing the quality of accesses against citations of scientific publications and calls into question the reliability of this bibliometric parameter.

Ultimately, the objective of the article "Using the impact factor and the H index to assess researchers and publications" was to show the readers how to calculate the Impact Factor and the H index, since these are the bibliometric indexes most commonly used by our development agencies and educational institutions, which justifies the dissemination of their knowledge².

Renato S. Assad

References

Referências (Resposta)

Manuscript received March 21, 2011; revised manuscript received March 21, 2011; accepted on March 24, 2011.

• 1. Thomaz PG, Assad RS, Moreira LF. Uso do Fator de Impacto e do Índice H para avaliar pesquisadores e publicações. Arq Bras Cardiol. 2011;96(2):90-3.
• 2. Fleischman M, Pons S, Hawkins M. Eletrochemically induced nuclear fusiono f deuterium. J Electroanal Chem. 1989;261:301-8.
• 3. U.S. Department of Energy . A Report of the Energy Research Advisory Board to the United States Department of Energy, Washington, DC: U.S. Department of Energ:1989. [Cited imn 2011 Mar 20]. Available from: http://www.ncas.org/erab/
• 4. National Health and Medical Research Council: Australian Government. [Cited in 2011 Mar 21] Available from: http://www.nhmrc.govau/_files_nhmrc/file/grants/peer/impact%20factors%20in20peer%20review.pdf
• 1. Coats AJ. Top of the charts: download versus citations in the International Journal of Cardiology. Int J Cardiol. 2005;105(2):123-5.
• 2. Thomaz PG, Assad RS, Moreira LF. Uso do Fator de Impacto e do Índice H para Avaliar Pesquisadores e Publicações. Arq Bras Cardiol. 2011;96(2):90-93

Publication Dates

• Publication in this collection
  30 Aug 2011
• Date of issue
  July 2011