How to evaluate and acknowledge a scientific journal peer reviewer: a proposed index to measure the performance of reviewers

Como avaliar e reconhecer um revisor de periódico científico: uma proposta de índice para medir o desempenho dos revisores

The peer review system is a worldwide process employed by the scientific community, and is one of the pillars of the ethical and technical scientific communication^(1,2). A network of researchers working as a growing team of collaborators in the peer reviewer process improves the quality of papers offered to the readers. Those collaborators are the journal’s secret treasure in the publishing competitive field and should be committed with the journal success in its area.

A reviewer might be acknowledged by qualities that can be documented with numbers, such as time taken in reviewing and the number of manuscript’s reviews concluded. Nonetheless, the editors can evaluate the reviewer deepness and broadness knowledge in the field of the manuscript and the reviewer clarity and capacity to propose constructive suggestions to the authors. Both variables, time and number, can be assessed by the editorial tools of several internet-based managing platforms, and the scientific quality of the review can be graded by means of a numeric score provided by the editor. Arquivos Brasileiros de Oftalmologia (ABO) adopted the ScholarOne template, where the score can go from grade 1 (minimum quality) to 3 (maximum quality).

Beyond those, there are reviewer’s qualities that are not detectable in the everyday work, but can be observed in their long-term work and relationship to the journal, such as exemption and dedication to science. The effort to properly credit the reviewers is an increasing concern of several journals and has resulted in the recent creation of the Publons (publons.com), a website that lists the completed reviews of their members, after crosschecking the database of the journal members⁽³⁾. It can be used to confirm the work done by reviewers worldwide and to find proper reviewers for each scientific topic.

Most scientific journals have faced two global challenges in the editorial process: a) to properly acknowledge their reviewers and reward the most dedicated, and b) to recruit and to teach new members, especially with the growing number of peer-reviewed journals available. The challenge is aggravated by the fact that peer review is an example of pro bono work and there is a huge imbalance between the number of scientific reviews performed and the number of reviewers really committed with the review system. A recent published estimate revealed that 5% of the researchers responded for almost 30% of the reviews performed in 2015⁽⁴⁾.

To overcome those challenges, the ABO has provided, for a long time, workshops in the ophthalmology meetings, encouraging mentors to invite new members to the crew. Every year, the list of reviewers that contributed to the ABO is published in its last issue. The most dedicated ones are recommended by their peers and editors to be part of the Editorial Board.

Recently, our editorial board strived for the recognition of the best among all ABO reviewers, and to do that; the editorial board accepted the suggestion of Dr. André M. V. Messias, M.D., PhD, in creating an index that considers the number of reviews performed, their quality and the time taken to submit those reviews. It takes into account measurable reviewing variables calculated using the outputs of the manuscript submission platform, as follows:

$\mathit{RI}\mathit{=}\frac{\mathit{n}\mathit{x}{\mathit{s}}^{\mathit{3}}}{\left(\mathit{T}\mathit{/}\mathit{7}\mathit{+}\mathit{5}\right)}$

where “RI” is the Reviewer Index; “n” is the number of reviews completed; “s” is the mean score given by the editor, and T is time spend from review acceptance and submission. Feeding data can be obtained from the tool provided by the platforms ScholarOne and Scielo to the ABO used to perform the editorial process of the manuscripts (https://mc04.manuscriptcentral.com/abo-scielo), where the time-length (in days) of each review is registered and the quality is scored as mentioned above. The equation weights editor score 3 times (cubic) and reduce the weight of a single day taken for submission, dividing by 7 (week) and adding the constant 5.

Using this method, if a researcher delivers five reviews per year, taking, on average, five days to submit the review, and achieving a mean score of 2.75: his index would be around 10, and it might be reasonable to state that this fellow reviewer has contributed immensely to the journal.

For the peer review process of the years 2016 and 2017, ABO received revisions from 273 reviewers. They completed between 1 to 11 (mean ± standard deviation: 1.8 ± 1.5) revisions each (Figure 1 A), and took between 0 to 58 (15 ± 10) days in the reviewing process (Figure 1 B), with quality scores ranging from 1 to 3 (2.3 ± 0.6) (Figure 1 C). As result, the reviewer index ranged from 0.1 to 45.8 (4.0 ± 5.0) (Figure 1 D).

Figure 1
Distributions of the number of reviews completed by each reviewer (A), average time taken from acceptance to submission (B), review quality score given by the associated editor (C), and final Reviewer index – RI (D).

Thereby, the ABO would like to thank all the reviewers, who served the ophthalmology and scientific community, dedicated their time and knowledge to improve and disseminate original and high-level information to the readers, in particular, Dr. Laurentino Bicas who won the first ABO Reviewer Award as the most dedicated ABO reviewer in 2016-2017.

REFERENCES

• ¹
  Muccioli C, Campos M, Goldchmit M, Dantas PE, Bechara SJ, Costa VP. Revisão de artigos científicos: privilégio para poucos; benefício para todos. Arq Bras Oftalmol. 2006;69(1):5.
• ²
  Moraes H, Rocha E, Chamon W. Operation and performance of the Peer-Review system. Arq Bras Oftalmol. 2010;73(6):487-8.
• ³
  Rajpert-De Meyts E, Losito S, Carrell DT. Rewarding peer-review work: the Publons initiative. Andrology. 2016;4(6):985-6.
• ⁴
  Kovanis M, Porcher R, Ravaud P, Trinquart L. The Global Burden of Journal Peer Review in the Biomedical Literature: strong imbalance in the collective enterprise. PLoS One. 2016;11(11):e0166387.

Publication Dates

• Publication in this collection
  Nov-Dec 2017

History

• Received
  24 Aug 2017
• Accepted
  24 Aug 2017