The peer-review process: doing good and doing well

In one of our past year’s editorials, we discussed the (critical) role played by reviewers in the publication process. At the time, we provided some guidance for inexperienced reviewers to perform good evaluations. We highlighted two main rules for them to perform a good job. First, they should clearly and objectively indicate the criteria for considering the article’s contribution to the literature, the quality and adequacy of the chosen methods, the quality of the logical arguments and the clarity of the language. Second, they should make recommendations or suggestions in a constructive and well-organized way (Saes & Hourneaux, 2018)[¹].

This time we would like to go further into this topic. We will discuss who should be the reviewers who would perform a helpful review, not only for the authors to have their feedback but also for the editors to make better decisions. Beyond, of course, being experts in the manuscript’s subject, which criteria should editors consider when choosing reviewers?

First of all, it is worthy to notice that there are two particular types of (good) reviewers. We nominate the first as the meticulous reviewer. This type of reviewer makes thorough comments, topic by topic, paragraph by paragraph, from the beginning to the end of the manuscript, drawing attention to each particular issue to be corrected and discussing it exhaustively. It will not be a surprise if his/her review is as lengthy as the article itself.

The other one is the generalist reviewer. She/he skims the manuscript to focus on the key points that need to be addressed, such as the relevance of the theme and the research questions, the coherence of the hypotheses, the adhesion of the methodology and the importance of the findings and conclusions. In this case, this reviewer raises some comprehensive problems of the manuscript, as well as makes recommendations. In general, we are talking about a senior researcher who can give fruitful suggestions and appraisal, if the paper presents actual innovative ideas.

Both approaches can convey good aspects. The meticulous reviewer enormously helps authors improve their paper. On the other hand, the generalist reviewer also makes a significant contribution for the editors to better decide on publishing or not the paper.

However, according to Romanelli (1996), both approaches fail, and they also are not quite as “efficient” as it seems. A meticulous reviewer would provide a long list of specific comments that may not allow the authors to understand what the key issues to be addressed are. The generalist review approach is good for evaluating the manuscript as a whole, by highlighting the key points, but sometimes not providing constructive and practical advice to authors (Frost & Taylor, 1996; Leblebici, 1996).

Of course, up to this point, we can say that an efficient reviewer should have the double competence of helping the editor decide and the authors improve their article. A halfway solution to combine both would be to recommend that reviewers present some general remarks at the beginning of their assessment (including answers to the questions proposed in the evaluation forms) and then go into some more specific questions.

Nevertheless, our experience shows that both types of reviewers can be more (or less) “efficient”, according to the nature of the manuscript: whether it is a conceptual-based or an empirical-based paper. In the first case, conceptual studies, the reviewer is supposed to prioritize the search for the main theoretical contributions in the paper. In general, a conceptual manuscript is good if it brings theoretical advances and, commonly, it should have at least one good and new idea to present to the academic community. Hence, a good guideline for the reviewer is to begin the assessment with a short statement discussing this new theoretical contribution. In some cases, there is a good contribution in the manuscript, but the reviewer can reject the article for other reasons (e.g. poor writing or communication of these ideas). If the idea is worthy (and the generalist reviewer can identify it), a recommendation for revision may solve those problems. Therefore, a generalist reviewer (a more experienced researcher), would be more suitable to do a good review in theoretical papers than a meticulous reviewer.

In opposition, for empirical studies, editors look for a thorough and detailed review. In general, these papers are standardized (introduction, literature review, methodology, results, discussion and conclusions) with each section having its characteristics and importance[²]. However, the “empirical” parts, i.e. methods, analyses and results, need special attention with a specific recommendation, considering: the technical merit (internal and external validity, craftsmanship (clarity in presenting the model and results) and significance (testing a theory in a non-trivial and redundant way, improves the literature and leads to public or private policy implications) (Schwab, 1985). In short, in empirical manuscripts, reviewers need to pay more attention to the techniques and methodology details, thus being better suited for meticulous reviewers.

Finally, we would like to emphasize two comments. First, as editors, we must choose reviewers who fit best with the manuscript (depending on the theme, methodology or any other technical criteria) in order to not lead them to an uncomfortable situation, risking their immediate refusal to review it. And, of course, editors should try to balance and mix the nature of manuscripts and the characteristics of reviewers, making it possible to define sets of reviewers with different characteristics and perspectives.

Finally, the authors wish to immensely thank the reviewers for their good work, as they have help improve this and other journals, and their knowledge on management science itself.

Notes

• 1.
  To read more about the criteria for evaluating a paper, see also Mendes-Da-Silva (2018).
• 2.
  Even empirical-oriented papers should have a solid literature review, around 30 per cent of the paper extension (Sun & Linton, 2014).

References

• Frost, P. J., & Taylor, M. S. (1996). Rhythms of academic life: Personal accounts of careers in academia, Thousand Oaks, CA: Sage.
• Leblebici, H. (1996). The act of reviewing and being a reviewer. In P. J. Frost, & M. S. Taylor, (Eds.) Rhythms of academic life: Personal accounts of careers in academia (pp. 269–274). Thousand Oaks, CA: Sage.
• Mendes-Da-Silva, W. (2018). Reconhecimento da contribuição do avaliador anônimo. Revista de Administração Contemporânea, 22, 1–13.http://dx.doi.org/10.1590/1982-7849rac2018180281
  » http://dx.doi.org/10.1590/1982-7849rac2018180281
• Romanelli, E. (1996). Becoming a reviewer: Lessons somewhat painfully learned. In P. J. Frost, & M. S. Taylor, (Eds.) Rhythms of academic life: Personal accounts of careers in academia, (pp. 263–268). Thousand Oaks, CA: Sage.
• Saes, M. S. M., & Hourneaux, F. Jr, (2018). The reviewers’ role in the publication process. RAUSP Management Journal, 53, 486–487. https://doi.org/10.1108/RAUSP-10-2018-047
  » https://doi.org/10.1108/RAUSP-10-2018-047
• Schwab, D. P. (1985). Reviewing empirically based manuscripts: Perspectives on process. In L. L. Cummings, & P.J. Frost, (Eds.), Organization Sciences, (pp. 171–181). Homewood, IL: Irwin.
• Sun, H., & Linton, J. D. (2014). Structuring papers for success: Making your paper more like a high impact publication than a desk reject. Technovation, 34, 571–573. https://doi.org/10.1016/j.technovation.2014.07.008
  » https://doi.org/10.1016/j.technovation.2014.07.008

Publication Dates

• Publication in this collection
  25 Nov 2019
• Date of issue
  Jul-Sep 2019