Editorial: Why should we care about preparing referee reports?

It is undeniable that the whole academic publication universe depends on the fundamental, but not necessarily enough recognized and rewarded, reviewers’ role. The fact is that without the reviewers, even if we consider the new open science practices, there is no way to guarantee the quality of the published research in journals worldwide (Bedeian, 2003; Gerhardt, Dal Pai, Gouveia & Azzolin, 2014).

Although authors have praised the role of anonymous reviewers in academic life (Hamermesh, 1992, 1994; McAfee, 2010; Taylor, 2009), there seem to be very few defined and consensual reasons for scholars to prepare referee reports promptly (Tosi, 2009). In fact, according to some studies, delays and missed deadlines are often a common feature of academic publishing today (Ellison, 2002; Card & Dellavigna, 2013; Hamermesh, 2013; Gerhardt et al., 2014).

If this is the general context and understanding, why should scholars care about writing referee reports? Starting from this doubt, this editorial aims at discussing the motivations behind the reviewers’ commitment to performing their tasks.

The reviewer’s dilemma

How does the process of preparing a report work? In general, preparing a referee report is an anonymous [¹] task, for referees are not known for the service they perform (Gerhardt et al., 2014). More than that, the preparation of such reports is unpaid and can be time-consuming for those involved in such a task (Chetty, Saez, and Sándor, 2014; Gerhardt et al., 2014). Thus, for example, when scholars must decide how to divide their time into various academic tasks such as class preparation, research activities or committee participation, among others, it makes sense to imagine that the preparation of referee reports may not be one of their priorities (Berk, Harvey & Hirshleifer, 2017; Hamermesh, 1992, 1994; Tosi, 2009).

Moreover, if the reviewers’ value perception of the evaluation process is not positive, there could be some distortions along the way (Bedeian, 2003; Tosi, 2009). The quote below reflects one example:

At a recent reviewer workshop I attended, one reviewer lamented that although providing such detail was not terribly difficult for one who knows the research domain, “Why should I do the author’s work for him?” A fair question, and one that a panelist answered simply, “Why not?” After all, a reviewer’s expertise is exactly the value she brings to the process (Schminke, 2002, p. 487).

In other words, in situations like this, an academic can ask the following question: “Will my actions have any significant impact on the remainder of the academic community?” If the answer is negative, this person is likely to have little incentive to cooperate with the collective undertaking as his/her actions are perceived as socially insignificant. One problem derived from such thinking is that if many people proceed in this way, it may be difficult for the academic community to achieve collective goals, such as advancing knowledge in the scientific community. After all, the prosperity of the scientific community is based on referee reports (Treviño, 2008; Gerhardt et al., 2014).

Then, what kind of incentives could matter?

Given this context, some questions arise: What would be the best way to encourage the preparation of referee reports for academic journals? Would it be worth paying reviewers for such a task? Moreover, what would be the proper monetary values to encourage the preparation of reviews with enhanced quality? Conversely, would there be alternative ways – besides monetary payments – to encourage this type of prosocial behaviour?

There are examples of situations in which the introduction of monetary incentives ended up generating unintended effects. For example, a day-care centre in Israel tried to establish a fine for parents of children who remained there after official departure hours. The initial intention of the day-care administration was to encourage parents to pick their children up on time. Ironically, what ended up happening after the introduction of the fine was an actual increase in the average time of delays (even after the end of the fine). That is, instead of reducing parents’ delays, the fine ended up raising them as some parents saw it as a price for lateness (Gneezy & Rustichini, 2000).

Despite these limitations of monetary incentives, why not run an experiment to test the best way to encourage the preparation of referee reports on time? That was precisely the proposal by Chetty et al. (2014), in which the authors ran an experiment with peer reviewers from an academic journal in the field of economics, the Journal of Public Economics (Chetty et al., 2014).

The authors started their experiment by randomly dividing the reviewers of the journal into four distinct groups:

1. a control group, with 45 days for the referee reports’ submission;

2. a first treatment group, subject to a shorter delivery time of 28 days;

3. a second treatment group, also subject to a 28-day term, but eligible to receive a cash incentive of US$100 per referee report handed on time; and

4. a third treatment group, subject to a social incentive, in which the reviewers would have their review preparation hours published on the journal’s website at the end of the year.

When evaluating their experiment results, the authors discovered interesting patterns arising from their experimental data. First, they noticed a reduction in the median time of submission of referee reports from 48 to 36 days approximately. Second, the introduction of a monetary incentive of US$100 per referee report reduced median review hours by additional eight days. Third, the treatment group subject to the social incentive (that is, which had their names published on the journal's website) presented a reduction of median review time of 2.5 days. This kind of incentive had a more significant impact in the case of tenured professors when compared to untenured referees.

Monetary incentives matter (as well as social incentives)

The most exciting aspect of the study by Chetty et al. (2014) is the result related to their experiment’s incentives. Unlike previous studies, in which monetary incentives ended up generating adverse effects in terms of behaviour (Gneezy & Rustichini, 2000; Squazzoni, Bravo & Takács, 2013), there was a substantial improvement in the time taken to prepare anonymous referee reports. In general terms, these results draw attention to the importance of incentives for referee reports, given the differences between individual incentives and social benefits in an academic setting. A recent body of evidence-based behavioural experiments suggests that people may care more about symbolic incentives – such as respect and status – than monetary gains (Ellingsen & Johannesson, 2007). It would be interesting to evaluate how referees respond to social and monetary incentives in real-world settings, given the differences between intrinsic and extrinsic motivations for collective action situations (Bénabou & Tirole, 2006).

Writing referee reports is so essential to developing a knowledge field that actions and interventions directed at the reviewer level “were associated with improved review quality compared with traditional methods” (Gaudino et al., 2021). Chetty et al.’s (2014) study focused on the monetary and social incentives and identified those gains.

If all those shreds of evidence are not enough to justify why we should care about writing referee reports, then the simple fact is that imperfect as it might be, there seems to be no better alternative to the peer review process at present. Therefore, we must continue to strive to improve it (London, 2021). For Gerhardt et al. (2014, p. 13), “the academic paper publishing process is implicitly based on a contract of trust between publishers and reviewers, authors, publishers and readers, and also on transparency and strictness about the procedures used to review papers”.

In a nutshell, “[…] reviewing is a professional responsibility” (Treviño, 2008, p. 8), and we must strive for academic progress despite all the problems and difficulties along the process.

Note

• 1.
  Because of the new open science practices, some referee reports now have the reviewers’ identification side by side with the authors’. This is becoming more and more common in the so-called “pre-print” format.

References

• Bedeian, A. G. (2003). The manuscript review process: the proper roles of authors, referees, and editors. Journal of Management Inquiry, 12(4), 331–338. Retrieved from: https://doi.org/10.1177/1056492603258974
  » https://doi.org/10.1177/1056492603258974
• Bénabou, R., & Tirole, J. (2006). Incentives and prosocial behavior. American Economic Review, 96(5), 1652–1678. Retrieved from: https://doi.org/10.1257/aer.96.5.1652 doi: 10.1257/aer.96.5.1652.
  » https://doi.org/10.1257/aer.96.5.1652» https://doi.org/10.1257/aer.96.5.1652
• Berk, J. B., Harvey, C. R., & Hirshleifer, D. (2017). How to write an effective referee report and improve the scientific review process. Journal of Economic Perspectives, 31(1), 231–244. Retrieved from: https://doi.org/10.1257/jep.31.1.231 doi: 10.1257/jep.31.1.231.
  » https://doi.org/10.1257/jep.31.1.231» https://doi.org/10.1257/jep.31.1.231
• Card, D., & DellaVigna, S. (2013). Nine facts about top journals in economics. Journal of Economic Literature, 51(1), 144–161. Retrieved from: http://doi.org/10.1257/jel.51.1.144 doi: 10.1257/jel.51.1.144.
  » https://doi.org/10.1257/jel.51.1.144» http://doi.org/10.1257/jel.51.1.144
• Chetty, R., Saez, E., & Sándor, L. (2014). What policies increase prosocial behavior? An experiment with referees at the journal of public economics. Journal of Economic Perspectives, 28(3), 169–188. Retrieved from https://doi.org/10.1257/jep.28.3.169 doi: 10.1257/jep.28.3.169.
  » https://doi.org/10.1257/jep.28.3.169» https://doi.org/10.1257/jep.28.3.169
• Ellingsen, T., & Johannesson, M. (2007). Paying respect. Journal of Economic Perspectives, 21(4), 135–149. Retrieved from: https://doi.org/10.1257/jep.21.4.135 doi: 10.1257/jep.21.4.135.
  » https://doi.org/10.1257/jep.21.4.135» https://doi.org/10.1257/jep.21.4.135
• Ellison, G. (2002). The slowdown of the economics publishing process. Journal of Political Economy, 110(5), 947–993, doi: 10.1086/341868. Retrieved from www.jstor.org/stable/
  » https://doi.org/10.1086/341868» www.jstor.org/stable/
• Gerhardt, L. M., Dal Pai, D., Gouveia, H. G., & Azzolin, K. D. E. (2014). Peer reviewer: what commitment is that? Revista Gaúcha de Enfermagem, 35(2), 12–3.8.–11. Retrieved from http://doi.org/10.1590/1983-14470201400200001 doi: 10.1590/1983-14470201400200001.
  » https://doi.org/10.1590/1983-14470201400200001» http://doi.org/10.1590/1983-14470201400200001
• Gneezy, U., & Rustichini, A. (2000). A fine is a price. The Journal of Legal Studies, 29(1), 1–17. Retrieved from https://doi.org/10.1086/468061 doi: 10.1086/468061.
  » https://doi.org/10.1086/468061» https://doi.org/10.1086/468061
• Hamermesh, D. S. (1992). The young economist's guide to professional etiquette. Journal of Economic Perspectives, 6(1), 169–179. Retrieved from http://doi.org/10.1257/jep.6.1.169 doi: 10.1257/jep.6.1.169.
  » https://doi.org/10.1257/jep.6.1.169» http://doi.org/10.1257/jep.6.1.169
• Hamermesh, D. S. (1994). Facts and myths about refereeing. Journal of Economic Perspectives, 8(1), 153–163. Retrieved from www.jstor.org/stable/2138156 doi: 10.1257/jep.8.1.153.
  » https://doi.org/10.1257/jep.8.1.153» www.jstor.org/stable/2138156
• Hamermesh, D. S. (2013). Six decades of top economics publishing: who and how? Journal of Economic Literature, 51(1), 162–172. Retrieved from http://doi.org/10.1257/jel.51.1.162 doi: 10.1257/jel.51.1.162.
  » https://doi.org/10.1257/jel.51.1.162» http://doi.org/10.1257/jel.51.1.162
• London, B. (Ed.) (2021). Reviewing peer review. Journal of the American Heart Association, 10(15), e021475. Retrieved from https://doi.org/10.1161/JAHA.121.021475 doi: 10.1161/JAHA.121.021475.
  » https://doi.org/10.1161/JAHA.121.021475» https://doi.org/10.1161/JAHA.121.021475
• McAfee, R. P. (2010). Edifying editing. The American Economist, 55(1), 1–8. doi: 10.1177/056943451005500101.
  » https://doi.org/10.1177/056943451005500101
• Schminke, M. (2002). From the editors. Academy of Management Journal, 45(3), 487–490. Retrieved from https://doi.org/10.5465/amj.2002.25891519 doi: 10.5465/amj.2002.25891519.
  » https://doi.org/10.5465/amj.2002.25891519» https://doi.org/10.5465/amj.2002.25891519
• Squazzoni, F., Bravo, G., & Takács, K. (2013). Does incentive provision increase the quality of peer review? An experimental study. Research Policy, 42(1), 287–294. Retrieved from https://doi.org/10.1016/j.respol.2012.04.014 doi: 10.1016/j.respol.2012.04.014.
  » https://doi.org/10.1016/j.respol.2012.04.014» https://doi.org/10.1016/j.respol.2012.04.014
• Taylor, T. (2009). An editor's life at the journal of economic perspectives. The American Economist, 53(1), 1–12. Retrieved from www.jstor.org/stable/40657775 doi: 10.1177/056943450905300107.
  » https://doi.org/10.1177/056943450905300107» www.jstor.org/stable/40657775
• Tosi, H. (2009). It's about time!!!!: What to do about long delays in the review process. Journal of Management Inquiry, 18(2), 175–178.https://doi.org/10.1177/1056492608330468 doi: 10.1177/1056492608330468.
  » https://doi.org/10.1177/1056492608330468» https://doi.org/10.1177/1056492608330468
• Treviño, L. K.(Ed.) (2008). Editor's comments: Why review? Because reviewing is a professional responsibility. Academy of Management Review, 33(1), 8–10. Retrieved from https://doi.org/10.5465/amr.2008.27744831
  » https://doi.org/10.5465/amr.2008.27744831

Publication Dates

• Publication in this collection
  10 Dec 2021
• Date of issue
  Oct-Dec 2021