Responsible Science Assessment: downplaying indexes, boosting quality

Abstract

Scientists are facing enormous pressures posed by growing scientific communities and stagnant/reduced funding. In this scenario, mechanisms of knowledge achievement and management, as well as how recruitment, progression and evaluation are carried out should be reevaluated. We argue here that knowledge has become a profitable commodity and, as a consequence, excessive academic quantification, individual output assessment problems and abusive editorial market strategies have reached unsustainable levels. We propose to reinforce existing guidelines and to establish new ones to overcome these issues. Our proposal, the Initiative for Responsible Scientific Assessment (IRSA), has the main goal to strengthen and expand previous movements in the scientific community to promote higher quality research assessment, focused on better Science.

Key words
Evaluation; impact factor; peer-review; policy; quantophrenia; university

IMPATIENT SCIENCE: HAS KNOWLEDGE BECOME A COMMODITY?

There was a time when universities were houses of free thought (Saint Victor 1961) where scholars and philosophers could contemplate and interrogate anything. Indeed, critical thinking lies at the heart of any scientific endeavor, since the inquisitive nature of a scientist’s mind can deal best with the intricacies and complexities of the Universe when allowed to explore ideas freely. However, the structure and culture in academia has changed radically over the years, and free thought and critical thinking are no longer core values in many Universities. Instead, management strategies, market and industry interests and red tape have transformed academic culture (Readings 1996). Modernization of universities is a necessity, closer interactions with society should be stimulated, and evaluating scientific progress is fundamental for the pursuit of true advances in Science, but these activities cannot corrode the central reason for the existence of Universities as sources and distributors of knowledge and culture (Readings 1996).

Research productivity undoubtedly must be evaluated to make decisions on the use of limited resources such as research funding and academic hiring. Escalating evaluation demands over the last decades have generated an increased use of parameters directly imported from the corporate-executive world. As a result, current academic assessment involves strong reliance on numerical criteria for scientific quality and quantity measurements, as well as a stimulus toward immediate knowledge production. Examples include the quantification of scientific publications, often normalized to the visibility of journals they are published in. Furthermore, many current grant proposals (used to request research funding) require the presentation of vast quantities of “preliminary results”, often proof that the project presented has in fact already been almost completely executed prior to its approval, thus ensuring immediate, quantifiable, scientific output (Readings 1996). We call this new reality “Impatient Science”, a term derived from “Impatient Capital”, coined by Bennett Harrison and describing a growing desire for quick financial return for investors (Harrison 1994).

ARE WE OUR OWN WORST ENEMIES?

Within this impatient framework in which scientists must publish or perish, and also publish quickly, often, and in high impact venues, it is not surprising that the rates of scientific publishing now increase close to 9 % every year (Larsen & von Ins 2010, Bornmann & Mutz 2015). This has created an avalanche of scientific information which has crushed the ability of specialists to dedicate the time and focus necessary to read and fully comprehend these publications (Rayner et al. 2016). Paradoxically, as human knowledge has become more available, reading by specialists is increasingly shallow. The lack of focus on scientific reading is worsened by an excessive amount of non-scientific activities performed by academics (Ziker 2014).

While direct production lines certainly benefit from quantitative assessments and strategic planning with specific/measurable/achievable goals, scientific endeavor is by nature much more unpredictable and less amenable to these restraints. Indeed, truly novel scientific findings cannot be strategically planned; or they would not be truly novel. Examples of breakthrough discoveries that were completely unplanned are too numerous to list here but include findings that led to the development of seminal modern conveniences as diverse as penicillin, X-rays, artificial sweeteners, gunpowder, plastics, anesthetics and so on. The corollary is that quality and originality in the production of new knowledge is inherently linked to unpredictable experimental outputs, their interpretations, and the construction of new hypotheses emerging from data obtained.

By decreasing serendipity, the growth of Impatient Science within the academic community is, we believe, resulting in a loss in overall research quality. One result is the production of more incremental, rather than fundamental, discoveries. Another much more worrisome aspect is that in the angst to publish, more unreliable data is being produced. Examples include increased frequency of retractions of scientific articles due to misconduct in high impact journals (Fang et al. 2012) and lack of reproducibility (Baker 2016), in which results from one group of researchers cannot be replicated by others. Finally, Impatient Science may well be at the center of the epidemic increase in scientific bullying, burnout syndromes, and other psychosocial illnesses among students and scholars in recent years (de Meis et al. 2003a, Evans et al. 2018).

In addition to impacting upon how scientists produce knowledge and the resulting quality of scientific endeavors, a side effect of excessive competitiveness and pressure to produce numbers of scientific publications is the unconstrained growth of the scientific editorial market. Scientific publishing is an enormously valued business, with profit margins well above most other economic sectors (Buranyi 2017). This is due to its unique characteristic in which researchers are not paid by editorial companies to produce their material; instead, they write for free to broadcast their scientific findings, which were supported by research grants from public and private sectors. The scientific papers are then evaluated prior to publication for their suitability by fellow scientists through a process known as “peer-review”. Although the peer-review process is very specialized and laborious, it is also typically done for free. The publication costs can then either be covered by fees for readers of the papers or, increasingly, prior to publication, by the authors themselves, within what has been named the Open Access model.

Open Access has the undeniable advantage of providing unrestricted reading ability to anyone worldwide once scientific knowledge is published but has also produced unwanted side effects. One undesirable consequence of the Open Access model is the recent proliferation of predatory publishing venues, in which the peer review process is completely absent or not selective, and authors basically pay to get any material published, irrespective of quality (Hern & Duncan 2018). Another consequence is that the true costs of open access publishing are poorly known and unregulated, so even reputable journals with selective publication policies can charge fees that are far superior to market value. Indeed, a number of highly expensive “brand-name” scientific journals have appeared over the last few years (Kowaltowski & Amigo 2018). Furthermore, since the sources of payment of these open access papers are the authors themselves, this model may distort scientific dissemination by scientists with more limited grant funds, forcing them to decide in which journals they would publish based on economic, rather than scientific, criteria (Kowaltowski & Oliveira 2019). Indeed, the current model of publishing payment, where research, publishing costs and even salaries (the “triple dipping” model) are financially covered by public funding agencies, represents a true anomaly and reinforces the corrosive nature of the Impatient Science culture (Romero 2018).

In essence, Impatient Science has helped foster the very profitable business of knowledge dissemination within the scientific community through scientific publications (Young et al. 2008). Importantly, while some editorial companies are backed by scientific societies and return their earnings in the form of conference organization funding, grants, and fellowships for scientists, most, including the largest growing editorial groups (Kowaltowski & Amigo 2018), do not have ties to these societies, and earnings are theirs to keep. Increasingly, also, journals have switched to the use of professional in-house editors instead of using active scientists for editorial decisions, which results in less input from front-line researchers as to which directions the journals should take in their scientific and financial decisions.

The solution for this situation, we believe, lies in two distinct and complementary actions by scientists: rethinking the scientific process and its evaluation, avoiding Impatient Science and refocusing on quality and true knowledge gains, as well as changing the way venues for publication of scientific data are chosen by scientists.

“ACADEMIC QUANTOPHRENIA” AND THE NEED TO RETHINK SCIENTIFIC EVALUATION

When rethinking the scientific evaluation processes, one must consider the current framework in which the qualification of scientific contributions through indexes has become common ground. Excessive focus on parameterization has shaped a new generation of scientists with high performance on indexes and not primarily concentrated on scientific significance. This phenomenon, which has found fertile ground to flourish in academia, is described by Ptirim Sorokin as ‘‘Quantophrenia’’ (Sorokin 1956).

The journal impact factor (JIF), created by Eugene Garfield (Garfield 1955) and managed today by Clarivate Analytics, is the most popular and controversial metric to quantify journal visibility. Despite the simplicity in which JIF is calculated (it reflects the average number of times each paper in a journal is referenced in other publications per year), strong questions regarding the lack of transparency, reproducibility and potential manipulation of impact data have been raised (Seglen 1997). Surprisingly, despite all these questions, the JIF is still seen by the scientific community as a proxy of quality for a given journal. We believe the problem is not the JIF itself, but rather the way in which we use it to measure individual performances or chose journals to submit work to. The hard truth is that the problem of excessive scientific parameterization and Quantophrenia has been perpetrated by scientists themselves (Johnston 2013).

Scientists have not completely ignored the problems with the JIF and have actively discussed alternatives, which included new metrics with the intent of overcoming JIF flaws (de Meis et al. 2003b, Seglen 1992, 1997, Hansson 1995). These include the h-index, g-index, eigenfactor, Scimago Journal Rank and others (Hirsch 2005, Egghe 2006, Bergstrom 2007, Radicchi et al. 2008, Falagas et al. 2008, Priem et al. 2012, Hutchins et al. 2016). Although we acknowledge the significant steps forward made by these newer scientometric tools, we would like to emphasize that Quantophrenia cannot be resolved by new metrics, since these metrics are the core of the problem of excessive quantification in detriment of quality evaluations. Instead, these indexes are important as accessory tools to facilitate global evaluative processes, insofar as they quickly provide numerical and seemingly objective indicators of scientific production. However, parameterization cannot per se define relevance, prestige, and quality (evaluating quality through quantitative parameters is, in fact, an oxymoron) of a researcher, nor determine their professional trajectory. This instead requires true understanding and qualitative assessment of papers and proposals by specialized peers. The difficulty lies in executing such quality-based evaluations during grant evaluation processes within funding agencies, when dozens or hundreds of proposals must be compared as rapidly and efficiently as possible.

The answer may lie in a fine balance of quantitative and qualitative assessment metrics to measure scientific relevance of projects, individuals or institutions. Peer-reviewing, even with all its imperfections, is still highly effective, has stood the test of time, and should be emphasized. Indeed, peer reviews can evolve in quality if reviewers are increasingly made aware of their seminal importance to Science. Their critical opinions must be central in funding decisions and analyzed jointly with higher quality metrics (such as those that focus on the individual´s output itself and not where it is published) and also on their subjective perception of relevance and originality. Indeed, the scientific community must understand that the subjective perception of a well-selected and specialized peer reviewer is a strong indicator of merit. This subjective perception can be aided by requesting qualitative information from the submitting parties such as their five most important papers (Finkel 2019) or a description of their main scientific findings. Value must be placed also on refining proposals based on concise and constructive feedback comments from these peer reviewers, a task that takes time. We are not advocating for slowing down the already onerous peer-review process (Vale 2015, Powell 2016), but rather for wiser use of the time already required in grant analysis, substituting useless and time-consuming documentation, formatting and indexing for quality revisions. This can be implemented by recognizing peer reviewers for their hard work, such as by creating mechanisms to cite and reward them (Stern & O’Shea 2019).

This strategy is well in line with recent high-profile statements backed by thousands of scientists, editors, journals, scientific societies and funding agencies (San Francisco Declaration On Research Assessment (DORA) 2012, Hicks et al. 2015, The Metric Tide 2015). These documents have in common proposals to boldly shift the way Science is evaluated, emphasizing the advance itself, rather than relying on objective indexes, and also the rational use of responsible metrics associated with subjective analyses. Although sluggish changes started in 2012 as a result of the DORA-Leiden-Tide declarations, they may have been overshadowed by growth in sectors of Impatient Science and Quantophrenia.

One Quantophrenic and Impatient Science aspect of scientific endeavors today relates to the second point we believe lies at the core of a change in scientific assessment and progress today: the choice of publication venues for papers describing research findings. While the DORA-Leiden-Tide intent is to focus on science produced and not where it is communicated, researchers worldwide have increasingly found that publishing their work in specific scientific journals with well-known “brand names” increases the immediate/impatient attention given to their publication, at least within the framework of headlines in social media (Kowaltowski & Amigo 2018). This has strongly encouraged the editorial market to create more such venues, mainly by creating “daughter journals” with the same brand name as a well-respected and established journal. Indeed, a well-respected journal has launched more than 50 “daughter journals” within its brand (Nature 2021). While a larger number of options in a market generally helps increase quality and decrease pricing, the specific niche characteristics of scientific publishing have produced the exact opposite: these “daughter journals” are usually highly priced and have in-house rather than active scientists on their editorial boards, which makes them more prone to promote market values over quality Science. Most importantly, the revenue from publication proceeds in these journals goes to the publishers, which are mostly private enterprises, unassociated with scientific societies.

To avoid both predatory, low quality, publications and quench the Quantophrenic trend, we urge our colleagues to rethink in which venues they publish their results. A feasible solution to expedite scientific communication is to fully embrace the culture of sharing original research through preprint servers before formal journal publication (Kaiser 2017). However, preprint sharing is not widespread enough in the Life Sciences community yet, and we urgently need to discuss its possibilities and benefits for students and senior researchers. In this sense, publication in a peer-reviewed format in scientific journals is still desirable, especially considering the benefits of peer-revision for manuscript quality.

In the current paradigm, in which the number of publication venues is increasing enormously, choosing where these papers are submitted for appreciation is ever more important. We believe market culture, Impatient Science and Quantophrenia cannot interfere with this decision process and urge researchers to carefully consider the journals they submit their work to, and not rely on JIF and brand names. A recent, very ambitious, plan (cOAlition S 2021) has proposed that the main focus for scientific publications should be to choose open access venues. While we agree limited access to scientific knowledge is a problem, we do not think it is the most pressing obstacle toward increased scientific quality and fear a fast push for open access can stimulate predatory and unfairly priced journals (Kowaltowski & Oliveira 2019). Instead, we believe researchers should rethink who benefits financially from the journals they publish, as well as who are the people shepherding this publication venue. Preference should be given to journals with ties to scientific societies, in which at least part of the publication profits revert to research-related causes. Strong links to societies also warrants steering committees with scientific, rather than market, interests. Finally, we urge our colleagues to give preference to journals in which Associate and Board Editors are active, well recognized scientists. This ensures that they will both steer the journal´s aims well and chose the best possible peer reviewers for each submission. In giving preference to society-backed journals with active scientists as editors, authors can change the current Quantophrenic tide.

A final note should be made regarding the consequences of Impatient Science on the process of knowledge acquisition and its effects on Basic Science, or the act of immersing into the uncertainties of the unknown, seeking to understand it for the sole purpose of understanding. Within the Impatient Science framework, researchers have increasingly been forced to provide evidence of immediate applicability of their ideas, even when seeking public funding for their research. Due to its inherently elemental characteristic, only basic research can bring true innovation to the various fields of Science. Therefore, basic research must be valued, prioritized and protected. To accomplish this, robust, stable, and funded programs to support Basic Science should be implemented and supported by governmental agencies. Scientists must rally and defend basic research openly to ensure this. Furthermore, grant evaluators and peer reviewers should recognize and highlight truly innovative basic science whenever possible.

THE INITIATIVE FOR RESPONSIBLE SCIENTIFIC ASSESSMENT (IRSA):

The scientific community is by nature creative. As such, it is time for this group of creative persons to search, together, for new solutions and new directions when appraising Science. The pioneering DORA-Leiden-Tide documents requesting changes in scientific output evaluation need to be expanded upon to guarantee the full potential of Science´s undeniable ability to promote development. These expansions involve modifications in both how scientists disseminate and promote their findings and, centrally, how scientific output and projects are analyzed. Most of all, this requires a change of attitude by researchers for the benefit of scientific endeavor. Within this framework, the synergistic engagement of society, policymakers, and researchers through scientific and innovation processes, as stated in the Responsible Research and Innovation movement (Responsible research and innovation (RRI) 2021), represents a valuable proposal to match scientific research and outputs with the society needs.

Overall, to change the current situation and allow research to reach its full contemporary potential, we propose a scientist-based movement to improve scientific assessment, the Initiative for Responsible Scientific Assessment (IRSA). This movement combines and expands upon previous initiatives, with the following guidelines:

• Endorse and spread DORA, the Leiden Manifesto, Responsible Research and Innovation movement, and the Metric Tide. Stimulate quality assessment over quantifiable metrics. Follow their guidelines and discuss these ideals within your academic community, scientific societies and funding agencies.

• Elevate the value of quality peer review. Provide the best and most balanced assessment possible when acting as a reviewer, focusing on improving the quality of the paper and Science in general. Make reviewer suggestions for your paper submissions mindfully, prioritizing critical specialists. Denounce questionable peer-review practices or evidence of predatory publishing actions. Develop and support policies to reward and qualify peer-reviews (including in available platforms; Publons 2021). Consider and discuss the advantages and disadvantages of open peer review.

• Recognize seminal findings by researchers. Cite primary literature, avoiding reviews as references. Recognize and give credit to scientists that made original discoveries. Value central scientific discoveries in funding and hiring decisions.

• Promote actions to associate quality assessment with more representative scientometric metrics in academic decisions. Assess work done and scientific findings above the impact factor of the journal it is published in. Discuss the flaws of metrics in evaluation processes as well as new ways to improve how Science is qualified with students and colleagues. When using metrics in association with quality assessment, give preference to metrics that focus on the work produced by the individual, and not those that focus on journal metrics.

• Submit manuscripts to journals with editors who are active scientists with recognized reputation in the field. Choose journals considering their track record and the qualifications of academic editors and the editorial board, which will ensure quality peer review. Do not support predatory or open access journals that prioritize profits over knowledge. Consider which entities the funds from page charges benefit and give preference to scientific societies over unassociated editors. Debate fair open access costs. Stimulate affordable open access by using pre-print depository platforms.

An online petition to endorse IRSA can be found at https://www.petitions.net/the_initiative_for_responsible_scientific_assessment_irsa.

REFERENCES

• BAKER M. 2016. 1,500 scientists lift the lid on reproducibility. Nature 533: 452-454.
• BERGSTROM C. 2007. Eigenfactor: Measuring the value and prestige of scholarly journals. College Res Libr News 68: 314-316.
• BORNMANN L & MUTZ R. 2015. Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. J Assoc Inform Sci Tech 66: 2215–2222.
• BURANYI S. 2017. Is the staggeringly profitable business of scientific publishing bad for science? The Guardian. Accessed March 18, 2019. https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science
  » https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science
• COALITION S. 2021. https://www.coalition-s.org/
  » https://www.coalition-s.org/
• DE MEIS L, VELLOSO A, LANNES D, CARMO MS & DE MEIS C. 2003a. The growing competition in Brazilian science: rites of passage, stress and burnout. Braz J Med Biol Res 36: 1135-1141.
• DE MEIS L, DO CARMO MS & DE MEIS C. 2003b. Impact factors: just part of a research treadmill. Nature 424: 723.
• DORA - SAN FRANCISCO DECLARATION ON RESEARCH ASSESSMENT. 2012. https://sfdora.org/
  » https://sfdora.org/
• EGGHE L. 2006. Theory and Practice of the g-Index. Scientometrics 69: 131-152.
• EVANS TM, BIRA L, GASTELUM JB, WEISS LT, & VANDERFORD NL. 2018. Evidence for a mental health crisis in graduate education. Nat Biotech 36: 282-284.
• FALAGAS ME, KOURANOS VD, ARENCIBIA-JORGE R & KARAGEORGOPOULOS DE. 2008. Comparison of SCImago journal rank indicator with journal impact factor. FASEB J 22: 2623-2628.
• FANG FC, STEEN RG, & CASADEVALL A. 2012. Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci USA 109: 17028-17033.
• FINKEL A. 2019. To move research from quantity to quality, go beyond good intentions. Nature 566: 297.
• GARFIELD E. 1955. Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas. Science 122: 108-111.
• HANSSON S. 1995. Impact factor as a misleading tool in evaluation of medical journals. Lancet 346: 906.
• HARRISON B. 1994. The Dark Side of Flexible Production. Nat Prod Rev 480-501.
• HERN A & DUNCAN P. 2018. Predatory publishers: the journals that churn out fake science. The Guardian. https://www.theguardian.com/technology/2018/aug/10/predatory-publishers-the-journals-who-churn-out-fake-science
  » https://www.theguardian.com/technology/2018/aug/10/predatory-publishers-the-journals-who-churn-out-fake-science
• HICKS D, WOUTERS P, WALTMAN L, DE RIJCKE S & RAFOLS I. 2015. Bibliometrics: The Leiden Manifesto for research metrics. Nature 520: 429-431.
• HIRSCH JE. 2005. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci USA 102: 16569-16572.
• HUTCHINS BI, YUAN X, ANDERSON JM, & SANTANGELO GM. 2016. Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level. PLoS Biol 14: e1002541.
• JOHNSTON M. 2013. We Have Met the Enemy, and It Is Us. Genetics 194: 791-792.
• KAISER J. 2017. The preprint dilemma. Science 357: 1344-1349.
• KOWALTOWSKI A & AMIGO I. 2018. Branding in Scientific Publishing. Redoxoma Newsletter http://redoxomanewsletter.iq.usp.br/node/86
  » http://redoxomanewsletter.iq.usp.br/node/86
• KOWALTOWSKI A & OLIVEIRA MF. 2019. Plan S: Unrealistic capped fee structure. Science 363: 461.
• LARSEN PO & VON INS M. 2010. The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics 84: 575-603.
• NATURE. 2021. https://www.nature.com/siteindex
  » https://www.nature.com/siteindex
• POWELL K. 2016. Does it take too long to publish research? Nature 530: 148-151.
• PRIEM J, GROTH P & TARABORELLI D. 2012. The Altmetrics Collection. PLoS ONE 7: e48753.
• PUBLONS. 2021. https://publons.com/dashboard/summary/
  » https://publons.com/dashboard/summary/
• RADICCHI F, FORTUNATO S & CASTELLANO C. 2008. Universality of citation distributions: Toward an objective measure of scientific impact. Proc Natl Acad Sci USA 105: 17268-17272.
• RAYNER K, SCHOTTER ER, MASSON ME, POTTER MC & TREIMAN R. 2016. So Much to Read, So Little Time: How Do We Read, and Can Speed Reading Help? Psychol Sci Public Interest 17: 4-34.
• READINGS B. 1996. The University in Ruins. Harvard University Press, Cambridge.
• ROMERO A. 2018. Academic Publishing is Big Business, And How Blockchain Can Make A Difference. Nasdaq website. https://www.nasdaq.com/articles/academic-publishing-big-business-and-how-blockchain-can-make-difference-2018-07-12
  » https://www.nasdaq.com/articles/academic-publishing-big-business-and-how-blockchain-can-make-difference-2018-07-12
• RRI - RESPONSIBLE RESEARCH AND INNOVATION. 2021. https://rri-tools.eu/
  » https://rri-tools.eu/
• SAINT VICTOR H. 1961. The Didascalicon of Hugh of Saint Victor: The medieval guide to the arts. Columbia University Press, New York.
• SEGLEN PO. 1992. The skewness of science. J Assoc Inform Sci Tech 43: 628-638.
• SEGLEN PO. 1997. Why the impact factor of journals should not be used for evaluating research. British Med J 314: 498-502.
• SOROKIN PA. 1956. Fads and foibles in modern sociology and related sciences, Praeger Publishers, Westport.
• STERN BM & O’SHEA EK. 2019. A proposal for the future of scientific publishing in the life sciences. PLoS Biol 17: e3000116.
• THE METRIC TIDE. 2015. https://responsiblemetrics.org/the-metric-tide/
  » https://responsiblemetrics.org/the-metric-tide/
• VALE RD. 2015. Accelerating scientific publication in biology. Proc Natl Acad Sci USA 112: 13439-13446.
• YOUNG NS, IOANNIDIS JPA & AL-UBAYDLI O. 2008. Why Current Publication Practices May Distort Science. PLoS Med 5: e201
• ZIKER J. 2014. The Long, Lonely Job of Homo academicus. Blue Review. Accessed January 22, 2021. https://www.boisestate.edu/bluereview/faculty-time-allocation/
  » https://www.boisestate.edu/bluereview/faculty-time-allocation/

Publication Dates

• Publication in this collection
  22 Feb 2021
• Date of issue
  2021

History

• Received
  10 Dec 2019
• Accepted
  28 Apr 2020