Responsible Science Assessment: downplaying indexes, boosting quality

: Scientists are facing enormous pressures posed by growing scientifi c communities and stagnant/reduced funding. In this scenario, mechanisms of knowledge achievement and management, as well as how recruitment, progression and evaluation are carried out should be reevaluated. We argue here that knowledge has become a profi table commodity and, as a consequence, excessive academic quantifi cation, individual output assessment problems and abusive editorial market strategies have reached unsustainable levels. We propose to reinforce existing guidelines and to establish new ones to overcome these issues. Our proposal, the Initiative for Responsible Scientifi c Assessment (IRSA), has the main goal to strengthen and expand previous movements in the scientifi c community to promote higher quality research assessment, focused on better Science.

There was a time when universities were houses of free thought (Saint Victor 1961) where scholars and philosophers could contemplate and interrogate anything. Indeed, critical thinking lies at the heart of any scientific endeavor, since the inquisitive nature of a scientist's mind can deal best with the intricacies and complexities of the Universe when allowed to explore ideas freely. However, the structure and culture in academia has changed radically over the years, and free thought and critical thinking are no longer core values in many Universities. Instead, management strategies, market and industry interests and red tape have transformed academic culture (Readings 1996). Modernization of universities is a necessity, closer interactions with society should be stimulated, and evaluating scientifi c progress is fundamental for the pursuit of true advances in Science, but these activities cannot corrode the central reason for the existence of Universities as sources and distributors of knowledge and culture (Readings 1996).
Research productivity undoubtedly must be evaluated to make decisions on the use of limited resources such as research funding and academic hiring. Escalating evaluation demands over the last decades have generated an increased use of parameters directly imported from the corporate-executive world. As a result, current academic assessment involves strong reliance on numerical criteria for scientifi c quality and quantity measurements, as well as a stimulus toward immediate knowledge production. Examples include the quantifi cation of scientifi c publications, often normalized to the visibility of journals they are published in. Furthermore, many current grant proposals (used to request research funding) require the presentation of vast quantities of "preliminary results", often proof that the project presented has in fact already been almost completely executed prior to its approval, thus ensuring immediate, quantifiable, scientific output (Readings 1996). We call this new reality "Impatient Science", a term derived from "Impatient Capital", coined by Bennett Harrison and describing a growing desire for quick financial return for investors (Harrison 1994).

ARE WE OUR OWN WORST ENEMIES?
Within this impatient framework in which scientists must publish or perish, and also publish quickly, often, and in high impact venues, it is not surprising that the rates of scientific publishing now increase close to 9 % every year (Larsen & von Ins 2010, Bornmann & Mutz 2015. This has created an avalanche of scientific information which has crushed the ability of specialists to dedicate the time and focus necessary to read and fully comprehend these publications (Rayner et al. 2016). Paradoxically, as human knowledge has become more available, reading by specialists is increasingly shallow. The lack of focus on scientific reading is worsened by an excessive amount of non-scientific activities performed by academics (Ziker 2014).
While direct production lines certainly benefit from quantitative assessments and strategic planning with specific/measurable/ achievable goals, scientific endeavor is by nature much more unpredictable and less amenable to these restraints. Indeed, truly novel scientific findings cannot be strategically planned; or they would not be truly novel. Examples of breakthrough discoveries that were completely unplanned are too numerous to list here but include findings that led to the development of seminal modern conveniences as diverse as penicillin, X-rays, artificial sweeteners, gunpowder, plastics, anesthetics and so on. The corollary is that quality and originality in the production of new knowledge is inherently linked to unpredictable experimental outputs, their interpretations, and the construction of new hypotheses emerging from data obtained.
By decreasing serendipity, the growth of Impatient Science within the academic community is, we believe, resulting in a loss in overall research quality. One result is the production of more incremental, rather than fundamental, discoveries. Another much more worrisome aspect is that in the angst to publish, more unreliable data is being produced. Examples include increased frequency of retractions of scientific articles due to misconduct in high impact journals (Fang et al. 2012) and lack of reproducibility (Baker 2016), in which results from one group of researchers cannot be replicated by others. Finally, Impatient Science may well be at the center of the epidemic increase in scientific bullying, burnout syndromes, and other psychosocial illnesses among students and scholars in recent years (de Meis et al. 2003a, Evans et al. 2018. In addition to impacting upon how scientists produce knowledge and the resulting quality of scientific endeavors, a side effect of excessive competitiveness and pressure to produce numbers of scientific publications is the unconstrained growth of the scientific editorial market. Scientific publishing is an enormously valued business, with profit margins well above most other economic sectors (Buranyi 2017). This is due to its unique characteristic in which researchers are not paid by editorial companies to produce their material; instead, they write for free to broadcast their scientific findings, which were supported by research grants from public and private sectors. The scientific papers are then evaluated prior to publication for their suitability by fellow scientists through a process known as "peer-review". Although the peerreview process is very specialized and laborious, it is also typically done for free. The publication costs can then either be covered by fees for readers of the papers or, increasingly, prior to publication, by the authors themselves, within what has been named the Open Access model.
Open Access has the undeniable advantage of providing unrestricted reading ability to anyone worldwide once scientific knowledge is published but has also produced unwanted side effects. One undesirable consequence of the Open Access model is the recent proliferation of predatory publishing venues, in which the peer review process is completely absent or not selective, and authors basically pay to get any material published, irrespective of quality (Hern & Duncan 2018). Another consequence is that the true costs of open access publishing are poorly known and unregulated, so even reputable journals with selective publication policies can charge fees that are far superior to market value. Indeed, a number of highly expensive "brand-name" scientific journals have appeared over the last few years (Kowaltowski & Amigo 2018). Furthermore, since the sources of payment of these open access papers are the authors themselves, this model may distort scientific dissemination by scientists with more limited grant funds, forcing them to decide in which journals they would publish based on economic, rather than scientific, criteria (Kowaltowski & Oliveira 2019). Indeed, the current model of publishing payment, where research, publishing costs and even salaries (the "triple dipping" model) are financially covered by public funding agencies, represents a true anomaly and reinforces the corrosive nature of the Impatient Science culture (Romero 2018).
In essence, Impatient Science has helped foster the very profitable business of knowledge dissemination within the scientific community through scientific publications (Young et al. 2008). Importantly, while some editorial companies are backed by scientific societies and return their earnings in the form of conference organization funding, grants, and fellowships for scientists, most, including the largest growing editorial groups (Kowaltowski & Amigo 2018), do not have ties to these societies, and earnings are theirs to keep. Increasingly, also, journals have switched to the use of professional inhouse editors instead of using active scientists for editorial decisions, which results in less input from front-line researchers as to which directions the journals should take in their scientific and financial decisions.
The solution for this situation, we believe, lies in two distinct and complementary actions by scientists: rethinking the scientific process and its evaluation, avoiding Impatient Science and refocusing on quality and true knowledge gains, as well as changing the way venues for publication of scientific data are chosen by scientists.

"ACADEMIC QUANTOPHRENIA" AND THE NEED TO RETHINK SCIENTIFIC EVALUATION
When rethinking the scientific evaluation processes, one must consider the current framework in which the qualification of scientific contributions through indexes has become common ground. Excessive focus on parameterization has shaped a new generation of scientists with high performance on indexes and not primarily concentrated on scientific significance. This phenomenon, which has found fertile ground to flourish in academia, is described by Ptirim Sorokin as ''Quantophrenia'' (Sorokin 1956).
The journal impact factor (JIF), created by Eugene Garfield (Garfield 1955) and managed today by Clarivate Analytics, is the most popular and controversial metric to quantify journal visibility. Despite the simplicity in which JIF is calculated (it reflects the average number of times each paper in a journal is referenced in other publications per year), strong questions regarding the lack of transparency, reproducibility and potential manipulation of impact data have been raised (Seglen 1997). Surprisingly, despite all these questions, the JIF is still seen by the scientific community as a proxy of quality for a given journal. We believe the problem is not the JIF itself, but rather the way in which we use it to measure individual performances or chose journals to submit work to. The hard truth is that the problem of excessive scientific parameterization and Quantophrenia has been perpetrated by scientists themselves (Johnston 2013).
Scientists have not completely ignored the problems with the JIF and have actively discussed alternatives, which included new metrics with the intent of overcoming JIF flaws (de Meis et al. 2003b, Seglen 1992, 1997, Hansson 1995. These include the h-index, g-index, eigenfactor, Scimago Journal Rank and others (Hirsch 2005, Egghe 2006, Bergstrom 2007, Radicchi et al. 2008, Falagas et al. 2008, Priem et al. 2012, Hutchins et al. 2016). Although we acknowledge the significant steps forward made by these newer scientometric tools, we would like to emphasize that Quantophrenia cannot be resolved by new metrics, since these metrics are the core of the problem of excessive quantification in detriment of quality evaluations. Instead, these indexes are important as accessory tools to facilitate global evaluative processes, insofar as they quickly provide numerical and seemingly objective indicators of scientific production. However, parameterization cannot per se define relevance, prestige, and quality (evaluating quality through quantitative parameters is, in fact, an oxymoron) of a researcher, nor determine their professional trajectory. This instead requires true understanding and qualitative assessment of papers and proposals by specialized peers. The difficulty lies in executing such qualitybased evaluations during grant evaluation processes within funding agencies, when dozens or hundreds of proposals must be compared as rapidly and efficiently as possible.
The answer may lie in a fine balance of quantitative and qualitative assessment metrics to measure scientific relevance of projects, individuals or institutions. Peer-reviewing, even with all its imperfections, is still highly effective, has stood the test of time, and should be emphasized. Indeed, peer reviews can evolve in quality if reviewers are increasingly made aware of their seminal importance to Science. Their critical opinions must be central in funding decisions and analyzed jointly with higher quality metrics (such as those that focus on the individual´s output itself and not where it is published) and also on their subjective perception of relevance and originality. Indeed, the scientific community must understand that the subjective perception of a well-selected and specialized peer reviewer is a strong indicator of merit. This subjective perception can be aided by requesting qualitative information from the submitting parties such as their five most important papers (Finkel 2019) or a description of their main scientific findings. Value must be placed also on refining proposals based on concise and constructive feedback comments from these peer reviewers, a task that takes time. We are not advocating for slowing down the already onerous peer-review process (Vale 2015, Powell 2016), but rather for wiser use of the time already required in grant analysis, substituting useless and time-consuming documentation, formatting and indexing for quality revisions. This can be implemented by recognizing peer reviewers for their hard work, such as by creating mechanisms to cite and reward them (Stern & O'Shea 2019).
This strategy is well in line with recent highprofile statements backed by thousands of scientists, editors, journals, scientific societies and funding agencies (San Francisco Declaration On Research Assessment (DORA) 2012, Hicks et al. 2015, The Metric Tide 2015. These documents have in common proposals to boldly shift the way Science is evaluated, emphasizing the advance itself, rather than relying on objective indexes, and also the rational use of responsible metrics associated with subjective analyses. Although sluggish changes started in 2012 as a result of the DORA-Leiden-Tide declarations, they may have been overshadowed by growth in sectors of Impatient Science and Quantophrenia.
One Quantophrenic and Impatient Science aspect of scientific endeavors today relates to the second point we believe lies at the core of a change in scientific assessment and progress today: the choice of publication venues for papers describing research findings. While the DORA-Leiden-Tide intent is to focus on science produced and not where it is communicated, researchers worldwide have increasingly found that publishing their work in specific scientific journals with well-known "brand names" increases the immediate/impatient attention given to their publication, at least within the framework of headlines in social media (Kowaltowski & Amigo 2018). This has strongly encouraged the editorial market to create more such venues, mainly by creating "daughter journals" with the same brand name as a wellrespected and established journal. Indeed, a well-respected journal has launched more than 50 "daughter journals" within its brand (Nature 2021). While a larger number of options in a market generally helps increase quality and decrease pricing, the specific niche characteristics of scientific publishing have produced the exact opposite: these "daughter journals" are usually highly priced and have in-house rather than active scientists on their editorial boards, which makes them more prone to promote market values over quality Science. Most importantly, the revenue from publication proceeds in these journals goes to the publishers, which are mostly private enterprises, unassociated with scientific societies.
To avoid both predatory, low quality, publications and quench the Quantophrenic trend, we urge our colleagues to rethink in which venues they publish their results. A feasible solution to expedite scientific communication is to fully embrace the culture of sharing original research through preprint servers before formal journal publication (Kaiser 2017). However, preprint sharing is not widespread enough in the Life Sciences community yet, and we urgently need to discuss its possibilities and benefits for students and senior researchers. In this sense, publication in a peer-reviewed format in scientific journals is still desirable, especially considering the benefits of peer-revision for manuscript quality.
In the current paradigm, in which the number of publication venues is increasing enormously, choosing where these papers are submitted for appreciation is ever more important. We believe market culture, Impatient Science and Quantophrenia cannot interfere with this decision process and urge researchers to carefully consider the journals they submit their work to, and not rely on JIF and brand names. A recent, very ambitious, plan (cOAlition S 2021) has proposed that the main focus for scientific publications should be to choose open access venues. While we agree limited access to scientific knowledge is a problem, we do not think it is the most pressing obstacle toward increased scientific quality and fear a fast push for open access can stimulate predatory and unfairly priced journals (Kowaltowski & Oliveira 2019). Instead, we believe researchers should rethink who benefits financially from the journals they publish, as well as who are the people shepherding this publication venue. Preference should be given to journals with ties to scientific societies, in which at least part of the publication profits revert to research-related causes. Strong links to societies also warrants steering committees with scientific, rather than market, interests. Finally, we urge our colleagues to give preference to journals in which Associate and Board Editors are active, well recognized scientists. This ensures that they will both steer the journal´s aims well and chose the best possible peer reviewers for each submission. In giving preference to society-backed journals with active scientists as editors, authors can change the current Quantophrenic tide.
A final note should be made regarding the consequences of Impatient Science on the process of knowledge acquisition and its effects on Basic Science, or the act of immersing into the uncertainties of the unknown, seeking to understand it for the sole purpose of understanding. Within the Impatient Science framework, researchers have increasingly been forced to provide evidence of immediate applicability of their ideas, even when seeking public funding for their research. Due to its inherently elemental characteristic, only basic research can bring true innovation to the various fields of Science. Therefore, basic research must be valued, prioritized and protected. To accomplish this, robust, stable, and funded programs to support Basic Science should be implemented and supported by governmental agencies. Scientists must rally and defend basic research openly to ensure this. Furthermore, grant evaluators and peer reviewers should recognize and highlight truly innovative basic science whenever possible.

THE INITIATIVE FOR RESPONSIBLE SCIENTIFIC ASSESSMENT (IRSA):
The scientific community is by nature creative. As such, it is time for this group of creative persons to search, together, for new solutions and new directions when appraising Science. The pioneering DORA-Leiden-Tide documents requesting changes in scientific output evaluation need to be expanded upon to guarantee the full potential of Science´s undeniable ability to promote development. These expansions involve modifications in both how scientists disseminate and promote their findings and, centrally, how scientific output and projects are analyzed. Most of all, this requires a change of attitude by researchers for the benefit of scientific endeavor. Within this framework, the synergistic engagement of society, policymakers, and researchers through scientific and innovation processes, as stated in the Responsible Research and Innovation movement (Responsible research and innovation (RRI) 2021), represents a valuable proposal to match scientific research and outputs with the society needs.
Overall, to change the current situation and allow research to reach its full contemporary potential, we propose a scientist-based movement to improve scientific assessment, the Initiative for Responsible Scientific Assessment (IRSA). This movement combines and expands upon previous initiatives, with the following guidelines: 1) Endorse and spread DORA, the Leiden Manifesto, Responsible Research and Innovation movement, and the Metric Tide. Stimulate quality assessment over quantifiable metrics. Follow their guidelines and discuss these ideals within your academic community, scientific societies and funding agencies. 2) Elevate the value of quality peer review.
Provide the best and most balanced assessment possible when acting as a reviewer, focusing on improving the quality of the paper and Science in general. Make reviewer suggestions for your paper submissions mindfully, prioritizing critical specialists. Denounce questionable peerreview practices or evidence of predatory publishing actions. Develop and support policies to reward and qualify peer-reviews (including in available platforms; Publons 2021). Consider and discuss the advantages and disadvantages of open peer review.

3) Recognize seminal findings by researchers.
Cite primary literature, avoiding reviews as references. Recognize and give credit to scientists that made original discoveries. Value central scientific discoveries in funding and hiring decisions. 4) Promote actions to associate quality assessment with more representative scientometric metrics in academic decisions. Assess work done and scientific findings above the impact factor of the journal it is published in. Discuss the flaws of metrics in evaluation processes as well as new ways to improve how Science is qualified with students and colleagues. When using metrics in association with quality assessment, give preference to metrics that focus on the work produced by the individual, and not those that focus on journal metrics. 5) Submit manuscripts to journals with editors who are active scientists with recognized reputation in the field. Choose journals considering their track record and the qualifications of academic editors and the editorial board, which will ensure quality peer review. Do not support predatory or open access journals that prioritize profits over knowledge. Consider which entities the funds from page charges benefit and give preference to scientific societies over unassociated editors. Debate fair open access costs. Stimulate affordable open access by using pre-print depository platforms.
An online petition to endorse IRSA can be found at https://www.petitions.net/ the_initiative_for_responsible_scientific_ assessment_irsa.