Relationships of field-normalized and non-normalized scientometrics with the Physical Education Web-Qualis grades

abstract This study aimed (1) to verify the relationship between the JIF, Eigenfactor, Citescore and SJR metrics of Physical Education journals throughout Qualis 2013-2016; and (2) to analyze their associations with the field-concerned Web-Qualis grades (based on the 2013-2016 quadrennium criteria). WebQualis grades, which refers to 2015-2018 data, were acquired by accessing officially available data. The scientometrics Journal Impact Factor (JIF), Citescore, Eigenfactor, and Scimago Journal Rank (SJR) for the years 2013 to 2016 and 2018 were extracted from official Clarivate Analytics and Elsevier sources. Approximately half of Physical Education journals included in the 2018 WebQualis database were indexed in Web of Science Collection and Scopus databases. The main results demonstrated high and stable appeared bivariate correlations between all scientometrics from 2013 to 2016. Both JIF, Eigenfactor, Citescore and SJR were associated with WebQualis grades. However, better adjustment of parameters was observed in the model that included SJR as the WebQualis predictor. The field normalized SJR appeared to better predict the WebQualis defined by the 2013-2016 Qualis criteria, which includes the adherence indicator. If researchers consider that distinct research areas are not equally weighted and require specific evaluation, as the authors suggest, use the normalized metrics to classify journals in Brazilian scenarium.


INTRODUCTION
Quality metrics in science are being increasingly used by the need to measure and analyze journal-level knowledge production and research numbers 1 .The criteria used to meet these metrics need to be accepted by the scientific community in each area of knowledge, being called bibliometrics 1,2 .Bibliometrics is a multidisciplinary field whose main focus is to analyze scientific production in a given context, such as in a country, an institution, a research area or a researcher 3 .In a special case, bibliometric indicators promoted by bibliometrics are used by universities to evaluate graduate programs, as in the Brazilian scenario 4 .Nowadays, the most commonly applied bibliometric indicators include the Journal Impact Factor ( JIF), the Eigenfactor, Citescore, and the Scimago Journal Rank (SJR) 5 .
In the Brazilian scenario, the quality metric criteria for the use of bibliometric indicators are organized by the Coordination for the Improvement of Higher Education Personnel (CAPES).According to CAPES, Brazilian scientific production is evaluated by a system called Qualis 6 .The Qualis system is a set of procedures used by CAPES to stratify the quality of intellectual production (i.e.scientific journals articles) of graduate programs at grade levels (e.g., A1, A2, B1, B2, B3, B4, B5, and C) [7][8][9] .It is important to highlight that the previous Physical Education Qualis system (2013-2016 quadrennium criteria) used of a single citation-based metric, the JIF.In this way, a previous concern emerges with the following question: What would be the best indicator to minimize the differences between areas?Field normalized metrics were proposed by both Scopus (SJR) and Clarivate Analytics (Eigenfactor) to advance on evaluating scientific journals.
Therefore, this study aimed (1) to verify the relationship between the JIF, Eigenfactor, Citescore and SJR metrics of Physical Education journals throughout Qualis 2013-2016; and (2) to analyze their associations with the field-concerned 2013-2016 Physical Education WebQualis.

Data sample
The previous Physical Education WebQualis grades, defined for the 2013-2016 quadrennium, were acquired by accessing officiall CAPES repository 10 .Scientometrics data (i.e., JIF, Citescore, Eigenfactor, and SJR) for the years 2013 to 2016 and 2018 were extracted from official Clarivate Analytics 11 and Elsevier sources 12 .These specific years were chosen to evaluate the data corresponding to the 2013-2016 Qualis quadrennium criteria and the most up-to-date WebQualis grades so far.
WebQualis grades data were treated by cleaning journal title and International Standard Serial Number (ISSN) entries (e.g., excessive spaces and grammatical variations).Thus, data from WebQualis grades and scientometrics were merged by matching journal title and ISSN.Duplicate journal entries (e.g., online and printed versions) were counted and merged to avoid bias.The final dataset included all journals reported in the Physical Education WebQualis and their respective WebQualis grade and scientometrics.

Statistical analysis
The absolute and relative frequencies were used to describe the data.The scientific metrics (i.e., JIF, Citescore, Eigenfactor, and SJR) were standardized to present an intuitive interpretation of findings (mean equal to zero and standard deviation [SD] equal to one).
The Spearman correlation coefficient was used to analyze each bivariate correlation between scientometrics according to evaluation year (i.e.; 2013 to 2016).The Spearman coefficient was chosen due to the known skewed distribution of outcomes.The magnitude of correlation coefficients was interpreted as moderate (0.50 to 0.70), high (0.70 to 0.90) and very high (0.90 to 1.00).This procedure included journals with complete data among all evaluation years.Thus, journals without information on at least one metric from 2013 to 2016 were excluded.
Multinominal logistic regression models were applied to evaluate the effect of each metric (i.e., JIF, Citescore, Eigenfactor, and SJR) on WebQualis grades.
McFadden's pseudo-R-squared and Akaike Information Criterion (AIC) were computed for model comparisons.Higher values of pseudo-R-squared and lower values of AIC indicates better model fit/prediction when comparing models.The model referring to the metric which better predicted the WebQualis grades was described by presenting the predicted probabilities of a journal being allocated to a grade (B4-A1).This procedure included journals with complete data for the reference year of 2018.Analyses were performed using the Stata SE, version 15 (StataCorp.College Station, TX: StataCorp LLC).

RESULTS
From a total of 2,219 journals listed in the 2013-2016 Physical Education WebQualis, 250 duplicates were identified, and 1,969 journals were included for analysis.The proportion of journals indexed on the Web of Science Collection and Scopus databases in 2018 was 44.2% and 54.5%, respectively.Only 42.6% of journals were indexed on both databases.
The magnitude of correlations for the relationships between JIF, Citescore, and SJR ranged from high to very high (r>0.85,p-value<0.001).High correlations were also observed for all metrics with Eigenfactor score (r≈0.75, p-value<0.001).The bivariate correlations between all scientometrics appeared to be stable from 2013 to 2016 (Figure 1).

4/8
Kons et al.Field normalized Scientometrics apply on Physical Education Webqualis JIF, Citescore, Eigenfactor, and SJR rankings were correlated with WebQualis grades in 2018 (Figure 2).A higher correlation coefficient was observed for the Scopus SJR metric (r=0.50)when compared to JIF (r=0.39),Citescore (r=0.40) and Eigenfactor (r=0.30).It is interesting to note that from 12 journals allocated to B3 WebQualis grade, three ranked over the 90 th sample percentile on all scientometrics (e.g., JIF>5.0).Four multinomial logistic regression models, one for each scientometric, were applied to further explore the associations between standardized metrics and the probability of being allocated on distinct WebQualis grades (Table 1).Better goodness of fit parameters was observed in the model that included Scopus SJR as the WebQualis predictor (higher McFadden's R 2 and lower AIC).However, the pattern of classification appeared to be similar among all predictors (see Supplementary Material).The probabilities of WebQualis classification according to the increase of Scopus SJR (z-score) are presented in Figure 3.The increase of SJR was associated with higher probabilities of a journal being allocated to higher WebQualis grades.The probability of B1 categorization increased until reaching 0.4 SD below the mean SJR, and then decreased.A similar pattern was observed for the allocation to A2 grade, with a probability increase until reaching 0.2 SD above the mean SJR.Higher probabilities of being a journal being allocated to the A1 grade were observed with the increase of the SJR score.

DISCUSSION
This study aimed to evaluate the magnitude of the correlations between scientometrics during the period of evaluation of the WebQuails system (2013 to 2016) and highlight the importance of considering normalized metrics for classifications in the Brazilian scenario.It is important to state that the 2013-2016 Qualis system applied an indiscriminate use of a single citation-based metric, the JIF, which scarcely represents scientific reality 6,13 .Impact factors are influenced by many aspects that are far from assessing the quality of production, such as the size of the scientific community of knowledge fields, the prestige of their subareas in the same field, the average number of authors per article, the language of publication and the country of origin of the authors 4,14 .For this perspective, field-normalized scientometrics, such as SJR and Eigenfactor, should be considered as they may improve evaluation criteria for the Brazilian system.
High to very high correlations were observed for between JIF, Citescore, SJR, and Eigenfactor over the years of 2013 to 2016.Recently, Waris et al. 5 considering journals of sports science verified high correlations between JIF and SJR (r= 0.90) and between JIF and Eigenfactor (r= 0.84).The authors advocate that SJR may be considered as an alternative to the JIF for sports science journals since the system is free and easily accessible for submission of manuscripts.Similar results were found considering other scientific areas, for example specific respiratory medicine 15 and orthopedic journals 16 .
We observed better goodness of fit parameters in the predictive model which included SJR as the WebQualis predictor, which was in line with our hypothesis.
It is worth mentioning that than some metrics (i.e.SJR and Eigenfactor) are field-normalized, and thus, allow the comparison between scientific areas.The SJR algorithm takes into account field differences, which are included in the 2013-2016 Qualis criteria by applying the field adherence criteria.However, small differences were observed between the effect of all the predictors and the findings should be interpreted with caution.
It is worth highlighting the main limitation of this study was that our analysis considered only the quadrienium from 2013 to 2016, as the evaluation of journals in the national Brazil scenario included the adherence criteria during this period.

CONCLUSION
We concluded that SJR, JIF, Citescore, and Eigenfactor are correlated between them over the years of 2013 to 2016, with strong correlations among SJR, JIF, and Citescore.The SJR was the metric that best discriminates the allocation of Physical Education journals to WebQualis Grades.It is suggested the use of field normalized scientometrics over metrics that account for only citations count to evaluate Brazilian scientific journals.

Figure 2 .
Figure 2. Distribution of journal rankings allocated on WebQualis Grades in 2018 according to scientometrics (n=847).Note: Journal rankings are presented in reverse order, with higher values suggesting higher metrics' scores.r: Spearman correlation coefficients.

Figure 3 .
Figure 3. Multinomial logistic regression model applied to evaluate the association between SJR z-scores and WebQualis Grades (n=847).

Table 1 .
Goodness of fit parameters of polynomial logistic regression models to predict 2018 WebQualis Grade (n=847).