The Journal Impact Factor (JIF) is an index annually published by The Journal Citation Reports (JCR). It was established by Eugene Garfield to compare investigators’ and journals’ research influence on its time (1). The JIF for a particular journal is calculated by the ratio of total number of citations received in a determined year to the total number of citable articles published by that journal in the previous 2 years (Figure-1) (2, 3). The non-citable articles criteria include editorials, news, meeting reports, etc. Original research and review articles are the only article types that meet the definition of citable articles according to the Institute of Scientific Information (3, 4). Web of Science (Thomson Reuters Inc.) is a citation service accessible through an indexing database and search engine, ISI Web of KnowledgeSM (WoK) (5–9).
The numerator of the JIF formula includes the citations of all articles of a given journal whereas its denominator excludes the number of non-citable articles published; thus making the formula potentially manageable. While this flaw has been regarded as insignificant by the index creator, it has also generated significant debate and skepticism among editors and authors who disagree. The creator himself has accepted the issue yet considering it “statistically significant only in rare cases” (2, 10–14). The exclusion of non-citable articles from the denominator can hypothetically improve the JIF by increasing the number of self-citations, non-citable articles or reducing the number of citable articles (15–17). Regardless of this situation, the JIF is held as the gold standard measure to judge journals’ quality with a general lack of awareness of its formula, purpose, and meaning. All of which is now coming under greater scrutiny (18–21, 14). This debate is not merely academic as many universities are now using “publishing in high JIF journals” as an evaluation criterion for promotion. Publication in high JIF journals also affects success in competing for extramural grant funding (22).
We sought to examine the JIF among nephrology and urology journals in order to elucidate the main predictors of the index while generating awareness among scientific community about the current formula used to calculate the JIF and its flaws. We also attempt to show that small discrepancies in the formula do influence the overall calculation and the subsequent ranking list.
MATERIALS AND METHODS
The top 40 journals ranked by JIF in the Journal Citation Reports for 2011 in the urology and nephrology category were selected for the study. We excluded journals that were not published in English and published articles of one specific kind of research design (basic science, retrospective studies, review articles). Twenty-three journals out of the 40 journals initially selected were excluded for containing only review (16) and basic research (7) publications.
All of the archived abstracts from each journal's website for years 2009-2010 were individually reviewed. This is the period used for JIF calculation that is reported for 2011. We categorized each of the articles according to research design (basic science, non-clinical experimental and translational research, clinical trial, retrospective study, prospective study, case report and case series, cross-sectional study, review article, meta-analysis, systematic review, and guideline). If the study design could not be ascertained from the abstract, the manuscript's methods section was read in order to obtain a more precise classification. Appendix 1 outlines the articles’ sorting criteria for research design (Appendix 1). The number of self-citations, citable and non-citable articles, and citations made to citable articles, and citations made to non-citable articles were also identified (15, 16). Other variables examined included society affiliation, US vs. non-US journals, number of articles published by the journal, average pages per issue, and US and non-US articles.
ISI Web of KnowledgeSM Citations Data Validation
Data values for JIF, self-citations, citable and non-citable articles published during 2009 and 2010, as well as the total number of citations used to calculate the 2011 JIF were retrieved and examined from the JCR web site (7, 8). The JCR website, however, did not individually disclose the citations made to citable and non-citable articles that were used to calculate the 2011 JIF. As an alternative, two other databases (Web of Science and MEDLINE) from the WoK website were used (5, 6, 9). The data for US and non-US articles was also acquired through the WoK website (6, 9).
Because the sum of citations to citable and non-citable articles (total number of citations) retrieved from the WoK website did not directly match the total number of citations acquired from the JCR website, we sought to validate the WOK website data by generating an alternative JIF utilizing the data extracted from the WOK website. The goal for this intermediate step was to accurately use the WOK website data a posteriori during our statistical analysis in lieu of the JCR website data (not individually disclosed).
The alternative JIF was calculated using the citations data acquired from the WoK website. This JIF was correlated with the official JIF published by JCR through the Spearman's rank-order correlation test. A correlation with a p-value less than 0.05 and a slope close to 1 were defined as the required criteria for validation.
The number of articles published during 2009 and 2010 issues published per year, average pages per issue, self-citations, issues published per year, average articles published per issue, citations made to citable articles, citations made to non-citable articles, and the categorical variables were analyzed as independent variables. All other continuous variables were analyzed as a ratio relative to the total number of publications in the period of interest because they depended on the total number of articles published.
The association between the variables and JIF was examined using the JIF as the dependent variable. A two tailed Welch's t-test and Spearman's rank-order correlation test were performed for categorical and continuous variables respectively. All variables were then included in a multivariate linear regression model using a stepwise variable selection method. An alternative multivariate model excluding the counts for citable articles was analyzed in order to account for co-linearity of other variables with the variable “citable articles”.
A hypothetical JIF (JIF’) was created with the data obtained from the JCR website, one that includes the non-citable articles in the denominator of the index calculation (Citations made to Citable and Non-citable articles/Citable and Non-citable articles). A Welch's t-test was also used to compare the JIF published by the JCR with the hypothetically created JIF. All computations were performed using SAS 9.3 (SAS Institute, Cary, NC). P-values of <0.05 were considered to be statistically significant. All the assumptions were verified for every test.
A total of 12/17 journals were society affiliated, 8 were US originated, and 9 were non-US originated. Table-1 summarizes all the variables used in the analysis (Table-1). A total of 23,012 article abstracts were examined with a median of 1,048 (range=78–6,342) articles per journal.
|Dependent Variable||Total Number||Median (Range)||–||–|
|Journal Impact Factor||17||3.22 (2.10-9.66)||–||–|
|Categorical Variables||Total Number||JIF Mean (CI-95%)||t -value (df) −0.18 (15)||p -value|
|US Journals||9||4.08 (2.01-6.16)||0.86 (15)||0.86|
|Non-US Journals||8||3.88 (2.20-5.55)||–||–|
|Society Affiliated Journals||12||4.46 (2.86-6.07)||-2.14 (13)||0.05*|
|Non-society Affiliated Journals||5||2.80 (2.06-3.54)||–||–|
|Continuous Variables||Total Number||Median (Range)||Rho (R2)||p -value|
|*Average Pages Per Issue||179.3||154.9 (31.54-424.58)||0.36 (0.13)||0.16|
|Total Number of Publications||23012||1048 (78-6342)||0.30 (0.09)||0.25|
|US Articles||7961||357 (20-1712)||0.46 (0.22)||0.06|
|Non-US Articles||15051||535 (58-4630)||-0.46 (0.22)||0.06|
|Citable Articles||9598||488 (31-1437)||-0.29 (0.08)||0.26|
|Citations to Citable Articles||36859||2497 (58-4630)||0.73 (0.53)||0.001*|
|Non-citable Articles||13595||467 (2-1437)||0.29 (0.08)||0.26|
|Citations to Non-citable Articles||1790||87 (0-234)||0.65 (0.42)||0.0046*|
|Journal Self-Citations||5255||240 (0-1339)||0.57 (0.32)||0.02*|
|Retrospective||2616||54 (14-613)||-0.51 (0.26)||0.03*|
|Clinical Trials||621||18 (6-108)||-0.01 (0.00)||0.96|
|Basic Science Research||2776||128 (5-287)||-0.05 (0.00)||0.84|
|Prospective||1659||64 (8-337)||-0.44 (0.20)||0.07|
|Case Report/Case Series||683||9 (2-199)||-0.47 (0.22)||0.05*|
|Cross-Sectional||313||7 (1-127)||-0.05 (0.00)||0.83|
|Review Articles||824||42 (2-117)||-0.03 (0.00)||0.89|
|Meta-analysis||53||1 (0-13)||0.03 (0.00)||0.92|
|Systematic Review||41||1 (0-10)||-0.18 (0.03)||0.50|
|Guidelines||45||0 (0-25)||0.23 (0.05)||0.37|
ISI Web of KnowledgeSM Citations Data Validation
While utilizing the WOK website data 11 journals increased their JIF, 5 decreased their JIF, and 1 was unaffected. The Journal of Urologic Oncology: Seminars and Original Investigations, and The Journal Neurourology and Urodynamics had an important citations count miss-match between databases, which resulted in a JIF decrement of 0.5 points or more. The JIF calculated using the WoK website data correlated well with the JIF calculated using the JCR website data (rho=0.91, p<0.001, m=0.96) (Figure-2). Once validated, the data gathered from the WoK website was used to calculate a hypothetical JIF excluding citations accounted to non-citable articles (Table-2). This exclusion represented a consistent drop in the JIF value throughout the 17 journals, promoting at the same time ranking position variations.
|JIF-JCR||JIF-WOK*||JIF'||NCA||CA||Total Cites-WOKΔ||Total Cites-JCR|
|J Am Soc Nephrol||9,66||9,71||7,49||143||493||4785||4764|
|Am J Kidney Dis||5,43||5,57||2,02||824||488||2718||2652|
|J Sex Med||3,55||3,58||1,24||1413||756||2707||2685|
|Nephrol Dial Transplant||3,40||3,46||2,77||272||1193||4126||4051|
|Urol Oncol-Semin Ori||3,22||2,20||2,61||46||199||438||640|
|Am J Nephrol||2,54||2,51||2,35||23||282||708||716|
|Prostate Cancer P D||2,42||2,39||2,21||11||114||272||276|
|Periton Dialysis Int||2,10||2,19||1,36||112||207||467||434|
JIF-JCR mean = 3.97 vs. JIF' mean = 2.34, t -Test (29) = 2.41, p -value= 0.02
JIF: Journal Impact Factor, WOK: Web of KnowledgeSM, CA: Citable Articles, NCA: Non-Citable Articles
*JIF using data extracted from Web of Knowledge.
JIF' calculated including Non-citable articles in the denominator; Formula: JIF=Total Cites-JCR/(NCA+CA)
ΔCalculated by adding up Citations to CA and NCA.
Journals with a society affiliation were associated with a higher JIF compared to journals without a society affiliation (mean JIF=4.46 vs. 2.80, p=0.05). US journal origin vs. non-US journal origin did not demonstrate an association with JIF (mean JIF=4.08 vs. 3.88, p=0.86) (Table-1). A significant difference was found while comparing the JIF published by the JCR with the hypothetical JIF including the non-citable articles in the denominator of the calculation (mean JIF=3.97 vs. mean JIF'=2.34, p=0.02) (Table-2).
The bivariate analysis also found some of the examined variables to be significantly associated with JIF: self-citations (rho=0.57, p=0.02), retrospective studies (rho=-0.51, p=0.03), citations to citable articles (rho=0.73, p=0.001), citations to non-citable articles (rho=0.65, p=0.0046), and case reports/case series (rho=-0.47, p=0.05) (Table-1). The remaining variables examined were not found to be good predictors of JIF.
The multivariate linear regression analysis demonstrated that citations made to citable articles (p<0.001) are the most important predictor of a high JIF, whereas the number of citable articles (p=0.015), total number of articles published (p<0.001) and case reports/case series (p=0.002) are the most important predictors of low JIF (Table-3). Appendix 2 summarizes the multivariate model excluding the variable citable articles.
|Multivariate Model||Estimate||Standard Error||Adjusted Estimate (t )*||p -value|
|Total Articles published||-0.0014||0.00022||-6.36||<0.001|
|Case Reports/Case Series||-28.03||7.3||-3.84||0.002|
|Citations to Citable Articles||0.0015||0.00016||9.38||<0.001|
*Adjusted estimate by variable scale (normalized estimate: Estimate/Standard Error).Model R2 = 0.9
The findings of this study emphasize the need to modify the JIF formula in order to make it more accurate and acceptable. No prior publications have focused on this topic in Urology, nor has a detailed statistical analysis been performed to weight the different variables that are included in the JIF. Variables such as citations to citable and non-citable articles, self-citations, and low number of citable articles published have been hypothesized as predictors of high JIF (15, 16, 17). These assumptions were held as true during this analysis and confirm them.
Regardless of having readily available counts of total number of citations the JCR website does not routinely disclose the individual counts for citations made to citable and non-citable articles, and such numbers can only be obtained indirectly through the WoK website (7, 8). The data acquired from the WoK and JCR websites were compared in order to assess the feasibility of the analysis. Regardless of being under the same umbrella, differences between the two datasets were found. Due to these discrepancies, we sought to validate the citations data retrieved from the WoK website against the data published in the JCR website. With the proper computing, a strong validation was indeed achieved with excellent similarity between the two data-sets (rho=0.91, slope=0.96) (Figure-2).
The validation process enabled us to prove that citations to non-citable articles do signifcantly improve the JIF (rho=0.65, p=0.0046) (15, 16). Also, a high proportion of retrospective studies and case reports/case series had a negative effect over the JIF. With few exceptions retrospective studies and case reports/case series are regarded as a resource of low evidence level, hence tend to have a low impact in the scientific community and do not receive as many citations as designs regarded as higher level of evidence (i.e. meta-analyses) (23). Therefore the inclusion of non-citable articles and exclusion of case reports/case series and retrospective studies may serve as positive predictors of JIF. Low citable article counts and high self-citation rates also appear to predict the JIF (Table-1). The latter echoes the result of a previous analysis including 6 anesthesiology journals (17).
Citations made to citable articles were the most important variable influencing the JIF. Articles reporting the results of large clinical trials, basic science research, cross-sectional studies, reviews, systematic reviews, and meta-analysis did not demonstrate a significant correlation with the JIF nor were they independent predictors of JIF when controlling for all other factors. This could be due to the fact that research design variables are included under the umbrella of citable articles making them collinear variables. However, a sensitivity analysis was performed in order to control for co-linearity among the research design variables by eliminating the citable articles variable. The results were not modified (Appendix 2).
Because citable articles are fundamental for the JIF calculation, such model was performed only to account for co-linearity and does not provide any other useful insight.
The journals with highest JIF in our cohort grossly were those that accepted the least number of retrospective studies and case reports/case series. The concern with this fact is that journal editors could place greater emphasis on the “citability” of an article rather than the quality and nature of the scientific question. Much of the literature in Urology focuses on certain subspecialties such as oncology, which makes these articles potentially more citable and getting higher priority for publication. The citability of an article could also be linked to the topical nature of the article or in other words, its conformation to the existing trend. In example, multiple articles on certain popular topics such as robotic surgery could be published and crowd out excellent work in basic science and less popular areas.
One other interesting finding was that the JIF of journals linked to professional societies had a higher value. This may represent the fact that members of the society are more likely to be aware of and cite articles from the journal of their society. This may be an unrecognized value to the journals and to the societies since publishing in higher JIF journals is becoming increasingly important to academicians and increasing JIF is important to journals.
When a hypothetical JIF including the non-citable articles in the denominator was created, a significant difference was found when compared to the JIF published by the JCR (p=0.02). This analysis contrasts with the belief of finding “significant discrepancies only in rare cases” (2). Moreover, this hypothetical modification of the JIF generated multiple rank changes among the journals. This issue builds a compelling case regarding the need to modify the formula to calculate the JIF since it is regarded as gold standard of journal quality and often looked at as a criterion for promotion or extramural grant funding (Table-2) (18-22).
The limitations of this study include the fact that we examined the correlation of articles to JIF over a two-year snapshot in time. Factors involved and associations could change over time and these data do not have the ability to capture that. We identified and analyzed a set of journals in a non-randomized manner. Hence we are making the assumption that these findings hold true for journals that are not the top 40 in the Urology/Nephrology category. These findings also apply only to English language journals. The biggest limiting factor lies on the lack of access to citations counts for each research design, which is essential to assess the influence of research designs over the JIF prediction. Despite these shortcomings, this is the first study to perform a detailed analysis of JIF among urology journals. These data can provide us with some understanding of what predicts JIF.
Variations to the formula to account for non-citable articles in the denominator do represent a statistical significant difference when compared to the JIF as it is currently calculated. A change in the formula for calculating JIF to include all types of published articles in the denominator would result in a more accurate and accepted representation. The JIF appears to be predicted by the number of citations made to citable articles and non-citable articles, self-citations, case reports/case series, retrospective studies and citable articles published. Number of citations made to each different research design is necessary in order to assess their influence on the JIF.