Factors influencing citations to systematic reviews in skin diseases: a cross-sectional study through Web of Sciences and Scopus^*

Abstract

BACKGROUND:

Disseminating information derived from systematic reviews is a fundamental step for translating evidence into practice.

OBJECTIVE:

To determine which features of dermatological SR are associated with systematic review dissemination, using citation rates as an indicator.

METHODS:

Dermatological systematic reviews published between 2008 and 2012 were obtained from Scopus, the ISI Web of Sciences and the Cochrane Skin Group. Bibliometric data of every systematic review were collected and analyzed.

RESULTS:

A total of 320 systematic reviews were analyzed. Univariable analysis showed that the journal impact factor, number of authors, and total references cited were positively associated with the number of citations. There was a significant difference in the median number of citations with regard to the corresponding author's country, type of skin disease, type of funding, and presence of international collaboration. Cochrane reviews were significantly associated with a lower number of citations. Multivariable analysis found that the number of authors, number of references cited and the corresponding author from United Kingdom were independently correlated with many citations. Cochrane systematic reviews tended to be independently associated with a lower number of citations.

CONCLUSIONS:

Citation number to systematic reviews may be improving by increasing the number of authors, especially collaborative authors, and the number of cited references. The reasons for the association of Cochrane SRs with fewer citations should be addressed in future studies.

Keywords:
Bibliometric indicators; Evidence-based medicine; Review; Utilization review

INTRODUCTION

Systematic reviews (SR) are considered the highest level of evidence and are massively used by policy makers, physicians and patients. The use of SR and meta-analysis (MA), when available, is crucial to make informed clinical decisions and, ultimately, to improve patient care.

There is a growing interest in undertaking SR by researchers of all medical specialties, though this endeavor requires be rewarding and justification in terms of the number of publications and citation rates. Equally, high citation rates and higher impact factors of publications are fundamental in increasing the chances of obtaining research funding and progressing in academia.

SR citation rates may be used as indirect measures of dissemination of the information they comprise.¹Knowing the factors correlated with a higher citation rate of dermatological SR may help reviewers to increase the chances of being cited. Furthermore, it may help policy makers, physicians and ultimately patients, by ensuring high quality evidence is better-known and disseminated. Hence, knowing what factors are associated with a higher number of citations would enable the elaboration of publication strategies, thereby maximizing citation rates for SR, ultimately helping to translate evidence into practice.

The databases, Web of Science (WOS) and Scopus (SC), are both widely used to obtain and analyze citations. Both require a paid subscription and provide strong coverage of selected peer-reviewed journals, including Cochrane Collaboration (CC)-SR.²While WOS has stricter criteria for journal inclusion, SC includes all Medline-indexed journals.

Previous studies have found that characteristics such as the Journal Impact Factor (JIF), number of authors and international collaboration, are positively correlated with a greater number of citations in General Medicine and Chinese SR.^3,4To our knowledge, there are no published studies on the bibliometric features of dermatological SR associated with the number of citations.

Therefore, our primary objective was to undertake a bibliometric analysis of dermatological SR and to determine which characteristics were associated with SR citations in skin diseases.

METHODS

In December 2013, we searched for dermatologic SRs and citations to them in SC and WOS. We decided to analyze both databases because they are comprehensive, accessible, with up-to-date differences, and include differences in retrieved number of citations.²The search was restricted to SRs published between January 2008 and December 2012, since it took at least a year to accumulate citations for the most recently published SRs. Searches were restricted to records with the words "meta-analysis" or "systematic review" in the title, as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement recommends that SRs should be identifiable via either term in the title.⁵No language restriction was applied. The search strategies used are displayed inchart 1. As CC-SRs are not described as "meta-analysis" or "systematic review" in the title, we performed a separate search of the Cochrane Skin Group (CSG) database in order to find SRs first published in the same time span. Bibliometrics of CSG-SR were obtained from WOS and SC.

Chart 1
Search strategies used for WOS, SC and CSG databases, variables collected for each SR and journal in which each SR was published

Search strategies

Scopus

#1 Title= ((systemat* AND review*) OR (meta-anal*) OR (metaanal*) OR (metanal*))

#2 Affiliation= (derm* OR skin* OR cutan*)

#1 AND #2

WOS

#1 Title= ((systemat* AND review*) OR (meta-anal*) OR (metaanal*) OR (metanal*))

#2 Research Area= dermatology

#3 WOS Category= dermatology

#4 Address= (derm* OR skin* OR cutan*)

#1 AND (#2 OR #3 OR #4)

Variables analyzed for each SR

·Number and countries for all authors and the corresponding author

·Funding (if reported or not, and the origin of funding if reported)

·Type of intervention and skin disease studied

·Presence of meta-analysis

·Mean number of cites per year available for citation in WOS in December 2013 (calculated as 2013 minus the publication year

·Clinical usefulness: SR was catalogued as clinically useful when data provided sufficient evidence to allow reviewers to support recommendations or suggestions in SR discussions or conclusion sections. If authors did not offer any clinica recommendations based on SR data, the review was catalogued as clinically useless

·If the analyzed SR was based on randomized controlled trials (RCT)

·Presence of cooperation (defined as at least one co-author from a different country than the corresponding author's country)

Variables for each journal

·Two-year JIF (Journal Citation Reports Science Edition 2012)

·If the journal was primarily devoted to dermatology (Cochrane Library was considered a non-dermatological journal)

The records and full texts obtained from the three searches (n=801) were gathered for analysis. In order to be included, the review had to meet the Database of Abstracts of Reviews of Effectiveness (DARE) criteria for SR.⁶Article inclusion and data extraction were performed independently by two reviewers, and disagreements were resolved through discussion. The variables collected for each SR and journal in which the SR was published are shown inchart 1.

After excluding duplicates, we obtained a final number of 320 assessed SRs, of which 265 were obtained from both the WOS and SC databases, while 19 were obtained solely through SC and 36 were obtained from the CSG.

Statistical analysis

Data were imported into the SPSS version 22 software (SPSS, Chicago, IL, USA). Since several, categorical variables had a large number of categories (with small numbers of cases resulting in some categories), some of them were combined. All statistical tests were two-tailed with the significance level set at 0.05. As all numerical variables were non-normally distributed, univariable statistical analysis was performed using Mann-Whitney U test, Spearman's correlation and Kruskal-Wallis test. Dunn's test was utilized for post-hoc analysis. Categorical variables were tested using Chi-square test. Multivariable analysis was performed using a general linear model.

RESULTS

Main characteristics of SRs and distribution of citations

The Scopus, WOS and CSG searches revealed 320 SRs published between 2008 and 2012, with a total of 3,879 WOS and 5,184 SC citations, respectively. The number of WOS citations per SR varied from zero to 133 (WOS citations/per year ranged from zero to 46 per SR), and the number of SC citations per SR varied from zero to 195 (the number of Scopus citations per year ranged from zero to 62 per SR).

Citations to the reviews were rather skewed: 10% of the top-cited reviews accounted respectively for 39% and 36% of the total WOS and SC citations, while 10% of the least-cited reviews accounted for 0.1% and 2.64% of the total WOS and SC citations, respectively. Nineteen reviews (6.5%) were not WOS-cited and twenty-seven SRs (8.4%) were not SC-cited. No CSG SRs featured in the 10% of top-cited reviews, while 25% of the 10% of least-cited SRs comprised CSG SRs.

The main characteristics of the SRs analyzed are shown intable 1.

Table 1
Characteristics of systematic reviews (N = 320) published between 2008 and 2012

CSG review (n.%)	36, 11.3%
WOS-indexed journal (n.%)	301, 94.1%
Medline-indexed journal (n.%)	302, 94.4%
Dermatological journal (n.%)	169, 52.8%
SR of RCTs (n.%)	116, 36.1%
Presence of international collaboration (n.%)	87, 27.2%
Presence of meta-analysis (n.%)	156, 48.9%
Country of corresponding author (n. %)
United States	67, 20.9%
China	43, 13.4%
United Kingdom	35, 10.9%
Netherlands	32, 10%
Other countries	143, 44.6%
Type of skin disease (n. %)
Psoriasis	52, 16.3%
Infections and infestatiions	37, 11.6%
Non-melanoma skin cancer	31, 9.7%
Atopic dermatitis	30, 9.4%
Other skin diseases	170, 53.1%
Intervention type (n. %)
Drug therapy (topical and/or systemic)	138, 43.1%
Epidemiology *	50, 15.6%
prevalence studies)
All other interventions	132, 41.2%
Funding (n. %)
Non funded	139, 43.4%
Funding non reported	47, 14.9%
Funded (government, industry, other sources)	134, 41.9%

Univariable analyses: correlation between citations and numerical bibliometric variables

Four of the variables (two-year JIF, five-year JIF-5, number of authors, and total cited references) were correlated (P < 0.05) with the number of citations per year in both databases. The highest correlation (Rho=0.259) was with the two-year JIF on the WOS database and the five-year JIF (Rho=0.295) on SC. The number of pages was slightly correlated with the number of citations per year on SC only (Rho= 0.138).

Univariable analyses: association between citations and categorical bibliometric variables

Country location of corresponding author

There was a significant difference in the median number of citations with regard to the country location of the corresponding authors in both databases (P = 0.000). Dunn's test showed that the following couples were significantly different: China vs. United States (p=0.034), China vs. France (p=0.000), and United Kingdom vs. France (p=0.005) in the WOS database; as well as China vs. United States (p=0.000), and China vs. France (p=0.000) in the SC database (Table 2).

Table 2
Number of citations per year of publication both in WOS and Scopus according to the corresponding author's country, skin disease type, intervention and funding. Data are displayed as median and range. Statistical analysis performed using Kruskal-Wallis test. (1) Drug therapy: topical and/or systemic (2) Epidemiology: aetiology, risk factor, disease impact, prevalence

Country of corresponding author	ISI-citations per year	Scopus-citations per year
United States	4.4 / 0 to 24.5	5.8 / 0 to 29
China	1.75 / 0 to 7	2 / 0 to 9
United Kingdom	2.5 / 0 to 44	4 / 0 to 62
Netherlands	3.5 / 0 to 33.25	5.5 / 0 to 48.8
France	6.75 / 0.25 to 17	8.83 / 0 to 20
Germany	5 / 0 to 46	7.33 / 1 to 50
Australia	4.38 / 0.5 to 22.8	4.25 / 0 to 26.4
Canada	2.8 / 0 to 14	3.7 / 0.5 to 20
All other countries	3.25 / 0 to 45	5 / 0 to 47.5
	p=0.000	p=0.000
Type of skin disease
Psoriasis	6 / 0 to 24.5	8 / 0 to 29
Infections and infestatiions	2 / 0 to 14	3.4 / 0 to 17
Non-melanoma skin cancer	4.1 / 0 to 44	4.83 / 0 to 62
Atopic dermatitis	3.67 / 0 to 23.5	5 / 0 to 29
Adverse drug reactions	4.2 / 0.5 to 14.6	5.5 / 1 to 17
Melanoma	5 / 0 to 46	6.1 / 1 to 50
Ulcers and wounds	2 / 0 to 33.3	2.5 / 0 to 48.8
Connective tissue disordes	3.2 / 0 to 11.8	5 / 1 to 12
All other skin diseases	2.5 / 0 to 45	3 / 0 to 47.5
	p=0.000	p=0.002
Intervention type
Drug therapy 1	4 / 0 to 46	6 / 0 to 50
Epidemiology 2	2.7 / 0 to 30	3.5 / 0 to 37
Laser and phototherapy	3.5 / 0 to 44	4.7 / 0 to 62
Diagnosis	3.3 / 0.2 to 22.8	4.5 / 0.5 to 26.4
Mixes	1.8 / 0 to 8.5	2 / 0 to 20.3
All other interventions	4 / 0 to 33.3	4.5 / 0 to 48.8
	p=0.088	p=0.191
Funding
Non funded SR	3 / 0 to 46	5 / 0 to 62
Funding non reported	2 / 0 to 33.2	3.5 / 0 to 48.8
Government	2.8 / 0 to 45	3.2 / 0 to 47.5
Mixed and universities (no industry)	2 / 0 to 9	3.5 / 0 to 20.3
Industry	6 / 0 to 17	8.7 / 0 to 23
Private foundations	4 / 0 to 22.8	5.3 / 0 to 26.4
	p=0.002	p=0.020

Type of skin disease

A significant difference (P = 0.000 in WOS and P= 0.002 in SC) was found in the median number of citations regarding skin disease type (Table 2). Dunn's test showed that the following couples were significantly different in the WOS database: infections and infestations vs. psoriasis (p=0.006), and Infections and infestations vs. melanoma (0.022). Meanwhile, the only significantly different couple in the SC database was infections and infestations vs. psoriasis (p=0.025).

Intervention type

No differences were found between the median number of citations and intervention type studied (P=0.088 and 0.191 for WOS and Scopus, respectively) (Table 2).

Funding

In both databses, there was a significant difference in the median number of citations between the presence and origin of the funding (P=0.002 and 0.020 for WOS and SC, respectively) (Table 2). Dunn's test demonstrated significant differences in the WOS database concerning the following: non-reported funding vs. industry (p=0.005), mixed and university funding vs. industry (p=0.002), and non-funded vs. industry (p=0.016). As regards the SC database, the only significantly different couple was mixed and university funding vs. industry (p=0.012).

Univariable analyses: other categorical bibliometric variables

The median number of citations per year to reviews which had international collaboration was significantly higher than for those with authors all from the same country, in both databases. Cochrane reviews were significantly correlated with a lower number of citation counts in both databases. SRs published in a WOS-indexed journal, along with the presence of meta-analysis, were not associated with the median number of citations, in both databases (Table 3).

Table 3
Number of citations per year of publication both in WOS and Scopus according to the categorical dichotomous variable analyzed. Data are displayed as median and range. Statistical analysis performed using Mann– Whitney U test

	ISI-citations per year	Scopus-citations per year
	Yes vs No (P)	Yes vs No (P)
Cochrane review	1.8 / 0 to 8.5 vs 3.7 / 0 to 46 (p=0.000)	2.3 / 0 to 20.3 vs 5 / 0 to 62 (P=0.006)
WOS-indexed journal	NA	NA
Medline-indexed journal	NS (p=0.068)	5 / 0 to 6.2 vs 1.5 / 0 to 5.2 (p=0.000)
Dermatological journal	NS (p=0.078)	5.5 / 0 to 62 vs 4 / 0 to 50 (p=0.021)
SR of RCTs	3 / 0 to 24.5 vs 3.7 / 0 to 46 (p=0.032)	NS (p=0.051)
International collaboration	4.5 / 0 to 46 vs 3 / 0 to 44 (p=0.007)	6 / 0 to 50 vs 4 / 0 to 62 (p=0.011)
Presence of meta-analysis	NS (p=0.937)	NS (p=0.810)

Multivariable analysis

We applied a general linear model using the number of citations per year in both databases as a dependant variable. After combining all the assessed bibliometric variables, the number of authors (corresponding authors from the United Kingdom) and the total number of cited references (WOS only) were independently correlated with the number of citations in the databases studied. Cochrane SRs were independently associated with a lower number of citations in both databases (Table 4).

Table 4
General linear model showing bibliometric variables that turned out to be independently associated with citations in multivariate analysis. (*) Non-significant. R2 of the model that included all the studied bibliometric variables

	Coefficients		Coefficients
	F	P	F	P
Number of authors	8.941	0.003	4.594	0.033
Total number of cited references	3.948	0.048	3.385	*0.067
Correspondence author from United Kingdom	4.03	0.046	7.478	0.007
Cochrane SR	5.518	0.02	5.976	0.015
	**R2: 0.299; p=0.000		**R2: 0.296; p=0.000

DISCUSSION

In this study, we examined several bibliometrical features of dermatological SRs and citations to them, one to four years after their publication. Citations to reviews were rather skewed: 10% of the top-cited reviews accounted for more than 35% of all citations in both databases, while 10 % of the least-cited reviews accounted respectively for 0.1% and 2.64 % of all WOS and SC citations. This asymmetrical distribution of citations to SR is similar to that reported in other studies in non-dermatological, medical journals.^7-9

One bibliometrical analysis of citations to original research and review articles from a wide range of WOS-indexed journals found that 12-18% of review articles accounted for 50% of the citations, and that this percentage did not vary markedly across journals of different JIF levels (8). Another bibliometrical study of citations to SRs regarding a wide range of diseases and interventions uncovered a similar trend: 17% of the reviews accounted for 50% of all citations and 1.6% of the reviews were not cited.⁹

A study on three medicine journals with high impact factors found that the following features were correlated with a greater citation number: industry funding, industry-favoring result, clinical category of article, group authorship, journal of publication and sample size.¹⁰Another study noted that a two-year JIP was the strongest determinant of citations in emergency medicine original research. Following adjustment of the JIF, the presence of a control group, subjective newsworthiness score and sample size were the next most important determinants of citations.¹¹All the aforementioned studies assessed original articles of any methodology relating to medical specialties other than dermatology. Our study found that two- and five-year JIFs, the corresponding author's country, type of skin disease studied and funding type were associated with the number of citations. Only when the corresponding author's country was the United Kingdom were results independently associated with the number of citations after multivariate analysis. International collaboration, the number of authors and of references cited in each article published were correlated with a greater number of citations in both databases. This finding also emerged in other non-SR and non-dermatological bibliometrical studies.^12-14Of these variables, only the number of authors turned out to be independently associated with the number of citations after multivariable analysis. A correlation was found between the number of citations and skin disease type. However, no skin diseases were independently associated with higher citation numbers after multivariate analysis. The type of intervention under study was not associated with the number of citations.

There was an association between the number of citations and the state and origin of funding. Specifically, industry-funded SRs had more WOS citations than SRs with non-reported funding, university funding, or non-external funding. However, the state and origin of financial support were not correlated with the number of citations after multivariate analysis.

Surprisingly, no CSG SRs featured among the 10% of top-cited reviews, while 25% of the 10% of least-cited SRs comprised CSG SRs. On the other hand, Cochrane SRs were associated with a lower number of citations in both databases studied. Moreover, Cochrane SRs were found to be independently correlated with a lower number of citations after multivariate analysis. These data are remarkable and unexpected mainly because of the wide access to the Cochrane Library and the fact that Cochrane reviews are of higher methodological quality than others.^15,16The following factors may explain the lower number of citations to Cochrane SRs: possible difficulties in reading Cochrane reviews and their complexity, the idea of many empty Cochrane reviews with no randomized controlled trials, the potentially narrow scope of Cochrane reviews, the supposed ambiguity of final conclusions in Cochrane reviews and the fact that some subjects selected by Cochrane reviews are not catalogued as priorities by clinicians.^17,18

This study's main strength is that it is the first bibliometric analysis of SRs in dermatology, based on a reproducible and non-language limited search strategy, including a wide range of factors possibly associated with citation numbers on the two most-used databases (ISI Web of Sciences and Scopus). Although our objective was not to develop a systematic review, a limitation of this study is that some SRs may have escaped from our search, as it was limited to three databases and to SRs containing the words "systematic review" or "meta-analysis" in the title, in accordance with the PRISMA Statement. This could also have introduced bias by selecting higher quality reviews, as well as publication bias, as no sources of grey literature were searched. Another limitation is that we did not include any analysis of self-cites to SRs, which may have increased the number of citations, especially where the number of authors was high.

Unanswered questions and future research

Other features of SRs may have affected the number of citations not included in this study. It would be interesting to assess whether report and methodological quality of the reviews, open access category of the journal were the SR is published, and number and design of included studies may also predict the number of received citations.

CONCLUSIONS

Citations to dermatological SRs are rather skewed. The WOS and Scopus citations-related bibliometrical factors affecting SRs in dermatology are: two- and five-year JIFs of the journal in which reviews are published, the number of authors, total references cited, country location of the corresponding authors, skin disease type studied, the state and origin of funding, not being a Cochrane review, and the presence of international collaboration.

Independent variables associated with the number of citations include: the number of authors, total number of references cited, and whether or not the corresponding author is from the United Kingdom. Contrary to expectation, Cochrane reviews were independently correlated with a lower citation number.

• Financial Support: None.
• How to cite this article: Manriquez J, Cataldo K, Harz I. Factors influencing citations to systematic reviews in skin diseases: a cross-sectional study through Web of Sciences and Scopus. An Bras Dermatol. 2015;90(5):646.52.
• *
  Work performed at the Department of Dermatology. Pontificia Universidad Catolica de Chile – Santiago, Chile.

REFERENCES

• ¹
  Shen J, Li Y, Clarke M, Du L, Wang L, Zhong D. Visualization of evidence-based medicine domain knowledge: production and citation of Cochrane systematic reviews. J Evid Based Med. 2013;6:34-42.
• ²
  Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA. 2009 ;302:1092-6.
• ³
  Royle P, Kandala NB, Barnard K, Waugh N. Bibliometrics of systematic reviews: analysis of citation rates and journal impact factors. Syst Rev. 2013;2:74.
• ⁴
  Du L, Chen Y, Huang J, Li Y. A citation analysis of systematic review and meta-analysis published in Chinese journals. J Evid Based Med. 2012;5:66-74.
• ⁵
  Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62:1006-12.
• ⁶
  Petticrew M, Song F, Wilson P, Wright K. Quality-assessed reviews of health care interventions and the database of abstracts of reviews of effectiveness (DARE): NHS CRD review, dissemination, and information teams. Int J Technol Assess Health Care. 1999;15:671-8.
• ⁷
  Opthof T, Coronel R, Piper HM. Impact factors: no totum pro parte by skewness of citation. Cardiovasc Res. 2004;61:201-3.
• ⁸
  Falagas ME, Zarkali A, Karageorgopoulos DE, Bardakas V, Mavros MN. The impact of article length on the number of future citations: a bibliometric analysis of general medicine journals. PLoS One. 2013;8:e49476.
• ⁹
  Weale AR, Bailey M, Lear PA. The level of non-citation of articles within a journal as a measure of quality: a comparison to the impact factor. BMC Med Res Methodol. 2004;4:14.
• ¹⁰
  Kulkarni AV, Busse JW, Shams I. Characteristics associated with citation rate of the medical literature. PLoS One. 2007;2:e403.
• ¹¹
  Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA. 2002;287:2847-50.
• ¹²
  Lokker C, McKibbon KA, McKinlay RJ, Wilczynski NL, Haynes RB. Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study. BMJ. 2008;336:655-7.
• ¹³
  Figg WD, Dunn L, Liewehr DJ, Steinberg SM, Thurman PW, Barrett JC, Birkinshaw J. Scientific collaboration results in higher citation rates of published articles. Pharmacotherapy. 2006;;26:759-67.
• ¹⁴
  Evangelou E, Siontis KC, Pfeiffer T, Ioannidis JP. Perceived information gain from randomized trials correlates with publication in high-impact factor journals. J Clin Epidemiol. 2012;65:1274-81.
• ¹⁵
  Badgett R. Why would physicians undervalue reviews by the Cochrane Collaboration? J Clin Epidemiol. 2008;61:419-21.
• ¹⁶
  Delaney A, Bagshaw SM, Ferland A, Laupland K, Manns B, Doig C. The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal. Crit Care Med. 2007;35:589-94.
• ¹⁷
  Greenhalgh T: Outside the box: Why are Cochrane reviews so boring? Br J Gen Pract. 2012;62:371.
• ¹⁸
  Laupacis A, Straus S. Systematic reviews: time to address clinical and policy relevance as well as methodological rigor. Ann Intern Med. 2007;147:273-4.

Publication Dates

• Publication in this collection
  Sep-Oct 2015

History

• Received
  25 Aug 2013
• Accepted
  04 Nov 2014