Psychometric properties of the Liebowitz Social Anxiety Scale in a large cross-cultural Spanish and Portuguese speaking sample

Abstract

Objective:

To examine the psychometric properties of the Liebowitz Social Anxiety Scale-Self Report (LSAS-SR) based on a large sample recruited from 16 Latin American countries, Spain, and Portugal.

Methods:

Two groups of participants were included: a non-clinical sample involving 31,243 community subjects and a clinical sample comprising 529 patients with a diagnosis of social anxiety disorder (SAD). Exploratory factor analysis (EFA), confirmatory factor analysis (CFA) and exploratory structural equation modeling (ESEM) were used in order to determine the psychometric properties of the LSAS-SR.

Results:

EFA identified five factors with eigenvalues greater than 1.00 explaining 50.78% of the cumulative variance. CFA and ESEM supported this 5-factor structure of the LSAS-SR. The factors included: 1) speaking in public; 2) eating/drinking in front of other people; 3) assertive behaviors; 4) working/writing while being observed; and 5) interactions with strangers. Other psychometric properties such as inter-factor correlations, invariance, reliability, and validity of the scale were also found.

Conclusion:

Psychometric data support the internal consistency and convergent validity of the LSAS-SR. It seems to be a valid and reliable measure of global social anxiety for Spanish and Portuguese-speaking countries, although when considering a multidimensional approach (factor-based assessment) it seems to be lacking some relevant social situations that are feared in those countries.

Social anxiety disorder; questionnaires; psychometrics; cross-cultural comparison

Introduction

Social anxiety disorder (SAD) or social phobia refers to a “marked fear or anxiety about one or more social situations in which the individual is exposed to possible scrutiny by others.”¹ SAD is among the most prevalent anxiety disorders, with 12-month prevalence rates of 6.8% in the United States of America.² Nevertheless, little is known about the prevalence of SAD in Spanish and Portuguese-speaking countries. Extant epidemiological surveys suggest a 12-month SAD prevalence rate of 0.60% in Spain, 2.8% in Colombia, 2.60% among Mexican women, and 1.40% among Mexican men.² The prevalence of this condition in Portugal and other Latin American countries remains unknown.

SAD has been associated with substantial functional and occupational impairment, a chronic, unremitting course, an elevated risk of comorbid depression,³ and lower social skills.⁴ Specifically, individuals with SAD are more likely to be single, of a lower socioeconomic status, and undereducated.³ The economic burden of this condition to both society and the individual is also significant.⁵ Because of its high prevalence and societal burden, SAD has attracted increasing attention from researchers in recent decades, leading to innovations in both the treatment and assessment of this condition. Numerous measures have been developed to assess this condition, ranging from semi-structured interviews to self-report inventories.

One of the most commonly used self-report SAD measures internationally is the Liebowitz Social Anxiety Scale (LSAS).⁶ The original 24-item semi-structured interview involves a two-factor model with separate subscales to assess fear and avoidance of situations involving social interaction and performance/observation by others. However, the two-factor model has been shown to provide an inadequate fit of the data, and a self-report version (LSAS-SR) divided into four subscales was proposed instead.⁷ -⁹ This four-dimensional factor structure has been used in many studies.¹⁰,¹¹ Nevertheless, different studies with various populations have detected a different number of factors – three,¹² four,⁸ five,¹³ six,¹⁴ or even eight.¹⁵ These studies usually only factor-analyze the fear or anxiety subscale,⁸,¹²,¹³ because factor analysis of the fear and avoidance subscales produces similar results.¹²,¹⁴ Thus the fear or anxiety subscale is usually the only one analyzed.

Regarding the psychometric properties of the LSAS-SR, the literature has reported good test-retest reliability, adequate internal consistency, and adequate convergent and discriminant validity,¹⁰,¹³ even in versions translated into other languages.¹⁵ -¹⁷ The LSAS-SR has also been used to establish the convergent validity of other self-report measures of social anxiety, such as the Social Phobia Inventory (SPIN),¹⁸ the Social Phobia and Anxiety Inventory (SPAI),¹⁹ the Social Interaction Anxiety Scale (SIAS) and the Social Phobia Scale (SPS),²⁰ and the short forms of the SIAS and the SPS.²¹ Furthermore, it is commonly used to assess treatment outcomes in SAD patients receiving pharmacological treatment⁶,²² or cognitive-behavioral therapy.²³

Although the psychometric properties of the LSAS-SR have been examined in different countries, few studies have been conducted in Spanish or Portuguese-speaking countries.¹⁷,¹⁹,²⁴,²⁵ The aim of this study was to examine the factor structure, invariance, internal consistency, and convergent validity of the LSAS-SR with clinical and non-clinical samples from Spain, Portugal, and 16 Latin American countries.

Methods

Participants

The first group of participants consisted of 31,243 non-clinical individuals (mean age [M] = 25.50 years, standard deviation [SD]= 10.13, range: 16-87 years) from 18 countries (22.68% Mexico, 18.19% Colombia, 12.30% Spain, 9.82% Peru, 7.75% Brazil, 3.84% Argentina, 3.35% Uruguay, 3.06% Venezuela, 1.75% Puerto Rico, 3.30% Portugal, 3.37% Chile, 1.91% Paraguay, 0.82% Costa Rica, 1.44% Honduras, 2.63% Bolivia, 1.92% El Salvador, 0.63% Dominican Republic, and 1.22% Guatemala). The sample included 56.71% women (M = 25.10 years, SD = 9.87) and 43.14% men (M = 26.00 years, SD = 10.43), with 0.15% of participants not reporting their gender. Regarding education and type of occupation, 58.89% were higher education students, 14.99% were workers with a college diploma, 9.58% were secondary education students, 8.69% were workers with no higher education, and 7.22% did not match any of the former categories (e.g., housewife, retired, or unemployed). No data on occupation were available for 0.63% of the participants.

The second group of participants consisted of 529 patients (M age = 31.73 years, SD = 11.96, range = 16-72) from 13 countries (164 Spain, 112 Mexico, 64 Brazil, 59 Argentina, 44 Colombia, 31 Peru, 27 Chile, 11 Portugal, 10 Uruguay, three Venezuela, two Bolivia, one Panama, and one Puerto Rico); there were 337 women (M = 32.61 years, SD = 12.14) and 192 men (M = 30.18 years, SD = 11.49). For inclusion in this group, patients had to meet a primary diagnosis of SAD according to the criteria of the DSM-IV-TR²⁶ or ICD-10.²⁷ Each center conducted its own diagnostic assessment of individual patients. Patients with a DSM-IV-TR or ICD-10 diagnosis of SAD were included in the study even if they had other disorders in addition to SAD (Table 1).

Table 1
Distribution of patients by psychiatric disorder

	n (%)
Psychiatric disorders	Women	Men	Total
SAD	119 (22.49)	81 (15.31)	200 (37.81)
SAD + other anxiety disorder	49 (9.26)	36 (6.80)	85 (16.07)
SAD + mood disorder	104 (19.66)	37 (6.99)	141 (26.65)
SAD + other anxiety disorder + mood disorder	6 (1.13)	6 (1.13)	12 (2.26)
SAD + avoidant personality disorder	1 (0.19)	4 (0.76)	5 (0.94)
SAD + personality disorder (except avoidant)	18 (3.40)	7 (1.32)	25 (4.72)
SAD + other anxiety disorder + personality disorder (except avoidant)	4 (0.76)	1 (0.19)	5 (0.94)
SAD + eating disorder	17 (3.21)	0 (0)	17 (3.21)
SAD + eating disorder + personality disorder (except avoidant)	4 (0.76)	0 (0)	4 (0.76)
SAD + substance use disorder	0 (0)	8 (1.51)	8 (1.51)
SAD + mood disorder + substance use disorder	3 (0.57)	3 (0.57)	6 (1.13)
SAD + one other disorder (not included above)	8 (1.51)	7 (1.32)	15 (2.83)
SAD + two other disorders (not included above)	2 (0.38)	1 (0.19)	3 (0.57)
SAD + three other disorders (not included above)	2 (0.38)	1 (0.19)	3 (0.57)
Total	337 (63.70)	192 (36.30)	529 (100)

• Data presented as n (%).
• SAD = social anxiety disorder.

A second inclusion criterion was a score ≥ 60 on the LSAS-SR.²⁸ It should be noted that although Mennin et al. used a score of 60 as indicative of generalized social anxiety disorder (GSAD) and of 30 as non-GSAD (NSAD), a previous Brazilian study found scores between 52 and 81 for moderate phobia.¹⁷ Furthermore, it has been found that while 21% of a Spanish non-clinical sample scored higher than 60 on the LSAS-SR, this percentage rose to 68% with a cutoff score of 30.²⁹ Given these results, a cutoff score of 60 was considered more appropriate than a score of 30 for the present sample.

Patients were excluded for several reasons (e.g., five or more unanswered items, presence of psychotic disorders, SAD not the primary diagnosis – which in fact was the main reason for exclusion). From a pool of 907 patients diagnosed with SAD, 529 satisfied all the inclusion criteria. Regarding occupation, 25.14% were workers with a college diploma, 20.60% were workers with no college diploma, 17.58% were higher education students from different majors, 7.75% were secondary education students, 1.89% were higher education psychology students, 0.76% were psychologists, and 23.63% did not match any of the former categories (e.g., housewife, retired or unemployed). No data on occupational status were obtained for the remaining 14 patients.

Measures

Liebowitz Social Anxiety Scale (LSAS)⁶

The LSAS is a 24-item interviewer-rated instrument that assesses fear/anxiety and avoidance of specific social situations. Each of the 24 items serves to assess both variables. Respondents are asked to rate their fear/anxiety (LSAS-anxiety subscale) on a four-point scale ranging from 0 (none) to 3 (severe), and avoidance (LSAS-avoidance subscale) on a four-point scale ranging from 0 (never) to 3 (usually). The total score is obtained by adding the scores obtained on both subscales. The LSAS has also been used as a self-report instrument (LSAS-SR) in the literature with these same characteristics.¹⁰,¹³ With regard to the psychometric characteristics of the Spanish version, one study reported a four-factor structure²⁵ and another study reported five-factors.¹⁹ A Portuguese version of the scale showed a five-factor structure.¹⁷ Reported reliability indices are adequate.¹⁹,²⁴,²⁵ In this way, these last three studies have found scores for internal consistency (Cronbach’s alpha) of 0.93, 0,83, and 0,87, respectively, for the LSAS-anxiety subscale. Split-half reliability coefficient (Guttman) found has been 0.90,¹⁹ and test-retest reliability 0.89.²⁵

Social Anxiety Questionnaire for Adults (SAQ)²⁹-³¹

The SAQ is a 30-item self-reported questionnaire which was empirically developed in Spanish and Portuguese speaking countries to assesses five dimensions of social anxiety: 1) speaking in public/talking with people in authority; 2) interactions with the opposite sex; 3) assertive expression of annoyance, disgust, or displeasure; 4) criticism and embarrassment; and 5) interactions with strangers. Each item is answered on a five-point Likert scale to indicate the level of unease, stress, or nervousness in response to each social situation: 1 = not at all or very slight; 2 = slight; 3 = moderate; 4 = high; and 5 = very high or extremely high. Cronbach’s alpha for the total scale has been shown to range from 0.88 to 0.93,²⁹ -³¹ with split-half reliability coefficients (Guttman) ranging from 0.90 to 0.93.³² Regarding the five dimensions, Cronbach’s alpha ranged from 0.74 to 0.90,²⁹ -³¹ with split-half reliability coefficients (Guttman) ranging from 0.57 to 0.95.³²

Procedure

Our CISO-A Research Team consists of researchers and psychologists from Spain, Portugal, and most Latin American countries. The LSAS-SR and the SAQ were administered to 529 patients with SAD and to 31,243 non-clinical participants from the community. Given that we did not find significant differences among Spain, Portugal, and most Latin American countries regarding assessment of social anxiety,²⁹,³¹ we grouped all these countries together for analysis. For the assessment of the clinical group, our collaborators administered the two questionnaires (LSAS-SR and SAQ) individually to patients. For the non-clinical sample, the questionnaires were administered to groups of subjects. Collaborators working in high schools, colleges, or universities administered the questionnaires to people in classes and meetings of teachers or professors. Those working in companies convened voluntary meetings for workers. No compensation was provided to participants.

The two questionnaires were sent to each collaborator and in order to derive the Portuguese version, both questionnaires were translated and back-translated from Portuguese to English (LSAS-SR) or Spanish (SAQ) until agreement was reached between translators. Both questionnaires were administered together, but the order of administration was random.

Ethical considerations

Participation in the study was voluntary and the questionnaires were filled out anonymously. Informed verbal consent was obtained from all respondents, who were free to withdraw at any time or to refuse to answer the questionnaires. The study and all its procedures were approved by Spain’s Ministry of Science and Technology. This study does not break the agreements of the Helsinki Declaration.

Data analysis

To cross-validate the factor structure of the LSAS-SR, the total sample of 31,243 individuals was randomly split in two halves (n₁ =15,566; n₂ =15,677). There were missing data in 2.19% of the responses to the LSAS-SR. Given the low rate of missing data, and also the absence of evidence incompatible with a missing completely at random structure, pairwise deletion was used to handle the missing data.³³,³⁴

First, we conducted an exploratory analysis with parallel analysis and exploratory factor analysis (EFA) to examine the internal structure of the LSAS-SR. Parallel analysis was implemented with LSAS-SR anxiety subscale data from subsample 1 using the Monte Carlo procedure with 1,000 replications. Parallel analyses compared the eigenvalues extracted from the observed correlation matrix to be analyzed with the eigenvalues obtained from uncorrelated normal variables (parallel components derived from random data). EFA (unweighted least squares with direct oblimin oblique rotation)³⁵ was computed on the first subsample of non-clinical participants. We conducted separate EFAs on the fear/anxiety and the avoidance items. Because we obtained similar results with both subscales and because their distinctiveness has been questioned,⁸,¹⁹ we decided to continue the analysis with only the fear/anxiety subscale. We also conducted an EFA with the clinical sample.

Then, we tested the factor structure of the LSAS-SR anxiety subscale unveiled by EFA results using confirmatory factor analysis (CFA)³⁶ and exploratory structural equation modeling (ESEM).³⁷ ESEM models have been recently developed to address a common limitation of CFA models, which often produce overly restrictive measurement models that do not provide acceptable goodness of fit for most psychological instruments.³⁸ The ESEM model is a special case of CFA in which the assumption that the cross-loadings are 0 is relaxed, so that both models can be considered nested and their fit comparable.³⁷ The CFA and ESEM were completed with the second non-clinical subsample of participants (n₂ =15,677) using weighted least squares with adjusted means and variances (WLSMV) estimation. Four models were tested: M1 = unifactorial model; M2 = two correlated factors model (social interaction and performance situations of the original model); M3 = four-correlated factors model (social interaction, public speaking, observation by others, and eating and drinking in public of the Safren’s model); M4 = five-correlated factors model; and M5 = ESEM. These same five models were also examined in the clinical sample.

To appraise overall model fit, a number of fit indices were examined, including the root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker-Lewis index (TLI). Values of the RMSEA exceeding 0.07 indicate poor fit, whereas values ≥ 0.90 would indicate acceptable fit for the CFI and TLI.

Subsequently, we estimated the correlations among the factors of the LSAS-SR, internal consistency (Cronbach’s alpha), and the convergent validity of the instrument with another self-report measure of social anxiety (i.e., the SAQ) for the non-clinical and clinical samples separately.

In the case of invariance by country, and given the large number of groups involved, Multiple Indicators Multiple Causes models (MIMIC)³⁹ were used, according to the procedure described in a former study.⁴⁰ In a MIMIC model, the grouping variable (previously dummy-coded) acts as a predictor of either the latent or the observed variables. Three nested MIMIC models were estimated:

1. A null model (M10) in which the paths between the grouping variable and the other terms of the model were set to 0 (zero); M10 reflects the null hypothesis that the country of origin has no effect either on thresholds (i.e., scalar invariance holds) or on latent variables (i.e., there is no country-dependent differences in LSAS-SR scores).

2. An invariant model (M11) in which the paths leading from the grouping variable to the factors were freely estimated, setting the paths to the observable variables at 0 (zero).

3. A saturated model (M12) in which the paths leading to factors were set to 0 (zero), but the paths from the grouping variable to all observable indicators were freely estimated (i.e., hypothesizing the non-invariance of measurement relative to the country of origin).

Once the three models were estimated, M10 was compared with M11 and M12. Since M11 and M12 are more parameterized than M10, they should tend to a better fit. However, if the M10 fit is not substantially worse than that of the other models, the hypothesis that the country has negligible influence on the measurement model would not be rejected. Conversely, if the M12 fit is substantially better than that of M10 and M11, possibly certain items would be at risk of differential functioning.

Finally, we analyzed the mean differences between clinical and non-clinical samples in the factors and subscales of the LSAS-SR, reporting their effect sizes (Cohen's d), as well as the differences in latent means expressed in SDs of the clinical group from the general group, as estimated from the scalar invariance model. These differences are directly interpretable as a Cohen's d.³⁶

All statistical analyses were performed using Statistica version 12,⁴¹ SPSS version 22,⁴² and MPlus version 7.4.⁴³

Results

Exploratory and CFA of the LSAS-SR

The results of parallel analyses of subsample 1 with the LSAS-SR anxiety subscale showed that the five-factor solution was the best fit for the data, given that only the eigenvalues of these five factors were greater than the randomly generated eigenvalues.

Furthermore, the EFA with the LSAS-SR anxiety subscale identified five factors with eigenvalues greater than 1.00, explaining 50.78% of the cumulative variance. The five factors were the following: 1) speaking in public (six items); 2) eating/drinking in front of other people (four items); 3) assertive behaviors (four items); 4) working/writing while being observed (two items); and 5) interactions with strangers (five items) (Table 2). All items loaded above 0.40 on only one factor except for items 7 and 13, which loaded above 0.40 on two factors. Thus, items 7 and 13 were assigned to factors 2 and 3, respectively, because those are the individual factors with which these items were most strongly associated. Items 1, 14, and 18 did not load above 0.40 on any factor. In order to empirically test the redundant contribution of the avoidance subscale, an EFA was also performed. We identified the same five factors explaining 48.58% of the cumulative variance. Eighteen items from the EFA of this subscale loaded above 0.40 on the same factors as the anxiety subscale. Item 7 loaded strongly on two factors and item 1 did not load above 0.40 on any factor.

Table 2
Factor loadings and item-total correlations in the first subsample

Representative items of each factor of the Liebowitz Social Anxiety Scale-Self	Factors
Report (LSAS-SR)	F1	F2	F3	F4	F5	r i-t
F1. Speaking in public (eigenvalue: 7.29; explained variance: 30.41%)
6. Acting, performing, or giving a talk in front of an audience* †	0.78	-0.03	0.04	0.10	0.10	0.48
16. Speaking up at a meeting* †	0.77	0.09	0.08	0.15	0.18	0.60
20. Giving a report to a group* †	0.61	0.14	0.17	0.18	0.17	0.56
5. Talking to people in authority* †	0.61	0.11	0.07	0.21	0.18	0.54
15. Being the center of attention* †	0.58	0.07	0.22	0.13	0.27	0.55
17. Taking a test	0.42	0.00	0.30	0.36	-0.03	0.42
F2. Eating/drinking in front of other people (eigenvalue: 1.66; explained variance: 6.93%)
3. Eating in public places* †	0.08	0.73	0.08	0.17	0.10	0.41
4. Drinking with others in public places* †	0.01	0.69	0.23	0.03	0.09	0.36
2. Participating in small groups* †	0.39	0.49	-0.14	0.13	0.27	0.47
7. Going to a party* †	0.00	0.46	-0.01	0.32	0.41	0.43
F3. Assertive behaviors (eigenvalue: 1.13; explained variance: 4.70%)
24. Resisting a high pressure salesperson* †	0.10	0.09	0.61	0.24	0.15	0.44
22. Returning goods to a store* †	0.12	0.16	0.59	0.14	0.25	0.47
13. Urinating in a public bathroom* †	0.09	0.43	0.53	0.04	-0.02	0.35
21. Trying to pick up someone	0.30	-0.04	0.45	0.06	0.37	0.47
F4. Working/writing while being observed (eigenvalue: 1.08; explained variance: 4.25%)
9. Writing while being observed* †	0.12	0.12	0.07	0.83	0.15	0.50
8. Working while being observed* †	0.22	0.08	0.12	0.81	0.12	0.55
F5. Interactions with strangers (eigenvalue: 1.02; explained variance: 4.25%)
11. Talking with people you don't know very well* †	0.26	0.14	0.14	0.11	0.72	0.61
10. Calling someone you don't know very well* †	0.15	0.03	0.24	0.15	0.64	0.53
12. Meeting strangers* †	0.25	0.18	-0.03	0.23	0.63	0.54
19. Looking at people you don't know very well in the eyes	0.29	0.24	0.21	0.15	0.44	0.55
23. Giving a party†	0.06	0.15	0.26	0.28	0.41	0.45

• Numbers in bold represent item loading > 0.40 on each factor.
• r_i-t = correlation item-total.
• ^*
  Items loading on the same factor in the clinical sample.
• ^†
  Items loading on the same factor in the LSAS-SR avoidance subscale.

The best factor solution for the clinical sample based on the scree test was again a five-factor structure with eigenvalues greater than 1.00, explaining 47.23% of the cumulative variance. The first factor, interactions with strangers (eigenvalue: 5.91, items: 1, 10, 11, 12), explained 24.61% of the variance. The second factor, speaking in public (eigenvalue: 1.84, items: 5, 6, 15, 16, 20), explained 7.68% of the total variance. Factor 3, working/writing while being observed (eigenvalue: 1.52, items: 8, 9), explained 6.33% of the variance. Factor 4, assertive behaviors (eigenvalue: 1.41, items: 13, 17, 18, 22, 24), explained 5.89% of the variance. Finally, factor 5, eating/drinking in front of other people (eigenvalue: 1.32, items: 2, 3, 4, 7), explained 5.51% of the variance. All items loaded above 0.40 on only one factor. Items 14, 19, and 21 did not load above 0.40 on any factor. Seventeen of the 24 items loaded on the same factors as observed for non-clinical subsample 1.

Table 3 shows the results obtained with the CFA of non-clinical subsample 2 and ESEM of subsample 1. The fit indices corroborated the five correlated factor model (M4) as the best fitting model. The fit of the ESEM model was better than that of the CFA, although this result can be expected given that the ESEM is a more parameterized model (with 64 degrees of freedom less than the equivalent CFA model). Table 3 also displays the results of these analyses in the clinical sample. M4 also received the most support as the best fitting model in the clinical sample.

Table 3
Results of CFA of the clinical sample and non-clinical subsample 2 and ESEM of non-clinical subsample 1

LSAS-SR	M1 1 factor	M2 2 correlated factors	M3 4 correlated factors	M4 5 correlated factors	M5 ESEM
Non-clinical sample
df	189	188	183	179	115
RMSEA	0.100	0.098	0.080	0.066	0.043
90%CI	0.099-0.101	0.097-0.099	0.079-0.081	0.065-0.067	0.041-0.044
CFI	0.837	0.844	0.898	0.933	0.982
TLI	0.819	0.826	0.883	0.933	0.967
Clinical sample
df	189	188	183	179	115
RMSEA	0.056	0.055	0.048	0.037	0.031
90%CI	0.050-0.061	0.050-0.061	0.042-0.054	0.031-0.044	0.022-0.039
CFI	0.830	0.833	0.878	0.928	0.969
TLI	0.811	0.814	0.861	0.915	0.943

• 90%CI = 90% confidence interval; CFA = confirmatory factor analysis; CFI = comparative fit index; df = degrees of freedom; ESEM = exploratory structural equation modeling; LSAS-SR = Liebowitz Social Anxiety Scale-Self Report; RMSEA = root mean square error of approximation; TLI =Tucker-Lewis index.

Correlations among LSAS-SR subscales and factors

The LSAS-SR total score was highly correlated with the anxiety and avoidance subscales (non-clinical sample, 0.93 and 0.93; clinical sample, 0.91 and 0.93 respectively). Anxiety and avoidance subscales were also highly correlated (r _non-clinical = 0.73: r _{clinical =} 0.70).

Correlations among the five factors obtained with the LSAS-SR in the non-clinical sample were moderate (from 0.37 to 0.58) and higher than those in the clinical sample, which also ranged from moderate to low (from 0.27 to 0.52).

Internal consistency of the LSAS-SR

Alpha coefficients for the LSAS-SR were high, ranging from 0.93 to 0.88 for the two subscales and the total score in the non-clinical sample and from 0.89 to 0.83 in the clinical sample. Alphas for the five factors were lower, ranging from 0.80 to 0.58 in the non-clinical sample and from 0.63 to 0.43 in the clinical sample. These results do not seem to support the consistency of some of the factors of the LSAS-SR in our sample. Only two factors had an alpha higher than 0.60 in clinical and non-clinical samples, interacting with strangers and speaking in public, the two factors with the highest correlations with similar dimensions of the SAQ (Table 4).

Table 4
Correlations among LSAS-SR anxiety subscale and SAQ factors

	SAQ factors/dimensions
LSAS-SR factors	Opposite sex	Criticism/embarrassment	Interaction with strangers	Speaking in public	Assertive expression	SAQ total
Clinical sample
F1. Speaking in public	0.22	0.30	0.25	0.59	0.27	0.44
F2. Eating/drinking	0.17	0.21	0.40	0.19	0.25	0.30
F3. Assertive behaviors	0.17	0.31	0.16	0.18	0.28	0.27
F4. Interacting with strangers	0.19	0.22	0.41	0.22	0.25	0.34
F5. Working/writing	0.21	0.29	0.33	0.29	0.22	0.33
Total score	0.34	0.40	0.48	0.37	0.38	0.50
Non-clinical sample
F1. Speaking in public	0.46	0.41	0.45	0.68	0.39	0.61
F2. Eating/drinking	0.26	0.21	0.40	0.27	0.27	0.36
F3. Assertive behaviors	0.44	0.38	0.37	0.32	0.39	0.48
F4. Interacting with strangers	0.43	0.35	0.55	0.42	0.38	0.54
F5. Working/writing	0.27	0.28	0.32	0.31	0.27	0.37
Total score	0.49	0.44	0.55	0.55	0.46	0.64

• All correlations were significant at p < 0.001.
• Bold font shows correlations between factors with similar names and total scores in both questionnaires.
• LSAS-SR = Liebowitz Social Anxiety Scale-Self Report; SAQ = Social Anxiety Questionnaire for Adults.

Invariance by country of origin

Regarding invariance by country of origin (Table 5), the null model (M10) presented better fit than both the invariant (ΔRMSEA = 0.008, ΔCFI = -0.014, ΔTLI = -0.018) and the saturated models (ΔRMSEA = 0.01, ΔCFI = -0.013, ΔTLI = -0.023). This suggests that: 1) the measurement model was invariant with respect to the country of origin; and 2) the latent mean differences between countries were not substantial. This was verified by the inspection of the parameters of M11 and M12. The standardized paths from the cluster variable to both the factors and items were close to zero (range 0.04-0.07 in M11, and range 0.00-0.011 in M12).

Table 5
Results of invariance analysis by country of origin

Contrast	RMSEA	RMSEA CI	CFI	TLI	df
Country
Null model	0.056	0.055-0.056	0.950	0.942	200
Invariant model	0.064	0.063-0.064	0.936	0.924	195
Saturated model	0.066	0.065-0.067	0.937	0.919	179

• CFI = comparative fit index; CI = confidence interval; df = degrees of freedom; RMSEA = root mean square error of approximation; TLI = Tucker-Lewis index.

Differences between clinical and non-clinical samples

Mean differences and their effect size were calculated between clinical (n=529) and non-clinical (n=31,243) samples in all the possible subscales of the LSAS-SR (Table 6). As can be seen, the effect sizes of the differences between these two samples are consistently large (Cohen’s d > 0.80).

Table 6
Scores obtained on several subscales of the LSAS-SR

	Non-clinical sample (n=31,243)		Clinical sample (n=529)
LSAS-SR and its subscales and factors	M	SD	M	SD	t	Cohen’s d
LSAS-SR
Total	44.07	21.88	88.65	18.41	-46.57	2.20
Anxiety subscale	22.74	11.88	46.38	9.58	-45.49	2.07
Avoidance subscale	21.31	11.61	42.26	10.90	-41.13	1.86
Anxiety performance	12.51	6.55	24.53	5.70	-41.92	1.96
Anxiety social interaction	10.23	6.00	21.86	4.88	-44.33	2.13
Avoidance performance	11.38	6.53	22.06	6.53	-37.26	1.63
Avoidance social interaction	9.93	5.86	20.20	5.46	-39.97	1.81
F1. Speaking in public	7.74	4.09	13.84	2.81	-	1.74
F2. Eating/drinking	2.04	2.16	5.90	2.92	-	1.50
F3. Assertive behaviors	4.07	2.66	7.55	2.42	-	1.37
F4. Working/writing	3.33	2.93	6.91	3.30	-	1.15
F5. Interacting with strangers	4.37	3.07	9.70	2.78	-	1.82

• LSAS-SR = Liebowitz Social Anxiety Scale-Self Report; M = mean; SD = standard deviation.
• Differences in latent means are expressed in standard deviations of the clinical sample with respect of the general sample (latent mean = 0); therefore, differences are directly interpreted as effect sizes (last column of the differences between factors, in bold).
• p < 0.001.

Convergent validity

Correlations between the LSAS-SR factors and the SAQ dimensions assessing similar constructs exhibited the highest associations (i.e., speaking in public, interacting with strangers) whereas the factor assertive behaviors had only a modest relationship with dimensions of the SAQ. The other two factors (i.e., eating/drinking in front of other people and working/writing while being observed) were not strongly associated with dimensions of the SAQ (Table 4).

Discussion

The aim of this study was to examine the psychometric characteristics of the LSAS-SR in a large cross-cultural sample, comprising clinical and non-clinical subjects from Spain, Portugal, and most Latin American countries.

Specifically, we investigated the factor structure of the LSAS-SR using exploratory analysis and CFA. The results indicate a five-factor structure as the best model in both clinical and non-clinical samples. Safren’s four-factor model¹³ exhibited good fit in the non-clinical sample, but not in the clinical sample. Although the four-factor model has been frequently used,⁷,⁸,¹⁵ it has been disputed.¹³,¹⁶ The single-factor or two-factor (performance and social interaction) models had poor fit, which is consistent with the results of prior research.⁷,⁹,¹⁵ It appears that for this specific group of Spanish and Portuguese-speaking countries, the five-factor model is the best solution for the LSAS-SR, a finding that is consistent with other studies with Spanish-speaking university students.¹⁹ Although some studies¹⁰,¹¹ have used the original theoretical structure of the LSAS-SR (anxiety and avoidance subscales, including performance and social interaction sections in each of them), the empirically derived factor structure obtained for this scale appears to be very different.⁸,⁹,¹² -¹⁴,¹⁷,¹⁹,⁴⁴ The findings of previous studies, of three to six factor structures, attest to considerable individual variability in the possible factor structure of the LSAS-SR.

The most commonly identified factors of the LSAS-SR have been speaking in public, eating/drinking in front of other people (seven studies),⁸,⁹,¹³ -¹⁵,¹⁷,⁴⁴ interactions with strangers (usually included as a minor part of the social interaction factor) (seven studies),⁸,¹² -¹⁵,¹⁷,⁴⁴ working/writing while being observed (usually included as a minor part of the observation or non-verbal performance factors) (seven studies)⁸,⁹,¹² -¹⁵,⁴⁴ and assertive behaviors (two studies).¹³,¹⁴ However, some studies include a major social interaction factor that is a mixture of heterogeneous, and sometimes unrelated, items.⁸,⁹,¹²,¹³,¹⁷,⁴⁴ Our interactions with strangers factor includes a smaller, but much more homogeneous, number of items, usually found under the social interaction factor in these former studies. Generally speaking, we can say that four of the five factors found in our study are also frequently found in many studies regarding the factor structure of the LSAS-SR, but the items composing the four factors of our study do not match the items composing the same four factors found in any of the studies reviewed for the present work. It seems that the selected Spanish and Portuguese-speaking countries have roughly the same dimensions of social anxiety as English speaking countries, but the behaviors through which these dimensions are expressed are partially different (only 15 out of 24 items loaded in the same factors found in the literature).

When comparing the factors of the LSAS-SR with the SAQ, a relatively new measure of social anxiety, empirically developed from 16 Latin American countries, Spain, and Portugal,²⁹ -³¹ some questions emerge regarding the validity of the LSAS-SR. Speaking in public and interactions with strangers seem to be dimensions of social anxiety that are well represented in the LSAS-SR (i.e., have good correlations with similar dimensions of the SAQ). The low correlation between the LSAS-SR and SAQ factors that are supposed to measure assertive behaviors could indicate that these factors reflect different constructs in each instrument. Finally, two LSAS-SR factors are not associated with any of the SAQ factors (i.e., eating/drinking in front of other people and working/writing while being observed). It might be that situations such as drinking in public places, writing while being observed or urinating in a public bathroom may be specific to US samples and less relevant for SAD patients from Latin American countries.²⁹ -³¹ Furthermore, items such as telephoning in public (1), entering a room when others are already seated (14), and expressing disagreement or disapproval to people you don't know very well (18) were not included in any factor of the LSAS-SR for the present Spanish and Portuguese-speaking samples. Item 14, the most problematic, has also been omitted from the factor solution in other studies.⁹,¹⁴,¹⁹ This is also the case for item 18¹⁴,¹⁹ and item 1.⁹,¹⁴ Finally, items such as urinating in a public bathroom (13), taking a test (17), trying to pick up someone (21) or giving a party (23) appear to load onto multiple factors⁹,¹² -¹⁴,¹⁹ and do not contribute to a clear factor structure of the LSAS-SR.

The present results suggest that the LSAS-SR includes items that may not be relevant to Spanish and Portuguese-speaking countries, at least the ones included in the present study. Although the LSAS-SR captures some features of the SAQ (e.g., interactions with the opposite sex and criticism and embarrassment), it does not do so comprehensively. For instance, the LSAS-SR only lists one single item measuring what appears to be an attempt at a sexual/romantic interaction (i.e., trying to pick up someone). One might argue that some LSAS-SR items touch upon fear of criticism or embarrassment (e.g., talking to people in authority, going to a party, speaking up at a meeting, or being the center of attention). However, these items do not form a single factor. Given that people with SAD typically report an extreme fear of criticism and embarrassment, this omission is surprising.

Our research did not find significant differences among the countries included in the study (invariance by country of origin). In this way, given the size of the samples of the study, the psychometric information included in Table 6 could be very informative for people using the LSAS-SR in the studied Spanish and Portuguese-speaking countries at clinical and non-clinical settings. However, the available literature suggests that the factor structure of the LSAS-SR has varied across individual studies, even when they are conducted within the same culture. It is thus not surprising that the current study supports a different factor structure than the original theoretical framework of the LSAS-SR for at least some Spanish and Portuguese-speaking cultures. Additionally, simple translation is insufficient to determine the construct validity of measures across cultures. Simple translation has been a common strategy among most questionnaires assessing social anxiety (including the LSAS-SR). However, new questionnaires have been developed to overcome this limitation (e.g., the SAQ).²⁹-³¹ Furthermore, psychometric properties can vary across dimensions of an assessment, and this is particularly true with the factors obtained in this study for the LSAS-SR. The internal consistency ranged from 0.80 to 0.43, the convergent validity ranged from 0.28 to 0.68, and the five-factor structure was different from what other studies have found.⁸,¹² -¹⁵

Despite these limitations, the LSAS-SR is a self-report measure of social anxiety associated with good psychometric characteristics and a five-factor solution. Although there has been some support for the convergent validity of this instrument, there may be a number of cultural differences that undermine the utility of this scale in Spanish and Portuguese speaking countries. Thus, the assessment of social anxiety may require judicious deliberation about the role of cultural factors, which can influence the expression of SAD. Although the results of the present study suggest a multifactorial solution of the LSAS-SR, it may not be an ideal instrument (including for poor internal consistency) to assess the various dimensions of social anxiety that are relevant to individuals from Spanish and Portuguese speaking countries. However, it could be a useful self-report instrument to measure general social anxiety in these countries, particularly when using the anxiety/fear subscale.

Acknowledgements

This study was partially supported by a grant from Spain’s Ministry of Science and Technology (reference BSO2003-07029/PSCE) and co-financed by the European Regional Development Fund (ERDF) as well as by a grant awarded to the first author by the Spain’s Ministry of Education, Culture and Sport (reference PRX16/00223). The manuscript’s translation was supported financially by the European Regional Development Fund (ERDF) and by the regional government of Andalusia as part of Universidad de Granada’s program Fortalecimiento de la I+D+i. Financial assistance from the Foundation for the Advancement of Clinical Behavioral Psychology (FUNVECA) is also acknowledged. SGH receives grant support from the National Institutes of Health/National Center for Complementary and Integrative Health (NIH/NCCHI; R01AT007257, R01MH099021, R34MH099311, R34MH086668, R21MH102646, R21MH101567, K23MH100259), and the Department of the Army.

CISO-A Research Team: Argentina: E. Barinaga, G. Bustamante, M. Correche, G. del Valle, L. Gómez, A. Grichener, M. Lipovetzky, L. Mariñelarena, M. Rivarola, P. Robles, S. Savoini, M. Tapia; Bolivia: D. Pinelo; Brazil: B. Donato, R. Lopes, L. Lourenço, C. Neufeld, M. Oliveira, M. Wagner; Chile: P. Álvarez, C. Guerra, S. Villanueva, C. Soto, M. Viveros; Colombia: M. Arias, L. Ariza, D. Barreto, C. Benavides, G. Cajiao, C. Carmona, F. Coconubo, Ó. David, P. Díaz, N. Estupiñán, V. Gómez, M. Lemos, R. Mazo, A. Meneses, M. Ortega, S. Ramírez, L. Rocha, M. Varela, D. Villa-Roel; Costa Rica: H. Cabezas; Dominican Republic: P. Beato, L. Blanco, Y. Ramírez; El Salvador: Ó. Olmedo, O. Olmedo M., A. Zúñiga; Guatemala: G. Aguilar, A. Musso; Honduras: R. Ardón; Mexico: S. Anguiano, M. Ascencio, P. Balcázar, M. Bonilla, Á. Camarena, I. Carrillo, R. del Pino, E. Erosa, G. García, M. García, C. González, M. González, G. Gurrola, D. Hernández, S. Hernández, M. Karam, R. Landero, M. Lomelí, J. Olvera, F. Páez, C. Reyes, M. Ríos, R. Robles, A. Tapia, J. Tejero, P. Vázquez, M. Villaseñor; Paraguay: C. Bettina, A. Caballero, R. Estigarribia, M. Silva; Peru: V. Barreda, L. Benites, L. Dueñas, A. Galli, M. Grijalva, R. Guzmán, J. Marchena, C. Merino, M. Salazar, C. Segura, G. Verme; Portugal: R. Barroso, F. Cardoso, P. Carvalho, M. Loureiro, E. Ribeiro; Puerto Rico: S. Fernández, N. Ortiz, J. Pons, E. Rosario, A. Santos, K. Suárez; Spain: C. Antona, B. Arias, P. Barbado, P. Bas, J. Delgado, M. Fernández, A. Goñi, V. Guillén, M. J. Irurtia, S. Lago, C. Loureiro, M. Muñoz, C. Rausell, S. Torrecillas; Uruguay: A. Conde, R. García, M. Golberg, M. Lagos, L. Moreira, A. Nassar, M. Reich, L. Scafarelli; Venezuela: L. Feldman, Z. Lugli, J. Pellicer, E. Vivas.

The first author is the director and the person responsible for the CISO-A Research Team. He also holds all rights for this team.

References

• ¹
  American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5). Arlington: American Psychiatric Publishing; 2013.
• ²
  Kessler RC, Üstün TB. The WHO Mental Health Surveys: global perspectives on the epidemiology of mental disorders. New York: Cambridge University; 2008.
• ³
  Grant BF, Hasin DS, Blanco C, Stinson FS, Chou SP, Goldstein RB, et al. (2005). The epidemiology of social anxiety disorder in the United States: results from the National Epidemiologic Survey on Alcohol and Related Conditions. J Clin Psychiatry. 2005;66:1351-61.
• ⁴
  Caballo VE, Salazar IC, Equipo de Investigación CISO-A España. Desarrollo y validación de un nuevo instrumento para la evaluación de las habilidades sociales: El “Cuestionario de habilidades sociales” (CHASO). Psicol Conductual. 2017;25:5-24.
• ⁵
  Gómez O, Casas C, Ortega-Ruiz R. Ansiedad social en la adolescencia: factores psicoevolutivos y de contexto familiar. Psicol Conductual. 2016;24:29-49.
• ⁶
  Liebowitz MR. Social phobia. Mod Probl Pharmacopsychiatry. 1987;22:141-73.
• ⁷
  Beard C, Rodriguez BF, Moitra E, Sibrava NJ, Bjornsson A, Weisberg RB, et al. Psychometric properties of the Liebowitz Social Anxiety Scale (LSAS) in a longitudinal study of African Americans with anxiety disorders. J Anxiety Disord. 2011;25:722-6.
• ⁸
  Oakman J, Van Ameringen M, Mancini C, Farvolden P. A confirmatory factor analysis of a self-report version of the Liebowitz Social Anxiety Scale. J Clin Psychol. 2003;59:149-61.
• ⁹
  Safren SA, Heimberg RG, Horner KJ, Juster HR, Schneier FR, Liebowitz MR. Factor structure of social fears: the Liebowitz Social Anxiety Scale. J Anxiety Disord. 1999;13:253-70.
• ¹⁰
  Fresco DM, Coles ME, Heimberg RG, Liebowitz MR, Hami S, Stein MB, et al. The Liebowitz Social Anxiety Scale: a comparison of the psychometric properties of self-report and clinician-administered formats. Psychol Med. 2001;31:1025-35.
• ¹¹
  Ha J, Lim SW, Shin YC, Oh KS. Comparison of anxiety-related traits between generalized and nongeneralized subtypes of social anxiety disorder. J Nerv Ment Dis. 2011;199:390-3.
• ¹²
  Romm KL, Rossberg JI, Berg AO, Hansen CF, Andreassen OA, Melle I. Assessment of social anxiety in first episode psychosis using the Liebowitz Social Anxiety scale as a self-report measure. Eur Psychiatry. 2011;26:115-21.
• ¹³
  Baker SL, Heinrichs N, Kim H, Hofmann SG. The Liebowitz social anxiety scale as a self-report instrument: A preliminary psychometric analysis. Behav Res Ther. 2002;40:701-15.
• ¹⁴
  Stein DJ, Kasper S, Andersen EW, Nil R, Lader M. Escitalopram in the treatment of social anxiety disorder: analysis of efficacy for different clinical subgroups and symptom dimensions. Depress Anxiety. 2004;20:175-81.
• ¹⁵
  Heeren A, Maurage P, Rossignol M, Vanhaelen M, Peschard V, Eeckhout C, et al. Self-report version of the Liebowitz Social Anxiety Scale: psychometric properties of the French version. Can J Behav Sci. 2012;44:99-107.
• ¹⁶
  Sugawara N, Yasui-Furukori N, Kaneda A, Sato Y, Tsuchimine S, Fujii A, et al. Factor structure of the Liebowitz Social Anxiety Scale in community-dwelling subjects in Japan. Psychiatry Clin Neurosci. 2012;66:525-8.
• ¹⁷
  Terra MB, Barros HM, Stein AT, Figueira I, Athayde LC, Gonçalves Mde S, et al. Internal consistency and factor structure of the Portuguese version of the Liebowitz Social Anxiety Scale among alcoholic patients. Rev Bras Psiquiatr. 2006;28:265-9.
• ¹⁸
  Connor KM, Davidson JR, Churchill LE, Sherwood A, Foa E, Weisler RH. Psychometric properties of the Social Phobia Inventory (SPIN). New self-rating scale. Br J Psychiatry. 2000;176:379-86.
• ¹⁹
  Caballo VE, Salazar I, Irurtia MJ, Arias B, Nobre-Sandoval L. The assessment of social anxiety through five self-report measures, LSAS-SR, SPAI, SPIN, SPS, and SIAS: a critical analysis of their factor structure. Behav Psychol. 2013;21:423-48.
• ²⁰
  de Beurs E, Tielen D, Wollmann L. The Dutch social interaction anxiety scale and the social phobia scale: reliability, validity, and clinical utility. Psychiatry J. 2014;2014:360193.
• ²¹
  Le Blanc AL, Bruce LC, Heimberg RG, Hope DA, Blanco C, Schneier FR, et al. Evaluation of the psychometric properties of two short forms of the social interaction anxiety scale and the social phobia scale. Assessment. 2014;21:312-23.
• ²²
  Bhogal KS, Baldwin DS. Pharmacological treatment of social phobia. Psychiatry. 2007;6:217-23.
• ²³
  Smits JA, Rosenfield D, McDonald R, Telch MJ. Cognitive mechanisms of social anxiety reduction: an examination of specificity and temporality. J Consult Clin Psychol. 2006;74:1203-12.
• ²⁴
  Bobes J, Badía X, Luque A, García M, González MP, Dal-Ré R. Validación de las versiones en español de los cuestionarios Liebowitz Social Anxiety Scale, Social Anxiety and Distress Scale y Sheehan Disability Inventory para la evaluación de la fobia social. Med. Clin. 1999;112:530-8.
• ²⁵
  González MP, Bobes J, García M, Badía X, Luque A, Dal-Ré R. Assessing social phobia. The Spanish validation of the “gold standard” clinical scales: the LSAS and the SADS. Eur. Neuropsychopharmacol. 1998;8:S259-S60.
• ²⁶
  American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR). Arlington: American Psychiatric Publishing; 2000.
• ²⁷
  World Health Organization (WHO). CIE-10 trastornos mentales y del comportamiento: descripciones clínicas y pautas para el diagnóstico. Madrid: Meditor; 1992.
• ²⁸
  Mennin DS, Fresco DM, Heimberg RG, Schneier FR, Davies SO, Liebowitz MR. Screening for social anxiety disorder in the clinical setting: using the Liebowitz Social Anxiety Scale. J Anxiety Disord. 2002;16:661-73.
• ²⁹
  Caballo VE, Arias B, Salazar IC, Irurtia MJ, Hofmann SG; CISO-A Research Team. Psychometric properties of an innovative self-report measure: the Social Anxiety Questionnaire for adults. Psychol Assess. 2015;27:997-1012.
• ³⁰
  Caballo VE, Salazar I, Arias B, Irurtia MJ, Calderero M, Graña JL, et al. Validation of the Social Anxiety Questionnaire for Adults (SAQ-A30) with Spanish university students: similarities and differences among degree subjects and regions. Behav Psychol. 2010;18:5-34.
• ³¹
  Caballo VE, Salazar IC, Irurtia MJ, Arias B, Hofmann SG; CISO-A Research Team. The multidimensional nature and multicultural validity of a new measure of social anxiety: the Social Anxiety Questionnaire for adults. Behav Ther. 2012;43:313-28.
• ³²
  Salazar IC. Fiabilidad y validez de una nueva medida de autoinforme para la evaluación de la ansiedad/fobia social en adultos [dissertation]. Granada: Universidad de Granada; 2013.
• ³³
  Brown TA. Confirmatory factor analysis for applied research. New York: Guilford; 2006.
• ³⁴
  Schafer JL, Graham JW. Missing data: our view of the state of the art. Psychol Methods. 2002;7:147-77.
• ³⁵
  Osborne JW. What is rotating in exploratory factor analysis? [Internet]. 2015 [cited 2018 01 30]. pareonline.net/getvn.asp?v=20&n=2
  » pareonline.net/getvn.asp?v=20&n=2
• ³⁶
  Brown TA. Confirmatory factor analysis for applied research. New York: Guilford; 2014.
• ³⁷
  Asparouhov T, Muthén B. Exploratory structural equation modeling (ESEM). Struct Equ Modeling. 2009;16:397-438.
• ³⁸
  Marsh HW, Morin AJ, Parker PD, Kaur G. Exploratory structural equation modeling: an integration of the best features of exploratory and confirmatory factor analysis. Annu Rev Clin Psychol. 2014;10:85-110.
• ³⁹
  Jöreskog KG, Goldberger AS. Estimation of a model with multiple indicators and multiple causes of a single latent variable. J Am Stat Assoc. 1975;70:631-9.
• ⁴⁰
  Morin AJ, Arens AK, Marsh HW. A bifactor exploratory structural equation modeling framework for the identification of distinct sources of construct-relevant psychometric multidimensionality. Struct Equ Modeling. 2016;23:116-39.
• ⁴¹
  Statistica [computer program]. Version 12. Tulsa: StatSoft, Inc.; 2013.
• ⁴²
  IBM SPSS Statistics [computer program]. Version 22.0. Chicago: IBM; 2013.
• ⁴³
  MPlus [computer software]. Version 7.4. Los Angeles: Muthén & Muthén; 2015.
• ⁴⁴
  Perugi G, Nassini S, Maremmani I, Madaro D, Toni C, Simonini E, et al. Putative clinical subtypes of social phobia: a factor-analytical study. Acta Psychiatr Scand. 2001;104:280-8.

Publication Dates

• Publication in this collection
  11 Oct 2018
• Date of issue
  Mar-Apr 2019

History

• Received
  16 Aug 2017
• Accepted
  11 Jan 2018