Psychometric properties of the Brazilian version of the Pittsburgh Sleep Quality Index Addendum for PTSD (PSQI-A)

Abstract

Objective:

Sleep disturbances play a fundamental role in the pathophysiology posttraumatic stress disorder (PTSD), and are not only a secondary feature. The aim of this study was to validate and assess the psychometric properties of the Brazilian version of the Pittsburgh Sleep Quality Index Addendum for PTSD (PSQI-A-BR), a self-report instrument designed to assess the frequency of seven disruptive nocturnal behaviors, in a sample of participants with and without PTSD.

Methods:

PSQI-A was translated into Brazilian Portuguese and applied to a convenience sample of 190 volunteers, with and without PTSD, who had sought treatment for the consequences of a traumatic event.

Results:

The PSQI-A-BR displayed satisfactory internal consistency (Cronbach's coefficient of 0.83 between all items) and convergent validity with the Clinician Administered PTSD Scale (CAPS), even when excluding sleep-related items (r = 0.52). Test-retest yielded high agreement in the global PSQI-A-BR, with good stability over time (r = 0.88). A global PSQI-A-BR cutoff score of 7 yielded a sensitivity of 79%, specificity of 64%, and a global score of 7 yielded a positive predictive value of 93% for discriminating participants with PTSD from those without PTSD.

Conclusion:

The PSQI-A-BR is a valid instrument for PTSD assessment, applicable to both clinical and research settings.

Sleep; posttraumatic stress disorder; tests/interviews, psychometric; stress; dreams

Introduction

Posttraumatic stress disorder (PTSD) is characterized by the persistence of dysfunctional stress reactions 1 month or more after exposure to an extreme traumatic event. The core triad symptom clusters in the DSM-IV-TR include intrusions, avoidance, and increased arousal.¹ In the DSM-5,² four distinct symptom clusters are defined, with a distinction between avoidance and distorted emotions and cognitions. According to the DSM-IV-TR¹ and DSM-5,² sleep disturbances are part of the definition of two clusters, namely intrusion (recurrent distressing dreams of the event) and increased arousal (difficulties in initiating or maintaining sleep).³ Some evidence suggests that sleep dysfunction plays an important role in the pathophysiology, clinical and treatment outcomes of PTSD, and is not only a secondary feature.⁴ Thus, it is important to accurately detect and intervene on sleep in the context of trauma-related reactions and disorders.⁴

Generally, patients with PTSD report a broad spectrum of unspecific sleep-related complaints, such as difficulty initiating and maintaining sleep, poor sleep quality, and frequent and lengthy nocturnal awakenings.⁵ Other disruptive nocturnal behaviors (DNBs) have been reported, and include trauma-related nightmares, nocturnal intrusive memories, distressing dreams not related to the trauma, sleep terrors, nocturnal panic attacks, dream enactment behaviors, and other complex motor behaviors.⁵,⁶

The Pittsburgh Sleep Quality Index (PSQI)⁷ is a worldwide used instrument developed to assess overall sleep quality.⁷,⁸ However, this instrument does not measure DNBs frequently observed in PTSD patients. The PSQI Addendum for PTSD (PSQI-A) was developed to address this need in trauma-exposed patients.⁵

Urban violence is currently endemic in Brazil; violent events are part of daily life.⁹ Exposure to traumatic events in São Paulo and Rio de Janeiro is highly prevalent: nearly 90% of people surveyed in these two cities have faced at least one lifetime traumatic experience. Over 59% of São Paulo residents and 63% in Rio de Janeiro reported a lifetime direct exposure to violence.¹⁰ The 1-year prevalence estimates of such events are 9.5% in São Paulo and 11.4% in Rio de Janeiro, with a high incidence of PTSD among civilians. The 1-year prevalence estimates of PTSD are 5% in São Paulo and 3.3% in Rio de Janeiro.¹⁰ Ribeiro et al. also estimated lifetime prevalence of PTSD as 10.2% in São Paulo and 8.7% in Rio de Janeiro.¹⁰

Therefore, the development and validation of reliable instruments for identification of clinically significant sleep disturbances related to PTSD and their severity in Brazilian samples is of paramount importance.⁹ To date, there is no validated instrument in Brazilian Portuguese to evaluate DNBs in trauma-exposed and PTSD patients. Thus, the validation of the Brazilian version of the PSQI-A was undertaken. In addition, the fact that the PSQI-A has also been used in other countries can further enable cross-cultural research.⁵,⁶,¹¹

The aim of this study was to validate and evaluate the psychometric properties of the Brazilian version of the PSQI-A (PSQI-A-BR), using criteria of validity, convergent validity, internal consistency and time sensibility.

Methods

The instrument

The PSQI Addendum for PTSD (PSQI-A) is a self-assessment scale, developed by Germain et al. in 2005, that consists of seven items to assess seven types of DNBs reported by PTSD patients.⁵ The seven items assess the frequency of hot flashes; general nervousness; memories or nightmares of the traumatic experience; severe anxiety or panic, not related to traumatic memories; bad dreams, not related to traumatic memories; episodes of terror or screaming during sleep without fully awakening; and episodes of acting out dreams, such as kicking, punching, running, or screaming. The person is asked to rate the frequency of these DNBs during the month preceding the completion of the questionnaire. Each item is rated on a 0-3 scale, with 0 = not in the past month, 1 = less than once a week, 2 = once or twice a week, and 3 = three or more times a week. A global score is obtained from the sum of these items, ranging from 0-21. In addition, three items are included for clinical and informative purposes only, regarding the frequency of anxiety and anger accompanying DNBs and the timing of these events during the night. These three items are not included in the total PSQI-A score, nor were they used in the analyses described herein. The instrument requires less than 5 minutes to complete. The complete PSQI-A-BR is available elsewhere.⁵

Translation procedures

Two English-Portuguese expert translators translated the instrument, resulting in two Portuguese versions, which were back-translated by two other independent translators, native speakers of English, unaware of the original English version. An expert panel and the author of the original English scale (AG) revised the back-translated English versions. Minimal semantic issues were found, e.g., the term “bed partner” was replaced by “partner,” as the term “partner” implicitly entails “bed partner” in Brazil; the term “had episodes of acting out your dreams” was roughly translated to Portuguese as “had episodes in which you made gestures and movements while sleeping”, as, in Brazilian Portuguese, the word “acting” could be misinterpreted as “pretending” (i.e., acting as in the work of an actor), with a negative connotation, resulting in underestimated results of this important item; “dream” in this sentence was replaced by “sleep,” which is a broader term. Our working group believes that Brazilians tend to interpret this sentence literally, not taking into consideration whether the person recalls his or her dreams but still acts them out. Therefore, this information may be obtained from the respondent's bed partner, even if the respondent does not recall the dream and acts it out.

A final version was developed after an expert panel worked to reach consensus on discrepancies and maximize the attainment of semantic, idiomatic, experiential, and conceptual equivalence between the Brazilian and the original versions.¹² To test the understandability of the instrument,⁶ a preliminary version was tested on 10 individuals, who were asked to pay close attention to the meaning and to judge whether there were any issues with the instrument. In this pretest, the mean age of the participants was 34±4.2 years, 80% were female, and different educational levels (6 to 30 years of formal schooling) were represented. All participants were recruited from the staff of the Universidade Federal de São Paulo (UNIFESP) Violence and Stress Program (PROVE). None reported any difficulty completing the instrument, and the final version was identical to the preliminary one.

Subjects

Individuals seeking treatment for trauma-related disorder in the UNIFESP PROVE program were screened, through an interview, for the pertinence of symptom permanence and need for treatment by a multidisciplinary team. When individuals were considered eligible for treatment, were aged 18 years or older, and had suffered from a traumatic event meeting criterion A of the PTSD DSM-IV diagnosis, they were invited to take part in the study.¹ All eligible participants based on the preceding two criteria were informed that participation in this study was voluntary and would not affect eligibility for treatment. All provided voluntary, written, and informed consent to participate in the study. The exclusion criteria were: pregnancy; rotating shift work; unstable major somatic or mental disorders, as assessed by a trained psychiatric interviewer;⁵ or refusal to participate. A convenience sample of 190 volunteers (73.2% female) was formed by consecutive patients over a period of 3 years, between 2009 and 2012. Of these, 154 developed PTSD (PTSD+), and 25 individuals exposed to trauma did not develop PTSD. The latter group formed the control group (trauma control group = PTSD-). Information regarding PTSD was missing for 11 participants. The mean age for the total sample (n=190) was 40.2±11.2 years. Mean age was 39.9±11.6 years for the PTSD group (n=154), and 41.1±10.5 for the control group (n=25). The two groups did not differ regarding age (t ₍₁₄₉₎ = 0.5, p = 0.62). Women represented the majority of participants in both groups (74.7% in the PTSD+ and 64% in the PTSD- group; chi-square [1] = 1.24, p = 0.33).

Procedures

Participants completed the Brazilian version of the Pittsburgh Sleep Quality Index (PSQI-BR)⁸ and the PSQI-A-BR at intake. The Brazilian version of the Clinician Administered PTSD Scale (CAPS) was also administered and used as the gold standard for PTSD diagnosis and symptom quantification.¹³ For retest purposes, 15 days after first assessment, 14 individuals from the PTSD+ group completed the PSQI-A-BR again (all individuals were invited to take part in the retest stage, however, only 14 were available to return to the center after 15 days). Six months after the intake assessments, the CAPS, PSQI-A, and PSQI-A-BR were administered again to all who had completed treatment.

Instruments

The characteristics of the PSQI-A-BR have been described previously.

The PSQI-BR is a self-assessment instrument developed to assess sleep quality over an 1-month period and was validated in Brazilian Portuguese by Bertolasi et al.⁸ It assesses subjective sleep quality, sleep latency, sleep duration, habitual sleep efficiency, sleep disturbances, use of sleeping medication, and daytime dysfunction, and yields one global score, which ranges from 0 to 21. A global PSQI score greater than 5 indicates major difficulties in at least two components or moderate difficulties in more than three components.⁷,⁸ Higher scores indicate poorer sleep quality.

The CAPS is a 30-item structured diagnostic interview designed to measure the frequency and intensity of PTSD symptoms and has been validated in Brazilian Portuguese by Pupo et al.⁹ The symptoms are scored on a 0-4 scale and the total score ranges from 0 to 136. Scores of 45 or greater are considered indicative of a positive PTSD screen.⁹

Statistical analysis

Data were analyzed in SPSS version 20 for Windows, with the exception of weighted kappa, which was calculated using MedCalc 12.7.4.0. The Kolmogorov-Smirnov test was used to test for normality of data. Global PSQI scores at intake and all PSQI-A-BR items (at both times) for both groups were significantly non-normal; global CAPS score, global PSQI-A-BR score, and age were distributed normally.

Student's t tests were used to compare total PSQI-A-BR scores between the PTSD and control groups. The nonparametric Mann-Whitney U test was used to compare total PSQI-BR scores and PSQI-A-BR item scores between the two groups. Cronbach's coefficient was used to determine PSQI-A-BR internal consistency in both study groups.

Pearson's correlations were used to determine convergent validity between PSQI-A-BR and CAPS scores from the PTSD group alone and with the control group. Spearman's correlations were used to determine convergent validity between non-normal (PSQI-A-BR items, PSQI-BR) and normal data; three sets of correlations were conducted: 1) correlations between the PSQI-A-BR and total scores of the other two instruments (with and without the control group); 2) correlations between the PSQI-A-BR and the total scores of the CAPS after subtracting sleep-related item scores (both groups); and 3) partial correlations controlling for age and for time elapsed from latest traumatic event (categorized into five groups): a) less than 1 month, b) between 1 month and 1 day but less than 6 months, c) more than 6 months but less than 1 year, d) more than 1 year but less than 5 years, and e) more than 5 years, with and without the control group, even though age did not differ between groups. This was conducted to evaluate the relationships between age and time elapsed since the latest traumatic event and sleep patterns. Sensitivity and specificity were determined using a receiver operator characteristic (ROC) analysis of PSQI-A global scores from the two groups. The positive predictive value was determined using the cutoff PSQI-A global score that best discriminated PTSD from trauma control, using the CAPS as the gold standard.

Sensitivity to time and treatment-related changes were assessed using data collected from subgroups of PTSD and control subjects after 6 months in treatment at PROVE. Both groups received the therapies available at the institution. Treatments were tailored to patients' individual needs (psychoeducational, psychotherapeutic, and/or pharmacological treatment). A paired-sample t test was used to compare score differences at baseline and 6 months later on the seven items and total PSQI-A-BR score, as well as total CAPS and PSQI scores.

For test-retest purposes, the intraclass correlation coefficient (ICC) was used for the global score, as it is a continuous variable. Quadratic weighted kappa was calculated for all items of the scale. The Cohen classification¹⁴ was employed to assess percent agreement for test and retest: 1) < 0, no agreement; 2) 0.0-0.20, slight agreement; 3) 0.21-0.40, fair agreement; 4) 0.41-0.60, moderate agreement; 5) 0.61-0.80, substantial agreement; 6) 0.81-1.00, almost perfect agreement.

Results

Sleep quality and severity of DNBs between groups

Data were available for 153 individuals with PTSD (one PSQI-A-BR score in the PTSD group was missing) and 25 individuals without PTSD. For time elapsed since last traumatic event, see Table 1. Respondents in the PTSD group reported poorer overall sleep quality (median = 14.0) on the PSQI-BR as compared with PTSD- subjects (median = 8) (U = 793.00, p < 0.001). The PTSD+ group also reported significantly higher PSQI-A-BR global scores than the PTSD- group (PTSD- = 5.5±4.6; PTSD+ = 11.4±5.4; t ₍₁₇₆₎ = -5.1, p < 0.001). Between-group differences on seven PSQI-A-BR items are summarized in Table 2.

Table 1
Distribution of time elapsed since last traumatic event

	PTSD- (n=25)	PTSD+ (n=154)
1 month > 6 months	29.2	28.4
6 months > 1 year	12.5	16.9
1 year > 5 years	20.8	24.3
> 5 years	37.5	30.4

• PTSD = posttraumatic stress disorder.

Table 2
Between-group differences in the PSQI Addendum (PSQI-A-BR) for PTSD in the PTSD- and PTSD+ groups (Mann-Whitney test)

	PTSD- (n=25)		PTSD+ (n=153)		U	p
	Median	IQR	Median	IQR
Hot flashes	0	1	2	3	885.5	<0.001
General nervousness	2	2	3	1	1,355.0	0.011
Memories/nightmares of trauma	1	2	3	2	989.5	<0.001
Anxiety/panic not related to trauma	0	2	2	2	1,186.0	0.001
Distressing dreams not related to trauma	1	1	1	3	1,422.5	0.034
Episodes of terror	0	1	1	2	1,159.5	0.001
Acting out dreams	0	0	1	2	1,167.5	0.001

• IQR = interquartile range; PSQI-BR = Brazilian version of the Pittsburgh Sleep Quality Index; PTSD = posttraumatic stress disorder.

Psychometric properties of the Brazilian version of the PSQI-Addendum

Internal consistency

All items of the PSQI-A yielded a Cronbach's alpha = 0.83, indicating a unitary underlying construct of DNB. Internal consistency was not improved by removing any of the seven PSQI-A-BR items. Furthermore, we found significant item-total correlations after controlling for the effect of age and time elapsed since last traumatic event (Table 3). The mean PSQI-A-BR item-total correlation was r = 0.7 and all were statistically significant (p < 0.001).

Table 3
PSQI-A-BR item global partial correlations adjusted for age and time elapsed since last traumatic event

	Global PSQI-A-BR minus item score*
Hot flashes	0.51
General nervousness	0.62
Memories/nightmares of trauma	0.65
Anxiety/panic not related to trauma	0.65
Distressing dreams not related to trauma	0.66
Episodes of terror	0.54
Acting out dreams	0.45

• PSQI-A-BR = Brazilian version of the Pittsburgh Sleep Quality Index Addendum for Posttraumatic Stress Disorder (PTSD).
• * All analyses resulted p < 0.001.

Convergent validity with PTSD scores

We found a significant positive correlation between PSQI-A-BR and CAPS global scores (r = 0.53, p < 0.001), between PSQI-A-BR and PSQI-BR scores (r_s = 0.67, p < 0.001), and between all PSQI-A-BR items and total CAPS and PSQI global scores. These correlations remained significant after adjusting for age and time elapsed since last traumatic event (Table 4).

Table 4
PSQI-A-BR partial correlations with CAPS and PSQI global scores adjusted for age and time elapsed since last traumatic event

	CAPS*	PSQI*
Hot flashes	0.408	0.553
General nervousness	0.397	0.552
Memories/nightmares of trauma	0.467	0.539
Anxiety/panic not related to trauma	0.345	0.473
Distressing dreams not related to trauma	0.415	0.498
Episodes of terror	0.448	0.447
Acting out dreams	0.328	0.365

• CAPS = Clinician Administered PTSD Scale; PSQI = Pittsburgh Sleep Quality Index; PSQI-A-BR = Brazilian version of the Pittsburgh Sleep Quality Index Addendum for Posttraumatic Stress Disorder (PTSD).
• * All analyses resulted p < 0.001.

After subtracting the sleep items from the total CAPS scores, the correlation for total sample remained significant (r = 0.52, p < 0.001), indicating that the relationships between global PSQI-A-BR and PTSD severity scores were not solely attributable to the weight of the sleep items on PTSD measures and the PTSD group did not attenuate the correlations.

Sensitivity, specificity, and positive predictive value

The area under the ROC curve was 0.79. ROC analysis revealed that a PSQI-A cutoff score ≥ 7 yielded a sensitivity of 79% and specificity of 64% for the PTSD group compared to the control group. A PSQI-A-BR score ≥ 7 yielded a positive predictive value of 93%.

Sensitivity to time and change

Data collected from 53 PTSD patients (mean age 41.1 years, 67% female) and 8 participants from the PTSD- group (mean age 45.8 years; 62.5% female) were available for this analysis. For the PTSD- group, there were no differences between total CAPS scores (-6.6±23.1, p = 0.44) or PSQI-A-BR scores (1.25±3.8, p = 0.38) after a 6-month period. However, the PTSD+ group showed a significant reduction in total CAPS (14.8±30.4, p < 0.001) and total PSQI-A-BR scores after 6 months of treatment (mean = 2.8±5.3, p < 0.001). There were no significant differences on the items distressing dreams not related to trauma, episodes of terror, or acting out dreams. Test-retest elicited slight agreement for the items hot flashes and anxiety/panic not related to trauma, fair agreement for acting out dreams, moderate agreement for general nervousness and episodes of terror, and substantial agreement for memories/nightmares of trauma and distressing dreams not related to trauma. The global score yielded almost perfect agreement on the basis of the ICC (Table 5).

Table 5
PSQI-A-BR test-retest reliability

	Weighted kappa	Quadratic weights
Hot flashes	0.018	Slight
General nervousness	0.452	Moderate
Memories/nightmares of trauma	0.767	Substantial
Anxiety/panic not related to trauma	-0.016	Slight
Distressing dreams not related to trauma	0.613	Substantial
Episodes of terror	0.477	Moderate
Acting out dreams	0.302	Fair
Global score*	0.879	Almost perfect

• PSQI-A-BR = Brazilian version of the Pittsburgh Sleep Quality Index Addendum for Posttraumatic Stress Disorder (PTSD).
• * Intraclass correlation coefficient.

Discussion

The PSQI-A-BR demonstrated satisfactory internal consistency and convergent validity with CAPS, the gold standard for PTSD evaluation. The Cronbach's coefficient of 0.83 indicates high internal consistency among all PSQI-A-BR items.

In this sample, a cutoff score of 7 yielded acceptable sensitivity and specificity and a high positive predictive value for discrimination between victims of violence who did not develop PTSD and those who did. The previous study of PSQI-A validation yielded a cutoff value ≥ 4.⁵ In the present study, the cutoff was higher, probably because the comparison group had also been exposed to traumatic events in the recent past. Nonetheless, our findings confirmed data from the validation study of the original English scale. The far higher cutoff value of this study may be used in studies conducted in Brazil (and elsewhere) involving controls recently exposed to trauma; evidence suggests that a lower cutoff may be used in studies whose control groups are primarily composed of individuals not exposed to trauma.⁵

This study design did not allow us to conduct an accurate assessment of test-retest reliability, since all retested patients had received treatment. Nonetheless, we found stability among all items, although some (namely hot flashes, anxiety/panic not related to trauma, and acting out dreams) exhibited low stability, probably due to treatment effect. However, there was high agreement in the global PSQI-A-BR score, with good stability over time. The high internal consistency and good sensitivity to change in the PTSD group suggest that the PSQI-A-BR is a reliable instrument. Despite the small control sample (n=8), the finding that PSQI-A-BR global scores did not change over a 6-month period in healthy subjects exposed to a traumatic event also points to the reliability of the instrument. Finally, the absence of detailed comorbidity measures limits our ability to speculate on their potential impact of the psychometric properties of the instrument.

Nevertheless, this newly validated Brazilian Portuguese instrument can be helpful in the study of PTSD-related sleep dysfunction, as this disorder is highly prevalent in Brazil. The instrument showed good psychometric properties, with an excellent positive predictive value, demonstrating its accuracy for measurement of DNBs in PTSD, which reinforces its utility for research. Since DNBs are a core symptom of PTSD and increasingly the object of study, it is essential that a valid instrument be made available to measure and evaluate these symptoms and their relationship with other PTSD symptoms, as well as with PTSD severity.

Acknowledgements

The authors would like to thank Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) for the scholarship granted. DS is the recipient of a Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) fellowship. The authors would like to thank the study participants and the staff and volunteers of the PROVE program.

References

• ¹
  American Psychiatric Association. Diagnostic and statistical manual of mental disorders - DSM-IV-TR¯. 4th ed. Washington: American Psychiatric Publishing; 1994.
• ²
  American Psychiatric Association. Diagnostic and statistical manual of mental disorders - DSM-5-TR¯. 5th ed. Arlington: American Psychiatric Publishing; 2013.
• ³
  Vieweg WV, Julius DA, Fernandez A, Beatty-Brooks M, Hettema JM, Pandurangi AK. Posttraumatic stress disorder: clinical features, pathophysiology, and treatment. Am J Med. 2006;119:383-90.
• ⁴
  Spoormaker VI, Montgomery P. Disturbed sleep in post-traumatic stress disorder: secondary symptom or core feature? Sleep Med Rev. 2008;12:169-84.
• ⁵
  Germain A, Hall M, Krakow B, Katherine Shear M, Buysse DJ. A brief sleep scale for Posttraumatic Stress Disorder: Pittsburgh Sleep Quality Index Addendum for PTSD. J Anxiety Disord. 2005;19:233-44.
• ⁶
  Farrahi J, Nakhaee N, Sheibani V, Garrusi B, Amirkafi A. Psychometric properties of the Persian version of the Pittsburgh Sleep Quality Index addendum for PTSD (PSQI-A). Sleep Breath. 2009;13:259-62.
• ⁷
  Buysse DJ, Reynolds CF 3rd, Monk TH, Berman SR, Kupfer DJ. The Pittsburgh Sleep Quality Index: a new instrument for psychiatric practice and research. Psychiatry Res. 1989;28:193-213.
• ⁸
  Bertolazi AN, Fagondes SC, Hoff LS, Dartora EG, Miozzo IC, de Barba ME, et al. Validation of the Brazilian Portuguese version of the Pittsburgh Sleep Quality Index. Sleep Med. 2011;12:70-5.
• ⁹
  Pupo MC, Jorge MR, Schoedl AF, Bressan RA, Andreoli SB, Mello MF, et al. The accuracy of the Clinician-Administered PTSD Scale (CAPS) to identify PTSD cases in victims of urban violence. Psychiatry Res. 2011;185:157-60.
• ¹⁰
  Ribeiro WS, Mari Jde J, Quintana MI, Dewey ME, Evans-Lacko S, Vilete LM, et al. The impact of epidemic violence on the prevalence of psychiatric disorders in sao paulo and rio de janeiro, Brazil. PLoS One. 2013;8:e63545.
• ¹¹
  Insana SP, Hall M, Buysse DJ, Germain A. Validation of the Pittsburgh Sleep Quality Index Addendum for posttraumatic stress disorder (PSQI-A) in U.S. male military veterans. J Trauma Stress. 2013;26:192-200.
• ¹²
  Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000;25:3186-91.
• ¹³
  Pupo MC, Jorge MR, Schoedl AF, Bressan RA, Andreoli SB, Mello MF, et al. The accuracy of the Clinician-Administered PTSD Scale (CAPS) to identify PTSD cases in victims of urban violence. Psychiatry Res. 2011;185:157-60.
• ¹⁴
  Cohen J. A Coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37-46.

Publication Dates

• Publication in this collection
  15 Apr 2014
• Date of issue
  Oct-Dec 2014

History

• Received
  1 Aug 2013
• Accepted
  28 Nov 2013