Reliability of a Questionnaire on Substance Use among Adolescent Students, Brazil

Confi abilidade de questionário sobre uso de drogas por escolares, Brasil ABSTRACT OBJECTIVE: To analyze reliability of a self-applied questionnaire on substance use and misuse among adolescent students. METHODS: Two cross-sectional studies were carried out for the instrument test-retest. The sample comprised male and female students aged 11–19 years from public and private schools (elementary, middle, and high school students) in the city of Salvador, Northeastern Brazil, in 2006. A total of 591 questionnaires were applied in the test and 467 in the retest. Descriptive statistics, the Kappa index, Cronbach's alpha and intraclass correlation were estimated. RESULTS: The prevalence of substance use/misuse was similar in both test and retest. Sociodemographic variables showed a " moderate " to " almost perfect " agreement for the Kappa index, and a " satisfactory " (>0.75) consistency for Cronbach's alpha and intraclass correlation. The age which psychoactive substances (tobacco, alcohol, and cannabis) were fi rst used and chronological age were similar in both studies. Test-retest reliability was found to be a good indicator of students' age of initiation and their patterns of substance use. CONCLUSIONS: The questionnaire reliability was found to be satisfactory in the population studied. Substance use and misuse and drug dependence constitute a major public health issue worldwide. Substance misuse usually begins during adolescence, but population based studies targeting adolescent students are relatively scarce in low and middle-income countries. The psychometric properties such as test-retest reliability of questionnaires addressing sociodemographic, behavioral and patterns of substance use/misuse depend on the characteristics of the population under evaluation , local social values and mores and the comprehen-sibility and cultural sensitivity of the instrument in a given context. 4 Some factors have been explored in recent literature including proper understanding of the phrasing used in the questionnaire, 25 respondents' willingness to provide reliable answers, 1,6 interest and time available to take part in the study and statistical power of the sample under analysis. In Brazil, previous studies applied standard questionnaires to assess substance use/misuse. These questionnaires usually consist of adaptations of instruments originally developed in English. There are few studies of questionnaire reliability on drug use in Brazil, with signifi cant methodology differences (in sampling and statistical analysis). RESULTADOS: A prevalência do uso/abuso das substâncias foi semelhante em ambas as avaliações. Para as variáveis sociodemográfi cas o índice kappa indicou concordância " moderada " a " quase perfeita " e a análise do alfa de …

Substance use and misuse and drug dependence constitute a major public health issue worldwide.Substance misuse usually begins during adolescence, but population-based studies targeting adolescent students are relatively scarce in low and middle-income countries.
The psychometric properties such as test-retest reliability of questionnaires addressing sociodemographic, behavioral and patterns of substance use/misuse depend on the characteristics of the population under evaluation, local social values and mores and the comprehensibility and cultural sensitivity of the instrument in a given context. 4me factors have been explored in recent literature including proper understanding of the phrasing used in the questionnaire, 25 respondents' willingness to provide reliable answers, 1,6 interest and time available to take part in the study and statistical power of the sample under analysis. 18,21,22 Brazil, previous studies applied standard questionnaires to assess substance use/misuse. 3,4,9,10These questionnaires usually consist of adaptations of instruments originally developed in English. 3,9,10ere are few studies of questionnaire reliability on drug use in Brazil, with signifi cant methodology differences (in sampling and statistical analysis). 3,4,10A RESUMO OBJETIVO: Analisar a confi abilidade de um questionário auto-aplicável sobre o uso e abuso de substâncias entre adolescentes escolares.

INTRODUCTION
Brazilian reliability study adapted an American questionnaire on psychoactive drug use in general population without specifying its performance among adolescents. 3nother study used a self-report instrument on drug use in a Brazilian sample of public elementary school children (from fi fth grade on), but the instrument's reliability was not reported. 9Considering that, the present study aimed to analyze reliability of a selfapplied questionnaire on substance use and misuse among adolescent students.

METHODS
Test-retest reliability was assessed in a comprehensive study in 2006.The self-applied questionnaire consisted of 93 questions on basic sociodemographic information, substance use and misuse, 7,19 related behaviors, the adolescents' relationship with school and their parents, and their opinions about the media and media campaigns to prevent tobacco, alcohol, and cannabis use.The questions were based on previous studies. 2,3,13e target population comprised adolescents (11  grades), and high schools (tenth, 11 th , and 12 th grades).A total of 715,877 students were enrolled in these schools: 538,364 in elementary and middle schools (467,961 in public and 70,403 in private schools), and 177,513 in high schools (156,784 in public and 20,729 in private schools).a Five schools (two state-run schools, two private schools, and one municipal school) were randomly selected by means of an electronic drawing using education census data.An initial database was created with names, number of classes and number of enrolled students in each class.Each school was stratifi ed according to type of school (public or private) and level (elementary, middle, or high school).
A total of three seventh grade, three eighth grade, three ninth grade, two tenth grade, two 11 th grade, and one 12 th grade class were selected.Students from each class were invited to participate in the study and were informed that the same questionnaire would be applied again two weeks later.
A minimum sample size of 466 students was estimated based on the following assumptions: inter-questionnaire agreement ≥0.80, with an estimated error of 0.03, 15 and a study power over 95% for the kappa index, the intraclass correlation and the Cronbach's alpha. 1,5e sample studied was greater than that estimated.In some schools, the number of students per class was higher than estimated in the study planning due to inaccurate school enrollment information.We opted to include them all to prevent interference in the school routine during data collection.If part of a class were selected, many students would be idle and could interfere with school activities of non-selected classes.
Exclusion criteria included the absence of the standard labels provided by the study coordination codes, being older than 19 or younger than 11, and incomplete questionnaires.Incomplete questions were excluded, as well as those with unmatched answers to the questions.
The questionnaire was applied to 591 students in the test and 467 in the retest with a two-week interval in-between. 20It was applied by medical students during regular school hours.
The students were informed about the study confi dentiality and that they could choose to participate or not, or skip questions they consider too sensitive or inappropriate.Two labels with a code were given to each student to be used in the test and retest.
The refusal rate was lower than 1.0% in the test and 1.7% of the questionnaires were excluded from the analysis (respondents were 19 years old or older).Among those who did not participate in the retest (124), 59.7% had lost their code labels or forgotten their confi dential identifi cation number, 32.3% missed class on the day of retest, and 8.1% refused to participate in the retest (Figure Of those who did not participate in the retest, 71% attended public schools, 60.5% were male, 73.4% were enrolled in elementary and middle schools with a mean age of 15.1 years (SD= 1.9).There were no statistical differences between those who participated and those who did not participate in the retest regarding type of school (p=0.14) and grade (p=0.268),but differences were seen regarding gender (p<0.001) and mean age (p<0.005).
A complete questionnaire was considered valid when it had a label with a valid code and was fi lled out with pen, as required.
A draft instrument was tested among junior students at a federal university aiming at adjusting the questionnaires' phrasing and structure.A revised version was then applied to sophomore students from a private high school to amend language usage issues and other problems found by the respondents.The respondents' questions and their receptiveness were recorded and examined.
The "bogus pipeline" technique in the form of a sham lie detector (fake question) was used to check for accuracy of responses. 17e study variables were classifi ed as follows: type of school (public, private), gender (male, female), grade (elementary, middle, or high school), ethnicity (Caucasian, African Brazilian, Native, or Asian), living situation (living with parents, grandfather and/ or grandmother, living with other relatives e.g.uncle/ aunt, living with friends), parents' education level (illiterate, complete elementary, middle, or high school, college, university education), and their mother's and/ or father's habits regarding smoking, alcohol, and/or cannabis use (yes/no). 2e study used the kappa index, Cronbach's alpha, and intraclass correlation (ICC) to measure the questionnaire's reliability.Due to limitations of the exclusive use of the kappa index in the assessment of reliability and the comprehensiveness of analyses, the three indices were used in a concerted manner. 11,12,16he criteria used in the interpretation of kappa index, intraclass correlation and Cronbach's alpha fi ndings are summarized in the Annex. 21There were estimated 95% confi dence intervals (95% CI) as well as the prevalence point for each variable studied.EpiData 6.0® was used for tabulating the data.Data were analyzed with SPSS 10 ® , EpiInfo 6.0 ® and Winpepi ® .
The ICC was calculated using a mixed-effects model and means after assessing putative interactions. 14,15 valence ratios and chi-square statistics were calculated for the different substances in order to assess if losses in the test-retest were signifi cantly associated with the use of each substance.
The study was approved by the Research Ethics Committee of Universidade Federal da Bahia (Protocol 01/04 - March 3, 2004).We obtained the consent of the principal of each participating school and they were asked to sign an informed consent form and schedule the application of the instrument in the classroom.Following the Research Ethics Committee guidelines it was not necessary to obtain a signed informed consent from each student individually, given the self-applicable and anonymous character of the instrument.1.

Lifetime
The prevalences of ever use of substances in the test and retest were similar for most substances.There was a non-signifi cant trend toward higher prevalences in the retest, except for the use of inhalants, which was lower than that reported in the test.The prevalence of use of most substances studied in the year and month prior to the questionnaire application was similar in the test and retest, except for alcohol and tobacco use, which was signifi cantly higher in the retest.The prevalence of drug use was inaccurate since some substances reported by the students (hashish, LSD, steroids, heroin, amphetamines, crack, and erectile dysfunction drugs) showed wide 95% CIs (Table 2).
Gender, ethnicity, living situation, parents' education level, cigarette smoking by the father and/or the mother, alcohol consumption by the father and/or the mother had a reliability that ranged from acceptable to quite good, with a Cronbach's alpha >0.6 (Table 1).
The kappa index for lifetime, prior year, and prior month use of substances in the test and retest ranged from "moderate" (0.43) to "almost perfect" (0.83) correlation for alcohol, tobacco, inhalants, appetite depressants, cocaine, ecstasy, and other drugs (such as herbal infusions with hallucinogenic properties).The kappa index was inaccurate for cannabis use (in the prior month), hashish (in the prior month and year), and other drugs (in the prior month and year).LSD, hashish, benzodiazepines, steroids, heroin, amphetamines, crack, and erectile dysfunction drugs were seldom reported and had values ranging from 0.21 to 1.0, with wide 95% CIs (Table 3).
Cronbach's alpha and ICC were considered acceptable and satisfactory for most substances, regardless of time of use (lifetime, in the prior year or month).ICC values ranged from 0.44 (satisfactory) to 0.94 (excellent) for lifetime substance use.On the other hand, regarding the use in the prior year, they ranged between 0.57 and 0.94, and in the prior month they ranged from 0.35 (unacceptable) to one.With respect to the use of benzodiazepines in the prior month, reproducibility was unacceptable (0.35) (Table 3).
The kappa index for alcohol, tobacco, and cannabis use (for different time of use) did not show any signifi cant association with type of school, gender, age, grade, parents' education level and parental substance use (alcohol, tobacco, and cannabis).
A signifi cant difference in the kappa index was seen for cannabis use in the prior month.The kappa index was 0.66 (95% CI: 0.04;1.00)among public school students and 0.31 (95% CI: 0.00;0.80)among private school students, with a kappa adjusted by sample size of 0.47 (95% CI 0.08;0.85).When stratifi ed by type of school (public vs. private), the kappa index, Cronbach's alpha, and ICC were not accurate among public school students regarding the use of ecstasy, crack, heroin, LSD, amphetamines, and erectile dysfunction drugs (regardless of time of use) (data not shown).Among private school students there was a clear higher prevalence of lifetime use of these substances (with the exception of cannabis), and use of these substances in the prior year and month (p<0.001).The proportion of students who reported the use of these substances was quite low with zero prevalence among public school students.
The students' mean age and mean age of initiation of substance use (tobacco, alcohol, and cannabis) were similar in the test and retest (Table 4).The ICC was excellent for age, age of initiation of substance use (tobacco, alcohol, and cannabis), and age of initiation of regular alcohol use.The coeffi cients were found to be inaccurate with respect to age of regular tobacco and cannabis use, and the number of days of heavy alcohol consumption in the prior month, with wide 95% CIs (Table 4).

DISCUSSION
Our results can be extrapolated to adolescents enrolled in schools in Salvador from the last year of elementary school to the last year of high school.However, they cannot be extrapolated to adolescents out of the educational system or in other contexts, outside the city of Salvador.
A questionnaire to be applied in a future comprehensive study on this population in Salvador was developed and reviewed through successive adjustments and improvements.The fi nal instrument was a scannable form allowing more reliable results and minimizing data entry errors. 3cording to the literature, 3,20 intervals of two to 14 days can be used between test and retest.In the present study, a two-week interval between assessments was used since smaller intervals seemed to increase memorization bias. 25A previous Brazilian study retested a questionnaire within an interval of 30 days and proved reproducible by kappa analysis. 3e sample size is a key aspect for epidemiological and test-retest studies. 1,15,25Overestimated samples are time and money consuming, and small samples may compromise accuracy of the fi ndings, yielding estimates with too broad confi dence intervals. 1,25In a test-retest assessment, sample size must be estimated for each specifi c statistics (kappa, Cronbach's alpha and the ICC), 1 and in the present study the sample had enough power to properly assess the three indices. 1,24ong those students who did not participate in the retest, statistically signifi cant differences were seen concerning gender and age, but no signifi cant difference was found associated with the use of any specifi c substance.
Different studies have shown that the use of a "bogus pipeline" method for anonymous instruments tends to increase the reliability of cross-sectional and prospective studies on drug use. 17This technique enhances the likelihood of positive answers when substance use and misuse is concerned. 26 the present study, the reported prevalences of substance use tended to be higher (with no statistical signifi cance) in the retest.This fi nding may be explained by: a favorable impact (motivation) of the previous successful test on retest; 15,17 students' increased trust on the research team after a prior positive experience; and the use of the "bogus pipeline" as a check for putative biases.The reported prevalence of use was too low for some substances, making such estimates inaccurate, 20,25 and precluding further analyses.
The kappa index had moderate to substantial correlation between sociodemographic variables for most substances (and different times of use), except for LSD, steroids, heroin, amphetamines, crack, and erectile dysfunction drugs.This inaccuracy seems to be secondary to a low prevalence of use of these substances. 8,23e fi rst study on substance use and misuse in Brazil used exclusively the kappa index for the assessment of reliability with a mean value of 0.79 (95% CI 0.50;1.00). 3Although this index was similar in the present study, the fi ndings from the two studies were not fully comparable.In the current study 95% CIs were provided, as well as the results for the ICC and Cronbach's alpha, but such information are absent in the previous one, compromising a comprehensive comparison of the psychometric properties of the respective questionnaires.
ICC was found to be satisfactory to excellent and Cronbach's alpha measures were acceptable.These indices are more accurate than the kappa index since they rely less on the prevalence and symmetry of the events under analysis and probe the internal consistency of the instruments under analysis. 4,16ICC showed high values for most continuous variables, except for age of initiation of regular use (tobacco and cannabis) and heavy use of alcohol, probably due to the low prevalence of such harmful habits. 1 Missing information and/or participant losses between test and retest may have biased fi ndings and may constitute a limitation of the present study. 25Even considering the limitations associated with underestimating or overestimating substance use, our questionnaire showed reliability among adolescent students in the city of Salvador, and may be a useful and culturally-sensitive instrument for the assessment of substance use and misuse in this population.
Figure.Flowchart of questionnaire test and retest among 591 adolescent students.City of Salvador, Northeastern Brazil, 2006.
, yearly or monthly substance use, with the exception of cannabis, was similar among the students who participated in the test and retest.
sample.The distribution of enrolled students per administrative unit was: 40.6% students were enrolled in state-run schools, 22.9% in municipal schools, and 36.4% in private schools(Figure).The distribution of sociodemographic variables in the test and retest is summarized in Table

Table 1 .
Frequency and reproducibility indicators (kappa, Cronbach's alpha, and intraclass correlation) according to sociodemographic variables among 459 adolescent students.City of Salvador, Northeastern Brazil, 2006.

Table 2 .
Frequency of psychoactive substance use (test and retest) for lifetime, in the prior year, and prior month by substance among 459 adolescent students.City of Salvador, Northeastern Brazil, 2006.

Table 3 .
Indicators of reproducibility (kappa, Cronbach's alpha, and intraclass correlation) of the frequency of psychoactive substance use (test and retest) for lifetime, in the prior year, and prior month by substance among 459 adolescent students.City of Salvador, Northeastern Brazil, 2006.

Table 4 .
Average and standard deviation (test and retest) and intraclass correlation (ICC) according to age, age of experimentation, age of regular use, and number of days of heavy alcohol use of 459 school-age adolescents.Municipality of Salvador, Northeastern Brazil, 2006.
a at least 5 times per month b 5 or more doses