Inconsistent reports of risk behavior among Brazilian middle school students : National School Based Survey of Adolescent Health ( PeNSE 2009 / 2012 )

This study assessed the consistency of self-reports of risk behavior (overall and within four specific domains: alcohol use, tobacco use, drug use, and sexual activity) in two editions of the Brazilian National School Based Survey of Adolescent Health (PeNSE): 2009 and 2012. The overall proportion of cases with at least one inconsistent response in the two editions was 11.7% (2.7% on the alcohol items, 2.1% for drug use, 4.3% for cigarette use, 3% for sexual activity) and 22.7% (12.8% on alcohol items, 2.5% for drug use, 4.3% for cigarette use, 4.1% for sexual activity), respectively. Such inconsistency was more prevalent among males, delayed students, those who reported having experimented with drugs, and those who did not have a cellphone. Because inconsistent responses were more prevalent among the students who claimed to have engaged in risky activities, removing inconsistent responders affected the estimated prevalence of all risk behaviors in both editions of the survey. This study supports the importance of performing consistency checks of selfreport surveys, following the growing body of literature on this topic. Risk-Taking; Self Report; Surveys; Methodology; Adolescent Correspondence D. O. Ramos Departamento de Epidemiologia, Instituto de Medicina Social, Universidade do Estado do Rio de Janeiro. Rua São Francisco Xavier 524, 7o andar, bloco D, Rio de Janeiro, RJ 20550-990, Brasil. dandararamos2@gmail.com 1 Instituto de Medicina Social, Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brasil. 2 McMaster University, Hamilton, Canada. 3 Instituto de Psicologia, Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brasil. ARTIGO ARTICLE doi: 10.1590/0102-311X00145815 Cad. Saúde Pública 2017; 33(4):e00145815 This article is published in Open Access under the Creative Commons Attribution license, which allows use, distribution, and reproduction in any medium, without restrictions, as long as the original work is correctly cited.


Introduction
The most widely reported measures of youth risk behavior are items from self-report surveys administered in schools 1 .Results from such surveys are commonly used for developing public policy, identifying problem behaviors within a particular school or district, and planning and evaluating prevention activities.
In Brazil, monitoring adolescents' health and risk factors has been an ongoing priority of the National School Based Survey of Adolescent Health 2,3 , also known as the PeNSE survey (PeNSE -Pesquisa Nacional de Saúde do Escolar).In the past two decades, as concern about youth health in Brazil has risen, so has the reliance on this survey's data for identifying risks and temporal trends, and for making policy decisions, especially by epidemiologists interested in identifying the prevalence of health risk behaviors and their associated factors 4,5,6,7,8,9,10,11,12,13,14,15,16 .There is a growing body of evidence about the validity of such self-reported survey data, indicating that (especially for surveys with young samples) estimates of the prevalence of various activities can sometimes be dramatically over, or underestimated 17,18 .The relative validity of dietary 19 and physical activity indicators 20 from the PeNSE questionnaire, as well as its sensitivity, specificity, and correct classification rate have been previously studied.Nevertheless, the data concerning adolescent risk behaviors from the PeNSE survey have apparently never been scrutinized for consistency.
The primary purpose of this study was to examine the consistency of self-reports of risk behavior, overall and within four specific domains: alcohol use, tobacco use, drug use, and sexual activity.A second purpose was to compare the two editions of the PeNSE survey, those of 2009 and 2012, with respect to methodology and sociodemographic aspects of the two years' sample, to document changes in the inconsistency rates between the two and explore possible causes of those changes.A third purpose was to assess how deleting inconsistent, self-contradictory responses would affect prevalence estimates.

Self-reported measures of risk behavior in survey data: quality issues and concerns
Research on the reliability and validity of self-report has been conducted since the 1940s, but for self-reports of risk behavior, the studies did not begin until the 1980s 21,22,23 and for epidemiologists, interest began a bit later 24 .
Because risk behaviors are often socially disapproved, one concern is that the young respondents may underreport actual behavior, despite the guaranteed anonymity of their responses.As discussed by Barnea et al. 25 , this may happen because of fear of admitting use of illegal substances, or because certain actions are considered shameful.Conversely, some adolescents may "brag" or over-report engaging in disapproved or risky behaviors, perhaps as a result of their desire to conform to the presumed norms of their peers 1,26,27,28,29 .
In addition to verifying these problems of distorted self-presentation, previous research on selfreport indicates other sources of error.Studies have shown that the individuals often fail to judge the frequencies accurately 23,30,31,32,33,34 , and that the extent of error associated with the report of frequency of past behaviors varies intensely among studies.Blair & Button 30 have established three factors that can affect the accuracy of self-reported frequency of past behaviors: the actual frequency of the event (more frequent behaviors are often reported by using estimation methods, rather than on the basis of actual episodic memory), the question wording (the use of "how many times" induces more inaccuracy than the use of "how many"), and the reference time frame (longer time frames increasing the chance of error).More recent research has shown that holding these three factors constant could help improve the accuracy of self-report data 32,35,36 .
Sometimes, the validity of self-reported risk behavior can be assessed by collecting additional, potentially contradictory responses and complementary evidence such as biochemical markers, collateral informant reports and medical interview.Studies of how the validity of self-report can affect the prevalence estimates of risk behaviors (especially for the age at first use of alcohol and drugs) have been intensely developed by epidemiologists and other health sciences researchers, with both community-based and school-based samples 37,38,39,40,41,42,43,44,45  ences were substantially reduced when the respondents who gave multiple inconsistent, or extreme responses to other survey items were screened out of the data.
Inaccurate measures of risk behavior not only bias prevalence estimates, but can also act as confounders for the study of morbidity, mortality, and the social and economic outcomes associated with risk behavior, especially for young samples.As the uncritical acceptance of findings from adolescent surveys can lead to erroneous conclusions, our aim is to analyze the consistency of self-reports of risk behavior on the 2009 and 2012 PeNSE surveys, some of the most commonly used data sources about adolescent health in Brazil.

Data sources and measures
The PeNSE is an ongoing school-based survey, conducted by the Brazilian Ministry of Health, together with the Brazilian Institute of Geography and Statistics (IBGE), to monitor the health of children and adolescents enrolled in the 9 th grade, from public and private schools.In 2009, a sample consisting of 62,910 students from 6,780 schools was drawn in such a way as to be representative of the student population of Brazil's 26 state capitals plus the Federal District.In 2012, the sample was expanded to be representative of Brazil's five regions, now including rural and urban areas of the South, Southeast, North, Northeast, Central and also the 26 state capitals and the Federal District, comprising a total of 109,104 students, from 2,842 schools 2,3 .
In both 2009 and 2012, all students in the selected classrooms who were present on the day of data collection were invited to participate.In 2009, out of the 63,411 students who were present in the classroom, 501 refused to participate (0.8%) and in 2012, from the total of 110,873 students invited, 1,651 refused to participate (1.5%).The 9 th grade was chosen because the students in this grade, mostly aged between 13 and 15 years old, have already acquired the necessary skills to answer self-applicable questionnaires.Other reasons for the choice of 9 th grade were because this group is prone to being exposed to several risk factors, and to permit comparability with various survey systems from other countries 46,47 .
The students answered the questionnaire using a personal digital assistant palmtop, similar to a smartphone.The 2009 questionnaire had 104 questions, involving items about socioeconomic status, social support, bullying, nutritional habits, body image perception, oral health, physical activity, substance use (alcohol, drugs and cigarettes), sexual activity, safety, accidents, and exposure to violence.In 2009, anthropometric measures were collected by the survey team for all the students.
In 2012, questions about asthma, hygiene habits, mental health, work activity, and use of health services were added, summing to a total of 127 questions.Some of the questions from 2009 were altered, and anthropometric measures were not collected in 2012.For further details, see the documentation of the 2009 and 2012 PeNSE survey 2,3 .

Analytical approach
In order to detect inconsistent responses, we created indicator variables for each behavior (called "flag variables"), which took on the value of 1 if a response was logically inconsistent with a previous statement about having ever engaged in that specific behavior.Using these domain-specific indicators of inconsistency, we then created a variable representing the total number of inconsistent responses given by each participant (2009's range = 0-19, 2012's range = 0-23).We also used these indicators to calculate the domain-specific and overall inconsistency rates.
Examples of inconsistencies found in the data included logical forms, such as a report of past use of alcohol at one period, followed by a report of never having drunk alcohol on a subsequent question.The opposite pattern of inconsistency was also considered, where participants who answered "yes" to the "have you ever" question then claimed to have never engaged in that behavior in a subsequent answer.Responses were also flagged as inconsistent when the participant reported a particular age at which he had first engaged in a behavior which was above his current reported age.A list of the Cad.Saúde Pública 2017; 33(4):e00145815 questions inspected for inconsistencies in the 2009 and 2012 editions of the PeNSE survey can be found in Table 1.
In order to explore the patterns of inconsistent responding and identify factors that might account for them, we conducted three types of analyses.First, we determined the rates of inconsistent responses among the participants in 2009 and 2012 for each type of behavior (sexual activity, alcohol, cigarettes and drug use), and across all of the questions.
Second, we calculated the prevalence of alcohol, drug and cigarette use, and sexual activity, after cleaning the data of all the cases with inconsistent responses on these domains, and then compared the results with the prevalences reported in previous studies based on the data from the 2009 4,9,13,14,15 and 2012 6,8,10,12,16 editions of the PeNSE survey.
Third, we estimated a logistic regression model, to measure the relationship between giving an inconsistent response, and various participant characteristics: age (categorized according to norms for the 9 th grade in Brazil 47 as"age appropriate" -ages 13 to 16, "accelerated" -ages 11 or 12, or "delayed" -ages 17 to 19), sex, type of school (private or public), possession of a cellphone, self-report of drug use, and self-reported level of difficulty in answering the questionnaire.As there are strong socioeconomic 48,49 and cultural differences 50,51,52 in risk attitudes across Brazil's five regions, we have also included regional dummies as potential"explanatory" variables.

Results
In 2009, 11.7% of the participants provided inconsistent responses for at least 1 of the 19 questions about risk behavior.In 2012, this percentage increased, with 22.7% of the participants giving an inconsistent response for at least 1 of the 23 questions.Limiting the analysis only to the identical questions in both surveys (19 items), the 2012 edition still showed an inconsistency rate of 22.2%.Table 2 shows the percentage of the participants providing at least one inconsistent response across all items, and for each of the four domains.
Inconsistency rates were higher for the most common behaviors: alcohol (2.7% in 2009 and 12.8% in 2012) and cigarettes (4.3% in 2009 and 2012).It is noteworthy that, in 2012, the question "How old were you when you had your first dose of alcohol?" was alone responsible for 46.6% of all the inconsistencies, but in 2009, inconsistencies on this question constituted only 10.2% of the total.
Only 1.3% of the participants in 2009, and 3.4% of the participants in 2012, were flagged in more than two domains.Inconsistency in one domain was not strongly associated with inconsistency in the others.Most inconsistent responders provided an inconsistent response in only one of the four domains (89.9% in 2009, and 89.1% in 2012).
To analyze how these inconsistency rates may have affected the prevalence estimates of risk behavior which have been published previously on the basis of only the "have you ever" questions 6,7,8,9,10,12,13,14,15,16 , we calculated the percentage of inconsistent responders in relation to "yes" or "no" responses to those "have you ever" questions.In both 2009 and 2012, there was a higher percentage of self-contradiction among the participants who said yes than among those who said no, except for alcohol experimentation.Table 3 shows the percentage of inconsistent responders in both cases.For example, 8% of the participants in 2009, and 9.6% in 2012, claimed to have experimented with drugs, but 23.1% of the 8% (in 2009) and 31.7% of the 9.6% (in 2012) did not corroborate those positive responses on subsequent questions about the frequency, or age at first use.
As shown in Table 4, the removal of inconsistent responders affects the estimated prevalence of these risk behaviors in different directions, and with different magnitudes.
Next, we examined the participant characteristics, and socio-demographic aspects of the sample, to determine which variables were significantly associated with giving an inconsistent response in at least one item.Table 5 shows the results from the final adjusted logistic regression model, as well as the unadjusted coefficients (univariate regression), and the percentages of inconsistencies within each of the independent variables.
Males were significantly more inconsistent than females in their responses, especially in the domain of sexual activity, where males constituted 64.7% of the inconsistent responders in 2009 and 63.1% in 2012.# In 2012 the question was changed to "How old were you when you had your first dose of alcohol?"; ## In 2012 the response options ranged from "7 years or younger" to "17 years or older".

Table 4
Comparative table of the prevalence of risk behaviors before and after removing cases with inconsistency within each domain and with at least one inconsistent response in any domain.Participants who were delayed for the 9 th grade had a higher rate of inconsistency than ageappropriate participants.In 2009, age-delayed participants were two times more inconsistent than age-appropriate respondents (95%CI: 1.96-2.39),and, in 2012, their odds of giving an inconsistent response were 1.13 times higher.Being accelerated (more than two years younger than the expected age for the 9 th grade) was not significantly associated with giving an inconsistent response.
As previous studies have shown that drug users are more prone to being inconsistent responders, we tested this association in the PeNSE data.Our results showed a strong association between selfreport of having experimented with drugs and inconsistent responding.In 2009, these participants were almost four times more likely to be inconsistent than the participants who did not report having ever experimented with drugs (OR = 3.97; 95%CI: 3.71-4.24),an association that decreased in 2012, but remained strongly significant (OR = 2.50; 95%CI: 2.48-2.52).Excluding the inconsistent cases from the explanatory variable (self-report of drug use) self-report of drug use was no longer a significant predictor of inconsistency in the 2009 data (OR = 1.11; 95%CI: 0.99-1.25),but remained significant for 2012.In this case, even when limiting the analysis only to the consistent responders, those who reported having experimented with drugs were significantly more inconsistent than those who have not (OR = 2.07; 95%CI: 1.91-2.25) .
Self-reported level of difficulty in answering the questionnaire was not significantly associated with being inconsistent, but not having a cellphone was.In 2009, there was a 21% lower likelihood of being inconsistent among cellphone owners than among those who had none.This contrast was in the same direction but much smaller in 2012, when having a cellphone was associated with only a 5% lower likelihood of being an inconsistent responder.
Finally, our multivariate results did not reveal strong regional differences in the odds of an inconsistent response.In both 2009 and 2012, the south and southeast regions were the ones with the highest percentages of inconsistent responders, but there were no statistically significant differences.

Discussion
The results of this study show that the majority of the participants in the PeNSE survey provided consistent reports, but that a sizable minority provided one or more self-contradictory responses.Consistent with previous analyses of other survey data, in both editions of the PeNSE survey, we found that the prevalence of inconsistency was higher for males 45,53,54,55 , delayed students 56 , and those who reported having experimented with drugs 26,44,56 .
Cad. Saúde Pública 2017; 33(4):e00145815 Considering that the PeNSE questionnaire is relatively long, and that the use of a smartphone could represent an extra source of difficulty for some of the participants, we tested the effect of possessing a cellphone, and also the self-reported level of difficulty in answering the questionnaire (assessed by the question: "How difficult did you find this questionnaire to answer? 1 -very easy; 2 -easy; 3 -neither difficult nor easy; 4 -difficult; 5 -very difficult).Interestingly, the self-report of difficulty was not significantly associated with being inconsistent, but not having a cellphone was.In 2009, when the percentage of Brazilian children and adolescents who had a cellphone was smaller 57 , there was a 21% lower likelihood of being inconsistent for those who had one.On the other hand, in 2012, when cellphones had become more widely available and hence familiar having a cellphone entailed a decrease of only 5% in the likelihood of being an inconsistent responder.This results suggest that familiarity with cellphone positively influence the consistency rates in both 2009 and 2012, a feature that needs to be taken into consideration in the next editions of the survey while recruiting the participants, by offering assistance to those who are not familiar with the technology.
Consistent with the literature 30,31,32,33,34,35,52,58 , inconsistency rates were higher for the most common behaviors: alcohol and cigarettes, but we found a low percentage of extreme cases of inconsistent responding, with most of the participants being flagged for inconsistency in only one of the four domains.Fewer than 4% of the participants were flagged in more than two domains, consistent with previous estimates of careless responding 59,60 .This finding may suggest that, in the case of the PeNSE survey, the inconsistencies are unlikely to derive from "complete indifference or pervasive carelessness" 60,61 .
Such inconsistency was much more prevalent in 2012, especially on the question about age of first use of alcohol.Inconsistent measures of alcohol consumption can bias the estimates of morbidity, mortality, and other associated outcomes.As pointed out by Kydd 55 , the validity and possible biases of self-report measures of alcohol consumption have been subjects of considerable research attention, and rates of inconsistency in this domain have been found to vary in relation to gender, ethnicity, and socioeconomic status 62,63,64 .
However, because the set of questions about alcohol was not identical in the 2009 and 2012 editions of the PeNSE survey, it is difficult to determine which factors may account for the increase in the inconsistency rate.Nonetheless, it is noteworthy that in the past years several changes were made in the Brazilian regulation for alcohol consumption, especially with the creation of the "Lei Seca" in 2008 -a dry law, different from the old American one -that makes the inspection of drivers stricter in the country, punishing people who are caught having alcohol in their blood when driving.As a consequence of the "Lei Seca", other changes were made on the alcohol regulation in Brazil, with the Law n. 5,502 from 2013 that made underage alcohol consumption illegal, as well as criminalizing the sale of alcoholic beverages to people under the age of 18.With these changes in the federal regulation, the debate about underage drinking gained the attention of the media and became a great topic of discussion in the popular opinion, which might have indirectly affected the students' attitudes in reporting their actual drinking habits.Because underage drinking in Brazil is generally socially disapproved, we believe that this may be affecting the youths' perception of this matter and therefore causing more inconsistencies.
As there was a bigger proportion of inconsistent responses among the participants who claimed that they had "ever" engaged in three of the four types of risk behaviors, it is important to consider how reliance on these "have you ever" questions may have affected estimates of the prevalence of these behaviors among Brazilian middle school students.This prominence of inconsistent responders among the self-proclaimed risk-takers is noteworthy, because excluding those cases changes the estimates of risk behaviors for most of the domains.Most importantly, the results demonstrated that even small percentages of inconsistent responses can change the estimate of reported risk behaviors.Some of our findings replicate prior studies, which have found that excluding the cases that fail consistency checks results in a reduction in estimated rates of risk behaviors 18,44,65 , but our finding that in the specific domains of alcohol use and sexual activity, the removal of inconsistent responders actually elevates estimated rates is apparently unprecedented.
Nevertheless, researchers should be cautious about removing inconsistent cases.The removal of all cases with inconsistency in any domain may increase the researcher's confidence that the reported behaviors are true behaviors, but it may also inadvertently remove valid data in domains where there are no inconsistencies, and thus create a bias towards more normative behavior.More generally, without further research employing methods of validation that are external to the survey itself, we simply cannot know whether the removal of data from respondents who have contradicted themselves generally tends to improve or damage the accuracy of survey-based prevalence estimates.
Our results suggest the need for more experimental studies about the assessment of risk behavior, via school-based surveys, and could be useful for the study of survey administration routines.
This study supports the importance of performing consistency checks of self-report surveys, following the growing body of literature on this topic.As suggested by Meade & Craig 59 , every survey could benefit from incorporating methods of data screening.Some of these methods entail inserting special items or scales (e.g social desirability, lie scales, bogus items) prior to the administration of survey, while others entail post hoc analysis of response patterns after the data collection.In the PeNSE survey, in particular, inconsistencies could be greatly reduced by the simple expedient of giving access to the risk behavior questions only to those participants who answer "yes" to the relevant "have you ever" question; now that the questionnaire is administered with an electronic device, this skipping over can easily be programmed.
Contributors D. O. Ramos, M. Daly and P. Nadanovsky participated in the study conception, data interpretation, write-up of the article and approval of final version for publication.M. L. Seidl-de-Moura and R. T. Jomar contributed in the article write-up and approval of final version for publication.Cad.Saúde Pública 2017; 33(4):e00145815 17Cross & Newman-Gonchar17have shown that the estimated prevalence of risk behaviors, antisocial behaviors, and victimization experi- Cad. Saúde Pública 2017; 33(4):e00145815

Table 1
Inconsistency rates on each question for the 2009 and 2012 data.
Have you ever tried drugs (marijuana, cocaine, crack, cola, loló/ lança-perfume (ether inhalants), ecstasy, oxy, etc)?Yes or No How many times have you used drugs such as marijuana, cocaine, crack, cola, loló/lança-perfume, or ecstasy in the last 30 days?None in the last 30 days; One or two times, Three to five times; Six to nine times; More than 10 times 126 (0.2) 2,241 (2.1) How old were you when you first tried drugs?I have never used drugs; 9 years or younger; 10; 11; 12; 13; 14; 15; 16; 17 or older *** 1,284 (2.1) 2,501 (2.3) How many times have you used marijuana in the last 30 days?** None in the last 30 days; One or two times, Three to nine times; More than 10 times -118 (1.3)How many times have you used crack in the last 30 days?** None in the last 30 days; One or two times, Three Cad.Saúde Pública 2017; 33(4):e00145815

Table 2
Inconsistency rates by domain and for at least one item across all domains (2009, N = 62,910; 2012, N = 109,104).

Table 3
Percentages of inconsistencies among the "positives" and "negatives" for each domain.

Table 1 (continued)
*** In 2012 the response options ranged from "7 years or younger" to "17 years or older" and the question for frequency of drug use was asked after the age of first use;

Table 5
Percentage of inconsistent responses, unadjusted and adjusted analysis of the prevalence of inconsistent responses in at least one item by sex, age, having a cellphone, self-reported drug use, self-reported level of difficulty in answering the questionnaire, type of school (private or public) and region.