Validation of the Online version of the Previous Day Food Questionnaire for schoolchildren

1 Universidade Federal de Santa Catarina, Centro de Ciências da Saúde, Programa de Pós-Graduação em Nutrição. Campus Universitário Trindade, 88040-900, Florianópolis, SC, Brasil. Correspondência para/Correspondence to: R ENGEL. E-mail: <raquelengel_nutri@hotmail.com>. Support: Ministério da Saúde do Brasil, Departamento de Ciência, Tecnologia e Insumos Estratégicos. Article based on the master’s thesis of R ENGEL, entitled “Comparação da avaliação entre as versões on-line e impressa do Questionário Alimentar do Dia Anterior para escolares”. Universidade Federal de Santa Catarina; 2014. Validation of the Online version of the Previous Day Food Questionnaire for schoolchildren


Objetivo
Avaliar a validade da versão baseada na Internet do Questionário Alimentar do Dia Anterior (Online) para escolares do segundo ao quinto ano do ensino fundamental.

I N T R O D U C T I O N
New technologies, such as computers, the Internet, and smartphones are promising alternatives for the collection of data on dietary intake, and present advantages over traditional data collection methods [1,2].
The application of computerized questionnaires to children allows the use of interactive and entertaining resources that motivate their participation [1,2].In addition, it enables questionnaire self-administration and interview standardization, decreases interviewer bias, allows the immediate storage of information in databanks, and decreases the costs associated with researcher training, and data collection and processing.These tools also recreate the context in which meals were performed, helping memory recall, and possibly improving the reliability of the answers obtained [1][2][3].Some computerized tools were developed in different countries with the aim of obtaining data on dietary intake from children, such as the Food Intake Recording Software System (FIRSSt) [4], the Automated Self-Administered 24-Hour Dietary Recall for Children (ASA24-Kids) [5] (USA), the Synchronized Nutrition and Activity Program (SNAP TM ) [6] (England), the Web-based Dietary Assessment Software for Children (WebDASC) [7] (Denmark), the Web-based Food Record (WebFR) [8] (Norway), the Portuguese Self-Administered Computerised 24-hour (PAC24) dietary recall [9] (Portugal), and the Food Intake and Physical Activity of School Children (WebCAAFE) [10,11] (Brazil).
Regardless of the advantages of computerized and Internet-based questionnaires for the evaluation of dietary intake of children and teenagers, validation studies have found these tools to have similar or lower validity than the paper-and-pencil versions [1].This indicates that self-reported errors in dietary intake (omissions and intrusions) remain a problem, unsolved by new technologies, and indicates the need for validation studies on the computer and internet versions.
Considering the importance of validated dietary intake data collection tools in population studies [12], the aims of the present study were: (a) to evaluate the validity of the Previous Day Food Questionaire (PDFQ) Online food questionnaire among school children aged 7 to 12 years old, enrolled in a public school in Florianópolis, in the state of Santa Catarina (SC), using the direct observation of food consumed during school meals as the reference; (b) to evaluate the influence of school children's characteristics on the disagreement of PDFQ Online reports.

M E T H O D S
A validation study was performed using a convenience sample of school children from second to fifth grade, enrolled in a public school within the state teaching network, Florianópolis (SC), Brazil.The study was conducted during November and December 2013 and March 2014.The school was chosen because it was the elementary public school with the best computer room, with a sufficient number of computers, and high-quality internet.It was a part-time school, offering one meal through the National School Feeding Program (mid-morning or afternoon snack), with a menu designed by nutritionists.However, the school children could also buy food from the school cafeteria, or bring food from home.
The sample size was determined a priori, using the method for diagnostic test studies described by Flahault et al. [13], and parameters obtained in a previous validation study of dietary intake data collection tools for school children [14]: an estimated sensitivity of 75%, a margin of error for the lower estimated sensitivity limit of 20%, and a prevalence of 50%.The estimated sample size was 124 individuals [13].Considering the need for stratified analyses (by gender and grade), 248 individuals would be needed.In order to reach this number of participants, 16 classes were selected via balloting, one for each grade (second to fifth) and school shift (morning and afternoon).Of the 476 school children invited to participate in the study, 452 returned the free and informed consent form.Of these, 140 students were excluded because they were not present on the day of observation or questionnaire administration.The final sample consisted of 312 school children.
The questionnaire PDFQ-3 [14] was adapted to an Online format by a web application written using JavaScript and PHP 5.3.Users interact with the tool through a graphic interface based on Adobe Flash technology.Data was stored using a database Management System (MySQL), making it possible to create files that could be exported to other software.
The layout of the PDFQ Online questionnaire is practically the same as the PDFQ-3 version [14], the only difference being the inclusion of a screen between each meal that highlights the type of meal, for easier visualization.This screen was included because, in the pilot study, the school children were observed to have difficulty in visualizing the type of meal.Both versions are composed of three sections: an initial identification section, one about dietary intake, and one about physical activities.The first screen was aimed at student identification, the second contained illustrations of the type of transportation used to go to school, and the subsequent pages presented information about the meals, ordered chronologically (breakfast, mid-morning snack, lunch, afternoon snack, dinner and evening snack).For each meal, illustrations of 21 foods or food groups were presented: bread and biscuits, chocolate milk, coffee with milk, milk, yogurt, cheese, rice, soft drinks, sweets, salty snacks, french fries, pizza and hamburger, fruits, beans, pasta, fish and seafood, beef and poultry, fruit juices, leafy vegetables, cooked vegetables, and vegetable soup.
Because Previous Day Food Questionnaire Online was designed as a Previous Day Food Questionnaire, the direct observation of the food consumed during school meals (midmorning or afternoon snacks) on the day before the questionnaire administration was used as a reference method for validation.This method is considered the gold standard for the validation of dietary intake questionnaires [15,16] because it is a direct measurement and not prone to recall bias [8,14,[15][16][17].
The observations followed a standardized protocol [14] and were performed by graduate students of Nutrition and Physical Education who had received training on how to conduct observations in a non-obstructive manner and the questionnaire administration protocol.A pilot study was then conducted with 69 school children from a municipal school, to assess the interobserver agreement of the protocol and the children's degree of difficulty in using the tool.The degree of agreement between observers and the 'gold standard' observer (the most trained observer) was 97.3%.
During the study, the researchers were placed strategically in the places where the meals took place (schoolyard, canteen, and cafeteria) and observed the children wearing name tags and colored bracelets.The types of food consumed were recorded in a standard form, not considering quantities.
The Previous Day Food Questionnaire Online was administered on the day following the observation, in the school computer room, under the guidance of a researcher who explained the aim of the study using appropriate vocabulary.
The filling of the questionnaire began with identification data (name, school, grade, and school shift) and the means of transportation to school.The researcher then explained each of the six meals, and the foods represented for each one, reminding the children that even if they liked a given type of food, they should only select the items they had in fact consumed on the previous day.
The students filled the PDFQ Online at the end of the school shift (morning or afternoon).
Anthropometric measurements were performed by graduate students of Nutrition and Physical Education, using standard techniques [18].Theoretical and practical workshops were performed previously, to standardize the measurements.Body weight was measured using an electronic scale (capacity of 180kg and readability of 100g; Marte LTDA, Belo Horizonte, Minas Gerais, Brazil).Height was measured using a portable stadiometer attached to the wall, with zero at ground level (readability of 0.5cm).Body Mass Index (BMI) was calculated using the body weight and height values (body weight in kg divided by the square of the height in m).
Gender specific BMI-for-age cut-off points proposed by the World Health Organization were used to classify the children as underweight, normal weight, overweight or obese [19].Two categories were considered in the analyses: non-overweight (including underweight and normal weight children) and overweight (including overweight and obese children).
Validation analyses were conducted considering the food items reported by the school children in the PDFQ Online and observed by the researchers during mid-morning or https://doi.org/10.1590/1678-98652017000500008afternoon snacks.The data for leafy vegetables, cooked vegetables (leafy vegetables/cooked vegetables); salty snacks and french fries (snacks/french fries); and milk, coffee with milk, yogurt, and cheese (dairy) were grouped together.Because preliminary observations in municipal schools indicated that most juices were not fresh (powdered and/or boxed juice), characterizing them as sugary drinks similar to soft drinks, the data on fruit juices and soft drinks were also grouped together (fruit juices/ soft drinks).According to the new food guide for the Brazilian population, both artificial juices and soft drinks belong to the group of ultraprocessed foods [20].
Validity was evaluated by calculating sensitivity and specificity [21], and the agreement rates (match, omission and intrusion rates) [10], using the direct observation of dietary intake at the school as the reference standard.Match rates were defined by the items reported in the PDFQ Online that were also observed directly; omissions as the directly observed items that were not reported in the PDFQ Online; and intrusions as the items reported in the PDFQ Online that were not observed [10].Match, omission and intrusion rates were calculated for each participant and food item/group, using the following equations: match rate = [matches/ (matches + omissions + intrusions)] x 100; omission rate = [omissions/(matches + omissions + intrusions)] x 100; intrusion rate = [intrusions/ (matches + omissions + intrusions)] x 100.
The influence of participant characteristics on omission and intrusion rates was evaluated by calculating the odds ratios and their respective confidence intervals, using multivariate multinomial logistic regression.The agreement rate (matches, omissions, and intrusions) was considered as the dependent variable, using the match rate as the reference.The independent variables included in the multivariate model were gender, age group (7 to 8 years old, 9 to 12 years old), grade (2nd and 3rd, 4th and 5th), school shift (morning, afternoon), and overweight (yes/no).
All analyses were performed using Statistical Package for the Social Sciences (SPSS Inc., Chicago, Illinois, United States) version 21 for Windows (IBM Inc., Armonk, New York, United States).The 95% Confidence Interval (95%CI) was determined for all analyses and used to determine statistical significance.
The study was approved by the Human Research Ethics Committee of the Universidade Federal de Santa Catarina (approval nº 2250/11).The parents signed a Written Consent Form, and the children gave their oral consent.

R E S U L T S
The sample was composed of 312 school children (57.4% girls).Of these, 59.6% were aged between 9 and 12 years, 51.0% were in the 2nd or 3rd grades of elementary school, 50.6% attended school in the afternoon, and 40.1% were overweight (Table 1).
Of the 21 foods or food groups presented in the PDFQ online, 12 were both directly observed and reported by the school children, five were only reported by the school children (coffee with milk, milk, french fries, pasta, fish/ seafood), and one was neither observed nor reported (vegetable soup).
Considering all food items/groups reported and/or observed, sensitivity varied between 66.4% and 72.5%, and specificity between 94.1% and 96.4%, for the independent variables.The stratified analysis according to participant characteristics yielded a sensitivity higher than 60.0%.Sensitivity was higher for children attending school in the morning than in the afternoon (72.5%; (95%CI=67.8-77.3)versus 61.8%; (95%CI=56.1-67.6).Specificity was lowest for the afternoon shift (94.1%), and higher than 95% for the remaining characteristics evaluated (Table 1).
The match, omission and intrusion rates for the 12 food items/groups reported by children in the PDFQ Online are presented in Table 2. Match rates (coincidences between PDFQ Online and direct observations) were around 47.7%, being highest for the group bread and biscuits, followed by beef and poultry, and lowest for the group snacks/french fries.The overall omission rate (items observed directly but not reported in the PDFQ Online) was 22.8%, but with some high variation.It should be highlighted that the omission rates for snacks/french fries and pizza/hamburger were five times higher than the median.This indicates some vulnerability of the PDFQ Online.In contrast, no omissions were observed for beans and leafy vegetables/cooked vegetables.Intrusion rates (items reported in the PDFQ Online but not directly observed) were higher for dairy and chocolate milk (overall intrusion rate of 29.5%).This is another possible critical point of the PDFQ Online.The multivariate logistic regression analysis showed that the probability of intrusion was higher for children attending school in the afternoon than in the morning.The probability of omissions was 21% higher for overweight than for non-overweight children, although this difference was not statistically significant (Table 3).

D I S C U S S I O N
The present study provided evidence on the validity of the PDFQ Online, using direct observation of consumed foods during school meals as the reference.PDFQ Online was found to have a sensitivity level (probability of correctly reporting a consumed food) of 67.7%, considered adequate for dietary intake data collection tools, and a specificity (probability of correctly reporting a non-consumed food) of 95.2%.PDFQ Online therefore possessed a https://doi.org/10.1590/1678-98652017000500008higher ability to detect non-consumed foods (specificity of 95.2%) than consumed foods (sensitivity of 67.7%).This result should be analyzed considering the reference method used, since direct observation was only performed during the meals taken at school (mid-morning or afternoon snack), which include a lower variety of foods than main meals (e.g., lunch or dinner).When all food items/groups were considered, moderate match rates (47.7%), and low omission (22.8%) and intrusion rates (29.5%) were observed for the dietary intake reports.Comparing the different food items/ groups, omission rates were highest for pizza/ hamburger, and intrusion rates highest for dairy.The probability of intrusions was higher for children attending school in the afternoon than in the morning.No significant differences in the probability of omissions or intrusions depending on gender, grade or weight status were observed.
On the positive aspects of the present study, the use of direct observation during school meals by trained observers as the reference for validation should be highlighted.Another positive aspect was the analysis of two different types of error commonly associated with the accuracy of dietary intake reports, i.e., omissions (underreporting) and intrusions (overreporting), and the characteristics of the participants associated with them.
On the study limitations, the fact that the tool's accuracy was evaluated based only on direct observation of food consumed at the school, therefore not encompassing all the meals included in the tool, should be highlighted.The use of multiple imputations, conditioned on information on gender, age, BMI and sociodemographic data, is a promising alternative to estimating the validity of all meals in 24-hour food recalls [22].
Comparing the obtained results with other dietary intake evaluation questionnaires is difficult due to the different experimental designs, evaluated meals, reference methods and statistical methods used.However, the resulting trend can be interpreted through comparison with data from validation studies of other tools such as FIRSSt [4] (for school children aged 8 to 13 years old) and ASA24-Kids [5] (for school children aged 9 to 15 years old).Both studies found a similar match (46% and 37%, respectively), intrusion (24% and 27%, respectively), and omission rates (30% and 35%, respectively) to those observed in the present study.
The validation study of PAC24 [9], conducted with 41 Portuguese school children aged 7 to 10 years old, also used direct observation of school lunches as the reference, and observed a match rate of 67.0%, intrusion rate of 11.5%, and omission rate of 21.5%.The highest omission rates were observed for the vegetables (39.7%) and sweets (15.1%), and the highest intrusion rate for beverages (38.2%) [9].In the present study, the highest omission and intrusion rates were observed for pizza/hamburger (53.2%) and dairy (67.4%), respectively.The sample of children used in the validation of PAC24 was relatively small (n=41) compared to the present study (N=312).In validation studies, it is important to describe the parameters used for sample size calculation.The sample size should be adequate to ensure the power of the statistical test to evaluate the agreement between compared measures [13].
A study of 117 Norwegian school children, 8 to 9 years old, observed higher food item omission rates for overweight children [8].This was not confirmed in the present study.
Overall omission and intrusion rates of 27% and 19%, respectively, were observed for WebFR [8].However, considerably lower omission (3%) and intrusion rates (14%) were obtained for WebDASC (on which WebFR was based) [7].It should be highlighted that for both WebFR [8] and WebDASC [7], the participating children answered the questionnaire on the day of the observation, and were helped by their parents.In the present study, the children answered the questionnaire one day after the observation, and without help, which may explain the higher intrusion rates observed (29.5%).It should also be noted that omission and intrusion rates were calculated differently in the three studies.
In the validation study of SNAP TM [6], with the participation of English school children with an average age of 10.7 years old, the results indicated that the children underreported foods, and the accuracy decreased with the increasing number of times food was selected.This was attributed to curiosity about the tool, leading children to select foods more than once, even without having consumed them.It should be highlighted that in the validation of SNAP TM , the comparison was performed using multiple step 24-Hour Dietary Recall (24HR).Although the 24HR is often used as the reference method, it presents limitations for validation studies, since it is subject to errors related to respondents' characteristics [12,15].The Brazilian tool WebCAAFE was validated in two Brazilian cities, Florianópolis (South of Brazil) and Feira de Santana (Northeast of Brazil) with school children aged 7 to 10 years and 7 to 15 years, respectively, enrolled in public schools, and using direct observation as the reference [10,11].In Florianopólis the agreement rates were similar to those found in the present study, with a match rate of 43%, intrusion rate of 29%, and omission rate of 28% [10].In Feira de Santana, the agreement rates were better compared to the present study, with a match rate of 81.4%, intrusion rate of 7.1%, and omission rate of 16.2% [11].These studies used the same equations for the calculation of agreement rates.
Comparable results were obtained for the validation of the paper-and-pencil version of the tool PDFQ-3 [14] and the PDFQ Online.PDFQ-3 possessed a sensitivity of 72.2% and specificity of 96.2% for all the food items combined [14].PDFQ Online possessed a similar specificity to PDFQ-3, but lower sensitivity (67.7%).It should be highlighted that the PDFQ-3 study differed from the present study in that the children only had access to food served by the school, and had their meals in the cafeteria, making memory recall easier [14], whereas in the present study meals could be had in different school areas.
Other studies [4,7,8,11,14,15] reported higher match rates for older children, probably because their cognitive capacities were more developed [1,14,15].In the present study, the grade was not related to agreement rates, in contrast with the school shift.It should be noted that PDFQ Online was applied during the last class of the school shift, which may have affected the agreement rates.A larger number of meals eaten on the day of questionnaire administration may interfere with the previous day recall [15], especially for children attending school in the afternoon.
The accuracy of dietary intake reports by individuals affects the inferences about dietdisease relationships.Errors in self-reporting may be random or systematic [12].Whereas random errors decrease the statistical power, thereby requiring bigger samples to obtain precise estimates of population characteristics and relationships between variables, systematic errors may affect those estimates [12].Systematic errors may be related to personality characteristics of the respondents, such as social desirability and social approval motivation [15,16].
In a validation study conducted with American children attending the fourth grade and using direct observation of school meals as the reference, the accuracy of dietary intake reports increased with increasing cognitive ability, the accuracy of the food energy reports decreased with increasing social desirability, and energy overreporting decreased with increasing BMI.The relationship between cognitive ability and accuracy of the dietary intake report was stronger for omission than for intrusion rates [16].
In order to accurately answer questions related to past behaviors, individuals should be able to appropriately recall the information required, and use it to formulate the answers.Memory is a component of cognitive ability [16,23].The accuracy of reporting may, therefore, be related to cognitive ability, which may be a source of systematic errors in data originating from dietary intake questionnaires [16].

C O N C L U S I O N
The results of the present study suggest that the PDFQ Online possessed satisfactory validity for the evaluation of dietary intake among school children 7 to 12 years old, attending a public school in Florianópolis (SC).Food item intrusion was the most prevailing error, especially for dairy and chocolate milk.The probability of intrusions was higher for children attending school in the afternoon.Future studies should evaluate the relationship between children's cognitive ability and the accuracy of dietary intake reports in computerized questionnaires.

A C K N O W L E D G M E N T S
The authors wish to thank the teachers, parents, and students of Escola de Aplicação do Instituto Estadual de Educação Florianópolis, Santa Catarina, who participated in the present study, and the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, for the scholarship grants.
C O N T R I B U T O R S R ENGEL was responsible for data collection, analysis and interpretation, and writing of the manuscript.MAA ASSIS developed the measurement instruments, participated in study design, supervised the study implementation and writing the manuscript.AS LOBO participated in data collection and analysis, and writing and revision of the manuscript.FGK VIEIRA and PF DI PIETRO participated in the critical revision of the manuscript.

Table 1 .
Sensitivity and specificity of Previous Day Food Questionnaire (PDFQ) Online according to the characteristics of the study participants (N=312).

Table 2 .
Agreement rates: comparison between foods reported in the Previous Day Food Questionnaire (PDFQ) Online and directly observed during the school meal (N=312).

Table 3 .
Multivariate multinomial regression: variables associated with intrusions and omissions in dietary intake reports by school children in the Previous Day Food Questionnaire (PDFQ) Online, using the match rates as a reference (n=312).