Validity and reliability of foods and beverages intake obtained by telephone survey in Belo Horizonte, Brazil

Abstracts

OBJETIVO: avaliar a reprodutibilidade e validade de indicadores de consumo de alimentos e bebidas levantados por meio de entrevistas telefônicas pelo Sistema de Vigilância de Fatores de Risco e Proteção para Doenças Crônicas por Telefone (VIGITEL). MÉTODOS: A reprodutibilidade foi aferida por meio de entrevistas repetidas com intervalos de 7 a 15 dias a partir da primeira entrevista original do sistema (n=258). A validade foi avaliada usando como padrão ouro 3 entrevistas correspondentes a recordatórios de 24h (R24h) em 3 dias da semana, após 7 a 15 dias da entrevista telefônica original (n=217). No estudo de reprodutibilidade a estatística kappa foi utilizada para medir a concordância entre os resultados das proporções obtidas na primeira e na segunda entrevista. Para a validação, a proporção de referência positiva de alimentos e bebidas na entrevista telefônica original foi comparada àquela obtida nos R24h e sua avaliação foi analisada por meio do cálculo da sensibilidade, especificidade e valor preditivo positivo. RESULTADOS: No estudo de reprodutibilidade, observou-se concordância quase perfeita para o indicador de consumo de leite com teor integral de gordura (0,86); concordância substancial para consumo de frutas (0,67), consumo de frutas e hortaliças, consumo de carne com gordura ou frango com pele e consumo abusivo de bebida alcoólica; concordância moderada para o indicador de consumo de refrigerante; e concordância regular para o indicador do consumo de hortaliças. Com relação ao estudo de validação, observou-se subestimação de indicadores de consumo de frutas, hortaliças e leite integral, exceto para consumo de carne/frango com gordura e bebida alcoólica, quando comparados com a frequência desse consumo estimada a partir dos três R24horas; as diferenças para o consumo de refrigerantes foi inexpressiva. CONCLUSÃO: Verificou-se que os instrumentos utilizados permitem a vigilância de fatores potencialmente associados a doenças não transmissíveis, entretanto, é prioritária a continuidade de estudos de validação em outras populações do sistema, com o objetivo de oferecer maior confiabilidade aos mesmos, sempre considerando as potenciais limitações.

consumo de alimentos; consumo de bebidas alcoólicas; validade dos testes; inquéritos nutricionais; vigilância nutricional

ARTICLES

^IPrograma de Pós-Graduação em Enfermagem - Universidade Federal de Minas Gerais (UFMG) - Belo Horizonte (MG), Brazil

^IIDepartamento de Enfermagem Materno Infantil e Saúde Pública - Escola de Enfermagem da Universidade Federal de Minas Gerais (UFMG) - Belo Horizonte (MG), Brazil

^IIIPost Graduation Program in Public Health of Universidade de São Paulo (USP) - São Paulo (SP), Brazil

^IVGeneral Coordination of Non Communicable Diseases and Conditions - Health Surveillance Secretariat - Ministry of Health - Brasília (DF), Brazil

Correspondence to

ABSTRACT

OBJECTIVE: To evaluate the validity and reproducibility markers of food and beverage intake obtained by means of a telephone-based surveillance system.

METHODS: Reproducibility was assessed by means of repeated interviews with a 7-15 day interval after the first interview of the system (n=258). Validity was analyzed based on three interviews corresponding to the 24 hour recall method, used as gold standard, 3 days a week, 7 to 15 days after the original telephone interview (n=217). In the study of reproducibility, kappa statistics was used to measure the similarity between the results of the proportions obtained in the first and second interviews. For validation, the proportion of positive reports regarding foods and beverages at the original telephone interview was compared to that obtained by the 24 hour recall method, and the results was analyzed by calculating sensitivity, specificity and positive predictive value.

RESULTS: In the reproducibility study, the agreement was almost perfect for the marker milk consumption (0.86), substantial for fruit consumption (0.67), intake of fruits and vegetables, intake of meat with fat and alcohol abuse. The agreement was moderate for the intake of soft drink, and fair for the consumption of vegetables. Regarding the validity, there was an underestimation of fruit, vegetable and milk consumption, except for meat and alcohol when compared with the consumption frequency estimated from the three 24-hours recalls, differences related to soft drink consumption was inexpressive.

CONCLUSION: The instruments enabled the surveillance of factors which were potentially associated with chronic diseases, however, it is a priority to continue with validation studies in other populations to provide greater system reliability, always considering possible limiting factors.

Keywords: food consumption; alcohol abuse; validity of tests; nutrition surveys; nutritional surveillance.

Introduction

Chronic non-communicable diseases (CNCD) have been regarded as importante health problems and their main risk factors are obesity, physical inactivity, alcohol abuse, smoking and poor eating habits. Obesity is one of the main nutritional issues in Brazil, being associated with the increased prevalence of cardiovascular diseases, type II diabetes and some types of cancer^1,2. Literature shows convincing evidence that inadequate eating habits and low rates of physical activity are among the main factors associated with obesity and chronic non-communicable diseases^3,4. Studies also show that the low intake of fruits and vegetables is associated with an increased risk of cancer⁵. The global strategy for healthy eating and physical activity of the World Health Organization (WHO) recommends the frequent intake of fruits and vegetables as sources of micronutrients and fibers^6,7. Thus, food intake constitutes one of the main modifiable risk factors and its assessment has been identified as an important target to prevent obesity and, therefore, other chronic non-communicable diseases^8,9.

Information from health surveillance systems is useful to systematically monitor the frequent intake of these foods by the population. In Brazil, the Telephone-based Surveillance of Risk and Protective Factors for Chronic Diseases (VIGITEL) was established in 2006. Its objective is to analyze the frequency of food and beverage intake and relates such frequency to risk and protective factors of non-communicable diseases in the adult population living in the Brazilian capital, among other issues. Despite being practical, studies of food and beverage intake in large populations may present errors, once the consumption reported by telephone may be subject to bias⁶. Thus, it is important to verify if the information collected by the surveillance systems is reliable in order to properly address the development of policies of health promotion and drug prevention.

The assessment of food intake is conducted with methods that enable the calculation of the nutritional composition of the diet by means of interviews. Among such methods, the 24-hour recall stands out for being fast, bringing about a recent memory related to consumption, besides being the method that is less prone to change eating behavior. When employed as a series, it efficiently estimates usual intake. However, the method has disadvantages such as the dependence on memory and the cooperation of the participant, besides the fact that one 24 hour recall test does not show the usual intake of the individual due to the varied daily food intake⁷.

Until this moment, only one study has been conducted in Brazil with the objective to analyze the validity of standard questionnaires used in telephone surveys of VIGITEL to assess and monitor behaviors and individual conditions as to food and beverage intake⁶, however, the identification of possible assessment error is necessary and justifies the constant validation of the system.

So, the objective of this study was to assess the performance of validity and reproducibility markers of food and beverage intake, conducted by telephone surveys by VIGITEL, in a sample of adults living in Belo Horizonte, Minas Gerais, Brazil.

Methods

This study was conducted with subsamples from the VIGITEl system, conducted with the residents of Belo Horizonte, Minas Gerais, Brazil. The sampling procedures used by the system aimed to obtain probability samples of the population aged 18 years or more, living in households with at least one telephone line in each of the 26 state capitals and the Federal District. More details in another publication¹⁰.

To sum up, telephone lines were drawn in each city and divided into 25 replicates; each one reproduces the same proportion of lines per region of the city or telephone prefix. At the end of each year, there is a sample of 2,000 individuals in each capital of the country¹⁰.

Two samples were used in this study: one with 258 participants (169 women and 89 men), corresponding to the study of reproducibility; and another with 217 participants (150 women and 67 men), corresponding to the validation analysis. The participants accepted to participate in new interviews, as demonstrated in Chart 1. Both samples were sufficient to conduct reproducibility and validation studies, according to Willett and Lenart¹¹, who indicate a sample of 100 to 200 individuals for this type of study.

Indicators of food and beverage intake proposed by VIGITEL¹⁰, presented in Chart 2, were chosen to analyze reproducibility and validity. The system uses indicators of unhealthy eating habits: daily or basically daily consumption of soft drinks, with unrestricted sugar (at least 1 can in >5 days a week); usual consumption (reported by the subject) of saturated fat (red meat with excess fat, without removing its visible fat, or chicken skin); usual consumption (reported by the subject) of whole milk and alcohol abuse (intake of more than 4 doses for women, or more than 5 doses for men, on the same day for the past 30 days, considering a can of beer, a glass of one or a dose of distilled beverage as one dose of alcohol).

Healthy eating indicators were: daily or basically daily consumption (>5 days a week) of fruits and/or natural juice; daily or basically daily (>5 days a week) intake of vegetables (including salads, except for tubers, such as potato and cassava); and daily or basically daily consumption of the markers regarding fruits and vegetables altogether.

The participants of the reproducibility study were contacted again by telephone 7 to 15 days after answering to VIGITEL. At the second interview, participants answered to the same questions regarding food and beverage intake, in the same order. In all cases, the second interviewer was different from the one who had conducted the original interview. Kappa statistics was used to measure the agreement between the results of the proportions obtained in the first and the second interviews. Values higher than 0.80 were considered as almost perfect agreement; from 0.61 to 0.80, substantial agreement; from 0.41 and 0.60, moderate agreement; from 0.21 and 0.40, fair agreement; values below 0.21 were considered as mild agreement¹².

As to validity, the performance of the original questionnaire was compared to the 24hR method, considered as gold standard in this study. The participants were recruited by telephone in up to two weeks (minimum of 7 and maximum of 15 days) after the original VIGITEL interview. They were contacted by a team of previously trained nutritionists. Three 24hR were performed in alternate days, two on week days and one on a weekend or holiday. The intake report in at least one of the 24 hour recalls of meat and/or chicken with fat, whole milk and alcohol (five doses for men, four doses for women) was considered as a positive reference to consumption. For fruits, vegetables and soft drinks, the reported consumption in two or more recalls (two days were equivalent to the consumption of five or more days a week) was also considered positive. When the participants did not spontaneously mention the studied food groups (fruits, vegetables, red meat, chicken, milk, soft drinks and alcohol), they were asked at the end of the interview about whether or not these were present or absent from their diet in the last 24 hours.

The proportion of positive reference of food and drinks in the original telephone survey was compared to that obtained in the 24 hour recall¹³, and its assessment of correct classification regarding the positive intake was analyzed by means of sensitivity and specificity calculations. Besides, the positive predictive value to verify the predictive capacity of VIGITEL questions in relation to three 24 hour recall tests was calculated. Statistical procedures were conducted with the software Stata (version 9.0).

This study was approved by the Research Ethics Committee of Universidade Federal de Minas Gerais, protocol n° 552/08.

Results

The sample consisted mostly of female patients, and 65.5% of the women were in the reproducibility analysis, while 69.1% participated in the validity study. For both studies, mean age was similar to that of the original sample of Belo Horizonte. The validation analysis showed no difference as to the average school years of the participants. The distribution of interviewees according to gender and age group were similar to that of the original VIGITEL interview in both cases. As to the anthropometric measurements in the reproducibility analysis, the height was similar, but the weight and Body Mass Index (BMI) were different from the original sample of VIGITEL. For the validation study, all anthropometric measurements were similar to VIGITEL, as demonstrated in Table 1.

In Table 2, the frequency of food and beverage intake reported in the reproducibility study is demonstrated, with indicators proposed by the system and calculated from the original interview and the second one, conducted after 7 to 15 days. The agreement was satisfactory by the analysis of the kappa coefficient for most of the indicators.

An almost perfect agreement was observed for the indicator "intake of whole milk"; as to the four indicators (intake of fruits, fruit and vegetables, meat with fat and chicken skin and alcohol abuse), the agreement was substantial; the intake of soft drinks presented moderate agreement; and fair agreement was observed for the indicator of intake of vegetables.

Table 3 presents the frequency of individuals exposed to food intake indicators, which was estimated from the original interview and the three recalls. An increased consumption frequency was noted when measured from the three 24 hour recalls, except for the indicator related to alcohol abuse, in which the intake frequency was lower than that reported at the original interview (54.0% versus 25.5%), and the one related to the intake of soft drinks with sugar, which barely varied (from 30.1% to 31.9%).

When the intake frequency was analyzed by gender, fewer women reported removing the fat from the meat or the chicken skin in the original interview (13.4%) when compared to the information reported in the 24 hour recall test (39.2%). Women reported an expressive reduction as to alcohol consumption at the recall analysis (58.9% versus 14.3%).

Sensitivity values of the indicators ranged from 45.2% (alcohol abuse) to 92.1% (intake of whole milk). The sensitivity of the instrument used by the original interview was close to or higher than 70% for most indicators, except for the ones regarding the intake of soft drinks with sugar (47.6%) and alcohol abuse (45.2%). Specificity ranged from 29.5% (intake of vegetables) to 87.5% (alcohol abuse). Positive predictive factors ranged from 22% to 100%, especially for indicators of consumption of vegetables (89%), milk (83%) and alcohol (83.3%).

Discussion

The monitoring of eating habits by means of indicators of fruit and vegetable consumption, considered to be healthy eating markers, is essential because of the convincing evidence regarding its effect on the reduction of morbidity and mortality risks at a population level, besides other benefits related to keeping a healthy weight³. Such activities may reveal important information as to intake trends of these markers, which enable the assessment and proposal of public policies. Results indicated that the questionnaire that analyzed food and beverage intake indicators as protective and risk factors proposed by VIGITEL had proper reproducibility; however, it was not accurate as to specificity. The analysis of positive predictive values, which are important regarding validity, showed that the chances of the telephone survey corresponding to the frequency of real consumption of vegetables, fruits, milk and alcohol was higher than 75% for the whole sample.

In general, validation studies about food intake do not present high performance levels, because some factors such as the memory of the participants, the ability to answer the questionnaire about eating habits and other conditions that occur during interviews may potentially impact the performance of the survey¹⁴.

In this study, the reproducibility analysis in two serial interviews of the VIGITEL system, with an interval of 7 to 15 days, was generally similar to that described by the study conducted in another Brazilian capital with the same objective¹³. In this study, participants were asked about their eating habits, not in terms of quantity, but as to the presence of some foods that are considered as protective or risk factors. Thus, reproducibility levels were substantial (kappa coefficient between 0.6 and 0.8) when related to the intake of meat/chicken, whole milk, fruits and alcohol. On the other hand, there were relatively significant differences between both studies concerning the reproducibility as to the consumption of vegetables and soft drinks. The comparison study showed that reproducibility was substantial for both indicators, and in this study it was regular and moderate for the consumption of vegetables and soft drinks, respectively.

When the frequency of adherence to the proposed indicators were compared in both interviews (the original and the one conducted after 7 to 15 days), results were close. As a consequence of this finding, it is possible to state that reproducibility levels are compatible with similar studies related to questionnaires of food intake patterns that do not assess the quantity, but only the frequency of consumption¹⁵. As to the validity analysis, trends of underestimation were detected for the consumption of fruits, vegetables, meat and milk when compared to the 24 hour recall test. However, overestimation bias was detected in relation to alcohol consumption. This last result may be explained by the questioned period of consumption in the first interview, related to the past 30 days (Figure 2). Another limiting factor is that the question about milk and meat from VIGITEL did not refer to a specific period, and the intake of fruits and vegetables, in some occasions, was not present in two 24 hour recall tests. Considering sensitivity and specificity patterns, there are usually higher values of sensitivity (84% to 92%) for fruits, vegetables and whole milk, and low values for drinks. The questionnaire suggests an accurate performance related to sensitivity in order to detect the adequate intake of healthy markers. However, it presented low sensitivity to detect unhealthy markers, such as the intake of meat/chicken with fat, whole milk, soft drinks with sugar and alcohol.

The fact that women were prevalent in this study can be seen as limiting factor, however, performance differences at VIGITEL regarding food and beverage intake in relation to reproducibility and validity were not important. The 24 hour recall tests performed by telephone was scheduled on the first day of contact, thus, the participant could have been more conscious about eating on the day prior to the interview, which can change patterns of eating habits¹⁶. On the other hand, the recall tests performed in a period of 7 days could make the adherence to the study more difficult, despite being the most adequate strategy. Out of the 522 available subjects, 144 completed only the first 24 hour recall test, and 105 completed two of these tests, especially due to trips, change in work schedule, occasional commitments or for being affected by an illness.

For validation studies, the choice of a reference method is critic. Concerning studies to validate food intake, serial 24 hour recall tests are usually chosen, but golden standard may not be the most accurate denomination, but intermethod validation¹⁷.

The conclusions of this study support the strategy used by VIGITEL to identify the trends of food and beverage intake indicators related to non-communicable diseases in Belo Horizonte, because the reported questions are reasonable as to reproducibility and validity. Thus, the instruments used enable the surveillance of factors that are potentially associated with non-communicable diseases. However, it is essential to continue with validation studies in other populations of the system in order to prove the system is reliable, always taking its potential limitations into account.

References

• 1. Renehan AG, Tyson M, Egger M, Heller RF, Zwahlen M. Body-mass index and incidence of cancer: a systematic review and meta-analysis of prospective observational studies. Lancet. 2008;371(9612):569-78.
• 2. Musaad S, Haynes EN. Biomarkers of obesity and subsequent cardiovascular events. Epidemiol Rev. 2007;29:98-114.
• 3. Rolls BJ, Ello-Martin JA, Tohill BC. What can intervention studies tell us about the relationship between fruit and vegetable consumption and weight management? Nutr Rev. 2004;62(1):1-17.
• 4. Swinburn BA, Caterson I, Seidell JC, James WP. Diet, nutrition and the prevention of excess weight gain and obesity. Public Health Nutr. 2004;7(1A):123-46.
• 5. Block G, Patterson B, Subar AF. Fruit, vegetables, and cancer prevention: a review of the epidemiological evidence. Nutr Cancer. 1992;18(1)1-29.
• 6. Philippi ST, Latterza AR, Cruz ATR, Ribeiro LC. Pirâmide alimentar adaptada: guia para escolha dos alimentos. Rev. Nutr. 1999;12(1):65-80.
• 7. EUA. Department of Health and Human Services and U.S. Department of Agriculture. Dietary Guidelines for Americans. Washington, DC: U.S. Government Printing Office; 2005.
• ⁸
  World Health Organization. Diet nutrition and theprevention of chronic diseases: report of a joint WHO/FAO Expert Consultation. Geneva: World Health Organization; 2002. (WHO Technical Report Series, 916).
• 9. Buzzard M. 24-hours dietary recall and food record methods. In: Willett WC. Nutritional Epidemiology. 2nd ed. Oxford: Oxford University Press; 1998. p. 50-73.
• 10. Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Secretaria de Gestão Estratégica e Participativa. VIGITEL Brasil 2009. Vigilância de fatores de risco e proteção para doenças crônicas por inquérito telefônico. Estimativas sobre frequência e distribuição sócio-demográfica de fatores de risco e proteção para doenças crônicas nas capitais dos 26 Estados brasileiros e no Distrito Federal em 2009. Brasília: Ministério da Saúde; 2010. (Série G. Estatística e Informação em Saúde).
• 11. Willett W, Lenart E. Reproducibility and validity of food-frequency questionnaires. In: Willett WC. Nutritional Epidemiology. 2nd ed. Oxford: Oxford University Press; 1998. p. 101-147.
• 12. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-74.
• 13. Monteiro CA, Moura EC, Jaime PC, Claro RM. Validade de indicadores do consumo de alimentos e bebidas obtidos por inquérito telefônico. Rev. Saúde Pública. 2008;42(4):582-9.
• 14. Block G, Hartman AM. Issues in reproducibility and validity of dietary studies. Am J Clin Nutr. 1989;50 Suppl 5:1133-8.
• 15. Nelson MC, Lytle LA. Development and evaluation of a brief screener to estimate fast-food and beverage consumption among adolescents. J Am Diet Assoc. 2009;109(4):730-4.
• 16. Buzzard IM, Faucett CL, Jeffery RW, McBane L, McGovern P, Baxter JS, et al. Monitoring dietary change in a low-fat diet intervention study: advantages of using 24-hour dietary recalls vs food records. J Am Diet Assoc. 1996;96(6):574-9.
• 17. Sklo M, Javier Nieto J. Epidemiology: Beyond the Basics. 2nd ed. Boston: Jones and Bartlett Publishers; 2007.

Validity and reliability of foods and beverages intake obtained by telephone survey in Belo Horizonte, Brazil

Larissa Loures Mendes^I; Suellen Fabiane Campos^I; Deborah Carvalho Malta^{II, III}; Regina Tomie Ivata Bernal^III; Naíza Nayla Bandeira de Sá^IV; Gustavo Velásquez-Meléndez^II

Publication Dates

• Publication in this collection
  05 Oct 2011
• Date of issue
  Sept 2011

History

• Received
  05 Jan 2011
• Accepted
  12 Mar 2011
• Reviewed
  28 Feb 2011