Development, content validity, and reliability of an instrument to assess the commercialization of food and beverages in school canteens in BrazilABSTRACT
Objectives: To develop, verify content validity, and evaluate the reliability of an instrument for assessing the commercialization of food and beverages in school canteens.
Methods: The preliminary version of the instrument was developed following a literature review, identification of food environment dimensions, item development, and structuring of the instrument, which was divided into three sections: identification and characterization of canteens, commercialized foods, and food advertising. The instrument was evaluated by experts, and the content validity index was calculated for each item (deemed satisfactory if >0.80). Reliability was assessed in a sample of canteens using inter-rater and intra-rater tests. Values of kappa, prevalence-adjusted bias-adjusted kappa, and intraclass correlation coefficient were calculated.
Results: The instrument included 945 items. The average content validity index score of the instrument, considering the evaluated aspects (relevance, clarity of item wording, and clarity of response options), was 0.95. Experts’ suggestions were reviewed, and items were added, removed, or modified when appropriate. Regarding reliability, 95.2% of the total analyzed items showed excellent, very good, or good agreement (≥0.6).
Conclusion: The instrument proved to be reliable for measuring the food environment in school canteens in the Brazilian context, showing satisfactory values for both content validity and reliability. Thus, the instrument contributes to advancing the assessment of food commercialization in canteens, an important aspect of the school food environment.
Keywords:
School meals; Environment; Reliability and validity; Food social space
RESUMO
Objetivos: Elaborar, verificar a validade de conteúdo e avaliar a confiabilidade de um instrumento de avaliação da comercialização de alimentos e bebidas em cantinas escolares.
Métodos: A versão preliminar do instrumento foi construída após revisão de literatura, identificação das dimensões do ambiente alimentar, elaboração dos itens e estruturação do instrumento, dividido em três seções, quais sejam: identificação e caracterização da cantina, alimentos comercializados e publicidade dos alimentos comercializados. O instrumento foi avaliado por especialistas e foi calculado o índice de validade de conteúdo (IVC) para cada item do instrumento (satisfatório >0,80). A confiabilidade foi examinada em uma amostra de cantinas por meio do teste interobservador e intraobservador. Foram calculados valores de kappa, PABAK e coeficiente de correlação intraclasse.
Resultados: O instrumento contemplou 945 itens. A média de pontuação do IVC do instrumento, considerando os aspectos avaliados (relevância, clareza da escrita dos itens e das suas opções de respostas) foi de 0,95. As sugestões dos especialistas foram avaliadas e os itens foram incluídos, retirados ou alterados quando pertinente. Com relação à confiabilidade, 95,2% do total dos itens analisados apresentou concordância excelente, muito boa ou boa (≥0,6).
Conclusão: O instrumento apresentou validade de conteúdo adequada e confiabilidade satisfatória no contexto das escolas estudadas para mensurar o ambiente alimentar de cantinas escolares. Dessa forma, contribui para o avanço da aferição da comercialização de alimentos em cantinas, um importante aspecto do ambiente alimentar escolar.
Palavras-chave:
Alimentação escolar; Ambiente; Confiabilidade e validade; Espaço social alimentar
INTRODUCTION
The environments where people are inserted influence food choices and food consumption in various ways, through a set of elements that encompass physical, economic, political, and sociocultural aspects of the environments1,2,3. Thus, the school food environment is defined as spaces, infrastructure, and conditions, inside and around schools, in which food is available and can be obtained, purchased, or consumed: canteens, cafeterias, vending machines, and street vendors, as well as any facility where food is commercialized. It also includes information, promotion (marketing, advertising, brands, labels, packaging) and pricing for food4.
In this context, school canteens stand out as spaces within the schools destined for the commercialization of food and beverages for the school community5. They influence food choices, being accessible to 31.4% of students from public schools and 88% of students from private institutions in Brazil, according to the 2019 National Survey of School Health (Pesquisa Nacional de Saúde do Escolar - PeNSE)6.
The commercialization of unhealthy foods in schools is associated with higher consumption of these foods among students7,8,9. In Brazil, an association was verified between the regular purchase of snacks in canteens and a higher chance of sugary drinks consumption by adolescents10 and the high prevalence of ultra-processed food commercialization in private schools11.
Taking this into consideration, the importance of instruments with good psychometric properties to evaluate food commercialization in school canteens is highlighted. Studies that validate or verify the reliability of these instruments are still scarce. Authors of studies conducted in Australia12,13,14,15, New Zealand16, South Africa17,18, and Norway19 assessed the validity and/or reliability of instruments applied to different aspects of the school environment. In Brazil, there are no validated instruments to accurately and reliably measure food commercialization in school canteens, making it difficult to compare studies20 and limiting the advancement of scientific knowledge21.
In view of this gap and the impact of the school food environment on food consumption of children and adolescents, in the present study, we aimed to develop, verify content validity, and evaluate the reliability of an instrument to assess the commercialization of food and beverages in these spaces.
METHODS
Developing the instrument
The instrument was developed within the context of the study Food Commercialization in Brazilian Schools (Comercialização de Alimentos em Escolas Brasileiras - Caeb), whose main objective was to evaluate the commercialization of food and beverages in private schools in Brazilian capitals22.
The first stage for developing the instrument was the review of existing instruments that evaluate the school food environment and encompass dimensions or items applicable to the Brazilian context. The review was conducted by two researchers in the Medical Literature Analysis and Retrieval System Online (MEDLINE, via the United States National Library of Medicine - PubMed) and the Scientific Electronic Library Online (SciELO) databases in 2022. The preliminary versions of the instrument were elaborated based on the identification of the dimensions, formulation of the items, and structuring of the instrument.
Among the dimensions identified are availability (which includes the variety of available foods), affordability (price), convenience (payment methods, opening hours)23, nutritional information1, food promotion (advertising)1,3,24,25,26, infrastructure for preparing meals, governance27, and policies3. The instrument was elaborated based on the following instruments: “Restaurant Observation Tool: ESAO-restaurants”28; “Food environment audit instrument based on the new food classification of the Dietary Guidelines (NOVA) - AUDIT-NOVA”29; “University food environment assessment instrument”30; the instrument of PeNSE6; and the instrument of the project “Food security in school canteens in the municipalities of the territory of citizenship of Noroeste Colonial/RS”31. Food-related items were elaborated based on the recommendations of the Dietary Guidelines for the Brazilian Population32 and the “University food environment assessment instrument”30.
The instrument was organized into two sections: Section 1, intended for the identification and characterization of canteens; and Section 2, focused on the commercialized foods. After developing the preliminary versions, experts evaluated the instrument. This process consists in the prototypical phase, essential for the refinement of the initial drafts and the creation of an appropriate instrument for the proposed measurement21.
The instrument was developed for use by researchers, civil society, the government, and other stakeholders, aiming to evaluate the commercialization of food in school canteens focusing on adequate and healthy diet. To this end, it is imperative to understand how canteens operate, as their organization and operation influence the availability of food.
The instrument does not generate a single score or a general classification, as each dimension can be analyzed separately according to the evaluated aspect. However, based on the results of Section 2 concerning the availability of food, the healthiness index has been calculated33. This index is obtained by the sum of the availability of fresh, minimally processed, processed foods, and culinary preparations based on these foods and the non-availability of ultra-processed foods and culinary preparations based on these foods. The final value ranges from 0 to 100, indicating greater adequacy the closer to 10033.
Experts’ evaluation
It is recommended that five to 20 experts evaluate the instrument34,35. In this study, 12 researchers in the area of nutrition and linked to different research institutions, specialized in Epidemiology, Public Health, Food Environments, Collective Eating, or Gastronomy, were invited to participate in the evaluation. All of them had experience in the study topics and/or in the development of instruments.
The preliminary version was analyzed at a single moment regarding the relevance of the items for the study objectives, clarity of writing, and absence of ambiguity, based on a four-point Likert scale, in which: 1=nonequivalent item; 2=item requires major review to evaluate equivalence; 3=equivalent item, requires minor changes; and 4=absolutely equivalent item36. The evaluators wrote down comments/suggestions. This stage, with a qualitative approach, integrates the prototypical phase of the elaboration of the instrument21.
Pretest
The pretest was carried out in a school with characteristics similar to the schools of the Caeb study sample, with the printed instrument, and was performed by a qualified researcher.
Reliability of the instrument
The reliability of the instrument was evaluated in a convenience sample composed of public and private schools of a capital and a municipality of its metropolitan region. The estimate of the sample of evaluated canteens (n=13) was performed based on the criteria proposed by Hulley et al.37 for calculating the correlation coefficient. A mean correlation of 0.70 was considered, which is classified as good agreement according to the criteria established by Byrt38. A 5% significance level and a test power of 80% were adopted. A total of 28 canteens were invited to participate in the study.
The collection took place between September and December 2023 via the Collect Box39 application and was applied by two qualified researchers with experience in school food environment. The canteens were evaluated three times40,41, namely: by the two researchers on the same day, with an interval of up to 30 minutes (inter-rater test), and by one of them once again after 21 days, on average (intra-rater test).
The collection combined different strategies: some pieces of information were obtained from an interview with managers or canteen employees, while others were obtained from direct observation (audit). To ensure the standardization of procedures, the collection handbook was used, also available as supplementary material.
Data analysis
After obtaining the experts’ feedback, the Content Validity Index (CVI) was applied for quantitative evaluation42. CVI measures the proportion of experts who agree on aspects concerning the items43 based on the aforementioned four-point Likert scale36. Items rated 1 or 2 should be reviewed or excluded. The CVI of each item is calculated by the ratio between answers 3 and 4 and the total answers, and an index of ≥0.80 is acceptable, preferably >0.9044.
For reliability, categorical variables were analyzed by percentage agreement (PA45), kappa and prevalence-adjusted bias-adjusted kappa (PABAK46), and continuous variables (variety, size, and prices) by the Intraclass Correlation Coefficient (ICC47). The PABAK was included as a complementary measure for correcting distortions in situations of extreme prevalence, conferring greater robustness to the interpretation of agreement. The values of PA, kappa, PABAK, and ICC were classified according to the criterion of Byrt38 as: excellent agreement (≥0.92), very good agreement (0.80 to 0.91), good agreement (0.60 to 0.79), reasonable agreement (0.40 to 0.59), superficial agreement (0.20 to 0.39), poor agreement (0.00 to 0.19), and no agreement (<0.00). Kappa and PABAK values were not calculated for items that were not commercialized in any canteen, as the statistics could not be computed due to absence of one or more levels in the cross-reference tables.
The analyses were performed using the Statistical Package for the Social Sciences (SPSS 21)48 software and the Software for Statistics and Data Science (Stata/SE 15)49. This study complied with the ethical precepts of the Declaration of Helsinki and Resolutions No. 466/2012 and 510/2016 of the National Council of Health, and was approved by the Research Ethics Committee of Universidade Federal de Minas Gerais (Opinion No. 5.240.459). The study participants were experts and managers of canteens who consented to participate and registered their acceptance in the Informed Consent Form.
Data Availability Statement:
The entire dataset that supports the results of this study was published in the article and in the “Supplementary Material” section.
RESULTS
Experts’ evaluation
83.3% (n=10) of the invited experts participated in content validity. The agreement was satisfactory, with a CVI mean of 0.9. The averages per section are presented in Table 1 of the Supplementary Material.
Based on the evaluation, the items were reviewed and adjusted according to the relevant suggestions, discussed by the study coordinators. The main changes were: inclusion of the question about the existence of the sales strategy, present in the dimension “food promotion”; inclusion of the item on the existence of meal offer and its price in the canteen; exclusion of four items related to the canteen’s identification; and reorganization of the size and price of foods and beverages - previously, the goal was to know the single size (when there was only one food/drink option), minimum and maximum, as well as the single price (when there was only one food/drink option), minimum and maximum. The final version included the size of the lowest-price food and the lowest price. The dimension of food advertising was reorganized and became the goal of the third section.
Experts recommended the elaboration of a comprehensive collection handbook, with guidance on the measurement of the variety, unit of measurement (weight, volume, or unit), and examples for the classification of foods as to their processing degree (according to the data collection handbook). The changes are described in Table 1 of the Supplementary Material.
Pretest
According to the pretest, the items were understood and properly filled out, without the need for changes. The sequence of the sections proved to be pertinent, in addition to the content. Hence, the instrument was programmed in an application, with items dynamically configured and conditioned to previous responses, which conferred greater practicality and reduced typing errors.
Final version of the instrument
The final version of the instrument considered the following dimensions: food and nutritional information, food infrastructure, convenience, affordability, availability, variety, and promotion. It comprised 945 items distributed in three sections:
-
1. Identification and characterization of canteens;
-
2. Commercialization, variety, size, value of the lowest price, and sales strategy for foods, beverages, and culinary preparations; and
-
3. Presence of advertising of these items.
The first section was composed of 95 items related to: administrative aspects, presence of nutritionist, type of meals, place of preparation, availability of menus and nutritional information, structure for consumption, commercialization of preparations for special diets (for example: allergies, intolerances, diabetes), availability of condiments, food criteria and prohibitions, incentives from suppliers, educational actions and materials on healthy diet, food price and sales strategies carried out by other members of the school community outside the canteen.
The second section was composed of 300 items encompassing the presence, variety, size of the lowest-price food, lowest price, and sales strategies (such as combos or promotions) for 50 foods and beverages. Of these, 21 are fresh, minimally-processed, or processed foods and culinary preparations based on these foods, and 29 are ultra-processed foods and culinary preparations based on these foods.
The third and last section concerns the possible marketing communication strategies of the 50 foods and beverages commercialized, comprising 550 items. 11 types of advertising strategies, such as banners, clothing, replicas, menus, among others, are verified. The complete instrument is available in the Supplementary Material.
Reliability of the instrument
The collection included 14 canteens, with average interview duration of 30 minutes in private schools and 20 minutes in public schools. The instrument was reapplied, on average, after 21 days, with an interval of 19 to 24 days. In section 1, we could not calculate kappa, PABAK, and ICC for 33 items due to the absence of one or more levels in the cross-reference tables, given the low number of occurrences. The same occurred in section 2, with 124 items. In addition, 50 items were not included in the sample, because ten foods from the instrument were not available in the canteens, making it impossible to evaluate variety, size, price, combos, and promotions. In section 3, no item allowed us the calculation of these measurements, but the percentage of agreement in the inter-rater and intra-rater test was excellent (100%). Thus, we evaluated the reliability of 188 items in total.
In the overall performance, 95.2% of the total items analyzed (n=188) showed excellent, very good, or good percentage agreement. In the inter-rater test, considering the analyses for categorical variables, among the 112 items evaluated, 90.1% had excellent agreement and 9.9% had very good or good agreement for kappa (ranging from 0.75 to 1.00). For the PABAK, 89.3% of the items had excellent agreement and 9.8%, very good or good agreement. We observed null agreement in only 0.9% of the items, specifically in the PABAK analysis in the item on commercialization of 100% fruit juice in box, can, or bottle (ranging from -0.28 to 1.00) (Table 1). According to the ICC calculated for the items “number of clients served per day” and for the variety, size, and price of foods/meals in the inter-rater analysis, we verified a variation from 0.0 to 1.0 (92.1%≥0.92; 2.6% between 0.6 and 0.91; 1.3% between 0.2 and 0.39 and 3.9%<0.0). Of the total items evaluated (n=76), 92.1% presented excellent agreement and 2.6%, very good or good agreement (Table 2). Only 1.3% of the items showed superficial agreement (pizza variety with ultra-processed filling) and 3.9% of the items showed no agreement (variety of fruit candies; size of fresh fruit, and variety of sandwich with ultra-processed filling).
In the intra-rater test, for the 76 items in which ICC was calculated, 97.3% presented excellent agreement, 2.7% presented very good and good agreement (ranging from 0.61 to 1.00), and no item had reasonable, superficial, poor, or null agreement (Table 2). Considering the analyses with categorical variables, of the total 112 items evaluated for kappa, 95.5% presented excellent agreement (≥0.92); 3.6%, very good and good agreement (ICC between 0.6 and 0.91); and 0.9%, no agreement (ICC<0.0). With regard to PABAK, 94.6% of the items were found to have excellent agreement and 5.4%, very good or good agreement, ranging from 0.8 to 1.0 (Table 3). Considering the kappa, we observed no agreement in 0.9% of the items, specifically in the item that evaluates permission according to municipal or state legislation as a criterion for food commercialization. Evaluation details can be seen in Tables 2, 3, and 4 of the Supplementary Material.
DISCUSSION
In this study, we developed, verified the content validity and the reliability of an instrument to evaluate the commercialization of food and beverages in school canteens. In the studied contexts, the instrument presented satisfactory content validity and reliability, and can be used in studies on food commercialization in school canteens.
The identification and characterization of canteens allows us to trace the profile of these establishments and understand the foods commercialized in this important organizational space. Authors who develop instruments to evaluate food environments organize items thoroughly, ensuring that each aspect to be evaluated is identified and that the dimensions of the food environment are clearly defined28,29,30. The foods present in the instrument were selected based on national research and the foods most frequently commercialized in canteens6,50,51, taking into account the NOVA classification52 and the current guidelines of the Dietary Guidelines for the Brazilian Population32. Information on variety, price, sales strategies, and advertising of commercialized foods allow a detailed assessment of the school food environment4,53, analyzing the obstacles and enablers that children and adolescents face when making decisions in these spaces.
Although over 90% of the items presented satisfactory reliability, a few items were poorly evaluated. Among them, we highlight those related to the commercialization of 100% juice in box, can, or bottle; the variety of pizzas with ultra-processed filling; the variety of fruit candies; the size of fresh fruit; the variety of sandwiches with ultra-processed filling; and to the criterion of permission to commercialize food based on municipal or state legislation, whose unsatisfactory reliability can be explained by differences in the interpretation and application of these laws between schools, an issue that emphasizes the inconsistency in the implementation of the rules.
In the inter-rater test, the differences can be attributed to the interpretation of the criteria by the evaluators, the lack of clarity or ambiguity in the instructions, or to variations in the observation and recording of the data, indicating the need for improvement of the training carried out. Other authors also encountered this scenario30,54,55, which demonstrates the complexity of measuring different dimensions of the food environment related to the characteristics of the items, which may involve subjectivity in the measurement or present variation in the availability of items.
In the intra-rater test, the results were even more satisfactory. Differences can be attributed to changes in the environment, such as availability and variety of foods, or to the behavior of participants between the two test applications. In the study conducted by Franco et al.30, whose objective was to evaluate content validity and reliability of an instrument for assessing the Brazilian university food environment, the differences found were due to variations in the environment, specifically in food offer. Borges et al.29, when assessing the reliability of a consumer food environment audit instrument, found small changes in the availability, price, quantity of brands, and advertising strategies of food. It is worth mentioning that the authors of the aforementioned study evaluated establishments such as supermarkets, hypermarkets, and markets, which show greater variation in price, brands, and advertising. This differs from the present study and from the study by Franco et al.30, who analyzed establishments such as school canteens and university cafeterias/restaurants, both inserted in the organizational environment, where the variety of the menu does not usually suffer as many changes and price readjustments are usually performed less often, in more specific moments such as the back-to-school period.
Still regarding the intra-rater test, authors of the study on the validity and reliability of the Perceived Nutrition Environment Measures Survey (NEMS-P) instrument56, translated and adapted to Brazil, also found acceptable ICC values. However, as observed in the present study, some dimensions showed lower performance - in the case of NEMS-P, issues related to prices, marketing in supermarkets, and food promotion, which showed low or even negative values. The authors attributed these results to the context of inflation in the country, the difficulties understanding the questions, and the limitations of psychometric methods to capture individual perceptions56.
Authors of a systematic review57, aimed at classifying and evaluating the quality of the methods for measuring the school food environment, identified that few studies carried out the stages of elaboration and evaluation of content validity and reliability of the instruments57. We emphasize the originality of the present study by taking all these steps for a specific instrument for measuring food commercialization in school canteens.
Among the strengths of the study, we mention content validity through qualitative and quantitative analysis and the evaluation of the reliability of the instrument in public and private schools, thus covering the two contexts of the school food environment. In addition, the instrument included a broad list of foods, based on the NOVA classification, and incorporated detailed information on the commercialization of food in canteens. The instrument includes questions for identifying the influence of the food industry on schools, enabling a more critical analysis of this context.
Among the limitations, we highlight the period of data collection, which lasted three months, not allowing the identification of seasonal differences. However, the study included two cities with distinct socioeconomic and demographic characteristics, which adds value to the analysis. For some of the items for which kappa and PABAK would be calculated, these statistics were not computed, which may be related to the reduced number of canteens evaluated, although the sample size has shown sufficient statistical power for the analyses. Furthermore, canteens of public schools were evaluated, where food commercialization is less frequent, as a result of the National School Feeding Program. In these schools, human resources limitations impact the structure of canteens, resulting in a high number of negative responses in the application of the instrument.
Within the context of the surveyed schools, the instrument proved to be reliable to measure the commercialization of food in school canteens, indicating applicability to similar contexts. Its use contributes to the advancement in assessing the food environment of these canteens, allowing detailed monitoring and providing subsidies for interventions that promote healthier environments and adequate diet for students.
Supplementary Materials
Supplementary material 1
Supplementary material 2
REFERENCES
-
1. Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: concepts and measures. Am J Health Promot 2005; 19(5): 330-3, ii. https://doi.org/10.4278/0890-1171-19.5.330
» https://doi.org/https://doi.org/10.4278/0890-1171-19.5.330 -
2. Story M, Nanney MS, Schwartz MB. Schools and obesity prevention: creating school environments and policies to promote healthy eating and physical activity. Milbank Q 2009; 87(1): 71-100. https://doi.org/10.1111/j.1468-0009.2009.00548.x
» https://doi.org/https://doi.org/10.1111/j.1468-0009.2009.00548.x -
3. Swinburn B, Sacks G, Vandevijvere S, Kumanyika S, Lobstein T, Neal B, et al. INFORMAS (International Network for Food and Obesity/non-communicable diseases Research, Monitoring and Action Support): overview and key principles. Obes Rev 2013; 14 Suppl 1:1-12. https://doi.org/10.1111/obr.12087
» https://doi.org/https://doi.org/10.1111/obr.12087 -
4. Food and Agriculture Organization of United Nations. FOA school food and nutrition framework [Internet]. Rome: Food and Agriculture Organization of United Nations; 2019 [cited on Mar. 21, 2024]. Available at: https://openknowledge.fao.org/server/api/core/bitstreams/6f3162ea-1c1f-4699-a4b1-59a041e5f113/content
» https://openknowledge.fao.org/server/api/core/bitstreams/6f3162ea-1c1f-4699-a4b1-59a041e5f113/content -
5. São Paulo. Portaria Conjunta COGSP/CEI/DSE, de 23 de março de 2005. Normas para funcionamento de cantinas escolares [Internet]. 2005 [cited on Mar. 29, 2024]. Available at: https://www.educacao.sp.gov.br/a2sitebox/arquivos/documentos/953.pdf
» https://www.educacao.sp.gov.br/a2sitebox/arquivos/documentos/953.pdf -
6. Instituto Brasileiro de Geografia e Estatística. Pesquisa nacional de saúde do escolar: 2019 [Internet]. Rio de Janeiro: IBGE; 2021 [cited on Feb. 10, 2024]. Available at: https://biblioteca.ibge.gov.br/visualizacao/livros/liv101852.pdf
» https://biblioteca.ibge.gov.br/visualizacao/livros/liv101852.pdf -
7. Leite MA, Azeredo CM, Peres MFT, Escuder MML, Levy RB. Availability and consumption of ultra-processed foods in schools in the municipality of São Paulo, Brazil: results of the SP-Proso. Cad Saude Publica 2022; 37(suppl 1): e00162920. https://doi.org/10.1590/0102-311X00162920
» https://doi.org/https://doi.org/10.1590/0102-311X00162920 -
8. Clinton-McHarg T, Janssen L, Delaney T, Reilly K, Regan T, Nathan N, et al. Availability of food and beverage items on school canteen menus and association with items purchased by children of primary-school age. Public Health Nutr 2018; 21(15): 2907-14. https://doi.org/10.1017/S1368980018001726
» https://doi.org/https://doi.org/10.1017/S1368980018001726 -
9. Azeredo CM, Rezende LFM, Canella DS, Claro RM, Peres MFT, Luiz OC, et al. Food environments in schools and in the immediate vicinity are associated with unhealthy food consumption among Brazilian adolescents. Prev Med 2016; 88: 73-9. https://doi.org/10.1016/j.ypmed.2016.03.026
» https://doi.org/https://doi.org/10.1016/j.ypmed.2016.03.026 -
10. Rocha LL, Pessoa MC, Gratão LHA, Carmo AS, Cordeiro NG, Cunha CF, et al. Characteristics of the school food environment affect the consumption of sugar-sweetened beverages among adolescents. Front Nutr 2021; 8: 742744. https://doi.org/10.3389/fnut.2021.742744
» https://doi.org/https://doi.org/10.3389/fnut.2021.742744 -
11. Carmo AS, Assis MM, Cunha CF, Oliveira TRPR, Mendes LL. The food environment of Brazilian public and private schools. Cad Saúde Pública 2018; 34(12): e00014918. https://doi.org/10.1590/0102-311X00014918
» https://doi.org/https://doi.org/10.1590/0102-311X00014918 -
12. Reilly K, Nathan N, Wolfenden L, Wiggers J, Sutherland R, Wyse R, et al. Validity of four measures in assessing school canteen menu compliance with state-based healthy canteen policy. Health Promot J Austr 2016; 27(3): 215-21. https://doi.org/10.1071/HE16053
» https://doi.org/https://doi.org/10.1071/HE16053 -
13. Nathan N, Wolfenden L, Morgan PJ, Bell AC, Barker D, Wiggers J. Validity of a self-report survey tool measuring the nutrition and physical activity environment of primary schools. Int J Behav Nutr Phys Act 2013; 10: 75. https://doi.org/10.1186/1479-5868-10-75
» https://doi.org/https://doi.org/10.1186/1479-5868-10-75 -
14. Finch M, Sutherland R, Harrison M, Collins C. Canteen purchasing practices of year 1-6 primary school children and association with SES and weight status. Aust N Z J Public Health 2006; 30(3): 247-51. https://doi.org/10.1111/j.1467-842x.2006.tb00865.x
» https://doi.org/https://doi.org/10.1111/j.1467-842x.2006.tb00865.x -
15. Finch M Begley A, Sutherland R, Harrison M, Collins C. Development and reproducibility of a tool to assess school food-purchasing practices and lifestyle habits of Australian primary school-aged children. Nutr Diet 2007; 64(2): 86-92. https://doi.org/10.1111/j.1747-0080.2007.00148.x
» https://doi.org/https://doi.org/10.1111/j.1747-0080.2007.00148.x -
16. Carter MA, Swinburn B. Measuring the ‘obesogenic’ food environment in New Zealand primary schools. Health Promot Int 2004; 19(1): 15-20. https://doi.org/10.1093/heapro/dah103
» https://doi.org/https://doi.org/10.1093/heapro/dah103 -
17. Faber M, Laurie S, Maduna M, Magudulela T, Muehlhoff E. Is the school food environment conducive to healthy eating in poorly resourced South African schools? Public Health Nutr 2014; 17(6): 1214-23. https://doi.org/10.1017/S1368980013002279
» https://doi.org/https://doi.org/10.1017/S1368980013002279 -
18. Bekker F, Marais M, Koen N. The provision of healthy food in a school tuck shop: does it influence primary-school students’ perceptions, attitudes and behaviours towards healthy eating? Public Health Nutr 2017; 20(7): 1257-66. https://doi.org/10.1017/S1368980016003487
» https://doi.org/https://doi.org/10.1017/S1368980016003487 -
19. Gebremariam MK, Henjum S, Terragni L, Torheim LE. Correlates of fruit, vegetable, soft drink, and snack intake among adolescents: the ESSENS study. Food Nutr Res 2016; 60: 32512. https://doi.org/10.3402/fnr.v60.32512
» https://doi.org/https://doi.org/10.3402/fnr.v60.32512 -
20. Bastos JL, Duquia RP, González-Chica DA, Mesa JM, Bonamigo RR. Field work I: selecting the instrument for data collection. An Bras Dermatol 2014; 89(6): 918-23. https://doi.org/10.1590/abd1806-4841.20143884
» https://doi.org/https://doi.org/10.1590/abd1806-4841.20143884 -
21. Reichenheim ME, Bastos JL. O quê, para quê e como? Desenvolvendo instrumentos de aferição em epidemiologia. Rev Saúde Pública 2021; 55: 40. https://doi.org/10.11606/s1518-8787.2021055002813
» https://doi.org/https://doi.org/10.11606/s1518-8787.2021055002813 -
22. Canuto R, Clark SGF, Borges LD, Castro Junior PCP, Tavares LF, Cardoso LO, et al. Aspectos metodológicos do estudo sobre Comercialização de Alimentos em Escolas Brasileiras. Cad Saúde Pública 2025; 41(5): e00167624. https://doi.org/10.1590/0102-311XPT167624
» https://doi.org/https://doi.org/10.1590/0102-311XPT167624 -
23. Caspi CE, Sorensen G, Subramanian SV, Kawachi I. The local food environment and diet: a systematic review. Health Place 2012; 18(5): 1172-87. https://doi.org/10.1016/j.healthplace.2012.05.006
» https://doi.org/https://doi.org/10.1016/j.healthplace.2012.05.006 -
24. Story M, Kaphingst KM, Robinson-O’Brien R, Glanz K. Creating healthy food and eating environments: policy and environmental approaches. Annu Rev Public Health 2008; 29: 253-72. https://doi.org/10.1146/annurev.publhealth.29.020907.090926
» https://doi.org/https://doi.org/10.1146/annurev.publhealth.29.020907.090926 -
25. Turner C, Aggarwal A, Walls H, Herforth A, Drewnowski A, Coates J, et al. Concepts and critical perspectives for food environment research: A global framework with implications for action in low- and middle-income countries. Glob Food Secur 2018; 18: 93-101. https://doi.org/10.1016/j.gfs.2018.08.003
» https://doi.org/https://doi.org/10.1016/j.gfs.2018.08.003 -
26. Downs SM, Ahmed S, Fanzo J, Herforth A. Food environment typology: advancing an expanded definition, framework, and methodological approach for improved characterization of wild, cultivated, and built food environments toward sustainable diets. Foods 2020; 9(4): 532. https://doi.org/10.3390/foods9040532
» https://doi.org/https://doi.org/10.3390/foods9040532 -
27. Castro IRR, Canella DS. Organizational food environments: advancing their conceptual model. Foods 2022; 11(7):993. https://doi.org/10.3390/foods11070993
» https://doi.org/https://doi.org/10.3390/foods11070993 -
28. Duran AC, Lock K, Latorre MRDO, Jaime PC. Evaluating the use of in-store measures in retail food stores and restaurants in Brazil. Rev Saude Publica 2015; 49: 80. https://doi.org/10.1590/S0034-8910.2015049005420
» https://doi.org/https://doi.org/10.1590/S0034-8910.2015049005420 -
29. Borges CA, Jaime PC. Development and evaluation of food environment audit instrument: AUDITNOVA. Rev Saude Publica 2019; 53: 91. https://doi.org/10.11606/s1518-8787.2019053001316
» https://doi.org/https://doi.org/10.11606/s1518-8787.2019053001316 -
30. Franco AS, Canella DS, Tavares LF, Pereira AS, Barbosa RMS, Oliveira Junior GI, et al. Content validity and reliability of a university food environment assessment instrument. Cien Saude Colet 2022; 27(6): 2385-96. https://doi.org/10.1590/1413-81232022276.13792021
» https://doi.org/https://doi.org/10.1590/1413-81232022276.13792021 -
31. Balestrin M, Brasil CCB, Bellei EA, Marchi ACB, Kirsten VR, Wagner MB. Cantinas survey: proposição e avaliação de um aplicativo para análise do risco sanitário e dos alimentos comercializados em cantinas escolares. Reciis 2022; 16(3): 704-18. https://doi.org/10.29397/reciis.v16i3.2367
» https://doi.org/https://doi.org/10.29397/reciis.v16i3.2367 - 32. Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Guia alimentar para a população brasileira. Brasília: Ministério da Saúde; 2014.
-
33. Tavares LF, Perez PMP, Passos MEA, Castro Junior PCP, Franco AS, Cardoso LO, et al. Development and application of healthiness indicators for commercial establishments that sell foods for immediate consumption. Foods 2021; 10(6): 1434. https://doi.org/10.3390/foods10061434
» https://doi.org/https://doi.org/10.3390/foods10061434 - 34. Lynn MR. Determination and quantification of content validity. Nurs Res 1986; 35(6): 382-5. PMID: 3640358.
-
35. Haynes B, Haines A. Barriers and bridges to evidence based clinical practice. BMJ 1998; 317(7153): 273-6. https://doi.org/10.1136/bmj.317.7153.273
» https://doi.org/https://doi.org/10.1136/bmj.317.7153.273 -
36. Coluci MZ, Alexandre NMC, Milani D. Construção de instrumentos de medida na área da saúde. Ciênc Saúde Coletiva 2015; 20(3): 925-36. https://doi.org/10.1590/1413-81232015203.04332013
» https://doi.org/https://doi.org/10.1590/1413-81232015203.04332013 - 37. Hulley SB, Cummings SR, Browner WS, Grady D, Hearst N, Newman TB. Designing clinical research: an epidemiologic approach. 2nd ed. Philadelphia: Lippincott Williams & Wilkins; 2001.
-
38. Byrt T. How good is that agreement? Epidemiology 1996; 7(5): 561. https://doi.org/10.1097/00001648-199609000-00030
» https://doi.org/https://doi.org/10.1097/00001648-199609000-00030 -
39. Collect Box. Coleta de dados de forma rápida e acessível [Internet]. 2019 [cited on Sept. 29, 2023]. Available at: https://www.collectbox.com.br/
» https://www.collectbox.com.br/ - 40. Polit DF, Beck CT. Fundamentos de pesquisa em enfermagem: métodos, avaliação e utilização. 7a ed. Porto Alegre: Artmed; 2011.
-
41. Heale R, Twycross A. Validity and reliability in quantitative studies. Evid Based Nurs 2015; 18(3): 66-7. https://doi.org/10.1136/eb-2015-102129
» https://doi.org/https://doi.org/10.1136/eb-2015-102129 -
42. Souza AC, Alexandre NMC, Guirardello EB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saúde 2017; 26(3): 649-59. https://doi.org/10.5123/S1679-49742017000300022
» https://doi.org/https://doi.org/10.5123/S1679-49742017000300022 -
43. Chen CM, Cano SJ, Klassen AF, King T, McCarthy C, Cordeiro PG, et al. Measuring quality of life in oncologic breast surgery: a systematic review of patient-reported outcome measures. Breast J 2010; 16(6): 587-97. https://doi.org/10.1111/j.1524-4741.2010.00983.x
» https://doi.org/https://doi.org/10.1111/j.1524-4741.2010.00983.x -
44. Polit DF, Beck CT. The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health 2006; 29(5): 489-97. https://doi.org/10.1002/nur.20147
» https://doi.org/https://doi.org/10.1002/nur.20147 - 45. Luiz RR. Métodos estatísticos em estudos de concordância. In: Medronho RA, Bloch KV, Luiz RR, Werneck GL, orgs. Epidemiologia. São Paulo: Atheneu; 2009. p. 343-70.
-
46. Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993; 46(5): 423-9. https://doi.org/10.1016/0895-4356(93)90018-v
» https://doi.org/https://doi.org/10.1016/0895-4356(93)90018-v - 47. Streiner DL, Norman GR, Cairney J. Reliability. In: Streiner DL, Norman GR, Cairney J, eds. Health measurement scales: a practical guide to their development and use. New York: Oxford University Press; 2015. p. 179-99.
- 48. IBM Corp. IBM SPSS Statistics for Windows [Software]. Versão 21.0. Armonk: IBM Corp; 2012.
- 49. StataCorp. Stata Statistical Software: Release 15 [Software]. College Station: StataCorp LLC; 2017.
- 50. Mendes LL, Canuto R, Tavares LF, Cardoso LO, Castro Júnior PCP, Borges LD, et al. Manual para coleta de dados: comercialização de alimentos em escolas brasileiras. Belo Horizonte; 2024
-
51. Souza LBO, Azevedo ABC, Bandoni DH, Canella DS. Características do ambiente escolar relativas à alimentação e atividade física: PeNSE 2015. Rev Saúde Pública 2021; 55: 115. https://doi.org/10.11606/s1518-8787.2021055003377
» https://doi.org/https://doi.org/10.11606/s1518-8787.2021055003377 -
52. Monteiro CA, Cannon G, Moubarac JC, Levy RB, Louzada MLC, Jaime PC. The UN Decade of Nutrition, the NOVA food classification and the trouble with ultra-processing. Public Health Nutr 2018; 21(1): 5-17. https://doi.org/10.1017/S1368980017000234
» https://doi.org/https://doi.org/10.1017/S1368980017000234 - 53. World Health Organization. Food marketing exposure and power and their associations with food-related attitudes, beliefs and behaviours: a narrative review. Geneva: World Health Organization; 2022.
- 54. Messias GM. Ambiente alimentar hospitalar: desenvolvimento e avaliação de confiabilidade de um instrumento na rede pública de uma metrópole brasileira [tese]. Rio de Janeiro: Universidade do Estado do Rio de Janeiro; 2019.
-
55. Winston CP, Sallis JF, Swartz MD, Hoelscher DM, Peskin MF. Reliability of the hospital nutrition environment scan for cafeterias, vending machines, and gift shops. J Acad Nutr Diet 2013; 113(8): 1069-75. https://doi.org/10.1016/j.jand.2013.04.004
» https://doi.org/https://doi.org/10.1016/j.jand.2013.04.004 -
56. Avelar BA, Hino AAF, Santos AP, Mendes LL, Carraro JCC, Mendonça RD, et al. Validity and reliability of the Perceived Nutrition Environment Measures Survey (NEMS-P) for use in Brazil. Public Health Nutr 2023; 27(1): e11. https://doi.org/10.1017/S1368980023002653
» https://doi.org/https://doi.org/10.1017/S1368980023002653 -
57. O’Halloran S, Eksteen G, Gebremariam M, Alston L. Measurement methods used to assess the school food environment: a systematic review. Int J Environ Res Public Health 2020; 17(5): 1623. https://doi.org/10.3390/ijerph17051623
» https://doi.org/https://doi.org/10.3390/ijerph17051623
-
HOW TO CITE THIS ARTICLE:
Borges LD, Tavares LF, Carmo AS, Cardoso LO, Canuto R, Castro Junior PCP, et al. Development, content validity, and reliability of an instrument to assess the commercialization of food and beverages in school canteens in Brazil. Rev Bras Epidemiol. 2026; 29: e260015. https://doi.org/10.1590/1980-549720260015
-
FUNDING:
This study was supported by the following Brazilian research foundations and institutions: MS-SCTIE-Decit/National Council for Scientific and Technological Development (CNPq) No. 26/2019; National Council for Scientific and Technological Development (CNPq) Call for Submission No. 4/2021 - Research Productivity Grant - PQ; ACT Promoção da Saúde; Instituto Brasileiro de Defesa do Consumidor (Idec); Instituto Ibirapitanga and Instituto Desiderata.
Edited by
-
ASSOCIATE EDITOR:
Roger Keller Celeste http://orcid.org/0000-0002-2468-6655
-
SCIENTIFIC EDITOR:
Maria Fernanda Tourinho Peres http://orcid.org/0000-0002-7049-905X
