Content validity and reliability of a university food environment assessment instrument

This study aimed to evaluate the content validity and reliability of an instrument for evaluating the university food environment. A checklist was developed to assess establishments that sell food and beverages in the university environment. The content validation encompassed the development of the instrument, expert evaluation and pretest performance. Reliability was evaluated using a convenience sample (n=64) of establishments distributed across seven campuses of three public universities and was carried out using interobserver (IO) and test-retest (TR) evaluations. Categorical and count variables were analyzed by calculating the percentage agreement (PA), kappa coefficient (k) and prevalence-adjusted, bias-adjusted kappa (ka), and continuous variables were analyzed by the intraclass correlation coefficient (ICC). The checklist consisted of 204 items distributed in seven domains. The instrument’s performance was considered excellent or very good for 91.3% (PA) of the items when evaluated. For IO, 68.3% (k) and 96.5% (ka) had excellent, very good or good agreement, while for TR, 65% (k) and 96.5% (ka) had excellent agreement. The instrument showed satisfactory content validity and reliability for characterizing the food environment at Brazilian universities.


Introduction
Efforts have been made to translate concepts and constructs about the food environment into measurement tools that can generate empirical evidence [1][2][3][4][5][6] .The consumer food environment 1 , which was the object of interest in the present study, includes factors related to the food itself, i.e., the way in which it is provided or presented and the price, nutritional quality and nutritional labeling (including nutrition claims), among other factors.The consumer's food environment may be inserted into organizational environments, such as companies and universities 1 .In the literature, the most commonly reported method for evaluating the consumer food environment is the auditing of establishments that sell food.Researchers evaluate the variety of products, nutritional information, healthy options, prices, advertisements and other aspects of food 2,[7][8][9] .
The university food environment, which was the scenario of interest of this study, can be characterized by the availability of foods, preparations and beverages (FPB), physical and financial accessibility, the promotion of FPB, nutritional information and advertising within the campus and may also include the campus surroundings.It is known that this environment influences the eating habits of the individuals who are exposed to it 10 , since it can act as a facilitator of or barrier to healthy choices 1,10 given that meals consumed at the university may be an important part of the diet of students, teachers and employees 11,12 .
As of 2021, Brazil had 302 public higher education institutions (HEIs) and 2,306 private HEIs, which together comprise 8,603,824 students and 386,703 teachers, 13 in addition to thousands of technical-administrative employees.Therefore, these locations should be valued as a strategic setting for promoting healthy eating.
The literature on the evaluation of the university food environment is still expanding.Countries such as the United States and Australia have published studies on the characterization of the food environment at universities and their surroundings 14,15 .In Brazil, to date, few such studies have been found [16][17][18][19] .
The selection or development of a data collection instrument is an essential step that, if not performed with care and methodological rigor, can be a source of error and misleading conclusions 20 .Although many studies have been conducted to evaluate the food environment, a small portion of them have assessed the psychometric properties of the instruments used in them 21,22 .
Among the studies on the university food environment, only two analyzed the reliability of the instrument used 14,16 , and none evaluated its validity.In addition, of the instruments that have been identified to date, none is able to fully capture the specificities of the Brazilian university food environment or highlight the extensive variety of convenience items and the existence of mixed establishments (that have characteristics of both restaurants and cafeterias).In addition, these instruments do not use the NOVA classification of foods, which has been adopted in the Brazilian guidelines for public policies on food and nutrition 23 and studies on food, diet quality and health 24 , to characterize commercial foods and preparations 25 .
Considering the importance and specificities of the university food environment, the scarcity of instruments for its evaluation and the scarcity of evaluations of the performance of these instruments in different contexts, the present study aimed to evaluate the content validity and reliability of an instrument for evaluating the university food environment.

methods
The present study is an initiative of the Collaborative Group for Studies on the University Food Environment (Grupo Colaborativo de Estudos sobre o Ambiente Alimentar Universitário -CALU, in Portuguese), which is composed of professors from public universities located in the state of Rio de Janeiro that have a course in nutrition, namely, the Universidade do Estado do Rio de Janeiro (UERJ), Universidade Federal do Rio de Janeiro (UFRJ), Universidade Federal do Estado do Rio de Janeiro (UNIRIO) and Universidade Federal Fluminense (UFF).

Content validation
The content validation of the instrument involved three stages: the development of the instrument, its evaluation by experts 26,27 and the performance of a pretest.

Development of the instrument
The preparation of the instrument was based on two theoretical frameworks: the approach of Caspi et al. 9 regarding access to food and the approach of Glanz et al. 1 regarding the consumer food environment.Of the five dimensions proposed by Caspi et al. 9 , those that could fully apply to commercial establishments located in universities were considered, namely, availability (including the variety of food available), affordability (including the price of food) and convenience (e.g., open hours, payment methods accepted).These dimensions were complemented by two components proposed by Glanz et al. 1 : nutritional information and food promotion.
The first stage of construction of the instrument involved mapping the existing instruments for the evaluation of the consumer's food environment and the selection of two that closely matched the characteristics of the food environments of Brazilian public universities 14,17 .The elaboration of the instrument encompassed three phases, which will be described in detail in the results section: the identification of the domains that should compose the instrument (e.g., characterization of the establishment, available foods, prices, advertisements), the preparation of the items and the structuring of the instrument.During this process, the inclusion of foods based on the recommendations of the Dietary Guidelines for the Brazilian Population 24 and international guidelines 28 , the NOVA classification of foods 23 and foods commonly found in universities 14,17 was guaranteed.The items included in the checklist comprised fresh or minimally processed foods, processed foods and culinary preparations based on these foods, as well as ultra-processed foods and culinary preparations containing these foods.Together, these are markers of healthy and unhealthy eating.

expert evaluation
For the evaluation of the instrument by experts, the literature suggests the participation of five to 20 individuals 26,27 .In the present study, 12 researchers in the field of nutrition from different institutions were invited.Their expertise was in epidemiology, public health, collective food or gastronomy, and they had experience in topics related to the study and/or construction of instruments.
The preliminary version of the instrument was qualitatively evaluated and was performed in two stages: in the first phase, the proposed domains were evaluated, and in the second phase, the items were evaluated, including their relevance and clarity.Regarding relevance, whether each item expressed the expected concept, its relevance and its adequacy for the proposed objectives were evaluated.Regarding clarity, whether the wording of each item expressed the concept in a way that was understandable and consistent with what was expected to be measured was evaluated.
The experts also evaluated the instrument as a whole, examining its scope, i.e., whether each domain was adequately covered by the set of items presented and whether the dimensions proposed by Caspi et al. 9 and the components of the food environment proposed by Glanz et al. 1 that were expected to be covered by the instrument had been adequately considered.

Pretest
The final step was a pretest of the instrument.The pretest was conducted by the first author of this article at a commercial establishment located near one of the universities studied that had characteristics similar to those of the establishments of the universities participating in this study.

Assessment of the reliability of the instrument
To evaluate the reliability of the instrument, a convenience sample of establishments with different characteristics distributed over seven campuses of three universities was studied, namely, UFRJ: Cidade Universitária, Praia Vermelha and Professor Aluísio Teixeira (Macaé) campuses; UFF: Gragoatá and Valonguinho campuses; and Unirio: Pasteur and Biomedical Institute campuses.Based on publications on the topic 14,17 , it was determined at least 50 establishments should be included.All establishments on the campuses of the universities that agreed to participate in the study were included.
Data collection was performed by 15 trained field researchers (nutrition and gastronomy students) using a printed form.The training lasted six hours and was performed in two stages: first, a theoretical stage, in which the purpose of the study, the instrument to be applied and the data collection manual were presented; and second, a practical stage, in which the instrument was applied by the researchers in at least two establishments under the supervision of the study coordinators.
The establishments included in the study were evaluated three times: two researchers (A and B) visited each establishment on the same day at a maximum interval of 30 minutes, and one of the researchers from each pair (A or B) performed a new data collection visit 15 to 30 days after the first evaluation.Data collection occurred between November 2015 and February 2016, which was an academic period at the universities.
Data entry was performed by double entry using Excel software version 2016, which was also used to evaluate the consistency of the database.
The reliability of the instrument was determined by interobserver and the test-retest evaluations. 20For the categorical and count variables, the stability of the instrument was evaluated by calculating the percent agreement (PA) 29 and the kappa index (k), kappa index adjusted for prevalence and interviewer bias (ka) (the prevalence-adjusted, bias-adjusted kappa (PABAK)) for all variables 30 .For food prices, which were analyzed as continuous variables, the intraclass correlation coefficient (ICC) 20 was calculated.The PA, k, ka and ICC values were classified according to the criterion of Byrt 31 : excellent agreement (≥0.92), very good agreement (0.80 to 0.91), good agreement (0.60 to 0.79), fair agreement (0.40 to 0.59), slight agreement (0.20 to 0.39), poor agreement (0.00 to 0.19), and no agreement (<0.00).
All analyses were performed using the Statistical Package for Social Science (SPSS 21) software.To calculate the PABAK and the ICC, it was necessary to create a specific macro.
The study was submitted to the Ethics Committee of the Pedro Ernesto Hospital of the State University of Rio de Janeiro and approved under opinion number 49988015.6.0000.5259.

Content validation
In general, the content validity of the developed instrument was considered satisfactory by the experts, considering that the proposed domains and items included the conceptual dimensions of interest and that these dimensions covered the essential elements of the consumer food environment in the university context.
Based on the experts' assessment, items were changed or added to or removed from the checklist.The main changes were inclusion of a new domain ("information"); the removal of a domain ("review of the menu"); reorganization of the "environmental observation" domain and the inclusion of 13 items in it; addition of the category "convenience items" to the "beverages, foods and preparations" domain and the inclusion of eight new types of ultra-processed beverages; inclusion of the form of presentation of advertisements in the "advertisements" domain.
The results of the pretest did not indicate a need for changes in the content or form of the instrument.All items were suitable for completion, and no situation of interest was identified that was not covered by the instrument, nor were any unnecessary items noted.The ordering of the modules in the instrument, as well as its formatting, was also shown to be adequate.
The final version of the developed instrument consisted of a checklist that covered the dimensions availability, affordability, convenience 9 , nutritional information and food promotion 1 and included 204 items distributed across seven different domains, namely, characterization of the establishment; observation of the environment; information made available; beverages, foods and preparations; prices and promotions; availability of healthy substitutions; and advertisements (full text of the instrument is available athttp://www.observatoriodeobesidade.uerj.br/?p=3294), as detailed below: • Characterization of the establishment: type and location of the establishment; types of food offered; days and hours of operation and most frequent customers; forms of payment.
• Observation of the environment: availability of specific items (microwave, structures for on-site consumption, open-access shelves and counters displaying certain items, display refrigerators and freezers, available water dispenser and free filtered water supply) and organization of buffets (order of presentation).
• Information available: availability of information to the customer (menu, price, nutritional information) and form of presentation of this information (on the preparation or product, table display, banner, internet).
• Beverages, food and preparations: preparations, meals, snacks and other: availability of complete meals (portioned or a la carte); brown rice; legumes; fruits and fruit salads; raw and cooked vegetables; natural seasonings and sauces; natural fruit juices and soft drinks; savory snacks and sandwiches; and other items (water, table sugar and sweetener).Convenience items: availability and variety of the following convenience items (i.e., different types of the same item): ultra-processed foods (candies, bonbons, sweet cookies with and without filling, packaged snacks, whole cookies, cereal bars, breakfast cereal), ultra-processed beverages (flavored water, coconut water, soft drinks, fruit juice or fruit nectar drinks, ready-made teas, isotonic beverages, energy tea, soy-based drinks, guarana soft drinks, flavored milk or milk or yogurt beverages, alcoholic beverages, self-service coffee) and elaborate sweets (honey bread, cake, brigadeiros).
• Prices and promotions: prices of the smallest portions available for each of the foods, or, when available, the price of meals by the kilogram; supply and prices of promotions/combos (e.g., foods/products plus an ultra-processed beverage), supply and price of larger portions of certain products.
• Supply of healthy substitutions: options to replace white rice with brown rice and French fries with salads or vegetables in meals or combos.
• Advertising: existence of advertisements for fruits and vegetables and ultra-processed beverages, ice cream and desserts; number, presentation and messages of advertisements.
The types of establishments were classified according to their predominant activity: by-thekilo restaurants (essentially sells meals and foods by weight); a la carte/portioned/executive-style restaurants (essentially sells meals at a fixed quantity and price); free/rotating restaurant (essentially sells meals using a free (unweighed) or rotating system (variable quantity/weight and a fixed price) system; snack bar (sells mainly snacks/ sandwiches and savory snacks and candy items (candy, sweets, ultra-processed beverages); bar (mainly sells alcoholic beverages, cigarettes, candies, snacks and food items); cafeteria (sells coffee and other beverages and, sometimes, foods that can be eaten quickly or light dishes); bomboniere (sells sweets, candies, chewing gum, chocolates, soft drinks, juices, sweets in general and ice cream); and mixed establishment: snack bar/cafeteria + meals by the kilo or snack bar/cafeteria + a la carte meals/portioned items/executive-style (sells different typical cafeteria products with portioned menu items or meals sold by weight) 32 .
To categorize the beverages included in the instrument, the grouping used by Euromonitor International 33 for sugary or sweetener-based beverages was used and adapted for the Brazilian market.The adopted categories were flavored (bottled) waters, carbonated or uncarbonated (they contain natural or artificial fruit flavors, usually do not contain artificial colors, may or may not contain sugar); soft drinks: cola and non-cola beverages, regular or low calorie; liquid or powder concentrates for reconstituted beverages; 100% fruit juices; juice-based beverages; nectars; ready-to-drink teas (e.g., iced-tea, green tea and mate with added sugar or other sweeteners); isotonic or replenishing beverages (e.g., sport drinks); energy drinks, generally those containing caffeine and taurine; guarana soft drinks; soy-based beverages, plain or flavored; coconut water; milk-based beverages: whey-based dairy beverages, flavored milks, and mixed milk and fruit preparations, with or without cereals or other items.
Food advertisements were categorized according to their form of presentation.Any word or phrase about the food/product that was presented in a banner/poster produced by the supplier, a banner/poster produced by the establishment, on employees' clothing, on the product (advertising added to product packaging), on a replica of the product, on display in the establishment or even on the menu was considered.

reliability of the instrument
A total of 64 establishments were included in the first phase of data collection, and 54 were included in the second round (10 refused to participate).The average duration of data collection at each establishment was 18 minutes.
Regarding the performance of the instrument, excellent or very good agreement was observed for 91.3% of the items evaluated.Regarding the interobserver test, for among the evaluated items (158), 68.3% and 96.5% had excellent, very good or good agreement according to the k (range of -0.51 to 1.00) and ka (range from 0.44 to 1.00), respectively (Table 1).Poor or null agreement was observed for 12% and 0.6% of the items according to the k and ka, respectively; these items were in the domains of characteristics of the establishment, healthy substitutions and nutritional information.For the items in the food prices domain, the ICC ranged from 0.05 to 1.00.Of the evaluated items (n=28), 88% showed excellent, very good or good agreement (Table 2).In this domain, 23% of the convenience items showed poor agreement.
In the test-retest evaluation, of the evaluated items (158), 65% had excellent, very good and good agreement according to k, which ranged from -0.05 to 1.00.Regarding the ka, which ranged from 0.37 to 1.00, 96.5% of the items showed excellent, very good or good agreement (Table 3).Considering k, there was poor or null agreement for 4.8% of the items, which were concentrated in the domains of environmental characteristics; beverages, foods and preparations; and information.For ka, there was no poor or null classification of any item.For the items in the food prices domain (n=28), the ICC ranged from 0.05 to 1.00; 77% showed excellent, very good or good agreement, and none of the items had poor or null agreement (Table 2).table 1. Interobserver reliability (kappa and prevalence-adjusted, bias-adjusted kappa) of an instrument for the assessment of the university food environment.Rio de Janeiro, 2016.12.6 2.9 0.0 0.6

Domain
(1) Of the 204 items on the instrument, 28 referred to price and were not included in the calculation of kappa and PABAK.Of the 176 items that were included in these calculations, for 18 of them, the statistics could not be computed due to the absence of data for one or more levels in the cross-reference tables; (

Discussion
The results indicated satisfactory content validity and reliability (interobserver and test-retest) of the instrument for characterizing the consumer food environment at universities in the Brazilian context.Even for the test-retest evaluation, for which some of the differences can be explained by real variations in the environment, particularly with regard to food supply, good performance of the instrument was observed.
Although the performance of the instrument was generally satisfactory, some items showed low performance, especially in the interobserver reliability evaluation.This result suggests the need to improve the training provided for the application of the instrument.A similar situation was observed in a study that evaluated a checklist applied in a hospital food environment 34 .The food environment is a complex context with significant diversity of establishments, foods, beverages and information, and thorough and standardized evaluation through the appropriate selection and training of evaluators is necessary 4 .Although the results of the present study were good, the duration of training was shorter than that reported for other studies 3,35 .
Few studies in the literature were identified that determined the content validity of instruments for assessing the food environment, and none involved the university food environment.A review on the assessment of the food environment that included 432 studies published between 2007 and 2015 showed that 28.1% evaluat-ed the validity of the instrument used, and 2.1% assessed the content validity of the instrument 22 .Among those studies, that of Pomerleau et al. 36 is noteworthy, as it conducted a content validation procedure similar to that used in the present study; for the development of the instrument it used (the EURO-PREVOB Community Questionnaire), a literature review, expert consultations and a pretest were performed 33 .
In the international and national literature, there are few studies on the reliability analysis of instruments for assessing the food environment in universities.In addition, because the existing studies use different instruments to evaluate the environment and because reliability is a context-dependent measure 37 , the existing studies are not fully comparable with the present evaluation.Nevertheless, it is worth commenting on their results.In Australia, Roy et al. 15 developed a checklist for the evaluation of food availability, physical accessibility (e.g., products near the checkout counter, free access to shelves) and food promotions.The authors performed an interobserver reliability analysis through percentage agreement for two establishments and found 96.0% agreement 15 .
In Brazil, Rodrigues et al. 17 adapted a checklist to evaluate the university food environment that included items related to the description of the establishments and the evaluation, the availability of foods/preparations and facilitators and barriers to healthy eating.The interobserver and test-retest reliability was measured by kappa for 64 establishments.For interobserver agreement,  (1) Of the 204 items on the instrument, 28 referred to price and were not included in the calculation of kappa and PABAK.Of the 176 items that were included in these calculations, for 18 of them, the statistics could not be computed due to the absence of data for one or more levels in the cross-reference tables; ( the kappa ranged from -0.04 to 1.0.For the test-retest agreement, a variation of 0.06 to 1.0 was observed.The mean kappa index was 0.54 for interobserver agreement and 0.72 for test-retest agreement 17 .It should also be noted that some instruments have been developed with the objective of evaluating the food environment in other scenarios, such as the community or other organizational spaces, such as work and school environments 3,37,38 .In Brazil, it is worth mentioning three studies conducted in the state of São Paulo that evaluated the reliability of instruments for evaluating the food environment of restaurants 3,4 and food retailers 3,37,38 .The authors evaluated the interobserver and test-retest reliability using the kappa and ICC and generally found good reliability for both tools 3,4,38 .Our study adds to these findings, providing evidence for an instrument that addresses the specificities of the university food environment, including establishments that sell food for immediate consumption (and thus have characteristics similar to those of restaurants and snack bars) but also sell items such as packaged snacks and cookies that are typically sold in supermarkets/markets/grocery stores and are not usually available in restaurants.The instrument evaluated in the present study assesses all types of food available at the university and avoids the need to use two different instruments.
Although most studies on the university food environment have been conducted in public HEIs, given the similarities between the contexts of the food environments of public and private universities, the instrument presented here may also be useful for the latter.
The study has some limitations.Because it used a convenience sample of public universities, its external validity is limited.In addition, although the elaboration of the instrument included consultation with experts in the field, quantitative measures were not used to verify the content validity.Another aspect that warrants mention is the fact that the data collection process lasted three months; therefore, it is possible that seasonal differences in food supply were not captured.Consequently, the possible implications of seasonality, if present, for the performance of the instrument are unknown.A fourth limitation is the fact that, for some of the items that were used to calculate kappa and PABAK, these statistics could not be computed due to the absence of one or more levels in the cross-referenced tables.However, this affected only a small portion of the items (approximately 10%).Thus, the generally good performance of the other items suggests that the performance of these items, if calculated, would not significantly impact the overall good performance of the instrument.
Among the strengths of this study is the evaluation of test-retest reliability, since most evaluations of audit tools only evaluate interobserver reliability at a single moment 15 .Another strength of the study is the broad scope of the adopted instrument.In addition to being a compilation and adaptation of elements from other instruments to better reflect the characteristics of the Brazilian university food environment, the present instrument was conceived and evaluated by researchers with different training profiles and who worked on university campuses with different characteristics, such as being located in the state capital vs. other municipalities and in an urban center vs. in a neighborhood outside the urban center; having two (morning and afternoon) vs. three (morning, afternoon and night) shifts; and having fewer vs. more structured establishments.Another aspect to highlight is the use of ka, which is still infrequently used in the health literature.In addition, the application of the instrument was tested on university campuses that had different characteristics.

Conclusion
The proposed instrument performed well in the context in which it was tested.It can support the characterization of the food environment in both public and private universities (and their surroundings, if the establishments have characteristics similar to those present on the university campuses), monitor their dynamics over time and support interventions for their improvement.

Collaborations
All authors have read and agreed to the published version of the manuscript.AS Franco, IRR Castro and DS Canella participated in the design.AS Franco, IRR Castro, DS Canella, AS Pereira, LF Tavares, RMS Barbosa, GI Oliveira Junior and CRB Santos participated in the planning.AS Franco, IRR Castro, DS Canella, LF Tavares and RMS Barbosa participated in the data analysis.AS Franco, IRR Castro and DS Canella participated in data interpretation.AS Franco, IRR Castro, DS Canella, AS Pereira, LF Tavares, RMS Barbosa, GI Oliveira Junior and CRB Santos participated in the writing, review and approval of the final version of the article.
Healthy substitutions (replacing potato chips with salad; white rice with whole grain; white bread with whole grain; soft drink mixes with natural juices) and prices on banners, displays, internet, etc.fruits, salads or juices; ultra-processed beverages;

table 2 .
Interobserver reliability and test-retest of prices for foods/beverages, preparations and convenience items.Rio de Janeiro, 2015-2016.

table 3 .
Intraobserver reliability (kappa and prevalence-adjusted, bias-adjusted kappa) of an instrument for the assessment of the university food environment.Rio de Janeiro, 2016.