Scale for evaluating food and nutrition education practices in Primary Health Care Escala de avaliação de práticas de educação alimentar e nutricional na Atenção Primária em Saúde

1 Universidade de São Paulo, Faculdade de Saúde Pública, Programa de Pós-Graduação em Nutrição em Saúde Pública. Av. Dr. Arnaldo, n. 715, Cerqueira César, 01246-904, São Paulo, SP, Brasil. Correspondence to: L.C. REIS. E-mail: <ligiacreis@hotmail.com>. 2 Universidade de São Paulo, Faculdade de Saúde Pública, Núcleo de Pesquisas Epidemiológicas em Nutrição e Saúde. São Paulo, SP, Brasil. Support: Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq, National Council for Scientific and Technological Development) (Process n. 408115/2017-3).


A B S T R A C T Objective
To develop, validate and analyse intra-and inter-raters' agreement and reliability of a scale for the evaluation of food and nutrition education practices in primary health care.

Methods
Three theoretical references were used for the development of items with answers on a 4-point scale (not observed, partially/satisfactorily/fully observed): the Brazilian Dietary Guidelines, the Interprofessional Collaborative Practice and the Family Health Strategy as models for organizing primary health care. The scale was submitted for content and face panel validation and it was further reevaluated online. Thirty specialists used the scale for the rating of 4 videos produced for the study showing food and nutrition education practices in primary health care settings. Intra-and inter-raters agreement and reliability were analyzed by test at one time point and retest 30 days later.

Results
The scale initially included 72 items but was reduced to 17 items after content and face validation, together with a fill-out manual that was reviewed by experts. In the re-evaluation, all 17 items were maintained, and content adjustments were made in the manual. Thirty raters applied the scale to the videos in the test, and 28 in the retest. All videos obtained combined kappa values of >0. 4. In the test-retest, 19 raters showed >80% agreement; in the inter-rater evaluation, 14 raters generated kappa values >0.75.

Conclusion
The scale underwent content and face validation and showed to be useful to diagnose the absence and the full addressing of the Brazilian Dietary Guidelines content with good intra-and inter-rater agreement and reliability in different settings of Primary Health Care.
Keywords: Dietary guidelines. Food and nutrition education. Health personnel. Primary health care.

R E S U M O
Objetivo Desenvolver e avaliar a validade e a confiabilidade de uma escala de avaliação das práticas de educação alimentar e nutricional na Atenção Primária em Saúde.

I N T R O D U C T I O N
Strategies to promote adequate and healthy eating make up a set of interventions considered by the World Health Organization (WHO) to be cost-effective [1]. In the current nutritional epidemiological scenario, Food and Nutrition Education (FNE) actions are acknowledged to be fundamental to influence people's knowledge, awareness, behavior, preferences and attitudes towards food and nutrition [2].
Brazil is considered a reference country with a food and nutrition agenda that incorporates FNE in Primary Health Care (PHC) [2,3]. As food has become an important determinant of chronic non-communicable diseases, FNE started to be valued in the Sistema Único de Saúde (SUS, the Brazilian National Health System) policies and dietary guidelines have been published to be meant as technologies to promote adequate and healthy eating [4].
Dietary guidelines are defined by the Food and Agriculture Organization of the United Nations (FAO) as the expression of the FNE principles used as a guidance for the population and the national food and nutrition policies [5]. The Dietary Guidelines for the Brazilian Population (Brazilian Dietary Guidelines) announced an innovative paradigm of healthy eating and adopted an expanded vision of sustainability [6][7][8].
For FAO/WHO, the development and implementation of dietary guidelines that cover sustainability in all its dimensions are initiatives capable of interfering in food systems [9]. In this framework, FAO emphasizes, as a priority strategy for the effective implementation of dietary guidelines, the training of the facilitators who will disseminate their recommendations [10].
While in Brazil there is a significant scarcity of continuous professional development actions in PHC, WHO considers as an unresolved challenge the access to tools that help health educators to engage in more effective actions [11,12]. Although many countries invest in the development of dietary guidelines, mechanisms for evaluating and monitoring their implementation are undervalued [10].
Considering that no validated instruments were found to assess the implementation of Brazilian Dietary Guidelines in PHC, the present study may contribute to overcome this gap in the literature, by proposing an assessment tool to evaluate FNE actions for the SUS' administrators and others involved in professional qualification to promote adequate and healthy eating. Thus, this study has aimed at developing and assessing the validity and reliability of an evaluation scale of FNE practices in PHC, guided by the Brazilian Dietary Guidelines.

M E T H O D S
The theoretical matrix behind the development of the scale items included chapters of the Brazilian Dietary Guidelines, the Interprofessional Collaborative Practice and the Family Health Strategy (FHS) as models of care organization in the PHC [6,[13][14][15][16]. The scale was developed for the evaluator, a nutritionist with expertise in the Brazilian Dietary Guidelines, to observe the health professionals or team performing FNE actions in order to diagnose the appropriateness of that publication approach.
Validation of the instrument included the analysis of content and face validity [17]. In content validation, the instrument was reviewed by a face-to-face panel composed of six Brazilian Dietary Guidelines experts, in different areas, to cover the entire theoretical matrix used in the development of the scale. Among these professionals, there was a psychometric specialist, two from the FNE area, one specialist in health promotion and in the PHC work process and two researchers in public health, whose study areas involved the Brazilian Dietary Guidelines.
Two validity criteria were used for content validation: representativeness and clarity. Representativeness was used as a measure of the item's applicability to the Brazilian Dietary Guidelines domains [17,18]. Thus, all the experts were instructed to relate each item on the scale to at least one chapter of the Brazilian Dietary Guidelines and to score those items from 1 to 4 for representativeness and clarity (1 = item not representative/clear; 2 = major revisions are required to make the item representative/clear; 3 = minor revisions are required to make the item representative/clear; 4 = the item is representative/clear). The experts also commented on repetition, difficulty, adequacy of the scale to the proposed objective and reasons for the lack of representativeness or clarity of the items.
The average representativeness and clarity scores were then calculated. Items with an average below 3.0 were excluded from the scale, and items with an average between 3.0 and 4.0 were https://doi.org/10.1590/1678-9865202033e190231 4 LC REIS & PC JAIME removed or re-edited according to the suggestions. Potential users of the scale were invited for the face validation, conducted with a focus group mediated by the researcher. Two nutritionists in the area of permanent education and three nutritionists supervising a multiprofessional residency agreed to participate in the study. Participants discussed the structure, convenient use, the scoring system, clarity of the items and usefulness of the scale.
After content and face validation, a new version of the scale was presented on an online platform (Google Forms) for these panels' participants to review again the representativeness and clarity of each item following the same methodology applied in the content validation. Additionally, the professionals were invited to comment on the items and the scale filling out manual, incorporated as a suggestion after face validation.
The agreement was assessed and intra-and inter-rater reliability was tested by inviting nutritionists, experienced in working with the Brazilian Dietary Guidelines in different areas of expertise in Public Health, to apply the scale at two different time points: test at one time point and retest after thirty days. Of the 30 raters invited, 24 (80.0%) worked in the FNE area, 23 (76.7%) as university professors, 11 (36.7%) in the area of permanent education and four (13.3%) with collective feeding. Considering the impossibility of obtaining the exact same situation in the real PHC scenario for comparison, at two different time points, it was decided to produce a series of videos simulating the situations experienced by health professionals, inspired by similar studies [19,20].
For didactic purposes, four videos with approximately seven minutes duration were scripted and produced, representing routine situations at PHC involving FNE practices. The contents of the Brazilian Dietary Guidelines were strategically distributed throughout the four videos so that the raters could apply the scale while watching the performances. This strategic distribution aimed at scripting the videos with complete and incomplete approaches of each chapter of the Brazilian Dietary Guidelines, in order to bring fiction closer to the PHC reality.
The elaboration of the scripts was under the responsibility of the researchers of this study with the support of communication professionals in order to present different contexts of the health care and nutritional care organization: individual care, shared care, home visit and health education group [21][22][23][24]. The video 1 (individual nutritional care) shows a nutritionist at the Unidade Básica de Saúde (UBS, Basic Health Unit) attending an overweight patient with decompensated diabetes, referred by the doctor [21]. This video covered the following contents of the Brazilian Dietary Guidelines: principle "diet is more than intake of nutrients" (chapter 1); how to recognize natural and minimally processed foods, examples of these foods and reasons for prioritizing them; how to recognize processed foods and examples of these foods; how to recognize ultra-processed foods, examples of these foods and reasons for avoiding them (chapter 2); examples of healthy meals (chapter 3); obstacles: information and advertising (chapter 5).
Video 2 (shared care) was produced to demonstrate a mother and daughter with a conflicting relationship, attended by a psychologist at the request of the community health agent [22]. In this video, the following contents of the Brazilian Dietary Guidelines were addressed: principle "diet is more than intake of nutrients" (chapter 1); reasons to prioritize the consumption of natural and minimally processed foods; examples of ultra-processed foods and reasons to avoid them (chapter 2); eat regularly and carefully, in appropriate environments and in company (chapter 4); obstacles: time, supply and cooking skills (chapter 5).
Video 3 (home visit) shows PHC health professionals visiting the home of a family living in their territorial assistance area, after the school reported to the UBS' social worker a case of a undernourished child, with aggressive behavior and many school absences [23]. The following contents of the Brazilian Dietary Guidelines were addressed: claiming the right to have access to adequate food (principle "dietary guidelines broaden autonomy in food choices"); examples of natural and minimally processed foods and reasons for prioritizing them; examples of ultra-processed foods and reasons for avoiding them (chapter 2); example of a healthy meal (chapter 3); eating in company (chapter 4); obstacles: supply and cost (chapter 5).
Video 4 (health education group) shows a health education meeting mediated by a nutritionist with adult UBS' users [24]. In this video, the following contents of the Brazilian Dietary Guidelines are addressed: principle "diet is more than intake of nutrients" and "healthy diets derive from socially and environmentally sustainable food systems" (chapter 1); how to recognize natural and minimally processed, processed and ultra-processed foods; examples of foods from each group; reasons to prioritize the consumption of natural and minimally processed food and reasons to avoid ultra-processed foods; consumption of processed foods as part of meals based on natural and minimally processed foods; use oils / fats, salt and sugar in small amounts (chapter 2); example of a healthy meal; how to prepare a food group (chapter 3); eat in company(chapter 4).
For data collection of the videos analysis using the scale, an online platform (Google Classroom) was created, divided into sections: presentation of the investigation project and of the Free and Informed Consent Form. Instructional video for applying the scale while watching the videos; questions to characterize the respondent; and presentation of the four videos with a scale to be filled out for each video.
All raters were instructed to write down notes of the observed situations in order to, at the end of the video, decide among the four answer options in each item of the scale. The respondent was allowed to complete the tasks in stages and to write comments about the video watched and the application of the scale.
After a period of 30 days, the raters were invited to reassess the video 1 (individual nutritional care) using the scale. We opted to use this video in the test and retest, assuming that the raters were more familiar with the analysis of individual nutritional care and to enable participants to adhere to the retest. Supplementary material 1 describes, for each video reviewed by the raters, the distribution of the responses in the 17 items of the scale [25].
The terms agreement and reliability were used following the Guidelines for Reporting Reliability and Agreement Studies [26]. For the reliability analysis among the raters in the ordinal items, evaluated in the four videos, the weighted Kappa was used. The STATA command "kap" was used to compute the Kappa among the 30 raters in each video. Such a function assumes that the number of raters is constant. The combined Kappa value is the weighted average of the individual Kappas.
The magnitude of the weighted Kappa is just like the commonly used unweighted Kappa, where values greater than 0.75 would indicate excellent Kappas and those lower than 0.4, poor reliability [27]. Subsequently, a test-retest assessment was performed. For each rater, self-agreement and reliability was assessed. All statistical analyses were performed using the STATA version 14 package and for the inferences derived from the kappa index and agreement hypothesis tests, the adopted level of significance was <0.05.
The inferential statistics of the hypothesis test for the kappa index and agreement is based on the Z value, measuring the difference between an observed statistics and its population parameter in standard error units (z). To test whether Kappa is statistically greater than zero, this measure is https://doi.org/10.1590/1678-9865202033e190231 6 LC REIS & PC JAIME divided by its standard error [28]. The resulting value is a one-tailed Z test. Further details on how the Kappa standard error is estimated can be obtained from Fleiss [29]. All analyses (kappa index and agreement) assumed the category nature of the scale responses as outcome.
This study was approved by the Research Ethics Committee of the School of Public Health at USP (CAAE 52548815.4.0000.5421 and CAAE 56303716.6.0000.5421). All participants expressed an interest in participating in the study by signing an informed consent form.

R E S U L T S
The scale was initially developed with 72 items covering all chapters of the Brazilian Dietary Guidelines. Out of chapter 1, only the principles that could be addressed directly by the health professional or the health team in health care situations were selected: "diet is more than intake of nutrients", "healthy diets derive from socially and environmentally sustainable food systems" and "dietary guidelines broaden autonomy in food choices".
The first version of the scale was presented to a panel of six experts for content validation, and was reduced to 25 items in the second version. The face validation panel was attended by five potential users of the scale. In addition to synthesizing the scale items and proposing semantic corrections, the group suggested a fill-out manual for the rater in charge of applying the scale.
The third version comprised 17 items accompanied by the manual [30,31] reviewed by four experts/researchers. The manual presents the contents of the Brazilian Dietary Guidelines expected to be identified in each question/item in order to help the rater to choose an answer: 0=not observed (no content addressed); +1 = partially observed (only 1 content addressed); +2 = satisfactorily observed (only 2 contents addressed); +3 = fully observed (all OR at least 3 contents were addressed).
Considering the relevant reduction in the number of items on the scale and the inclusion of the fill-out manual, the third version of the scale was reevaluated. Out of the 11 professionals invited, six participated: four of the content validation panel and two of the face validation panel. The 17 items were maintained and some adjustments to the manual were made, obtaining the fourth and last version [30,31]. Thirty raters were asked to use the final version of the scale after watching the videos. One month later, the thirty specialists were invited to reassess video 1 (individual nutritional care) applying the scale; 28 completed the reevaluation. Table 1 shows the agreement for each response category in the 17 items in each of the videos. It can be noted that the response categories "not observed" and "fully observed" were those that the 30 raters most agreed with each other in videos 1, 3 and 4 for the 17 items. In the case of video 2, those categories in which the kappa values were highest were "not observed", once again, and "satisfactorily observed". Therefore, the first response category ("not observed") obtained greater reliability among the raters in the set of 17 items in all videos. In terms of overall inter-rater reliability for each video, they all obtained combined kappa values greater than 0.4. Table 2 shows the agreement (expected and observed) and kappa values for all 28 raters in the test-retest. It should be noted that only one of the raters had a self-agreement below 40%, with 19 raters showing agreement above 80%. In the test-retest, only two raters had kappa values below 0.4.  [24]. * Z was obtained from the inferential statistics of the hypothesis test for the kappa index, measuring the difference between an observed statistic and its population parameter expressed in units of standard error. Note: * Z was obtained from the inferential statistics of the hypothesis test for the kappa index and agreement, measuring the difference between an observed statistic and its population parameter expressed in units of standard error. https://doi.org/10.1590/1678-9865202033e190231

D I S C U S S I O N
The present study aimed to develop and evaluate the validity and reliability of a scale for assessing FNE practices in PHC guided by the Dietary Guidelines for the Brazilian Population. To our knowledge, this is the first instrument to assess the implementation of a food guide by PHC health professionals.
Two technologies were produced to support the dissemination of a food guide's recommendations (scale and set of videos) with the potential to support the implementation and evaluation phases of the public policy cycle [32]. It is understood that in addition to the validated scale, the videos produced can be useful tools in processes of permanent education and professional training for the promotion of adequate and healthy eating. This audiovisual material illustrates how the Brazilian Dietary Guidelines' messages can be worked out on everyday PHC situations. Besides that, the videos can be used as tools to support training on health professionals to implement the Brazilian Dietary Guidelines and raters interested in applying the validated scale.
The content and face validation steps contributed to substantial changes in the items, scale format and extension, demonstrating the importance of incorporating these methodological procedures into the validation process, as shown in other studies [19,33]. The format of the final instrument enables the diagnosis of the occurrence of nonexistent, complete or incomplete approaches of the Brazilian Dietary Guidelines application by PHC health professionals. Thus, it is emphasized that the professional responsible for employing the scale must pay attention to the context evaluated and the limits of the Dietary Guidelines' approach considering the reality of the PHC (duration of visits, users' demand, established priorities, etc.).
Therefore, the instrument can allow a diagnosis of the appropriateness of the approach to the contents of the Brazilian Dietary Guidelines (content approached correctly/incorrectly) and whether the evaluated professional could have expanded his/her educational intervention, making connections between the chapters of the publication with more effective communication strategies. The scale can further be used in the assessment of the health professionals' skills needed to apply the Brazilian Dietary Guidelines after training for qualifying nutritional care in PHC.
Just like in other studies [19,20], intra-and inter-rater reliability was reviewed with raters applying the scale by watching videos. This resource seems to allow a more consistent assessment of the validated tool, making it possible to eliminate potential confounding factors from the results [19].
Regarding the agreement for each response category in the 17 items and four videos, it was observed that the first ("not observed") showed greater agreement among the raters. This category is probably the one that is most clearly identifiable along the 17 parameters evaluated, regardless of the situations presented in the videos. This finding suggests the instrument adequacy to diagnose the absence of the Brazilian Dietary Guidelines application by health professionals, regardless of the situation observed.
The "fully observed" response category may not have shown greater agreement among the raters in video 2, as seen in the other videos, given the characteristics of the situation that was presented (shared care between psychologist and community health agent) or just the script itself. The video in question explored the social and cultural dimensions of healthy eating without the presence of a nutritionist on the scene. Although the raters are experts in the Brazilian Dietary Guidelines and Brazil is experiencing the expansion of the FHS, the country still faces the challenge of changing the conceptual paradigm of healthy eating and FNE practices in PHC, in addition to incorporating collaborative interprofessional practice within this framework [34][35][36][37].
Among the response categories in which there was little agreement, one can consider excluding them in future analyses to provide evidence of validity based on the internal structure. These findings suggest that the scale could have been developed with fewer response categories, which can be interpreted as a limitation of this study, signaling the need for other researchers to continue the instrument validation process.
Reviewing globally the performance of the scale, its suitability is suggested to diagnose the absence of the Brazilian Dietary Guidelines approach, as well as their full approach. It can also be assumed that the categories "partially observed" and "satisfactorily observed" may have not been clear enough for the raters or captured subtle elements, which lead them to disagree with each other.
The overall reliability of the scale for all 4 videos was considered good, and the highest reliability concerned the first video. This video, which addressed individual nutritional care, was chosen for the test-retest as a strategy to increase adherence to the study, and may have overestimated the scale's performance with regard to its reliability because it is a care model with which nutritionists tend to be more familiar. Despite the expansion and qualification of the FHS in Brazil and its potential effects on the health of the population, the rationale of a curative model still persists in the country, in which the activities carried out within the doctor's office (individual consultation) tend to be prioritized [16,[38][39][40].
For Veliz-Rojas et al., the biomedical model has been established worldwide as a health model, disregarding the culture, beliefs, traditions and customs of the community in the health care process [41]. Therefore, the approach proposed by the Brazilian Dietary Guidelines is in line with this demand for the transformation of health education practices.
Considering that the scale proposed in this study had its reliability assessed with the use of audiovisual material, we recommend its application by different Brazilian Dietary Guidelines experts to assess the implementation of this document in FNE actions carried out routinely by PHC health professionals. Thus, the performance of the proposed instrument can be evaluated in terms of the validity of its internal structure. Additionally, it is worth mentioning the operational limits of the scale in view of the wide diversity of educational strategies implemented at PHC and its format, which does not allow capturing operational obstacles experienced by health professionals (shortage of time, focus given by the professional in addressing the contents of the Brazilian Dietary Guidelines in view of the priority established and the users' demands, etc.).
Although these are limitations of the instrument presented in this study, it is worth highlighting its unprecedented character, given the lack of scales available in the literature to assess the adequacy of the implementation of Brazilian Dietary Guidelines in educational actions developed in PHC. This finding is especially important considering that the expansion of the FHS has consolidated the evaluation as an essential component to guide the health practices set up and reformulation processes, favoring especially the qualification of professional performance [38].

C O N C L U S I O N
This study developed, validated and tested the reliability of a scale for the assessment of FNE practices carried out by PHC professionals. The scale was submitted to content and face validation and proved to be adequate to diagnose the absence of the Brazilian Dietary Guidelines approach as well as their full approach, with good agreement and intra and inter-rater reliability in different health care situations.

A C K N O W L E D G E M E N T S
To all nutritionists that participated in the data collection of this study for their valuable contribution. To André Vessoni, Technical Support Scholarship Holder, for his help with data collection, scripting and video production. To Studio FICs for their efforts in producing the videos mentioned in this publication.

C O N T R I B U T O R S
LC REIS performed the data collection, analyzed and interpreted the results and wrote the manuscript; PC JAIME designed the study, interpreted the results and revised the manuscript. Both authors read and approved the final manuscript.