Four levels of evaluation nurse training program on regional food

Objectives : to verify the effectiveness of a regional nutrition training program for nurses based on four levels of assessment: response, learning, behavior, and conclusions. Methods : this was a longitudinal, almost experimental before and after type of study, carried out in 2014 with eight nurses and 62 mothers attended in the Primary Health Care Units of the rural area of Ceará, Brazil, in a training program using educational technology on regional food. Results : the evaluations showed a positive reaction from the nurses; a very good post-test average (92.3%); a change in nurses’ behavior, verified by 30 educational activities after training; and satisfactory results, verified by improvement in knowledge, attitude and practice of the mothers about regional food and food safety after participating in the educational interventions carried out by the trained nurses. Conclusions : the training program was considered effective, showing positive results in the four levels of evaluation. Descriptors: Educational Assessment; and Nutritional realizadas por las enfermeras entrenadas. Conclusiones : el programa de entrenamiento ha sido considerado efectivo, evidenciando resultados positivos en los cuatros niveles de evaluación. Descriptores: Evaluación Educacional; Educación Alimentar y Nutricional; Capacitación; Promoción de la Salud; Enfermería.


INTRODUCTION
The National Strategy for the Promotion of Breastfeeding and Healthy Complementary Feeding has as its objectives the qualification of breastfeeding and healthy complementary feeding promotion actions as well as the improvement of the competencies and skills of health professionals as a routine activity of the Basic Health Units (UBS) (1) .
Studies have emphasized the need for health professionals' improvement, because, in a systematic review regarding counseling on food for children under 2 years of age, it was discovered that out of 29 studies conducted in Brazil between 1998 and 2011, 18 were focused on training, highlighting the importance of those studies for improving the performance of professionals (2) .
The Health professionals informed difficulties in the development of food and nutrition educational actions (3) . Thus, a training program with Primary Health Care nurses on regional food can be an effective alternative for improving the professionals, in order to enable them to promote healthy and safe nutrition in families, ensuring adequate child growth and development.
Regional food is also encouraged by the Ministry of Health in the search for healthy eating, through the instruction manual "Brazilian Regional Foods", which provides a variety of fruits, vegetables, beans, roots, cereals and herbs available in each region, as well as suggests culinary preparations for the population (4) .
Therefore, it is appropriate not only to develop training programs, but also to verify their effectiveness. According to the Kirkpatrick model, evaluation of training programs can be carried out at four levels: reaction (evaluates participants' satisfaction with the training), learning (evaluates the change in knowledge), behavior (evaluates the professional's performance in his/her work environment) and outcome (verifies the tangible results of the program) (5) .
National and international surveys have been implementing training programs with nurses on various topics and conducting evaluations of reaction (6) , learning (7)(8) , behavior (9) and results (10) .
It should also be noted that the use of more than one level of evaluation can show more consistent and comprehensive results. For example, in a training held in São Paulo with nurses on the systematic evaluation of pain in patients with morphine use, two levels of evaluation were used at the same time, and it was found that in addition to the change in the nurse's decision process for the dispensation of morphine (behavior), there was an impact on patients' pain relief (result) (11) .

OBJECTIVES
To verify the effectiveness of a regional feeding training program for nurses based on four levels of assessment: response, learning, behavior, and results.

Ethical Aspects
The study was preceded by the approval of the Research Ethics Committee of the Federal University of Ceará and the signature off all nurses and mothers who participated in the study in the Free and Informed Consent Term.

Design, period, and place of study
Almost experimental, longitudinal, before-and-after-type study conducted in the Primary Health Care Units (UAPS) of a rural area in a municipality of the State of Ceará, Brazil, from September 2014 until March 2015.
The choice of the study site took into account that only 12% of this population living in rural areas had food security (12) , a concept defined as: constant access to food of sufficient quality and quantity by a population.

Population or sample: inclusion and exclusion criteria
The study population consisted of all 12 nurses working in the UAPS in the rural area of this region and the mothers of children assisted in these UAPS who participated in the educational intervention.
Among the 12 nurses, 9 were selected according to the following inclusion criteria: attend the first meeting of the training program. Participation of professionals in previous regional food training was adopted as exclusion criteria, with three nurses excluded. Also, one nurse was excluded by criterion of discontinuity since she asked to quit the UAPS where she worked after starting the program. Thus, the sample consisted of eight nurses.
Regarding the mothers, the sample was estimated for convenience, and from six to ten mothers were invited by each nurse to be on her group, so as to not compromise the effectiveness of the educational intervention. Thus, it was obtained a sample of 62 mothers, which met the following inclusion criteria: living in area of the study; having at least one child between 6 months and 6 years of age; and being attended at the UAPS where the nurses worked. Those who did not have telephone contact were excluded, since it was impossible to continue the evaluation after the educational intervention.

Study protocol
The data collection included stimulatingly the training program with the nurses (September to December 2014), composed of feedback, learning, and behavior evaluations; and the evaluation of results from the mothers who participated in the educational interventions conducted by the trained nurses (September 2014 to March 2015).
The nurses' training program followed the methodology of oral exposure dialogue and was developed in five meetings according to the schedule below.
Most of the meetings lasted an average of 50 minutes, with the exception of educational technology workshops, which lasted four hours, and the educational interventions, which lasted an average of 25 minutes. The educational technology used in the program was the validated serial album "Regional foods promoting food security", which consists of seven illustrations, which should be exposed to the target audience, and six worksheets, which guide the professional and address the themes: food and nutritional security; regional food; food hygiene; and regional recipes for cashew, banana, pumpkin and seriguela [cirguela] (13) .
of Four levels to evaluate a nurse training program on regional food Ferreira AMV, Sabino LMM, Sandoval LJS, Lima FET, Martins MC, Almeida PC, et al.
It is noteworthy that the workshops -tasting of regional recipes, demonstration of the application of technology and simulation activity (first meeting); and educational interventions performed by trained nurses (second meeting) -were developed by the researcher in a previous study (14) . However, only two levels of training evaluation were utilized: reaction evaluation and evaluation of the nurses' behavior regarding their performance and aptitude for application of educational technology.
In turn, in this study, the training program was continued to allow the use of the four levels of evaluation. The reaction´s evaluation occurred again to verify the nurses' satisfaction with the program as a totality (12 weeks), being carried out based of a form containing seven open questions that addressed: workload adopted, methodology used in all meetings, satisfaction with the educational technology used and suggestions for its improvement.
In the learning evaluation of nurses, the pre-test and post-test were knowledge previously validated with content validity index of 0.95 and 0.94, respectively (15) , which are composed of 13 questions addressing the themes of the serial album. The pre-test has an objective character, in which each question has four answer options, and only one alternative must be marked as correct. And the post-test presents affirmatives that should be classified by the nurse as "correct" or "incorrect". In the case of questions considered incorrect, the nurse should correct them on the corresponding lines.
For the classification of the theoretical knowledge of nurses, the following parameters were established: less than 50% of the questions -not acceptable; between 50% and 75% -acceptable; between 76 and 90% -good; and greater than 91% -very good (11) .
The evaluation of nurse's behavior was done by reading the monitoring journal follow-up, aiming to see the change in behavior of professionals in their daily lives on the long term. The journal was built by the researcher to be used by nurses for a period of one month, and was composed by topics that directed professionals to note relevant information about the use of educational technology in their work environment, such as the number of participants, and advantages and disadvantages presented.
Finally, the educational interventions substantiated the evaluation of results. For this purpose, two trained collaborators applied the sociodemographic form and the Knowledge, Attitude and Practice survey (KAP) with the mothers before the intervention, immediately after the intervention and one month, three months and six months after the intervention, by telephone contact.
It is worth noting that, in case of difficulty in establishing telephone contact, the mothers should return the call daily up to 45 days after the intervention, otherwise after this period, the mothers who did not answer were removed from the subsequent stages of result evaluation. Therefore, at the first moment, the sample consisted of 62 mothers, had a progressive decrease of this quantity within one month (n = 44), three months (n = 28) and within six months (n = 23) of follow-up.
The current KAP survey on regional foods was prepared and validated by two experts in the area and a pilot test performed with 15 family members in previous research. It consisted of 10 questions divided into the three axes (knowledge, attitude and practice), which can be classified as appropriate or inappropriate after response from the participants (13) .
For the axes to be taken as applicable, each one should be considered as specified in Chart 2. -Application of pre-test knowledge in nurses.
-Workshop for the use of educational technology on regional food.
2 nd meeting (2 nd to 5 th week) -Field activity: Educational intervention carried out by each nurse trained at UAPS in which she worked with the selected mothers. Chart 2 -Description of responses considered applicable in the regional food knowledge, attitude, and practice axes

Axes Applicable
Knowledge -Have heard of regional foods and know what they are for; -Cites at least three types of those foods; -And mentions at least two types of preparation options using those foods.

Attitude
-Knows the importance of introducing regional foods into a child's food routine; -Highlights the benefits of this diet: preventing malnutrition and obesity; promoting food security; and/or being a new food alternative.

Practice
-Mention having already offered the child some regional food; -Accepts that this food may be included whenever it is considered necessary.

Analysis of results and statistics
The Statistical Package for the Social Sciences (SPSS Inc., Chicago, United States), version 20.0 was used to analyze the information. The data were organized in tables, and the exploratory analysis occurred by absolute and relative frequencies. The Wilcoxon test was used to analyze the difference in scores between pre-test and post-test nurses' knowledge. Cochrane's Q statistical test was performed to verify the association and/or correlation of mothers' knowledge, attitude, and practice within the timing of the KAP survey.

RESULTS
All eight trained nurses were female. The average age was 32+10.4 years. Regarding academic title prevailed broad sensu specialization (6; 75%) and Primary Health Care (5; 62.5%). Although they did not have previous training in the same subject, six nurses (75%) reported having some knowledge about regional food; and four (50%), on food and nutritional safety.
of Four levels to evaluate a nurse training program on regional food Ferreira AMV, Sabino LMM, Sandoval LJS, Lima FET, Martins MC, Almeida PC, et al.
Regarding the nurses' reaction by reading their reports, there was a positive reaction to all the meetings of the training program. The nurses found it appropriate to develop the program in five meetings, covering 13 weeks.
Regarding the technique of group discussion held between the nurses and the researcher at the third and fourth meetings, all nurses agreed that this teaching strategy enabled better learning about the proposed content and provided an opportunity to exchange personal experiences.
The nurses were pleased to apply the serial album "Regional foods promoting food safety" with the mothers in the UAPS, highlighting in their perceptions that educational technology promoted greater integration, participation and learning of mothers during the activity. Only two participants in the program made suggestions for improvement: inclusion of more recipes and a broader approach on food hygiene.
The learning evaluation pointed to important changes in nurses' knowledge after the training, although not statistically significant (Table 1). educational interventions should be highlighted. It was noted that mothers aged between 20 and 34 years (77.4%), married (74.2%), with more than ten years of study (56.5%), only one child aged between 6 months and 6 years (83.9%), with extra domiciliary work (66.1%) and with a monthly income higher than a minimum wage (65%).
To evaluate the results, the KAP Survey was applied to mothers before and after the educational intervention (Table 2).   Table 2 shows that the educational intervention provided an increase in the axes of knowledge, attitude, and practice of mothers on regional food, with statistical significance in the axes of practice (p = 0.022).
In the knowledge axes, no mother presented adequate knowledge before the intervention, but after the intervention this number increased gradually until it reached all mothers in the sixth month of evaluation, which made it impossible to calculate the level of significance between evaluations.
Regarding the attitude axes, it is noted that there was no statistically significant difference between the evaluations before and after the educational intervention (p = 0.186), however it is noted that, at all times after the intervention, this number was higher than 86.3%, showing improvement in the attitude of mothers.
Finally, in the practice axes, a statistically significant difference was found between the evaluations before and after the educational intervention (p = 0.022). It is worth noting that, before the intervention, only 22.6% of the mothers presented the adequate practice, while this number reached 65.2% six months after the intervention.

DISCUSSION
In this survey, the evaluation according to reaction showed the nurses' satisfaction with the training program based on the positive evaluation regarding the methodological proposal and the educational technology adopted.
About the duration of the training, the nurses considered it sufficient to conduct it in 13 weeks. It is noticeable that there is a wide variation of time in the training sessions, from hours (16) to weeks (6) . However, in three trainings developed at Brazilian health regulatory agency, the professionals though the training hours (between 8 hours and 20 hours) insufficient for the complexity and amount of content (16) .
It is noticeable that nurses presenting good average scores already in the pre-test (10.8±1.9 questions and 83.6%±15.8%), obtained an improvement of knowledge in the post-test (12±0.9 questions and 92.3%±7.1%), reaching a very good average scores.
The evaluation of nurses' behavior was checked from the follow-up journal. It was attested that the trained nurses took the educational technology, as they performed a total of 30 activities using the serial album, an average of 3.7 activities per nurse in a month, ranging from 2 to 8 for each nurse. Individual activities also prevailed (25; 83.3%) in comparison to group activities (5; 16.7%).
The nurses reported in the journal's follow-up and during the round of conversation that the individual interventions provided greater proximity and interaction between nurses and mothers. However, some mothers were anxious and with greater urgency to complete the educational activity.
In the case of group activities, the advantages of better use and discussion of the themes were highlighted, as well as the greater acceptance by some mothers who were more inhibited, who would not participate actively if they were carried out individually. On the other hand, the noise and dispersion of some participants were stressed as barriers, due to the presence of mothers with their children.
Before presenting the evaluation of the results of the program, the characteristics of the mothers who participated in the of Four levels to evaluate a nurse training program on regional food Ferreira AMV, Sabino LMM, Sandoval LJS, Lima FET, Martins MC, Almeida PC, et al.
Thus, it is inferred that, although there is no precise recommendation on the workload, it is important to plan the courses considering the complexity of its contents, using more than one meeting whenever necessary.
In the present research, the strategies utilized in the field of continuing education in food and nutrition have included educational actions in groups and the use of active methodologies (17) . The professionals consider the inclusion of exercises in the workshops, demonstrations of practical examples and group discussions as a facilitator to the teaching and learning process (6,16) .
Concerning the educational technology used in this study, the nurses emphasized that the series album "Regional foods promoting food security" is an innovative resource that favored the greater involvement and interaction of the users participating in the educational activity. This is due to the fact that the material in question was built in light of the liberating pedagogy of Paulo Freire, which encourages the use of problematizing and dialogic teaching techniques between educator and student (13) .
Thus, the importance of using problematizing pedagogies in the construction of educational technologies is reinforced, since, as shown by a concept analysis made with 13 articles on nursing educational technologies, the use of problematizing pedagogies is still insufficient since only one study adopted this teaching perspective (18) .
It is important to remember that in our first study, nurses were evaluated on their performance and aptitude to use the serial album in educational activities, the majority (87.5%) being considered fit to perform effective interventions (14) .
In addition, the nurses suggested deepening the content of the serial album used in the training to address more recipes and aspects related to food hygiene. However, it is believed that such an increase may become stressful in the realization of educational activities, since, in this training, the nurses developed the activities in an average duration that has already reached 25 minutes.
On the other hand, it is considered relevant to update the serial album with regard to food hygiene, in view of the relevance of the theme of the families' food security. Furthermore, in a study developed in the Federal District, it was observed the low knowledge of mothers about food hygiene, because they reported using more often only water and soap (19) .
In the learning evaluation of this study, the nurses presented satisfactory results, considering that they achieved an incredibly good average score after the training. This corroborates recent research that developed training programs with nurses and also found positive results in learning evaluation (7) , although not always statistically significant (8) .
In the State of Minas Gerais, two types of learning evaluations were used before and after a theoretical-practical course in Basic Life Support, with 32 professionals, among them doctors and nurses. They had their theoretical knowledge evaluated by a test with 20 objective questions; and their technical skills, evaluated by a checklist in simulated care, so that the average score showed significant increase after training (7) .
In a survey conducted in Kentucky with ten nursing interns, it was found that the average rate of learning and confidence of participants also improved after a training program on postoperative complications in an intensive care unit, but did not show statistically significant changes (8) .
Moreover, it is worth mentioning that, before the training of nurses in the present study, a good average score was already obtained. It is concluded that this previous satisfactory knowledge in nutrition can be influenced by factors such as the number of courses held during training, number of years working as a nurse, updating course on nutrition after graduation and receiving other kind of training (20) . Therefore, even in the existence of a previous knowledge about regional foods and food and nutritional safety, an increase was verified with the application of the post-test, which demonstrates the importance of the implemented training program. Also, after analyzing the follow-up journals of the nurses, it was found that there was a change in their behavior, since they incorporated the educational practices in their work at the UAPS.
In line with this study, a quasi-experimental research developed at a university hospital in Turkey highlighted the extent of the behavior change in nurses after their participation in a training program on the need for greater use of the ventrogluteal area in safe drug administration. It was shown that, before training, 76.5% of nurses used the dorsal gluteal area; and only 7.4%, the ventrogluteal area. Already four months after the training, the use of the dorsal gluteal area had decreased (48.1%); and the ventrogluteal area had increased (34.6%) (9) .
In view of these findings, it is opportune to emphasize the breaking down of some barriers in the incorporation of health education activities in the area of food and nutrition in Primary Care, because, according to the report of professionals who work in the UAPS in other studies, educational practices are generally related to chronic diseases and sexual and reproductive health (21) ; and gives priority to age groups at higher risk, such as the elderly and pregnant women (22) .
Regarding the type of approach in the educational practices performed by nurses, although the individual approach prevailed in the present study, an integrative review on nursing practices with health education was found that nurses routinely perform health education activities in both individual and collective forms (23) . At the same time, it is opportune to clarify that the utilization of serial albums in educational activities can be effective both in individual interviews (24) and in group sessions (25) .
Finally, the results of the nurses' training program were evaluated based on the impact on the population, because it is known that the success of the professionals' training must be directly related to improvements in their care practice (26) . Thus, the positive impact of the training program on the care practice of trained nurses was perceived by the improvements in the mothers' knowledge, attitude, and practice on regional food after the educational intervention implemented were verified, and in the evaluation of results.
A study conducted in Ethiopia corroborates these discoveries, since it has found significant increases in the knowledge, attitude, and practice scores of mothers after nutritional education on complementary feeding based on the addition of locally grown cereals and vegetables. In addition, significant improvements in the nutritional status of the children occurred in the intervention group (10) .
It is important to stress out that knowledge and adequate access to food determine healthy eating behavior (27) , but it is also necessary to consider the attitude of the individual, which refers to the intention to put knowledge into practice.
After the educational intervention in this study, as a high level of knowledge and appropriate attitude of mothers was obtained, of Four levels to evaluate a nurse training program on regional food Ferreira AMV, Sabino LMM, Sandoval LJS, Lima FET, Martins MC, Almeida PC, et al. a statistically significant increase in the practice of using regional foods was achieved. These results confirm that a high level of knowledge, especially of nutrition, considerably improves the criteria for food selection and forms the basis for an appropriate attitude towards healthy eating and later safe practice (28) . Therefore, the training program has proved to be relevant as it has provided the opportunity for trained nurses to develop effective educational interventions that modify mothers' knowledge of regional foods, improving their understanding of the importance of introducing these foods into the child's dietary routine and directly influencing their increased use in infant food.

Limitations of the study
The absence of a control group is understood as a methodological limitation, as it is a quasi-experimental study. However, this was a conscious choice, considering the need to train all nurses working in the rural area of the municipality chosen.

Contributions to the area of Nursing, Health or Public Policy
It is believed that the training proposal undertaken will serve as a model for the development of other training programs in various topics aimed at the continuing education of professionals.

CONCLUSIONS
It is concluded that the training program was effective because the nurses obtained positive feedback, better learning and behavior change, and the mothers presented satisfactory results in the program based on adequate knowledge, attitude and practice after the intervention with the educational technology.

FUNDING
This research has received support from the National Council of Scientific and Technological Development (CNPq).