Evaluation of the online management course from the perspective of former students

Objective: To evaluate the online course from the perspective of e-learners as well as the relation between variables. Method: A quantitative, descriptive and exploratory study. Results: After three years, the satisfaction rates in the three listed categories presented an average rate higher than 75%. The coefficients indicated a high consistency of the questionnaire. Considering the overall rates in the three years period, the Instructor Performance category presented the highest rate. Strong associations between Self-Assessment and Instructor Performance, Self-Evaluation and Program of the Course and Instructor performance and Program of the course were identified. There was no association between the three categories mentioned with any other variables existing in the study. Conclusion: E-learners expressed satisfaction with the course that means favored the interaction and the promotion of collective knowledge in nursing management. Also aspects need to be improved, especially the training of the instructor to mediate discussions and encourage student involvement throughout the course. DeScriPtOrS Education, Distance; Training; Education, Nursing, Continuing; Evaluation. Correspondence Addressed to: Geisa Colebrusco de Souza Av. Doutor Enéas de Carvalho Aguiar, 419, Cerqueira César ZIP Code: 05403-000 – São Paulo, SP, Brazil geisacole@usp.br USP


INTRODUCTION
The advance of Information and Communications Technologies (ICT) and the increasingly access to them have contributed to social changes in all areas and, thus, in the educational field.The facilitated access to ICT enabled the development of learning styles based on the intensive use of computer and of the web, breaking the geographic-temporal barriers of access to formal and non-formal education (1) .
Distance education entails countless benefits, additionally to geographic-temporal facilities like convenience, easier access to teaching material, dynamic environment, exchange of experiences among agents participating in the process (2) .
Such break of barriers makes room for new ways of consumption of information in the educational scenario, mainly marked by the traditional forms of on-site classroom.Similarly, the education of human resources in the field of health and permanent education are mainly conditioned to that traditional scenario, restricted to courses developed in the working environment for a large number of workers, but that hardly promotes interaction among professionals experiencing different working realities.
It is worth mentioning that permanent education in health bears a crucial peculiarity that should be considered, i.e., students are adults, professionals living in a social context where perceptions and education are permeated by experiences in continuous process of transformation and interchange with other professionals or with the population targeted by their work (3) .
There is a need for developing more creative learning means to keep pace with the quick scientific development, not disregarding the dynamic context of work in health.ICT-based education in distance courses is an option that enhances continued education, the exchange of experiences, access to scientific knowledge and consistent upgrading.
Online training or course, as it is referred to herein, can be understood as a set of educational actions systematically planned in the light of improving and acquiring knowledge, skills and attitudes by students, with space and time flexibility between professors and students and synchronous and asynchronous activities that facilitate interaction and interactivity among agents (4) .Those educational actions aim at acquiring competences to fill in the work performance gaps and prepare professionals to take on new roles (5) .
Distance Education (DE) is not a novelty in Brazil.Throughout years, different educational models have been implemented with some variations and combinations.The most prominent ones are the teleaula, videoaula and web (6) .
The official concept of DE by the Ministry of Education is (7) "an educational modality where didactic-pedagogic mediation in teaching and learning processes takes place through the use of information and communications means and technologies, where students and tutors develop educational activities at different places or times".
In the last few years, education through internet-based distance courses became very popular in the national and international scenarios, despite the questioning about its quality and validity as educational activity.According to a survey by the Brazilian Association of Distance Education (ABED, in Portuguese) amongst 3.5 million students assessed in 2011, 77% used to take some kind of free course for professional supplementation and upgrading (8) .In the same survey the evaluation of courses regarding their quality started to be mentioned as one of the main obstacles posed to the DE (8) .
In face of the aforementioned, there is an urgent need for developing and applying consistent mechanisms to evaluate quality and satisfaction regarding the course, mainly by the target audience, i.e., the students.Thus, evaluation aims at observing if the course objectives are being fulfilled, and providing guidance to the team engaged in the course design to find more pertinent forms of dynamic as these provide the information required to trigger changes and promote continuous improvement and upgrading.
Courses can be assessed from different theoretical perspectives.Here, the guiding referential comes from Social and Labor Psychology that adds to the four levels described by Kirkpatrick, the analysis of other labor environment components (9) and the articulated analysis of behaviors to provide organizational support and to help the transfer and impact of training on the job (10) .
The opinion of distance course participants is crucial to evidence quality, employing pre-established criteria such as structure, layout and online environment, tutor's attitude and teaching material (11) .
Therefore, the evaluation of courses in the light of former students has driven the realignment of it; thus, this study bears the objectives of evaluating the online Nursing Management course in the light of former students, as well as checking the existence of links between the variables of the study.

METHOD
It is a study of quantitative nature, descriptive and exploratory, applied to participants concluding three versions of an online refreshing course in Nursing Management.
This course was designed in the light of Andragogytheory of significant learning -and of dialectic methodology facilitated by ICTs with synchronous and asynchronous activities scheduled.Resulted from an international partnership, it was led by professors and tutors of two institutions of higher education in nursing -one in Brazil and one in Portugal -additionally to technical and management tutors.Students were nurses of both nationalities.Because of the course design, activities were mainly online.
The virtual learning environment was the Moodle software (Modular Object-Oriented Dynamic Learning Environment) and, for online classes, it was the Adobe Connect® software that enabled managing and monitoring distance education.There were 90 class hours delivered in 12 weeks, with about 8 weekly hours of activities.
Activities were distributed by topics on nursing management, and aimed at providing support to the nurse's managerial dimension practice.Additionally to the contents, topics were made up by case studies in group, reading and interactive activities, forums, chats, synchronous class-es, among others.Learning evaluation was through case studies and, by the end of the course, an individual case study was applied.
The sampling technique for this survey was non-probabilistic upon agreement on participating by the end of each version of the course.Of the 98 graduates of the three versions, 77 nurses agreed on answering the questionnaire prepared to evaluate students' satisfaction, after signing the Free and Informed Consent Term (FICT).The survey was approved by the Committee of Ethics in Research of the proposing institution through protocol # 1062/2011/CEP-EEUSP -SISNEP CAAE: 0068.0.196.000-11, and the ethical and legal precepts of research were respected.
Therefore, population was composed by 77 graduates of the three courses delivered from 2011 to 2013.The statistical analysis excluded respondents that have not completely filled in the instrument and, thus, n in each treatment considered only the valid cases.
To gather data, a questionnaire was applied with eight socio-demographic and 52 items to evaluate the course through the Likert scale with 5 agreement levels (from 1 to 5), as shown in Figure 1.
The questionnaire was built inspired by the model proposed by the researcher who expanded the reaction evaluation technique, named evaluation of satisfaction as it expresses the participant's satisfaction in relation to the variables of the three levels of evaluation: reaction, learning and results (12) .
The 52 items were divided into three theoretical categories of analysis; Self-evaluation with 22 items; Tutor's performance with 11; and, Course schedule with 19.
Responses were organized in data sheets, using the Ex-cel® software.Data analysis was supported by the Statistical Package for Social Sciences SPSS® software.

totally disagree totally agree
The closest to the left-hand side, the less you agree with the content of the item.
The closest to the right-hand side, the more you agree with the content of the item.Data was submitted to descriptive and inferential statistical analysis, assuming p≤0.05 as significance.The students' satisfaction index was obtained through data standardization.
The instrument reliability was checked through category of analysis, using the Cronbach Alpha that shows the degree of co-variance of items, pointing out the internal consistence; the highest the reliability coefficient, more internally consistent the measure is (13) .
Following were the coefficients achieved: Self-evaluation 0.911; Tutor's performance 0.945; and, Course schedule 0.940, pointing out high consistency of the scale, evidencing its reliability to evaluate the course.

RESULTS
When characterizing students we found that 67.5% of nurses that completed the survey lived in Brazil and 32.5% in Portugal; 89.6% were females.The average age of subjects was 39 years old (SD±9.0).Of the nurses responding the questionnaire 94.8% held formal labor link and worked for 12.61 years (SD±10.3)on average.The prevailing work site was the hospital setting (55.8%), 20.8% in primary care and 5.2% in education institutions.Specialization (59.7%) was the highest education level of respondents followed by undergraduate nursing course (23.4%), while Master's and Doctors' degrees accounted for 16.9%.
Of the offices, 40.3% worked as assistance nurse, 39% as unit chiefs, and 5.2% were heading health services.
Regarding the level of computer skills 41% said to be very good, 35.1% said to be good, and 9.1% said to be excellent, an indication that 85.7% of students recognized satisfactory level of computer skills.
The course location was: 62.3% at home; 22.1% on job; and, 7.8% both locations.Only six respondents said they took the course in other site than home and work.
Of the course participants, 63.6% responded it was not their first experience with distance course, although the instrument failed in checking if previous experiences were with online courses or other formats of course.
Tables 1, 2 and 3 reported the students' satisfaction indexes in relation to the course by item evaluated, considering 0 to 100 intervals, the average and standard deviation from 2011 to 2013.In the self-evaluation category, of the 19 propositions evaluated 13 (68.4%)reached 75% as minimum satisfaction degree.The best indexes were found for items on acquisition of new knowledge after the course (87.99); delivery of papers on the deadline (87.01); and, regular follow-up of messages sent by the course teaching staff (85.39).On the other hand, the lowest indexes were regarding the students' consultation to course library (recommended reading) (64.47), organization of time to take the course at ease (68.51), and previous knowledge facilitating the learning process (71.75).
Regarding the tutor's performance analysis, all propositions evaluated reached the minimum satisfaction level of 80%, except for the proposition regarding tutor's capacity of sparking interest on the topic (79.22).
The propositions with best averages in the category of course schedule were easiness of subscribing to the course (92.43), if the equipment used by the student was responsive to that required by the course (91.56) and if the course objectives presented in instructions were clear (91.56).In the same category, the worst averages were found in the following items: if propositions in the discussion list have fostered the debate and assisted learning (71.67); if time assigned to perform the tasks proposed was enough (72.40), and if the course was structured in  such a way as to keep students motivated until the conclusion (77.60).Table 4 presents the students' satisfaction indexes with the course, by year and by category of analysis.
We have observed that indexes for three years and three categories presented results above 70.5% and that the lowest indexes were found for 2013.Considering the total index, the Tutor's performance category reached the highest index.For the investigation on association between variables, we could observe there was no association between the satisfaction index averages and the following variables: sex, age, country of residence, schooling level, degree of computer skills, site where the course was taken (home, work, others), if the student had taken distance course before, time of professional exercise, office and work place.
In the categories of analysis, the Pearson's correlation coefficient displayed strong associations between: Self-evaluation and Tutors' performance (0.819); Self-evaluation and Course schedule (0.787); Tutor's performance and Course schedule (0.882).
When comparing categories by year of the course, we found that in 2012 only the Self-evaluation category (ANOVA=0.032)presented average significantly higher than in 2013 (p=0.03).

DISCUSSION
The reaction evaluation tools should approach instructional and administrative aspects of the training program, as these aspects are sensitive to the participants' reactions.Most of the reaction evaluation instruments prioritize the adoption of close-ended questions, and could result in immobility regarding the perception of unforeseen and particular occurrences in the training.On the other hand, it increases the reliability of results as they allow the re-use and replication of the instrument in different situations (14) .
The results of a training course should produce favorable reactions among participants (10) , and this was recognized in the course evaluated.
This study found no association between satisfaction index and the variables selected, similarly to the finding in a satisfaction survey with web-based course that showed no link between satisfaction and learning with gender, age, learning styles, time spent with the course, interactions, course activities and asynchronous conferences.However, it pointed out that those more skilled in computer were more satisfied with the course, although it has not influenced on the student's successful completion of the course (15) .
Self-evaluation showed that new knowledge was acquired after the course.Comparative surveys between online and on-site courses were not unanimous regarding significant differences in relation to both kinds of course, as both can lead to changes on behavior and gain of knowledge (16) .Some studies point out better performance in the carrying out of activities by those learning in virtual environments, either through blended learning or e-learning (17)   .Other study has ratified the conclusion that clinical evidence-based decisions made by physicians attending online course were more likely to happen than those with no access to interactive activities (18) .
A recent systematic review recognizes online nursing education as an alternative to education; however, no statistical difference was found between on-site and online education groups regarding impact, knowledge, skill and satisfaction.Of the 11 articles selected, four disclosed improvement associated to online education compared to the traditional learning techniques, although differences were not statistically significant.A study disclosed weak impact on the online education capability, although with no significant statistical difference (19) .The tutor's performance reached the highest satisfaction indexes; a positive correlation between this category and the self-evaluation category was also found, meaning that the highest satisfaction regarding the tutor's performance, better the student's evaluation regarding their performance.A study points out that tutors' attitude entails significant effects on the satisfaction perceived by students.Tutors have played a core role in the learning process in online environments; when students perceive positive attitude in relation to the tutor there is significant improvement in relation to the student's satisfaction (11) .
Regarding the index interest on the topic sparked by tutors, as this topic achieved the lowest index in the tutor's performance category, it is understood that tutor's education should be continuous, including the use of new technologies.The professors' pedagogical practice assumes continued pursuit for methodologies to nourish critical, creative and transformer students, what presupposes a provoking educator that proposes new ways of learning and of getting adjusted to technologies (20)(21) .
Likewise, the distance education purpose is to let students gradually develop self-suggestion skills, where professor is the mediator of the knowledge-building process (21) .Literature consistently points out three important elements to the satisfaction perceived by teachers delivering online course in relation to students, tutors and the institution.Student is the most important factor mentioned regarding the influence of satisfaction, leading us to believe that in that light students play a core role in online learning (22) .
The course objectives were presented to students in a clear way, in line with literature, as it enables students to gradually evaluate if objectives are being achieved.Moreover the statement of educational objectives and skills to be developed is crucial to build a comprehensive evaluation process (23) .
In general, results point out that organization, scope, design, linkage and materials used are satisfactory to students.Therefore, it ratifies the findings of surveys that point out that the pleasant interface and simple handling of synchronous and asynchronous communication tools of the course facilitate interaction among students, content and professor, thus helping students in the performance of the activities scheduled (21) .
The degree of computer skills of former students was a favorable characteristic for online education in this course.Corroborating a survey that evaluated the pedagogical and psychological barriers in online university education, technology was not recognized as barrier in online courses, electronic study materials and learning activities were considered as mild barrier, and barriers re-lated to communications between fellows and tutor were considered as significant obstacles (24) , and this could be considered yet another positive aspect of this course, as the index about interaction among tutors, professors and students was evaluated as satisfactory.
It is worth mentioning that the course was based on the premise of building a collaborative online learning room aimed at social interaction, maximizing the interchange of ideas, knowledge and experiences between students-students and students-tutors of both nationalities.Also considering that in this study students are nurses with peculiar professional experiences, the approximation of students mediated by the tutor enhanced the discussion based on the theoretical fundamentals of nursing management, seeking for solutions to the everyday issues found in nurses' managerial work.

CONCLUSION
The former students' satisfaction index showed that the course has generally met the requirements described in literature for online courses, and also recognized aspects that should be improved, notably the tutor's training to mediate discussions and encourage the engagement of students throughout the course.
The association among the three categories of analysis, namely Self-Evaluation and Tutor's performance; Self-evaluation and Course schedule; and, Tutor's performance and Course schedule has disclosed the interdependence among these, ratifying the intrinsic link among the evaluation variables.Therefore, we could state that tutor's performance and the course schedule categories, under the responsibility of the course supporting team, have influenced on former students' satisfaction with tutor's performance.
No links between socio-demographic variables and satisfaction index were found in the three categories of analysis.
The evaluation tool is not the focus of this study but, to provide reliability to the course evaluation, we observed internal consistency and accuracy in the three categories of analysis, although being tested in a population shorter than that established for validation, which could be considered to be a limitation of the study.The delivery of further courses may enable the confirmatory factorial analysis and a new test of reliability.
The results achieved allow us to conclude that the course has enabled the development of a virtual community of practice and learning in cyberspace, favoring the interaction and collaboration in the building, promotion and application of collective knowledge in management in nursing.reSUMO Objetivo: Avaliar curso online na perspectiva do egresso e verificar a relação entre variáveis.Método: Estudo quantitativo, descritivo e exploratório, aplicado aos participantes no final de três versões de um curso de atualização online na temática de Gerenciamento em Enfermagem.Resultados: Os índices de satisfação nas três categorias elencadas, em três anos, apresentaram resultados acima de 75,0%.Os coeficientes obtidos indicaram alta consistência do questionário.Considerando o índice total, a categoria Desempenho do tutor foi a de índice mais alto.Fortes associações entre Autoavaliação e Desempenho do tutor, Autoavaliação e Programa do curso e Desempenho do tutor e Programa do curso foram identificadas.Não houve associação entre as três categorias referidas com as demais variáveis do estudo.Conclusão: Os egressos demonstraram satisfação com o curso, que favoreceu a interação e a promoção do conhecimento coletivo no gerenciamento em enfermagem.Foram reconhecidos, também, aspectos que carecem de melhorias, com destaque à capacitação do tutor para mediar discussões e estimular o envolvimento do estudante ao longo do curso.

Table 4 -
Students' satisfaction index with the course, by year of the course and category of analysis -São Paulo, SP, Brazil, 2013.