Effect of an app on students’ knowledge about diabetes during the COVID-19 pandemic

Abstract Objective: to analyze the effect of an app on Nursing students’ knowledge about diabetes during the COVID-19 pandemic, as well as their self-assessment and satisfaction level. Method: a quasi-experimental study carried out with 40 Nursing students from the Brazilian Northeast region. The E-MunDiabetes® app was used to assess the participants’ knowledge at the pre-test, immediate post-test and after 15 days, as well as their self-assessment and satisfaction level in relation to using the app. The analysis was performed by means of descriptive and inferential statistics (binomial test, Intraclass Correlation Coefficient and Wilcoxon’s test). Results: the comparison of the medians of correct answers in the three periods revealed a significant increase in the post-test. The self-assessment and satisfaction items presented an Agreement Index > 80%, with a total Agreement Index of 96.3% and an Intraclass Correlation Coefficient of 0.91. Conclusion: the app was considered satisfactory and promoted a significant increase in the students’ knowledge, therefore being suitable for its intended use.


Introduction
Since the first global records on COVID-19, a high percentage of those affected had Diabetes Mellitus (DM), especially in the cases with a fatal outcome.
Globally, COVID-19 became a major concern for the DM community (1) . At the same time, it triggered concerns and efforts in knowledge construction for governmental entities, teaching and research institutions, academic classes and health professionals, with a view to its management and coping. It so happens that, in line with the advances in knowledge about COVID-19, health professionals have faced difficulties in clinical management and in supporting the heavy workload in the health systems (2) .
In an attempt to alleviate the burden of frontline health professionals in coping with the pandemic, managers sought help from health students, particularly in the Nursing area (3)(4) . The global crisis installed by

COVID-19 requires Nursing professionals and students
to improve their knowledge about the disease and adapt to new and rapid teaching-learning methods, which have exerted a positive impact on knowledge construction and dissemination to better target the professional practice towards the urgent health needs imposed by the emerging pandemic situation (5) .
Understanding as a premise that Nursing students' learning requires innovation, subsidies were sought in the Problem Based Learning (PBL) method to develop the proposal described in this study. Thus, the current study adds knowledge to the literature, as effective use of PBL embedded in mobile devices provides consistent information to enrich the learning experience, promoting strong practical motivation and cognitive stimulation for creative solutions to problem cases (6) . It is believed that PBL represents the teaching-learning perspective anchored in knowledge (re)construction, where the process is centered on the students and on their ways of interpreting, researching and seeking answers to solve a case or answer questions. Furthermore, it may present favorable results in the remote education period, due to the suspension of school activities during the pandemic (7) .
The COVID-19 pandemic is a public health event and, even in the face of the administration of anti-COVID-19 vaccines, existing and available at the moment, it is also necessary to implement preventive measures and education in health (8) . In view of this, it is indispensable that Nursing students are trained and updated regarding education and reinforcement of guidelines on COVID-19 management together with DM control, in line with the scientific updates released constantly and on a large scale through digital technologies.
In this scenario, the use of educational technologies in the digital environment stands out for the possibility of promoting communication and meeting the receiver's needs more satisfactorily, providing reliable information for health maintenance, thus fostering technological advance in favor of the care process (9) . Therefore, it is understood that digital technologies emerge to mediate the teachinglearning process and to make teaching flexible in order to overcome geographical barriers and allow for dynamic learning (10) . Apps stand out among the educational digital technologies.
A number of studies on the use of apps for Nursing students' teaching and learning, with approaches in different care contexts, have shown positive effects on knowledge acquisition (11)(12)(13) . A randomized clinical study showed that apps can facilitate the contemporary teaching-learning process, calling upon professionals to develop, validate and use teaching tools (14) .
In this perspective, it is up to nurses to develop research studies based on a broad theoretical Nursing framework and that include the production of apps for different care dimensions in the current conditions of the health systems (9) . It is noteworthy that there is currently no app about diabetes education during the pandemic imposed by the new coronavirus. In view of this, the "E-MunDiabetes ® " app on the care of people with DM in global emergencies, such as the COVID-19 pandemic, was developed and validated by judges in the areas of Nursing and Informatics. The body of professionals who participated in face and content validation and in the technology usability assessment considered that the app is adequate (15) .
In relation to apps with an emphasis on the theme of diabetes, there were no studies on this type of tools specifically designed to improve the knowledge of Nursing students or of those attending courses in the health area.
This research aims at filling this gap and at developing, in Nursing students, the understanding of the concepts linked to the new coronavirus, as well as informing about diabetes care during the pandemic. Thus, the objective of the current study is to analyze the effect of an app on Nursing students' knowledge about diabetes during the COVID-19 pandemic, as well as their self-assessment and satisfaction level.

Type of study
A quasi-experimental and non-randomized study of the before-and-after type.

Study locus, period and population
The study was carried out between August and September 2021 in a public Higher Education Institution in the northeastern Brazil. The study population consisted of students from the eighth (n=27) and ninth (n=30) semesters of the undergraduate Nursing course at the aforementioned university, totaling 57 students.

Sample definition
The study sample was calculated from the formula for studies with comparative groups, as follows: n = (p1. Thus, the sample was initially calculated with 36 students. The students included were those with an email address, telephone number and availability to participate in the study during extracurricular hours. The students that did not answer the invitation within 10 days were excluded. The participants were selected for convenience and based on the listed criteria. Among the 57 students, 15 did not answer the invitation and two did not complete the required stages (pre-and post-test). Thus, 40 Nursing students comprised the study sample.

Data collection instruments
Three collection instruments were used to obtain the data, namely: 1) A questionnaire containing the sociodemographic variables (identification of the Nursing students, experience in care and in the use of the Internet and of mobile apps as learning tools); 2) The Quiz instrument to perform the knowledge assessment test; and 3) A questionnaire for self-assessment and satisfaction regarding the app.
The Quiz instrument was inserted into the app to assess the level of prior knowledge and that acquired  (20) , which states that a number of six to twenty specialists is recommended for the validation process.
The questionnaire regarding self-assessment and satisfaction level related to the use of the "E-MunDiabetes ® " app contains twenty items, subdivided into four domains: organization, writing style, appearance and motivation. The following scale was adopted to assess the items: 0: I disagree; 1: I partially agree; and 2: I totally agree. The instrument's total score is calculated as the sum of all the domains. In addition to that, the following questions were asked: How often have you used this app? How much time did you spend using this app?
Would you like to suggest something else to improve the app? Did you have any difficulty using it? Would you like to suggest something to improve it? Did you have any difficulties using it, if so, which one?

Data collection
Data collection was carried out in two phases, both virtually, in response to the isolation condition imposed documents were sent to the students who agreed to participate: invitation letter with the information and the link to download the "E-MunDiabetes ® " app, the Quiz instrument for the the pre-test and the immediate post-test, and the REDCap link to access the FICF and the data collection instruments (characterization, selfassessment and satisfaction with the app).
"E-MunDiabetes ® " is a tool that was elaborated and validated based on the guidelines for diabetes management and on scientific references that aims at assisting students in the development of education on diabetes during world emergencies, such as COVID-19.
The app is compatible with smartphones and tablets that operate with the iOS and Android technologies and can be found from the search tools of these platforms, by typing the name "E-MunDiabetes ® " (15) . For its download, the user needs Internet access, although the app will be available for offline use after saving it to the smartphone or tablet memory.
The "E-MunDiabetes ® " app consists of five screens. 2) Answering the characterization questionnaire and pretest quiz. When they installed the app, the students were directed to the Quiz instrument and it was only after completion that they were granted access to its content. Immediately after handling the technology, the students answered the Quiz. At the end, they were asked to enter their name and email address. Google Forms were sent (the same instrument of the pre-test and immediate post-test was applied at P2). The literature considers that there is a decline in knowledge and skills over time (21) . Thus, verification occurred on the 16 th day, considering that results obtained immediately after educational strategies can be influenced by short-term memorization, which requires a second verification after some time interval to investigate apprehension of the content over the days.
A maximum period of ten days was determined to return the evaluation, being extended for the same period. Five days after sending the material, the researcher forwarded a reminder by email and telephone, in order to reinforce completion of the instruments and confirm scheduling of the late posttest.

Study variables and data treatment and analysis
The variables were stored in an electronic database For self-assessment and satisfaction with the app, a minimum Agreement Index (AI) of 0.80 was adopted (22) . The binomial test, with a significance level of p>0.05, was also employed to verify agreement proportion equal to or greater www.eerp.usp.br/rlae 5 Negreiros FDS, Flor AC, Cestari VRF, Florêncio RS, Moreira TMM. than 80%, and the Intraclass Correlation Coefficient (ICC) was used as a reliability measure, considering values < 0.40 as low reliability, from 0.40 to 0.75 as moderate reliability, and values > 0.75 as excellent reliability (23) .
The outcome adopted was learning (acquired knowledge) after handling the app from the comparison of the tests performed. The literature reveals that knowledge means remembering specific facts or the ability to apply specific facts to solve problems or issue concepts with the understanding acquired about a given event (24) .
Knowledge about COVID-19 and diabetes was analyzed in time frames: pre-test (P0 -Immediately before using the app), immediate post-test (P1 -Immediately after using the app) and late post-test (P2 -15 days after using the app     The participants' self-assessment and satisfaction with the app showed an AI value > 80%, with an overall AI of 96.3%, meaning significant agreement among the students. The overall ICC was 0.91 ( Table 2). The suggestions made by the students were analyzed and accepted, when pertinent.

Discussion
The current study showed that the use of the "E-MunDiabetes ® " app by Nursing students resulted in a significant increase in knowledge about diabetes during the COVID-19 pandemic and that there was high agreement of satisfactory answers to the items related to self-assessment and satisfaction with the use of this technology.
The opening of communication channels in health through information culminates in a new learning proposal, which is more active and participative in a technological era that evidences interactivity (25) . In this context, the use of "E-MunDiabetes ® ", developed from the active Problem Based Learning method, arises from the need to build bridges between the theoretical and scientific knowledge about diabetes care in the pandemic context imposed by the new coronavirus. This method is employed by several authors who seek holistic and dynamic learning, and is indicated as an effective and highly efficient teaching approach widely applied in educational systems in several countries (26) .

Thus, using validated techniques such as Problem
Based Learning, which works in knowledge construction in the face of the discussion of a problem, the scholars argue that the remote learning processes in force, with the social isolation measures proposed, fosters the use of varied active methodologies, due to the repercussions caused by the new coronavirus pandemic. In this context, widely used in the practical field as a method to approach a clinical condition in a unique way, the PBL teaching-learning strategy adapts to the conditions recommended during the pandemic and contributes an adapted resource to promote self-knowledge and the acquisition of interactive knowledge in favor of qualifying the care process (27) .
The study in question revealed a significant increase Nursing students (p<0.001) when using the Nursing Education Progressive Web Application (NEPWA) (12) .  (6) . Accordingly, a randomized experimental research study with pre-and post-test conducted in Taiwan with 100 Nursing students used a mobile app for clinical care learning and detected that the experimental group had significantly higher knowledge scores and greater satisfaction levels than the control group (13) . There were correlated findings in a controlled experimental study conducted in Turkey with 122 Nursing students, which pointed out that the post-test of the experimental group that used an app on injection practices had a positive effect on the knowledge levels (p<0.05) (11) .
The current study revealed an increase in the medians of correct answers from P0 to P2. These same results were found in a randomized controlled clinical trial conducted in northeastern Brazil, which pointed out that the mean of the scores after the intervention increased, especially in the intervention group, showing an effect on the knowledge of students who used an app on therapeutic communication (14) .
Nurse educators and researchers should collaborate in the development of virtual learning resources to support Clinical Nursing teaching (13) . Given the suggestions, it is recommended, in a future time, to verify the repercussions of using the app in the clinical practice on various health issues and with larger samples and longer follow-ups (14) .
Not all studies with apps to mediate the teachinglearning process had similar methodological designs (some were randomized and others were not), or similar sample sizes; however, they showed a significant improvement in the students' performance when handling educational apps (6,11,(13)(14)28) . Thus, the literature corroborates the positive influence of this type of technology on Nursing students' knowledge, when compared to the traditional teaching method. Therefore, it is necessary to design more effective learning materials in digital environments that may encourage learning.
Therefore, it is reiterated that the use of mobile apps plays an essential role as a learning tool for nurses and Nursing undergraduates, enhancing the clinical practice (6) .
It can be inferred that the "E-MunDiabetes ® " app had an effect on Nursing students' learning, which can have showed that the Nursing students were satisfied when using the app in their learning process (12) . In another study, the Nursing students stated that use of the technology increased their motivation and self-confidence and reduced their concerns (11) .
Similarly to our findings, in a research study conducted in the Philippines on the evaluation of an educational app, the Nursing students attested to its quality through parameters such as ease of use, organization of the content and structuring of the screens.
Therefore, based on the results evidenced, the students considered the app developed as an acceptable, reliable and effective tool to be used by nurse educators and Nursing students, in order to improve and enhance the quality of Nursing education (28) .
In this premise of enhancing undergraduate learning, diabetes educators need to appropriate such technologies, assisting in the dissemination of appropriate and reliable information in learning. In studies on the theme, it can be seen that the use of mobile apps plays an essential role as a learning tool for nurses and Nursing undergraduate students, in addition to facilitating the care provided to the patients in self-management of their disease by disclosing the diverse information acquired within the scope of the technology used (6,29) .
In this understanding, the app is configured as a viable alternative for active in-person and/or remote teaching and learning, in an instrument for training students, favoring the discovery of a new knowledge source (30) . Among these areas in which education can for the course of action, aiming at knowledge acquisition by Nursing professionals.
In the health education field, the use of mobile apps is relevant in the teaching-learning process in Nursing undergraduate studies, as it provides opportunities for the exchange of experiences and information between individuals with different realities, expanding access to the content, enabling engagement, limiting geographical barriers and adapting to specific realities (32) . With regard to the Nursing area, the "E-MunDiabetes ® " app constitutes technological innovation in health, as it turns out to be an app grounded on the needs of compiling diverse information based on scientific recommendations.
In addition, this app has added basic and advanced operational functions to enable the user to employ a contemporary, valid and satisfactory product.
The following stand out among the study limitations: the fact that the students were only followed-up for 15 days, due to the need to finish the research, and the difficulty presented by some students to download the link to access the app, which resulted in the withdrawal of some participants.
As for the implications for the advancement of scientific knowledge, it is believed that "E-MunDiabetes ® " is a technological innovation for health because it is the first mobile app produced in Brazil that is based on a methodological and quasi-experimental study, aimed at Nursing students and enabling quick access to accurate information about the new coronavirus consonant with the care of people with diabetes. Therefore, it is an updated scientific, educational and professional tool that assists both students and nurses, anywhere and 24 hours a day, favoring the apprehension of diverse scientific knowledge and clinical reasoning. In addition, this app may come to contribute to the routine of Nursing teaching and practice since, considering the complexity of the contents, it offers Nursing students and professionals easy access to an updated tool that will guide them in performing diabetes care.

Conclusion
This study allowed evidencing a statistically significant increase in the knowledge of Nursing students who used the "E-MunDiabetes ® " app related to diabetes care during the COVID-19 pandemic. The results of the study support the finding that, at the end of the intervention, there was an improvement in learning through the increase in the median of correct answers in the pre-test, immediate post-test and late post-test.
In addition, the self-assessment and satisfaction with the app items presented significant agreement among the participants. Elaboration and subsequent use of this technology may come to fill a global knowledge and market gap. It was verified that the students' knowledge remained higher even after fifteen days after using the app, confirming the effectiveness of this teaching instrument.
In a future time, it is recommended to verify the outcomes of using the app in the practice and in different performance settings, together with a higher number of participants and with an extended longitudinal follow-up.