Acessibilidade / Reportar erro

Mental health in adolescence: Elaboration and validation of an educational technology for health promotion

Salud mental en la adolescencia: Construcción y validación de una tecnología educacional para promover la salud

ABSTRACT

Objective:

to build and validate the appearance of a digital educational technology to promote the mental health of school adolescents.

Methods:

methodological study, which included the elaboration and validation of the “Conect@dos com a S@ude” online course based on the Galvis-Panqueva methodological framework. The elaboration was based on a literature review and followed the Theory of Meaningful Learning. The validation was performed by a total of 21 adolescent students, using an instrument adapted for the study. A quantitative analysis was performed from the Content Validity Index and descriptive of the suggestions pointed out by the target audience.

Results:

the analysis of the Content Validity index ranged from 0.8 to 1 in all evaluated items. The adolescents presented some suggestions for improvement for the course, most of them were accepted.

Conclusion:

the course was validated by the target audience.

Descriptors:
Adolescence;; Mental Health;; Educational Technology;; Health Promotion;; Validation Study

RESUMEN

Objetivo:

construir y validar la apariencia de una tecnología educativa digital para promover la salud mental de los adolescentes escolares.

Métodos:

se trata de un estudio metodológico, que incluyó la construcción y validación del curso en línea “Conect@dos con la S@lud” a partir del referencial metodológico de Galvis-Panqueva. La construcción se baseó en una revisión de literatura y siguió la Teoría del Aprendizaje Significativo. La validación se llevó a cabo entre 21 adolescentes escolares, utilizando un instrumento adaptado para el estudio. Se realizó un análisis cuantitativo basado en el Índice de Validez de Contenido y una descripción de las sugerencias señaladas por el público objetivo.

Resultados:

el análisis del índice de Validez de Contenido varió de 0,8 a 1 en todos los ítems evaluados. Los adolescentes presentaron algunas sugerencias de mejoras para el curso, que se cumplieron en su mayoría.Conclusión: el curso quedó validado por el público objetivo.

Descriptores:
Adolescente; Salud Mental; Tecnología Educacional; Promoción de la Salud; Estudio de Validación

RESUMO

Objetivo:

construir e validar a aparência de uma tecnologia educativa digital para promoção da saúde mental de adolescentes escolares.

Métodos:

estudo metodológico, que incluiu a construção e validação do curso online “Conect@dos com a S@úde” a partir do referencial metodológico de Galvis-Panqueva. A construção baseou-se em uma revisão de literatura e seguiu a Teoria da Aprendizagem Significativa. A validação foi realizada por 21 adolescentes escolares, por meio de instrumento adaptado para o estudo. Realizou-se análise quantitativa a partir do Índice de Validade de Conteúdo e descritiva das sugestões apontadas pelo público-alvo.

Resultados:

a análise do índice de Validade de Conteúdo variou de 0,8 a 1 em todos os itens avaliados. Os adolescentes apresentaram algumas sugestões de melhoria para o curso que em sua maioria foram acatados.

Conclusão:

o curso foi validado pelo público-alvo.

Descritores:
Adolescência; Saúde Mental; Tecnologia Educacional; Promoção da Saúde; Estudo de Validação

INTRODUCTION

In Brazil, the Ministry of Health follows the recommendations of the World Health Organization, understanding adolescence as the period from 10 to 20 incomplete years old and youth from 15 to 24 years old, in order to establish that adolescents and young people are in the age group of 10 to 24 years old(11 Ministério da Saúde (BR). Secretaria de Atenção à Saúde. Departamento de Ações Programáticas e Estratégicas. Diretrizes Nacionais para atenção integral à saúde de adolescentes e jovens na promoção, proteção e recuperação da saúde[Internet]. Brasília: Ministério da Saúde. 2010[cited 2021 Jan 7]. 132 p. Avaliable from: http://bvsms.saude.gov.br/bvs/publicacoes/diretrizes_nacionais_atencao_saude_adolescentes_jovens_promocao_saude.pdf
http://bvsms.saude.gov.br/bvs/publicacoe...
).

According to data from the 2010 Census, this audience represented 51402821 (36.89%) of Brazilians, a population seen as essentially healthy(22 Ministério da Saúde (BR). Secretaria de Atenção à Saúde. Departamento de Ações Programáticas e Estratégicas. Proteger e cuidar da saúde de adolescentes na Atenção Básica[Internet]. Brasília: Ministério da Saúde . 2017[cited 2018 Jul 18]. 234 p. Avaliable from: http://189.28.128.100/dab/docs/portaldab/publicacoes/saude_adolecentes.pdf
http://189.28.128.100/dab/docs/portaldab...
). However, mental health problems in adolescence worldwide are estimated at 16% of the burden of illness and injury; being present in 10% to 20% of this population(33 Organização Pan-Americana Da Saúde (OPAS). Folha informativa: Saúde mental dos adolescentes [Internet]. Set 2018. [cited 2020 Oct 7]. Available from: https://www.paho.org/bra/index.php?option=com_content&view=article&id=5779:folhainformativa-saude-mental-dos-adolescentes&Itemid=839
https://www.paho.org/bra/index.php?optio...
).

In addition, 50% of conditions related to mental health express themselves from the age of 14 years old. It justifies the relevance of mental health promotion actions aimed at this audience, in order to offer protective factors and reduce risk factors, through various platforms, which include the school, community, health environments and digital media(33 Organização Pan-Americana Da Saúde (OPAS). Folha informativa: Saúde mental dos adolescentes [Internet]. Set 2018. [cited 2020 Oct 7]. Available from: https://www.paho.org/bra/index.php?option=com_content&view=article&id=5779:folhainformativa-saude-mental-dos-adolescentes&Itemid=839
https://www.paho.org/bra/index.php?optio...
).

Thus, health promotion contributes to the reduction of vulnerabilities, as recommended by the National Health Promotion Policy, favoring the autonomy of adolescents and their families. This is done through health information, which allows them to make an appropriate decision in the direction of improving their quality of life(44 Ministério da Saúde (BR). Secretaria de Vigilância em Saúde. Política Nacional de Promoção da Saúde: Portaria nº 687 MS/GM, de 30/3/2006[Internet]. Brasília: Ministério da Saúde . 2010. [cited 2020 Dec 10]. 60 p. Avaliable from: https://bvsms.saude.gov.br/bvs/publicacoes/politica_nacional_promocao_saude_3ed.pdf
https://bvsms.saude.gov.br/bvs/publicaco...
-55 Tomé G, Matos MG, Gomes P, Camacho I, Gaspar T. Promoção da Saúde mental nas escolas- Projeto ES’COOL. J Clin Child Adolesc Psychol[Internet]. 2017 [cited 2020 Dec 9];8:173-84. Available from: https://www.researchgate.net/publication/327106302_Promocao_da_Saude_mental_naS_eScolaS-Projeto_eScool
https://www.researchgate.net/publication...
).

Information and Communication Digital Technologies (ICDT) are characterized as digital resources for the creation of blogs, educational software and Virtual Learning Environments (VLE)(66 Fontana FF, Cordenonsi AZ. TDIC como mediadora do processo de ensino-aprendizagem da Arquivologia. Ágora (Rio J.) [Internet]. 2015 [cited 2020 Jan 7];25(51):101-31. Available from: https://agora.emnuvens.com.br/ra/article/view/548
https://agora.emnuvens.com.br/ra/article...
), and can represent an efficient strategy to promote health in adolescence. In addition to being based on the concepts of Distance Education (DE), which is due to the affinity of adolescents with technological resources.

Distance education in Brazil is a modality provided for in the Law of Directives and Bases of Education (LDB) since 1996. However, the legislation provides that this “distance” is only physical and/or temporal, not constituting a solitary education, but favoring collaborative learning between teachers and students, in addition to having a multidisciplinary team for the production and management of this educational modality(77 Joye CR, Moreira MM, Rocha SSD. Distance Education or Emergency Remote Educational Activity: in search of the missing link of school education in times of COVID-19. Res Soc Dev. 2020;9(7):1-29. https://doi.org/10.33448/rsd-v9i7.4299
https://doi.org/10.33448/rsd-v9i7.4299...
). It is noteworthy that there are several types of DE, in this study Blended Learning or B-learning was chosen, which represents a form of hybrid learning, combining virtual and face-to-face elements(88 Bacich L. Ensino Híbrido: Personalização e Tecnologia na Educação. Tecnol, Soc Conhecimento [Internet]. 2015 [cited 2020 Aug 2];3(1):100-3. Available from: https://www.nied.unicamp.br/revista/index.php/tsc/article/view/152/138
https://www.nied.unicamp.br/revista/inde...
).

The literature presents some strategies for the insertion of ICDT for the promotion of mental health and converging themes, both in the school environment and in health units. Thus, several digital tools are used, namely: web radio(99 Francisco DJ, Barros RA. Saúde mental e web rádio: processo de inclusão digital. Educação. 2015;38(3):369-78. http://doi.org/10.15448/1981-2582.2015.3.21781
http://doi.org/10.15448/1981-2582.2015.3...
) ; online chat using social networks(1010 Mauch AGD, Costa JEM, Silva KM, Andrade LBSO, Almeida LL, Araújo SL. The use of digital social networks in child and adolescent psychosocial care, in the face of the Covid-19 pandemic. Health Resid J [Internet]. 2020[cited 2020 Dec 8];1(2):1-18. Available from: https://escsresidencias.emnuvens.com.br/hrj/article/view/12
https://escsresidencias.emnuvens.com.br/...
); game-based online viral marketing campaign(1111 Ip P, Lam T-H, Chan SS-C, Ho FK-W, Lo LA, Chiu IW-S, et al. Use of Internet Viral Marketing to Promote Smoke-Free Lifestyles among Chinese Adolescents. PLoS ONE. 2014;9(6):e99082. https://doi.org/10.1371/journal.pone.0099082
https://doi.org/10.1371/journal.pone.009...
); VLE(1212 Cavalcante RB, Ferreira MN, Maia LLQCM, Araújo A, Silveira RCP. Uso de Tecnologias da Informação e Comunicação na educação em saúde de adolescentes escolares. J Health Inform [Internet]. 2012 [cited 2020 Dec 9];4(4):182-6. Available from: http://www.jhi-sbis.saude.ws/ojs-jhi/index.php/jhi-sbis/article/view/197/142
http://www.jhi-sbis.saude.ws/ojs-jhi/ind...
). However, there is a gap in knowledge about this theme to the detriment of others much more explained, such as sexuality in adolescence.

Therefore, this article presents the construction and appearance validation of an outreach course in distance education in a VLE, which aims to promote mental health in adolescence, through understanding about health and disease, in addition to contributing to the reduction of prejudices, stigmas and negative opinions related to the mental health field, using David Ausubel’s Theory of Meaningful Learning as a reference, whose main characteristic is the valorization of the apprentices’ previous knowledge(1313 Ausubel DP. Aquisição e retenção de conhecimentos. Lisboa: Plátano; 2003.).

OBJECTIVE

To elaborate and validate the appearance of a digital educational technology to promote the mental health of school adolescents.

METHODS

Ethical aspects

The study was approved by the institutional ethics committee, having respected all ethical aspects established for research with human beings.

Design, study setting and period

This is a methodological study that allows the construction, validation and subsequent evaluation of research instruments and techniques, in order to produce quality materials, capable of replication in other contexts(1414 Polit DF, Beck CT. Fundamentos de pesquisa em enfermagem: avaliação de evidências para a prática da enfermagem. Porto Alegre: Artmed; 2011. 670 p.). The design of a methodological study included the construction of the online course: “Conect@dos com a S@ude” (Connected with Health) and its validation by the target audience using the Galvis-Panqueva framework(1515 Galvis-Panqueva A, Mendoza P. Virtual learning environments: una methodology for creation. Inform Educ. 1999. 12(2):295-317.), in its 5 phases, namely: analysis, design, development, evaluation and administration.

The process of building the course, covering the phases of analysis, design and development(1515 Galvis-Panqueva A, Mendoza P. Virtual learning environments: una methodology for creation. Inform Educ. 1999. 12(2):295-317.) took place from March 2018 to July 2019, subsequently the process of validating the appearance of adolescents at a public school in the city of Sobral, Ceara, covering the evaluation and administration phases that took place from August to October 2019.

Population, inclusion and exclusion criteria

The study was developed in a state high school, which has a total of 1250 students regularly enrolled, distributed in three shifts: morning, afternoon and night.

For the selection of participants, the following inclusion criteria were used: being regularly enrolled in the School; being in the age group of 10 to 19 years old, 11 months and 29 days; having availability and interest for online participation in the study in the extracurricular period. The following exclusion criterion was determined: failure to complete the process of validating the course and transferring the school.

The sample was of the non-probabilistic and intentional type(1414 Polit DF, Beck CT. Fundamentos de pesquisa em enfermagem: avaliação de evidências para a prática da enfermagem. Porto Alegre: Artmed; 2011. 670 p.) composed of 90 students, enrolled in two classes of 2nd year of the morning shift, indicated by the school principal, who considered the schedule of classes and activities of the students. It is noteworthy that of these, 30 accepted to participate in the study, however, the final sample was composed of 21 students who completed all the planned steps, which included the resolution of the validation questionnaires and the performance of all course activities.

Study protocol

Considering the steps proposed by Galvis-Panqueva(1515 Galvis-Panqueva A, Mendoza P. Virtual learning environments: una methodology for creation. Inform Educ. 1999. 12(2):295-317.), initially a careful analysis was carried out in the literature, which justifies the need to create a course on the proposed theme and, furthermore, in distance education. In addition, the objectives and subjects to be addressed, the target audience, the study environment and the technologies to be used were defined.

The definition of the themes took place through an integrative literature review, which aimed to know the factors that impact the mental health of Brazilian adolescents, in addition to an exhaustive search for literature and distance education courses that came close to the proposal of this course on mental health, in order to support the choice of subjects to be addressed in the course modules, which were divided into 6, namely: 1- Introduction to the Course 2- What is Health? 3- Health is mental health! 4- Sexuality in Adolescence: A subject with a lot to talk about... 5- The practice of Bullying in adolescence 6- I already know what affects my mental health ... What now?

The instructional design of the course was then developed, which includes the design of the didactic material offered in the modules. The navigation path on the virtual platform and the educational technology interface, TAS by David Ausubel(1313 Ausubel DP. Aquisição e retenção de conhecimentos. Lisboa: Plátano; 2003.), was used to support the construction of didactic material and the choice of the Modular Object-Oriented Dynamic Learning Environment (Moodle) resources, for integrate the course. For Galvis Panqueva(1515 Galvis-Panqueva A, Mendoza P. Virtual learning environments: una methodology for creation. Inform Educ. 1999. 12(2):295-317.), the environment must be pleasant to the eye, objective, well-structured, comfortable and efficient. In addition, he/she points out that the VLE tools are essential in favoring the teaching and learning process, being related to the type of VLE, target audience and teaching methodologies.

It is noteworthy that the creation and maintenance of the course in Moodle version 3.0.3 was carried out by the team at the Distance Education Center of Vale do Acarau State University (NEaD- UVA).

Still, Galvis Panqueva(1515 Galvis-Panqueva A, Mendoza P. Virtual learning environments: una methodology for creation. Inform Educ. 1999. 12(2):295-317.) points out the need for an evaluation/validation phase of the system, in general, which takes place based on the judgment of content specialists and technicians, so that they can collaborate with the improvement of the VLE, as well as by the target audience, which in this case are school adolescents. In this study, the evaluation/validation by the target audience was developed.

For data collection, a questionnaire was used to characterize the sample and the questionnaire Software validation module adapted from Batista(1616 Batista SCF.SOFTMAT: Um repositório de softwares para matemática do Ensino Médio - um instrumento em prol de posturas mais conscientes na seleção de softwares Educacionais [Dissertação] [Internet]. Campos dos Goytacazes: Centro de Ciência e Tecnologia da Universidade Estadual do Norte Fluminense- UENF; 2004[cited 2018 Apr 7]. 201 f. Available from: http://www.geogebra.im-uff.mat.br/biblioteca/dissertacao-batista-2004.pdf
http://www.geogebra.im-uff.mat.br/biblio...
), with a total of 23 questions about the software, educational proposal, interactive environment and computer-assisted teaching, organized on a Likert scale of 1 to 5 (1- No, 2-Partially, with MANY restrictions, 3-Partially, 4- Partially, with FEW restrictions and 5- Yes), made available through the Google forms link.

At the end, a space was made available for participants to express themselves freely, presenting suggestions for improvement, positive and negative points of the course and its importance in helping teaching and learning about mental health.

Analysis of results

The data collected through the validation questionnaire were exported from google forms to the Excel 2010 software, in which they were organized and subsequently processed, as well as analyzed in the statistical software R, version 3.5.0.

To analyze the responses of the adolescents, the Content Validity Index (CVI) was used, which is used to quantify the extent of agreement between specialists, also among members of the population. To reach the CVI, three mathematical equations were used: S-CVI/Ave (average of the content validation indexes for all scale indexes); S-CVI/UA (proportion of a scale that reaches scores: 1 - Yes and 2 - partially with few restrictions) and I-CVI (content validity of individual items)(1414 Polit DF, Beck CT. Fundamentos de pesquisa em enfermagem: avaliação de evidências para a prática da enfermagem. Porto Alegre: Artmed; 2011. 670 p.). The item that showed agreement among the members of the population, equal to or greater than 0.80 was considered valid(1717 Rubio DM, Berg-Weger M, Tebb SS, Lee ES, Rauch S. Objectifying content validity: conducting a content validity study in social work reseach. Soc Work Res [Internet]. 2003 [cited 2020 Nov 30];27(2):94-111. Available from: https://www.researchgate.net/profile/Susan_Tebb/publication/265086559_Objectifyng_content_validity_Conducting_a_content_validity_study_in_social_work_research/links/558d3ab008ae591c19da8b51/Objectifyng-content-validity-Conducting-a-content-validity-study-in-social-work-research.pdf
https://www.researchgate.net/profile/Sus...
). Thus, the data were organized based on the responses and discussed according to the relevant literature.

RESULTS

Course construction

The construction of the course included the phases of analysis, design and development. In the analysis, the following were determined: the themes and methodologies for preparing the VLE, through an integrative literature review; the target audience defined, considering the need to know and discuss mental health; educational objectives, workload and necessary technologies for the administration and maintenance of the virtual environment, in an exhaustive search for courses offered in this modality.

The main objective of the course was to bring information about the importance of understanding and taking care of mental health. Chart 1 describes the modules, followed by their learning objectives, methodologies used and workload.

Chart 1
Modules, objectives, methodologies and workload of the "Conect@dos com a S@ude" course, Sobral, Ceara, Brazil, 2019

The second phase included the design process, in which all didactic material was elaborated, with the best media to be used in each module being defined, considering the themes and learning objectives, defined in the previous step. For the elaboration of the contents, the Theory of Meaningful Learning was used as a reference(1313 Ausubel DP. Aquisição e retenção de conhecimentos. Lisboa: Plátano; 2003.). For this purpose, manuals from the Ministry of Health, the National Anti-Drug Secretariat (SENAD), from the Ministry of Justice, Ministry of Education, articles available on the internet, YouTube channels, with emphasis on the Video Health Distribution Channel of Fiocruz, websites and blogs on health and adolescence, among other sources.

All material was organized in the form of interactive books, containing: hypertexts, which are texts with links that lead to pages on the Internet, videos and animations; infographics and figures; in addition to discussion forums; complementary activities and materials.

Each module has a personalized label with an image bearing the logo of the course, which is the silhouette of the human head with the name of the course inside, in addition to an image referring to the theme addressed, as can be seen in Figure 1.

Figure 1
Graphic organization of the “Conect@dos com a S@ude” modules

In Module 1, below the label, a small welcome text was placed, the tools chosen were: The File, in Portable Document Format (PDF), with the course schedule; the URL that leads to links, in Google Drive, of the Informed Consent Form (ICF) and to a pre-test questionnaire, whose data will not be discussed in this article; the Forum, in which the participants introduced themselves and expressed their expectations regarding the course; the Glossary of words and terms to facilitate adolescents’ understanding of the themes.

In general, all modules, starting with Module 2, have similar structure and tools, namely: the Book, which contains hypertexts with links, figures, videos, audios and animations; the Task or URL, which directs you to the survey questionnaires and/or activities; the Forum, in which course participants discuss among themselves and with facilitators about the themes of each module; o PDF file of complementary material, which includes suggestions for movies, videos, documentaries, Internet pages, passbooks, digital books and other materials, to deepen the module’s theme.

In the development stage, the texts were made available in the VLE using the Hyper Text Markup Language (HTML). We tried to use a simple, direct and objective vocabulary that would facilitate the adolescents’ understanding, in order to favor their learning. Activities involving access to social networks were included, such as: Facebook, Instagram and WhatsApp; in addition to the media already mentioned above.

Course validation

A total of 21 adolescents participated in the course validation process. Most participants were female (66.6%), 14 women. The ages ranged between 15 and 18 years old and the family income of the most predominance was 1 to 2 minimum wages, since 14 adolescents (66.6%) claimed to have this income.

All participants had access to the internet at home, but only 8 (38.09%) had a computer at home, thus, most of them had access via their smartphones. The VLE used has support for access on smartphones and tablets with different operating systems, which facilitated access.

The course went through an appearance validation stage with the target audience. Table 1 shows the average of the validation indexes for each item (S-CVI/Ave), as well as the validation index per item (I-CVI) and the proportion of a scale that reaches satisfactory scores (1 -Yes and 2 - partially, with few restrictions) per block of the questionnaire (S-CVI/UAB), obtained by adding the individual CVI of each block and dividing it by the number of questions in it; in addition to the global one (S-CVI/UAG), obtained by adding all individual CVIs and dividing them by the total number of questions.

Table 1
Content Validity Index Of course Evaluation Items, Sobral, Ceara, Brazil, 2019

The data presented in table 1 show the validity of the course according to the adolescents’ evaluation, considering that the average of all items evaluated (S-CVI/Ave) was between 1 and 2, showing the tendency of the respondents to the answers 1- Yes and 2- Partially, with few restrictions; the CVI per item (I-CVI), per block of questions (S-CVI/UAB) and global (S-CVI/UAG) presented a minimum value of 0.8 and a maximum of 1, validating the course for presenting values equal to or greater than 0.8 in all evaluated items.

Despite the good results found, the adolescents presented some suggestions for improvement for the course, which were organized according to the questionnaire evaluation blocks and items, as shown in Chart 2.

Chart 2
Adolescents' suggestions per blocks and evaluation items, Sobral, Ceara, Brazil, 2019

At the end of the validation questionnaire, participants also pointed out positive and negative points of the course, figure 2 shows what most appeared in the participants’ responses.

Figure 2
Positive and negative points of the “Conect@dos com a S@ude” course

Course Administration

The administration stage comprises the installation and configuration of the system(1515 Galvis-Panqueva A, Mendoza P. Virtual learning environments: una methodology for creation. Inform Educ. 1999. 12(2):295-317.), therefore, it will be constantly updated in terms of content and operating system, with the support of NEaD-UVA. It is noteworthy that the course will not undergo a second round of evaluation with the target audience that has already validated it but is subject to changes proposed by the adolescents who will take the course later, as well as by the judges who will participate in the validation step with experts.

DISCUSSION

The “Conect@dos com a S@ude” course falls into the category of outreach courses, which is characterized by having more specific learning objectives. It is worth mentioning that it is not always related to a degree and that it seeks to bring relevant content to its audience at any level of education, through new educational strategies(1919 Silva ARLS, Rebelo S, Santos JVV, Nunes CS, Spanhol FJ. Modelos utilizados pela educação a distância: uma síntese centrada nas instituições de ensino superior brasileiras. Rev GUAL [Internet]. 2011 [cited 2019 Oct 7];4(3):153-69. Available from: https://periodicos.ufsc.br/index.php/gual/article/view/1983-4535.2011v4n3p153/21984
https://periodicos.ufsc.br/index.php/gua...
).

The internet has an ambiguous role in the adolescents’ life, as its excessive use can cause psychological problems and behavioral impact in their life(2020 Salgado PG, Boubeta AR, Tobío TB, Mallou JV, Couto CB. Evaluation and early detection of problematic Internet use in adolescentes. Psicothema. 2014;26(1):21-6. https://doi.org/10.7334/psicothema2013.109
https://doi.org/10.7334/psicothema2013.1...
). However, it also represents a tool for searching for information, health-related ones, and can be useful both for coping with diseases and for promoting health(2121 Ferreira EZ, Oliveira AMN, Medeiros SP, Gomes GC, Cezar-Vaz MR, Ávila JA. Internet influence on the biopsychosocial health of adolescents: an integratitive review. Rev Bras Enferm. 2020;73(2):e20180766. https://doi.org/10.1590/0034-7167-2018-0766
https://doi.org/10.1590/0034-7167-2018-0...
). Thus, the Blended Learning or B-learning type of distance learning was chosen due to the proximity between digital technologies and adolescents, as a way to favor health education with the target audience.

B-learning is a type of hybrid learning that combines pedagogical practices of classroom teaching and distance learning. This type of distance education is characterized by its flexibility in teaching and learning methods and processes, which allow the development of more personalized strategies and focused on students’ learning needs(88 Bacich L. Ensino Híbrido: Personalização e Tecnologia na Educação. Tecnol, Soc Conhecimento [Internet]. 2015 [cited 2020 Aug 2];3(1):100-3. Available from: https://www.nied.unicamp.br/revista/index.php/tsc/article/view/152/138
https://www.nied.unicamp.br/revista/inde...
). It is noteworthy that the course load was 30H/A, divided into face-to-face moments, totaling 7.5 H/A, and the distance, with 22.5 online H/A.

The course had David Ausubel’s Meaningful Learning as a theoretical framework, in order to combine the learning of concepts to reach propositions. Concerning the interaction of new knowledge with the apprentice’s previous knowledge, combinatorial learning was emphasized, due to the scarce approach to mental health in the daily lives of adolescents, which makes it difficult to obtain specific subsunitors, but does not exclude the possibility of more general ideas about the thematic. It was also sought to develop the progressive differentiation that starts from the approach of more general knowledge, in order to reach more specific knowledge(1313 Ausubel DP. Aquisição e retenção de conhecimentos. Lisboa: Plátano; 2003.).

Therefore, before entering the mental health content, the course addresses the broad concept of health, re-signifying the importance of the Unified Health System - Sistema Único de Saúde (SUS) for the Brazilian population, going through the problems that can affect the health of adolescents and mental health. Until the end, with the presentation of the Mental Health Care Network at national and local level.

The didactic material of the course was built with the perspective of presenting the adolescents with different paths that would lead them to the significant construction of learning. The hypertexts with links to internet pages and the complementary material presenting options for movies, videos, documentaries, among others, allow the student to choose how to learn the content. It is noteworthy that the modality of distance education has been increasingly valued, which has been related to quality teaching materials, with clear, interactive and personalized objectives based on the style of the course and the preferences of the target audience(2222 Rosalin BCM, Cruz JAS, Mattos MBG. A importância do material didático no ensino a distância. RPGE- Rev Pol Gestão Educ. 2017;21(n.esp.):814-30. https://doi.org/10.22633/rpge.v21.n.esp1.out.2017.10453
https://doi.org/10.22633/rpge.v21.n.esp1...
).

It is understood that the validation with the target audience is an important step in methodological studies, considering that the agreement of those to whom the products are destined is decisive for its effectiveness. The CVI is shown as a widely used method to verify this validity, both for printed and digital resources based on health education: educational video(2323 Rosa BVC, Girardon-Perlini NMO, Gamboa NSG, Nietsche EA, Beuter M, Dalmolin A. Development and validation of audiovisual educational technology for families and people with colostomy by cancer. Texto Contexto Enferm. 2019;28:e20180053. https://doi.org/10.1590/1980-265x-tce-2018-0053
https://doi.org/10.1590/1980-265x-tce-20...
), booklet(2424 Medeiros JRR, Lima MA, Araújo LL, Galiza FT, Felipe GF, Caetano JÁ. Validation of educational technology for care in hemodialysis. Rev Enferm UFPE. 2016;10(11):3927-34. https://doi.org/10.5205/1981-8963-v10i11a11474p3927-3934-2016
https://doi.org/10.5205/1981-8963-v10i11...
), manual(2525 Ribeiro LCC, Oliveira TC, Moreira S, Paula FA. Construção e validação de manual sobre Burnout em professores. Rev Enferm Cent-Oeste Min. 2017;7:e1317. https://doi.org/10.19175/recom.v7i0.1317
https://doi.org/10.19175/recom.v7i0.1317...
), course(2626 Costa IKF, Tibúrcio MP, Melo GSM, Leite JEL, Dantas RAN, Torres GV. Construction and validation of a distance Basic Life Support Course. Rev Bras Enferm. 2018;71(Suppl6):2698-705. https://doi.org/10.1590/0034-7167-2018-0122
https://doi.org/10.1590/0034-7167-2018-0...
), application(2727 Saboia DM, Vasconcelos CTM, Oriá MOB, Bezerra KC, Vasconcelos Neto JA, Lopes MHBM. Continence App: construction and validation of a mobile application for postnatal urinary incontinence prevention. Eur J Obstet Gynecol Reprod Biol. 2019;240:330-5. https://doi.org/10.1016/j.ejogrb.2019.07.026
https://doi.org/10.1016/j.ejogrb.2019.07...
), among others.

It can be seen, through box 2, that the adolescents suggested changes referring only to Block E, which evaluates the software according to its educational proposal. Block E is divided into two excluding parts, Block E-1 related to software such as Interactive Learning Environments and Block E2- to software such as Computer-Assisted Teaching. Thus, E-1 is directed to a more constructivist educational proposal and E-2 to the traditional teaching model(1616 Batista SCF.SOFTMAT: Um repositório de softwares para matemática do Ensino Médio - um instrumento em prol de posturas mais conscientes na seleção de softwares Educacionais [Dissertação] [Internet]. Campos dos Goytacazes: Centro de Ciência e Tecnologia da Universidade Estadual do Norte Fluminense- UENF; 2004[cited 2018 Apr 7]. 201 f. Available from: http://www.geogebra.im-uff.mat.br/biblioteca/dissertacao-batista-2004.pdf
http://www.geogebra.im-uff.mat.br/biblio...
).

The Conect@dos com a S@ude course is more focused on a constructivist perspective, however, it was decided to select some questions also from block E-2, which evaluated items considered important in this research. It is noteworthy that the CVI, per block and per item, was the lowest in Block E-2 (table 1). What may be related to this distance from the course of a more traditional education.

Regarding the suggestions presented, they were brought to the end of the validation questionnaire, through an open question, it is highlighted that only 9 (42.85%) of the adolescents presented suggestions. The remaining 12 (51.14%) did not respond or they said that the course was already adequate and had no new ideas to improve it.

Studies that include validation with the target audience generally present validation questionnaires with open questions that allow more free responses and opinions of their respondents, as observed in other studies that presented results similar to this study in having achieved a positive evaluation by the target audience, who also collaborated with descriptive suggestions for improving studies(24, 27-29 ).

In block E-1, the participants suggested that the VLE offer other possibilities for virtual interaction besides the forum, and also the inclusion of forms of face-to-face interaction among the participants. It is noteworthy that although the forum was also pointed out as a negative point by an adolescent, who considered that it could cause embarrassment, the tool will be maintained, as it has proved useful for the debate and exposure of ideas by the course participants.

As a form of virtual interaction, chat will be included, which allows users to chat in real time, which can be saved and viewed later by those who participated and by others who were not present in the chat, being ideal for virtual meetings of a class of students who are not available to meet in the same place(3030 Núcleo de Educação a distância da Universidade Estadual Vale do Acaraú- NEaD-UVA. Ambiente Virtual de Ensino à Distância [Internet]. 2019 [cited 2020 Apr 18]. Available from: http://ead2.uvanet.br
http://ead2.uvanet.br...
). In addition, interaction through social networks, especially Instagram and Whatsapp, which has already proved to be efficient with the students who participated in the validation, will also be encouraged.

A study that aimed to analyze the role of interaction tools in VLE Moodle as a tool for interactivity and collaborative learning in an DE discipline(3131 Oliveira JKC. Ambiente virtual de aprendizagem: elementos e ferramentas que influenciam a interação online. REDOC - Rev Doc Cibercult. 2018;2(2):184-96. https://doi.org/10.12957/redoc.2018.31393
https://doi.org/10.12957/redoc.2018.3139...
) showed that forums, chats and e-mail were the tools used and that they contributed significantly to students’ learning, in addition to pointing out Whatsapp as an external tool that also fulfilled a role similar to those already mentioned, being the preferred means of interaction for students.

With regard to face-to-face interactions among the participants, it is noteworthy that because it is a B-learning course, this type of interaction is valid and interesting. Therefore, there will be more moments in which all the course participants can be together in person, which in this validation happened only once, at the closing meeting, mainly due to infrastructure issues at the research site.

Regarding the suggestion that the course should be freer to expose and debate more, it is believed that the format of the course already allows this debate, but that the inclusion of chat and other online tools such as social networks, which provide more moments with all the students of the course, even if in a virtual way, can help to reaffirm this characteristic. It is noteworthy that the union between the virtual and the face-to-face is already a reality in distance education courses, as it is understood that this combination has a lot to contribute to the students’ learning, presenting satisfactory results with regard to their satisfaction and performance(3232 Valente JA. Blended learning e as mudanças no ensino superior: a proposta da sala de aula invertida. Educar Rev. 2014;Ed. Esp(4):79-97. https://doi.org/10.1590/0104-4060.38645
https://doi.org/10.1590/0104-4060.38645...
).

In block E-2, the participants brought the inclusion of games, reports, and dialogues to hold their attention, which will be carried out as far as possible, considering the specifics of Moodle. Considering that the interest of adolescents in digital games can be used as a health promotion strategy, as evidenced in a study that proposed to validate the digital educational game “DECIDIX”(3333 Monteiro RJS, Oliveira MPCA, Belian RB, Lima LS, Santiago ME, Gontijo DT. DECIDIX: meeting of the Paulo Freire Pedagogy with the serious games in the field of health education with adolescents. Ciênc Saúde Colet. 2018;23(9):2951-62. https://doi.org/10.1590/1413-81232018239.12782018
https://doi.org/10.1590/1413-81232018239...
), which simulates the interface of a virtual chat, in which an adolescent talks to a group of friends (adolescents who are currently playing) about an affective/sexual relationship they are living in. The study provides recommendations for investing in the validation of other digital educational games as facilitators of health education, in view of the positive impacts of these tools for health actions with adolescents, due to their proximity to digital games.

Regarding the suggestions to increase the workload, the authors did not consider it feasible to include other subjects and go deeper, considering the way the course is offered now, as an extracurricular activity for adolescents, considering that it was carried out during regular classes. Therefore, increasing the workload could compromise the course in this format that has been developed.

As positive points, the participants highlighted issues related to VLE, about the accessibility of the system, presentation, and relevance of the contents. They also highlighted the attention and availability of the facilitators, virtual and in person.

Finally, it is noteworthy that despite considering that the course has negative points, all adolescents affirmed its importance to help teaching and learning in mental health. Thus, it was considered that the course achieved its objectives.

Limitations of the study

The course is valid in appearance by the target audience, but there is still a need to carry out validation with expert judges.

Contributions to the area

It is hoped that the course can bring adolescents closer to health professionals, collaborating with the debate about mental health, through the union between health education and ICDT, to value students’ prior knowledge as recommended by TAS, and resembling the SUS holistic care model, which considers the individuals’ knowledge and autonomy as a decisive factor for their well-being. Thus, enabling adolescents to access reliable information that allows them to know more about their health and that of their peers, also contributing to a reduction of prejudices, stigmas and negative attitudes related to the field of mental health.

CONCLUSION

This research made it possible to construct and validate the appearance of a health education tool, in the form of a distance course, in line with the specificities and preferences of adolescents, to produce meaningful learning, considering their health needs.

The CVI reached the pre-established values, being considered valid by the target audience and apt for the validation of appearance and content by the specialists. It is suggested that further studies on the development of digital educational technologies be carried out, as they are powerful tools to promote health, especially in adolescence.

  • ACKNOWLEDGMENT We are grateful to all members of the Laboratory for Social Research, Transformative Education and Collective Health (LABSUS) who participated directly or indirectly in this study.

REFERENCES

  • 1
    Ministério da Saúde (BR). Secretaria de Atenção à Saúde. Departamento de Ações Programáticas e Estratégicas. Diretrizes Nacionais para atenção integral à saúde de adolescentes e jovens na promoção, proteção e recuperação da saúde[Internet]. Brasília: Ministério da Saúde. 2010[cited 2021 Jan 7]. 132 p. Avaliable from: http://bvsms.saude.gov.br/bvs/publicacoes/diretrizes_nacionais_atencao_saude_adolescentes_jovens_promocao_saude.pdf
    » http://bvsms.saude.gov.br/bvs/publicacoes/diretrizes_nacionais_atencao_saude_adolescentes_jovens_promocao_saude.pdf
  • 2
    Ministério da Saúde (BR). Secretaria de Atenção à Saúde. Departamento de Ações Programáticas e Estratégicas. Proteger e cuidar da saúde de adolescentes na Atenção Básica[Internet]. Brasília: Ministério da Saúde . 2017[cited 2018 Jul 18]. 234 p. Avaliable from: http://189.28.128.100/dab/docs/portaldab/publicacoes/saude_adolecentes.pdf
    » http://189.28.128.100/dab/docs/portaldab/publicacoes/saude_adolecentes.pdf
  • 3
    Organização Pan-Americana Da Saúde (OPAS). Folha informativa: Saúde mental dos adolescentes [Internet]. Set 2018. [cited 2020 Oct 7]. Available from: https://www.paho.org/bra/index.php?option=com_content&view=article&id=5779:folhainformativa-saude-mental-dos-adolescentes&Itemid=839
    » https://www.paho.org/bra/index.php?option=com_content&view=article&id=5779:folhainformativa-saude-mental-dos-adolescentes&Itemid=839
  • 4
    Ministério da Saúde (BR). Secretaria de Vigilância em Saúde. Política Nacional de Promoção da Saúde: Portaria nº 687 MS/GM, de 30/3/2006[Internet]. Brasília: Ministério da Saúde . 2010. [cited 2020 Dec 10]. 60 p. Avaliable from: https://bvsms.saude.gov.br/bvs/publicacoes/politica_nacional_promocao_saude_3ed.pdf
    » https://bvsms.saude.gov.br/bvs/publicacoes/politica_nacional_promocao_saude_3ed.pdf
  • 5
    Tomé G, Matos MG, Gomes P, Camacho I, Gaspar T. Promoção da Saúde mental nas escolas- Projeto ES’COOL. J Clin Child Adolesc Psychol[Internet]. 2017 [cited 2020 Dec 9];8:173-84. Available from: https://www.researchgate.net/publication/327106302_Promocao_da_Saude_mental_naS_eScolaS-Projeto_eScool
    » https://www.researchgate.net/publication/327106302_Promocao_da_Saude_mental_naS_eScolaS-Projeto_eScool
  • 6
    Fontana FF, Cordenonsi AZ. TDIC como mediadora do processo de ensino-aprendizagem da Arquivologia. Ágora (Rio J.) [Internet]. 2015 [cited 2020 Jan 7];25(51):101-31. Available from: https://agora.emnuvens.com.br/ra/article/view/548
    » https://agora.emnuvens.com.br/ra/article/view/548
  • 7
    Joye CR, Moreira MM, Rocha SSD. Distance Education or Emergency Remote Educational Activity: in search of the missing link of school education in times of COVID-19. Res Soc Dev. 2020;9(7):1-29. https://doi.org/10.33448/rsd-v9i7.4299
    » https://doi.org/10.33448/rsd-v9i7.4299
  • 8
    Bacich L. Ensino Híbrido: Personalização e Tecnologia na Educação. Tecnol, Soc Conhecimento [Internet]. 2015 [cited 2020 Aug 2];3(1):100-3. Available from: https://www.nied.unicamp.br/revista/index.php/tsc/article/view/152/138
    » https://www.nied.unicamp.br/revista/index.php/tsc/article/view/152/138
  • 9
    Francisco DJ, Barros RA. Saúde mental e web rádio: processo de inclusão digital. Educação. 2015;38(3):369-78. http://doi.org/10.15448/1981-2582.2015.3.21781
    » http://doi.org/10.15448/1981-2582.2015.3.21781
  • 10
    Mauch AGD, Costa JEM, Silva KM, Andrade LBSO, Almeida LL, Araújo SL. The use of digital social networks in child and adolescent psychosocial care, in the face of the Covid-19 pandemic. Health Resid J [Internet]. 2020[cited 2020 Dec 8];1(2):1-18. Available from: https://escsresidencias.emnuvens.com.br/hrj/article/view/12
    » https://escsresidencias.emnuvens.com.br/hrj/article/view/12
  • 11
    Ip P, Lam T-H, Chan SS-C, Ho FK-W, Lo LA, Chiu IW-S, et al. Use of Internet Viral Marketing to Promote Smoke-Free Lifestyles among Chinese Adolescents. PLoS ONE. 2014;9(6):e99082. https://doi.org/10.1371/journal.pone.0099082
    » https://doi.org/10.1371/journal.pone.0099082
  • 12
    Cavalcante RB, Ferreira MN, Maia LLQCM, Araújo A, Silveira RCP. Uso de Tecnologias da Informação e Comunicação na educação em saúde de adolescentes escolares. J Health Inform [Internet]. 2012 [cited 2020 Dec 9];4(4):182-6. Available from: http://www.jhi-sbis.saude.ws/ojs-jhi/index.php/jhi-sbis/article/view/197/142
    » http://www.jhi-sbis.saude.ws/ojs-jhi/index.php/jhi-sbis/article/view/197/142
  • 13
    Ausubel DP. Aquisição e retenção de conhecimentos. Lisboa: Plátano; 2003.
  • 14
    Polit DF, Beck CT. Fundamentos de pesquisa em enfermagem: avaliação de evidências para a prática da enfermagem. Porto Alegre: Artmed; 2011. 670 p.
  • 15
    Galvis-Panqueva A, Mendoza P. Virtual learning environments: una methodology for creation. Inform Educ. 1999. 12(2):295-317.
  • 16
    Batista SCF.SOFTMAT: Um repositório de softwares para matemática do Ensino Médio - um instrumento em prol de posturas mais conscientes na seleção de softwares Educacionais [Dissertação] [Internet]. Campos dos Goytacazes: Centro de Ciência e Tecnologia da Universidade Estadual do Norte Fluminense- UENF; 2004[cited 2018 Apr 7]. 201 f. Available from: http://www.geogebra.im-uff.mat.br/biblioteca/dissertacao-batista-2004.pdf
    » http://www.geogebra.im-uff.mat.br/biblioteca/dissertacao-batista-2004.pdf
  • 17
    Rubio DM, Berg-Weger M, Tebb SS, Lee ES, Rauch S. Objectifying content validity: conducting a content validity study in social work reseach. Soc Work Res [Internet]. 2003 [cited 2020 Nov 30];27(2):94-111. Available from: https://www.researchgate.net/profile/Susan_Tebb/publication/265086559_Objectifyng_content_validity_Conducting_a_content_validity_study_in_social_work_research/links/558d3ab008ae591c19da8b51/Objectifyng-content-validity-Conducting-a-content-validity-study-in-social-work-research.pdf
    » https://www.researchgate.net/profile/Susan_Tebb/publication/265086559_Objectifyng_content_validity_Conducting_a_content_validity_study_in_social_work_research/links/558d3ab008ae591c19da8b51/Objectifyng-content-validity-Conducting-a-content-validity-study-in-social-work-research.pdf
  • 18
    Unicef Brasil. Projeto Caretas: Uma Experiência entre a ficção e a realidade [Internet]. 2019 [cited 2019 Oct 20]. Available from: https://www.unicef.org/brazil/projeto-caretas
    » https://www.unicef.org/brazil/projeto-caretas
  • 19
    Silva ARLS, Rebelo S, Santos JVV, Nunes CS, Spanhol FJ. Modelos utilizados pela educação a distância: uma síntese centrada nas instituições de ensino superior brasileiras. Rev GUAL [Internet]. 2011 [cited 2019 Oct 7];4(3):153-69. Available from: https://periodicos.ufsc.br/index.php/gual/article/view/1983-4535.2011v4n3p153/21984
    » https://periodicos.ufsc.br/index.php/gual/article/view/1983-4535.2011v4n3p153/21984
  • 20
    Salgado PG, Boubeta AR, Tobío TB, Mallou JV, Couto CB. Evaluation and early detection of problematic Internet use in adolescentes. Psicothema. 2014;26(1):21-6. https://doi.org/10.7334/psicothema2013.109
    » https://doi.org/10.7334/psicothema2013.109
  • 21
    Ferreira EZ, Oliveira AMN, Medeiros SP, Gomes GC, Cezar-Vaz MR, Ávila JA. Internet influence on the biopsychosocial health of adolescents: an integratitive review. Rev Bras Enferm. 2020;73(2):e20180766. https://doi.org/10.1590/0034-7167-2018-0766
    » https://doi.org/10.1590/0034-7167-2018-0766
  • 22
    Rosalin BCM, Cruz JAS, Mattos MBG. A importância do material didático no ensino a distância. RPGE- Rev Pol Gestão Educ. 2017;21(n.esp.):814-30. https://doi.org/10.22633/rpge.v21.n.esp1.out.2017.10453
    » https://doi.org/10.22633/rpge.v21.n.esp1.out.2017.10453
  • 23
    Rosa BVC, Girardon-Perlini NMO, Gamboa NSG, Nietsche EA, Beuter M, Dalmolin A. Development and validation of audiovisual educational technology for families and people with colostomy by cancer. Texto Contexto Enferm. 2019;28:e20180053. https://doi.org/10.1590/1980-265x-tce-2018-0053
    » https://doi.org/10.1590/1980-265x-tce-2018-0053
  • 24
    Medeiros JRR, Lima MA, Araújo LL, Galiza FT, Felipe GF, Caetano JÁ. Validation of educational technology for care in hemodialysis. Rev Enferm UFPE. 2016;10(11):3927-34. https://doi.org/10.5205/1981-8963-v10i11a11474p3927-3934-2016
    » https://doi.org/10.5205/1981-8963-v10i11a11474p3927-3934-2016
  • 25
    Ribeiro LCC, Oliveira TC, Moreira S, Paula FA. Construção e validação de manual sobre Burnout em professores. Rev Enferm Cent-Oeste Min. 2017;7:e1317. https://doi.org/10.19175/recom.v7i0.1317
    » https://doi.org/10.19175/recom.v7i0.1317
  • 26
    Costa IKF, Tibúrcio MP, Melo GSM, Leite JEL, Dantas RAN, Torres GV. Construction and validation of a distance Basic Life Support Course. Rev Bras Enferm. 2018;71(Suppl6):2698-705. https://doi.org/10.1590/0034-7167-2018-0122
    » https://doi.org/10.1590/0034-7167-2018-0122
  • 27
    Saboia DM, Vasconcelos CTM, Oriá MOB, Bezerra KC, Vasconcelos Neto JA, Lopes MHBM. Continence App: construction and validation of a mobile application for postnatal urinary incontinence prevention. Eur J Obstet Gynecol Reprod Biol. 2019;240:330-5. https://doi.org/10.1016/j.ejogrb.2019.07.026
    » https://doi.org/10.1016/j.ejogrb.2019.07.026
  • 28
    Interaminense INCS, Oliveira SC, Linhares FMP, Guedes TG, Ramos VP, Pontes CM. Construction and validation of an educational video for human papillomavirus vaccination. Rev Bras Enferm. 2020;73(4):e20180900. https://doi.org/10.1590/0034-7167-2018-0900
    » https://doi.org/10.1590/0034-7167-2018-0900
  • 29
    Sabino LMM, Ferreira ÁMV, Mendes ERR, Joventino ES, Gubert FA, Penha JC, et al. Validation of primer for promoting maternal self-efficacy in preventing childhood diarrhea. Rev Bras Enferm. 2018;71(Suppl 3):1412-19. https://doi.org/10.1590/0034-7167-2017-0341
    » https://doi.org/10.1590/0034-7167-2017-0341
  • 30
    Núcleo de Educação a distância da Universidade Estadual Vale do Acaraú- NEaD-UVA. Ambiente Virtual de Ensino à Distância [Internet]. 2019 [cited 2020 Apr 18]. Available from: http://ead2.uvanet.br
    » http://ead2.uvanet.br
  • 31
    Oliveira JKC. Ambiente virtual de aprendizagem: elementos e ferramentas que influenciam a interação online. REDOC - Rev Doc Cibercult. 2018;2(2):184-96. https://doi.org/10.12957/redoc.2018.31393
    » https://doi.org/10.12957/redoc.2018.31393
  • 32
    Valente JA. Blended learning e as mudanças no ensino superior: a proposta da sala de aula invertida. Educar Rev. 2014;Ed. Esp(4):79-97. https://doi.org/10.1590/0104-4060.38645
    » https://doi.org/10.1590/0104-4060.38645
  • 33
    Monteiro RJS, Oliveira MPCA, Belian RB, Lima LS, Santiago ME, Gontijo DT. DECIDIX: meeting of the Paulo Freire Pedagogy with the serious games in the field of health education with adolescents. Ciênc Saúde Colet. 2018;23(9):2951-62. https://doi.org/10.1590/1413-81232018239.12782018
    » https://doi.org/10.1590/1413-81232018239.12782018

Edited by

EDITOR IN CHIEF: Antonio José de Almeida Filho
ASSOCIATE EDITOR: Alexandre Balsanelli

Publication Dates

  • Publication in this collection
    30 Aug 2021
  • Date of issue
    2021

History

  • Received
    18 Oct 2020
  • Accepted
    14 Jan 2021
Associação Brasileira de Enfermagem SGA Norte Quadra 603 Conj. "B" - Av. L2 Norte 70830-102 Brasília, DF, Brasil, Tel.: (55 61) 3226-0653, Fax: (55 61) 3225-4473 - Brasília - DF - Brazil
E-mail: reben@abennacional.org.br