Expert assessment of the “Neonatal Pain Assessment Program” online course

Evaluación de expertos del curso en línea “Programa de evaluación del dolor neonatal”

ABSTRACT

Objectives:

to assess layout quality, visual identity and content of the “Neonatal Pain Assessment Program” (Programa de Avaliação da Dor Neonatal) online course.

Methods:

a descriptive exploratory study. The course was assessed by 24 experts in pain, neonatology and education. A form containing 20 questions on technical, interface and educational aspects was used. For each item, the score ranged from 0 to 1. Mean scores above 0.7 were considered indicative of high quality of the items. Descriptive statistics were used for data analysis.

Results:

navigation, clarity, ease of localization, content relevance, contextualization, content correction, multiple windows, ease of use, ease of return, ergonomics, esthetics, special brands, audiovisual resources, information and portability were assessed. All aspects obtained a mean ≥ 0.70 and no changes were required.

Conclusions:

the Neonatal Pain Assessment Program is considered as quality educational technology and promising strategy for health education.

Descriptors:
Distance Education; Educational Technology; Infant; Neonatal Nursing; Pain Measurement

RESUMEN

Objetivos:

evalúe la calidad del diseño, la identidad visual y el contenido del curso en línea “Programa de Evaluación del Dolor Neonatal” (Programa de Avaliação da Dor Neonatal).

Métodos:

estudio exploratorio descriptivo. El curso fue evaluado por 24 especialistas en las áreas de dolor, neonatología y educación. Se utilizó un formulario con 20 preguntas sobre aspectos técnicos, de interfaz y educativos. Para cada ítem, los puntajes variaron de 0 a 1. Los puntajes promedio por encima de 0.7 se consideraron indicativos de alta calidad de ítems. Se utilizaron estadísticas descriptivas para el análisis de datos.

Resultados:

se evaluaron: navegación libre, claridad, facilidad de localización, relevancia de contenido, contextualización, corrección de contenido, ventanas múltiples, facilidad de uso, facilidad de retorno, ergonomía, estética, marcas especiales, recursos audiovisuales, información y portabilidad. Todos los aspectos obtuvieron una media ≥ 0,70 y no se requirieron cambios.

Conclusiones:

el Programa de Evaluación del Dolor Neonatal se considera una tecnología educativa de calidad y una estrategia prometedora para la educación para la salud.

Descriptores:
Educación a Distancia; Tecnología Educativa; Recién Nacido; Enfermería Neonatal; Dimensión del Dolor

RESUMO

Objetivos:

avaliar a qualidade do layout, da identidade visual e do conteúdo do curso online “Programa de Avaliação da Dor Neonatal”.

Métodos:

estudo exploratório descritivo. O curso foi avaliado por 24 especialistas nas áreas de dor, neonatologia e educação. Foi utilizado um formulário contendo 20 questões sobre aspectos técnicos, de interface e educacionais. Para cada item, a pontuação variou de 0 a 1. Escores médios acima de 0,7 foram considerados indicativos de alta qualidade dos itens. Utilizou-se estatística descritiva para análise dos dados.

Resultados:

foram avaliados: navegação livre, clareza, facilidade de localização, pertinência do conteúdo, contextualização, correção de conteúdo, múltiplas janelas, facilidade de aprendizagem, eficiência de utilização, facilidade de retorno, ergonomia, estética, marcas especiais, recursos audiovisuais, informações e portabilidade. Todos os aspectos obtiveram média ≥ 0,70, não sendo necessárias modificações.

Conclusões:

considera-se o Programa de Avaliação da Dor Neonatal como tecnologia educacional de qualidade e estratégia promissora para a educação em saúde.

Descritores:
Educação a Distância; Tecnologia Educacional; Recém-Nascido; Enfermagem Neonatal; Avaliação da Dor

INTRODUCTION

Neonatal pain is a complex, subjective and multidimensional phenomenon making its assessment a challenge, which is compounded by the impossibility of verbal reporting in this age group⁽¹⁾. Thus, behavioral, physiological indicators, endocrine-metabolic changes, contextual and neurophysiological components are important for the multidimensional neonatal pain assessment. To this end, specific pain assessment tools for the Newborn (NB) have been constructed and validated based on these indicators^(2-3).

Currently, the literature offers a wide range of tools for neonatal pain assessment, although none of these tools is considered ideal, since they consider subjective indicators for their assessment, such as behavioral and contextual parameters⁽⁴⁾. However, some of these have adequately explored and established psychometric properties, and have been widely used in clinical practice and research. For this reason, they are recommended by agreement and guidelines^(2,5-6).

Although some health professionals, mainly nursing, recognize the occurrence of pain in the neonatal population, their assessment is still performed empirically in the care settings, especially by isolated assessment of physiological and behavioral parameters, predominantly crying^(7-9). The lack of specificity in the assessment of pain in NB reinforces the importance of adopting systematic methods to identify pain in this population⁽¹⁰⁾. To do this, training and qualification of health teams are necessary, in order to offer knowledge and develop skills in assessment performance, depending on the variety of tools available and the different scoring methods^(4,11).

Moreover, Information and Communication Technologies (ICTs) in education are a useful alternative for the teaching of students and training of health professionals, since they facilitate articulation between theory, clinical practice and research. Thus, it enables the development of a critical-reflexive behavior^(12-13). ICTs offer conveniences such as temporal flexibility⁽¹⁴⁾, greater ease of interaction⁽¹⁵⁾, focused and self-directed study, and active learning^(16-18). In addition, they allow interactivity and inclusion of audiovisual resources (audios, images, videos, animations, quizzes, texts, among others) as a form of educational support⁽¹⁹⁾. It is also pointed out that the use of technology in education has demonstrated significant cognitive gain for users, as observed in traditional teaching^(20-21).

Evidence shows the increasing use of ICTs in student education and training of health professionals, although online teaching in the context of neonatology is still incipient and requires investments for the use of new educational strategies^(22-23).

Considering the challenges related to neonatal pain assessment in clinical practice, as well as the advantages related to the use of ICTs in professional training process, an online course was developed, called Neonatal Pain Assessment Program (PAD-Neo - Programa de Avaliação da Dor Neonatal). The course aims to contribute to the teaching of health students and improve the professionals’ knowledge on pain assessment and measurement, neonatal pain recognition skills and, finally, assist in the selection and use of pain assessment tools in clinical practice.

In its initial version, developed between 2009 and 2010, PAD-Neo was hosted on the Modular Object-Oriented Dynamic Learning Environment (Moodle^®) platform and organized into eight modules. This version was assessed from the users’ perspective, which included professionals and undergraduate and graduate students in health. They were satisfied or very satisfied with aspects such as time, content organization and applicability, Virtual Learning Environment (VLE) and audiovisual resources. Additionally, a significant difference was observed between the number of hits observed in the post-test (16, ± 2.41 mean) compared to the pre-test (13 ± 1.88 mean), p = 0.006⁽²⁴⁾.

Due to the need for revision and updating, the course remained hosted on the Moodle^® platform, and its content was extended from eight to ten weeks of activities. The main modification involved neonatal pain assessment tools, with prioritized internationally recommended scales, recently revised and adequately translated into Portuguese (Brazil). The Neonatal Facial Coding System (NFCS)^(25-26), which was validated and translated in Brazil as Sistema de Codificação da Atividade Facial Neonatal ⁽²⁷⁾ and the Neonatal Infant Pain Scale (NIPS)⁽²⁸⁾, translated as Escala de Dor no Recém-Nascido ⁽²⁹⁾. Premature Infant Pain Profile (PIPP)⁽³⁰⁾ has been replaced by its upgraded version, the Premature Infant Pain Profile - Revised (PIPP-R)⁽³¹⁾ was translated as Perfil de Dor no Recém-Nascido Pré-Termo - Revisado ⁽³²⁾. There was also a need to replace the Crying Requires oxygen for saturation above 95%, Increased vital signs, Expression, Sleepless (CRIES)⁽³³⁾ by the Behavioral Indicators of Infant Pain(BIIP)⁽³⁴⁾, validated for Portuguese as Indicadores Comportamentais de Dor no Recém-Nascido ⁽³⁵⁾, since CRIES only assesses neonatal pain in the postoperative period. The other tools can be used for this purpose. Finally, the scale assessing the prolonged pain Échelle Douleur Inconfort Nouveau-Né (EDIN)⁽³⁶⁾, translated in Brazil as Escala de Dor e Desconforto do Recém-Nascido ⁽³⁷⁾.

Audiovisual resources such as interactive videotapes, texts for reading, forums, exercises using images and videos and links to websites made up the resources developed for the course initial version. Use and satisfaction of this first version were assessed by users, through pre-test and post-test, and satisfaction questionnaire, respectively.In spite of the positive results obtained from the assessment of the initial version of the PAD-Neo, a course quality assessment was considered necessary and pertinent from the perspective of experts in the areas of interest as part of the process of improving the use of ICTs in health education.

OBJECTIVES

To assess layout quality, visual identity and content of the PAD-Neo online course.

METHODS

This is an exploratory descriptive study that set out to describe the process for assessing the quality of layout, visual identity and content of the PAD-Neo online course.

Ethical aspects

The research project was assessed and approved on February 11, 2015, by the Research Ethics Committee of Universidade de São Paulo’s School of Nursing, CAAE (Certificado de Apresentação para Apreciação Ética - Certificate of Presentation for Ethical Consideration) 41048915.8.0000.5392, opinion number 1,596,346.

Place of study and period

The present study was conducted between July 2015 and July 2017, and data were collected between July and December 2016 at Universidade de São Paulo’s School of Nursing.

Population and sample

An expert committee was composed of professionals working in neonatal and/or pediatric pain, neonatal care and education (with emphasis on the development and assessment of digital educational resources), considering eight experts per area⁽³⁸⁾. The selection criteria of pain experts were: to hold a Doctor’s degree and to have research projects and/or Master’s or Doctorate guidelines related to pain in neonatology and/or pediatrics. For neonatology professionals inclusion, it was adopted as a criterion neonatal care performance for a time equal to or greater than three years and/or hold Master or Doctor title in neonatology. Finally, there was inclusion of experts in educational technology higher or who held Doctor’s degree and/or research projects, Master’s or Doctorate guidelines in the area of development and assessment of digital educational resources.

The non-random selection of convenience was selected through non-random selection of contacts and partnerships with leaders and members of research groups registered in the CNPq Research Groups directory and linked to Universidade de São Paulo’ School of Nursing.

Data collection

Experts were invited by electronic means. Upon accepting to participate as an evaluator, a new message was sent containing the course’s electronic address, instructions for visitor access and password, as well as the electronic form developed in Google Docs^® to record the profile characterization data of evaluator and course quality assessment. The online course assessment tool was developed based on a tool proposed by the Central Coordination of Distance Education (CCEAD - Coordenação Central de Educação a Distância, PUCRIO)⁽³⁹⁾.

The assessed aspects included: free navigation (control of the sequence of use by the user); information clarity (clarity of information and emphasis on one concept at a time); information localization (features that make it easier to locate); content pertinence (logical presentation and intuitive navigation); contextualization (consistency and appropriateness of content); content correction (scientific rigor); multiple windows (overlapping windows); ease of learning in interaction (understanding of interaction); use efficiency (information and commands for the use); ease of return (return to previous page); ergonomics (maintenance of interaction and standardization of presentation); esthetics (appropriate interface standards); special marks (use of marks facilitating recognition); audiovisual resources (adequate variety and quality); references (presentation of reliable sources); interactivity (interaction with the course); error management (error reduction or prevention); users help (information and help resources); information quality (information compatible with the public) and portability (functionality on different devices and browsers)⁽³⁹⁾. There is a section for observations and suggestions from evaluators.

There were six scoring options for each item (0, 0.25, 0.5, 0.75, 1, and not applicable). The score 0 represented the worst assessment and 1, the best assessment of the item. In order to establish the standard of online course quality assessment, subsidies of the standard of quality and performance of technologies, ABNT (The Brazilian Association of Technical Standards - Associação Brasileira de Normas Técnicas - is the body responsible for technical standardization in Brazil, providing inputs to the Brazilian technological development) ISO/IEC 14598-6⁽³⁸⁾. This standard also determines that the expected mean for the assessed characteristics is greater than or equal to 70%. Thus, values were calculated for each item, and those with an overall mean lower than 0.70 would be modified by the research team, according to expert assessment and suggestion⁽³⁸⁾.

Analysis of results procedures

The data obtained electronically through the Google Docs^® form was transported to Microsoft Excel for Windows^® for analysis. Descriptive statistics was used, where arithmetic mean and relative frequency of variables of interest were calculated. The results were presented in a descriptive way.

RESULTS

Twenty-four experts, all of them female, participated in the study. The mean age was 43 years, with the majority of the participants being graduated in nursing (18 - 75%). Of these, four were considered as pain experts, seven as experts in neonatology and seven as experts in education. The committee was also comprised of four psychologists working in the area of pain, a physiotherapist working in neonatal care and a participant with a degree in instructional design in education.

Two professionals with PhD (8.3%), fifteen professionals with doctor degree (62.5%), six with master’s degree (25%) and one instructional design expert (4.2%) were included. As to position and function performed, ten evaluators are university professors (41.6%). Five (20.8%) experts work at neonatal or joint care units as nurses. Four work in the area of education (16.6%). Two are graduate students (8,3%). One expert is an instructional designer (4.2%), one is a people management nurse (4.2%) and one is a supervisor of the physiotherapy service (4.2%). Regarding course quality assessment, Table 1 shows the means obtained by the overall areas of expertise.

Table1
Mean values of the assessed items, according to the areas of specialties pain (Pain), neonatology (Neo) and education (ED) and overall assessment, São Paulo, São Paulo, Brazil, 2017

Aspects (n)*	Pain	Neo	ED	Overall
Free navigation (n=24)	0.94	1.0	0.81	0.92
Information clarity (n=24)	0.97	1.0	0.94	0.97
Ease of locating information (n=24)	0.91	0.97	0.81	0.89
Content pertinence (n=24)	0.94	1.0	0.94	0.96
Contextualization (n=23)	0.87	1.0	0.96	0.94
Content correction (n=22)	0.97	1.0	0.96	0.98
Multiple windows (n=24)	0.97	0.97	0.97	0.97
Ease of learning in interaction (n=24)	1.0	1.0	0.87	0.96
Use efficiency (n=24)	0.94	0.94	0.97	0.91
Ease of return (n=24)	0.94	0.94	1.0	0.81
Ergonomics (n=24)	0.97	0.94	0.91	0.94
Esthetics (n=24)	1.0	0.97	0.97	0.96
Special marks(n=24)	0.97	0.91	0.91	0.93
Audiovisual resources (n=24)	0.87	1.0	0.69	0.85
References (n=24)	0.94	1.0	0.84	0.93
Interactivity (n=24)	0.84	0.97	0.66	0.82
Error management (n=17)	0.94	0.87	0.69	0.85
Users help (n=19)	0.90	0.89	0.68	0.81
Information quality (n=23)	1.0	1.0	0.86	0.96
Portability (n=20)	0.96	1.0	0.96	0.97

• Note: * Refers to the total number of valid responses, as some experts attributed “not applicable” to certain items.

All aspects obtained a mean above 0.70 in the committee overallassessment, which constitutes adequate quality of the PAD-Neo (Figures 1 and 2). The four aspects assessed by education experts, whose means were below 0.70 (audiovisual resources, interactivity, error management and help to users), were reviewed in order to adapt them to the requirements.

Figure 1
Interactive classroom model present in Neonatal Pain Assessment Program, São Paulo, São Paulo, Brazil, 2017

Figure 2
Interactive classroom model present in Neonatal Pain Assessment Program, São Paulo, São Paulo, Brazil, 2017

Space for suggestions and observations on the form was used by 17 experts who expressed satisfaction with presentation and organization of the online course and made suggestions for course improvement:

Objective, precise, pleasant, clear content, relevant, easy to use. (Education expert)

The course is very complete, easy to understand and uses materials with consistent content and based on updated literature. You will certainly have a great contribution in the practice of neonatology care. (Pain expert)

Excellent online learning opportunity. (Neonatalogy expert)

(The course setting) is linear and makes it difficult for the student. (Education expert)

Most of the complementary reading articles are in English, as the course should include nurses and students from all locations in Brazil, I consider having balance of articles in English/Portuguese. (Pain expert)

DISCUSSION

The increasing production of evidence on the use of technologies in the teaching and training of students and health professionals, especially nursing professionals^(20-22). Studies in this subject are essential to give validity and quality to educational proposals, such as the PAD-Neo, which, through assessment, offers subsidies to develop and improve the teaching-learning process in the virtual environment^(20-22).

The linear format of the course was considered by some experts as a limitation to the user’s free navigation, mainly by experts in education (0.81 mean). The literature shows that nonlinear navigation can affect user control, however, there is no significant influence on user learning⁽⁴⁰⁾. It is worth noting that the first modules of the course cover basic themes (Pain: definitions and basic concepts of physiology, Components of pain assessment in NB, and Principles of measurement and pain assessment in NB). Linearity was necessary in these steps for the user’s adequate understanding of the content. On the other hand, modules referring to neonatal pain assessment tools have independent content and, for this reason, will be offered in the future in an open way so that the user can develop their activities according to their preference, thus accepting experts’ suggestion.

Information clarity was assessed positively by all experts. The language used in an online course should be clear, objective and accessible in order to promote the understanding and learning process of the Distance Education user (EAD - Educação a Distância)⁽⁴¹⁾.

In order to promote user comprehension, a virtual environment should allow self-centered learning, favoring autonomy and independent navigation⁽⁴²⁾. The use of resources to locate information helps to understand the functioning of the course, from the tutorials and menus. For this reason, special identification marks were developed for activities such as classes, exercises, forum, among others. The main highlights and orientations of each unit are arranged from a short document guide in Portable Document Format (PDF) format, as well as the interactive classes developed with Articulate Storyline^® Software, among others.

Regarding content pertinence, this criterion refers to the logical and clear presentation of the information, and to the navigation as it is intuitive to the user⁽³⁹⁾. Adding to this, contextualization, when assessing adequacy and consistency in content, as well as scientific rigor, assesses correctness of content. From the initial version, the PAD-Neo was developed on the basis of high-quality scientific evidence⁽²⁴⁾. For the current version, the material content was updated by the research team, in order to include new contents and references, which is reflected in the positive assessment made by experts in these three aspects.

However, the number of articles in Portuguese available as a complementary reading resource in the PAD-Neo was an observation pointed out by two experts. Furthermore, in spite of the scarcity of scientific articles pertinent to the course theme and published in Portuguese, efforts were made to follow the suggestions and expand the references made available in the course.

Regarding the use of multiple windows, this feature was considered by experts as preferably the user’s individual, as it may facilitate navigation or confuse. Some of the features featured in PAD-Neo are viewed from new window openings because it is hosted on Moodle^®. On the other hand, hosting PAD-Neo on the Moodle^® platform may have favored the positive assessment of the ease of learning in the interaction. It is considered that this platform allows a greater participation and interaction of users with the virtual learning environment⁽⁴³⁾.

Ergonomics assesses uniformity in the presentation of online course content and resources. Its assessment is fundamental in order to improve technical and usability aspects that involve the teaching-learning process⁽⁴⁴⁾. In this way, adapting the online course to ergonomic criteria favors the understanding and use by users⁽⁴⁴⁾. Overall, experts pointed out that the course presents an organization favorable to the understanding of the user, configuration of the fonts that facilitate readability and that the icons, screens and resources are presented in a uniform way.

Evaluators considered “esthetic” and “use of special brands” as high quality. Visual information influences learning, since the organization of the interface can either encourage the individual, favor teaching or be a distracting resource⁽⁴⁵⁾. As part of the course update, a new visual identity was developed for PAD-Neo. According to experts, the selected colors and font facilitated content presentation and readability.

As to audiovisual resources and interactivity, four specialists judged the variety of resources as limited. These resources are considered as fundamental in the interaction between the user and the virtual environment, since its use benefits the learning, making the process more interesting and pleasurable⁽⁴⁶⁾. In addition, interactivity in online courses is characterized as an important tool for the active participation of the user, guaranteeing their satisfaction and avoiding their discontinuity⁽⁴⁷⁾. By offering videotapes, interactive lessons, exercises based on videos and photos, and discussion forums, it is believed that the PAD-Neo presents a considerable variety of audiovisual resources. Due to the overall mean attributed by “audiovisual resources” experts, no changes were made to this item.

Regarding errors management, along the course, icons and guidelines indicative of the following activities and/or steps are available, in order to minimize navigational errors. As for the following item, referring to user help, experts suggested creating a user doubts forum, which was included on the main page.

As to information quality, experts assessed the item in a positive way, since the references, workload and vocabulary used were considered adequate and pertinent to the intended target audience. Despite the success of experts accessing the course from different browsers, some of them reported difficulties in watching the course and its content on mobile devices, such as mobile phones and tablets. This is considered an important limitation and will be met in future course updates.

PAD-Neo can be considered as an educational tool of adequate quality, considering the rigor with which it was developed and assessed. The vast majority of ICTs are developed and implemented without assessment. However, expert assessment with specific competences is determinant and necessary for the quality of the proposed teaching⁽⁴⁸⁾. It is also worth noting the pioneering and innovative character of the PAD-Neo in the Brazilian educational setting, given the scarcity of studies on the development and assessment of online courses in neonatal care, including in relation to pain⁽²³⁾.

In addition, studies on the course’s clinical practice of health professionals repercussions, such as the development of institutional protocols for neonatal pain assessment and increase of records of pain assessment in medical records are necessary.

Study limitations

As a limitation of this study, some of the scores attributed below 0.50 were not adequately justified by experts, according to guidelines previously highlighted in the assessmenttool. This made it difficult to assess the course quality accurately and prevented the team from making improvements in order to adapt the items to the proposed criteria. It is also observed that the tool used in the present study may have contributed to failure and/or difficulty of interpretation of some items, thus generating meaning ambiguity of descriptions.

Contributions to Health

It is expected that the availability of the PAD-Neo for undergraduate and graduate students in health, through its classroom applicability, extension courses, and improvement and professional training in continuing and permanent education, contributes to disseminate knowledge about neonatal pain.

CONCLUSIONS

This study assessed the quality of PAD-Neo from the perspective of experts from different areas that present interface in relation to the subject addressed. PAD-Neo is considered as an educational technology of adequate quality and multi-applicability contributing to the teaching of undergraduate and graduate students in health and qualification of health professionals. Expert assessment contributed to the early identification and adjustment of failures that could negatively influence the user experience. The quality and innovative character of PAD-Neo make this initiative promising in the Brazilian setting.

REFERENCES

• ¹
  Hall RW, Anand KS. Pain management in newborns. Clin Perinatol. 2014;41(4):895-924. doi: 10.1016/j.clp.2014.08.010
  » https://doi.org/10.1016/j.clp.2014.08.010
• ²
  Cong X, McGrath JM, Cusson RM, Zhang D. Pain assessment and measurement in neonates: an updated review. Adv Neonatal Care. 2013;13(6):379-95. doi: 10.1097/ANC.0b013e3182a41452
  » https://doi.org/10.1097/ANC.0b013e3182a41452
• ³
  Harrison D, Bueno M, Reszel J. Prevention and management of pain and stress in the neonate. Res Rep Neonatol. 2015;5:9-16. doi: 10.2147/RRN.S52378
  » https://doi.org/10.2147/RRN.S52378
• ⁴
  Witt N, Coynor S, Edwards C, Bradshaw H. A guide to pain assessment and management in the neonate. Curr Emerg Hosp Med Rep. 2016;4:1-10. doi: 10.1007/s40138-016-0089-y
  » https://doi.org/10.1007/s40138-016-0089-y
• ⁵
  Lee GY, Yamada Janet, Kyololo O, Shorkey A, Stevens B. Pediatrics. 2014;133(3):500-515. doi: 10.1542/peds.2013-2744
  » https://doi.org/10.1542/peds.2013-2744
• ⁶
  American Academy of Pediatrics, Committee on Fetus and Newborn and Section on Anesthesiology and Pain Medicine. Prevention and management of procedural pain in the neonate: an update. Pediatrics. 2016;137(2):e20154271. doi: 10.1542/peds.2015-4271
  » https://doi.org/10.1542/peds.2015-4271
• ⁷
  Efe E, Dikmen S, Atlas N, Boneval C. Turkish pediatric surgical nurses' knowledge and attitudes regarding pain assessment and nonpharmacological and environmental methods in newborns' pain relief. Pain Manag Nurs. 2013;14(4):343-350. doi:10.1016/j.pmn.2011.08.003
  » https://doi.org/10.1016/j.pmn.2011.08.003
• ⁸
  Capellini VK, Daré MF, Castral TC, Christoffel MM, Leite AM, Scochi CGS. Conhecimento e atitudes de profissionais de saúde sobre avaliação e manejo da dor neonatal. Rev Eletr Enf. 2014;16(2):361-9. doi: 10.5216/ree.v16i2.23611
  » https://doi.org/10.5216/ree.v16i2.23611
• ⁹
  Hatfield LA, Ely EA. Measurement of acute pain in infants: a review of behavioral and physiological variables. Biol Res Nurs. 2015;17:100-11. doi: 10.1177/1099800414531448
  » https://doi.org/10.1177/1099800414531448
• ¹⁰
  Riddell RP, Fitzgerald M, Slater R, Stevens B, Johnston C, Campbell-Yeo M. Using only behaviours to assess infant pain: a painful compromise? Pain. 2016;157(8):1579-80. doi: 10.1097/j.pain.0000000000000598
  » https://doi.org/10.1097/j.pain.0000000000000598
• ¹¹
  Sposito NPB, Rossato LM, Bueno M, Kimura AF, Costa T, Guedes DMB. Assessment and management of pain in newborns hospitalized in a Neonatal Intensive Care Unit: a cross-sectional study. Rev Latino-Am Enfermagem. 2017;25:e2931. doi: 10.1590/1518-8345.1665.2931
  » https://doi.org/10.1590/1518-8345.1665.2931
• ¹²
  Buxton EC, De Muth JE. Pharmacists' perceptions of a live continuing education program comparing distance learning versus local learning. Res Social Adm Pharm. 2013;9(2):230-235. doi: 10.1016/j.sapharm.2012.05.003
  » https://doi.org/10.1016/j.sapharm.2012.05.003
• ¹³
  Rochman DL, Sheehan MJ, Kulich RJ. Evaluation of a pain curriculum for occupational therapists: experiences from a master's-level graduate program over six years. Disabil Rehabil. 2013;35(22):1933-1940. doi: 10.3109/09638288.2013.766273
  » https://doi.org/10.3109/09638288.2013.766273
• ¹⁴
  Fricton J, Anderson K, Clavel A, Fricton R, Hathaway K, Kang W, et al. Preventing chronic pain: a human systems approach - results from a massive open online course. Glob Adv Health Med. 2015;4(5):23-32. doi: 10.7453/gahmj.2015.048
  » https://doi.org/10.7453/gahmj.2015.048
• ¹⁵
  Alvarez AG, Dal Sasso GTM, Iyengar MS. Persuasive technology in teaching acute pain assessment in nursing: results in learning based on pre and post-testing. Nurse Educ Today. 2017;50:109-114. doi: 10.1016/j.nedt.2016.12.019
  » https://doi.org/10.1016/j.nedt.2016.12.019
• ¹⁶
  Brody AA, Groce-Wofford TM. Feasibility of implementing a web-based education program in geriatric pain and depression for home health care nurses. Home Health Care Manag Pract. 2013;20(10):1-5. doi: 10.1177/1084822313494785
  » https://doi.org/10.1177/1084822313494785
• ¹⁷
  Habich M, Letizia M. Pediatric pain assessment in the emergency department: a nursing evidence-based practice protocol. J Pediatr Nurs [internet]. 2015 [cited 2017 Dec 15];41(4):198-202. Available from:http://www.medscape.com/viewarticle/850009.pdf
  » http://www.medscape.com/viewarticle/850009.pdf
• ¹⁸
  Slaughter AL, Frith K, O'Keefe L, Alexander S, Stoll R. Promoting best practices for managing acute low back pain in an occupational environment. Workplace Health Saf. 2015;63(9):408-414. doi: 10.1177/2165079915589034
  » https://doi.org/10.1177/2165079915589034
• ¹⁹
  McKerlich R, Ives C, McGreal R. Measuring use and creation of open educational resources in higher education. IRRODL. 2013;14(4):90-103. doi: 10.19173/irrodl.v14i4.1573
  » https://doi.org/10.19173/irrodl.v14i4.1573
• ²⁰
  Lahti M, Hätönen H, Välimäki M. Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis. Int J Nurs Stud. 2014;51:136-149. doi: 10.1016/j.ijnurstu.2012.12.017
  » https://doi.org/10.1016/j.ijnurstu.2012.12.017
• ²¹
  McCutcheon K, Lohan M, Traynor M, Martin D. A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. J Adv Nurs. 2015;71(2):255-70. doi: 10.1111/jan.12509.
  » https://doi.org/10.1111/jan.12509
• ²²
  Cogo ALP, Pedro ENR, Silva APSS, Alves EATD, Valli GP. The use of digital educational technologies in nursing education. Cienc Enferm. 2013;19(3):21-9. doi: 10.4067/S0717-95532013000300003
  » https://doi.org/10.4067/S0717-95532013000300003
• ²³
  Freire LM, Paula MA, Duarte ED, Bueno M. Distance education in neonatal nursing scenarios: a systematic review. Rev Esc Enferm USP. 2015;49(3):515-521. doi: 10.1590/S0080-623420150000300021
  » https://doi.org/10.1590/S0080-623420150000300021
• ²⁴
  Bueno M, Duarte ED, Marques RL, Freire LM, Castral TC. Neonatal pain assessment program II: an innovative strategy to increase knowledge translation. Case report. Rev Dor. 2014;15(2):152-5. doi: 10.5935/1806-0013.20140029
  » https://doi.org/10.5935/1806-0013.20140029
• ²⁵
  Grunau RE, Craig KD. Pain expression in neonates: facial action and cry. Pain. 1987;28(0):395-410. doi: 10.1016/0304-3959(87)90073-X
  » https://doi.org/10.1016/0304-3959(87)90073-X
• ²⁶
  Grunau RE, Oberlander T, Holsti L, Whitfield MF. Bedside application of the neonatal facial coding system in pain assessment of premature neonates. Pain. 1998;76(3):277-86. doi: 10.1016/S0304-3959(98)00046-3
  » https://doi.org/10.1016/S0304-3959(98)00046-3
• ²⁷
  Pereira ALST, Guinsburg R, Almeida MFB, Monteiro MC, Santos AMN, Kopelman BI. Validity of behavioral and physiologic parameters for acute pain assessment of term newborn infants. São Paulo Med J. 1999;117(2):72-80. doi: 10.1590/S1516-31801999000200005
  » https://doi.org/10.1590/S1516-31801999000200005
• ²⁸
  Lawrence J, Alcook D, McGrath P, Kay J, MacMurray SB, Dulberg C. The development of a tool to assess neonatal pain. Neonatal Netw. 1993;12(6):59-66. doi: 10.1016/0885-3924(91)91127-U
  » https://doi.org/10.1016/0885-3924(91)91127-U
• ²⁹
  Motta GDCP, Shardosim JM, Cunha MLC. Neonatal Infant Pain Scale: cross- cultural adaptation and validation in Brazil. J Pain Symptom Manage. 2015;50(3):394-401. doi: 10.1016/j.jpainsymman.2015.03.019
  » https://doi.org/10.1016/j.jpainsymman.2015.03.019
• ³⁰
  Stevens B, Johnston C, Petryshen P, Taddio A. Premature Infant Pain Profile: development and initial validation. Clin J Pain. 1996;12(1):13-22. doi: 10.1097/00002508-199603000-00004
  » https://doi.org/10.1097/00002508-199603000-00004
• ³¹
  Stevens BJ, Gibbins S, Yamada J, Dionne K, Lee G, Johnston C, et al. The premature infant profile-revised (PIPP-R): initial validation and feasibility. Clin J Pain. 2014;30(3):238-43. doi: 10.1097/AJP.0b013e3182906aed
  » https://doi.org/10.1097/AJP.0b013e3182906aed
• ³²
  Bueno M, Forni E, Costa T, Kimura AF. Adaptação Transcultural e Validação do Premature Infant Pain Profile - Revised para o português (Brasil). In: Anais do 67ª Congresso Brasileiro de Enfermagem [Internet]. 2015 out. 27-30; São Paulo. ABEn- Seção -SP; 2015 [cited 2017 dec 15]. Available from:https://icongresso.itarget.com.br/useradm/anais/?clt=abe.3&lng=P
  » https://icongresso.itarget.com.br/useradm/anais/?clt=abe.3&lng=P
• ³³
  Krechel SW, Bildner J. CRIES: a new neonatal postoperative pain measurement score. Initial testing of validity and reliability. Paediatr Anaesth. 1995;5(1):53-61. doi: 10.1111/j.1460-9592.1995.tb00242.x
  » https://doi.org/10.1111/j.1460-9592.1995.tb00242.x
• ³⁴
  Holsti L, Grunau RE. Initial validation of the behavioral indicators of infant pain (BIIP). Pain. 2007;132(3):264-72. doi: 10.1016/j.pain.2007.01.033
  » https://doi.org/10.1016/j.pain.2007.01.033
• ³⁵
  Bueno M, Castral TC, Kimura AF, Holsti L. Adaptação transcultural e validação de conteúdo do Behavioral Indicators of Infant Pain (BIIP) para o português (Brasil). In: Anais do V Congresso Brasileiro de Enfermagem Pediátrica e Neonatal [Internet]. 2013 out29- nov01; Gramado, RS. Gramado, SOBEP; 2014 [cited 2017 Dec 15]. Available from: https://sobep.org.br/v-congresso/images/stories/arquivos/5Congresso/anais_enfermagem.zip
  » https://sobep.org.br/v-congresso/images/stories/arquivos/5Congresso/anais_enfermagem.zip
• ³⁶
  Debillon T, Zupan V, Ravault N, Magny JF, Dehan M. Development and initial validation of the edin scale, a new tool for assessing prolonged pain in preterm infants. Arch Dis Child Fetal Neonatal Ed. 2001;85(1):36-41. doi: 10.1136/fn.85.1.F36
  » https://doi.org/10.1136/fn.85.1.F36
• ³⁷
  Dias FSB, Marba STM. Avaliação da dor prolongada no recém-nascido: the evaluation of prolonged pain in the newborn: adaptation of the EDIN scale for brazilian culture. Texto Contexto Enferm. 2014;23(4):964-70. doi: 10.1590/0104-07072014002100013
  » https://doi.org/10.1590/0104-07072014002100013
• ³⁸
  Associação Brasileira de Normas Técnicas. ISO/IEC 14598-6. Engenharia de software: avaliação do produto. Parte 6: Documentação de módulos de avaliação. 2004
• ³⁹
  Campos GHB, Martins I, Nunes BP. Instrumento para a avaliação da qualidade de objetos de aprendizagem. Coordenação Central de Educação A Distância PUC-Rio [Internet]. 2008; p.8 [cited 2017 Dec 15]. Available from:
• ⁴⁰
  Katuk N, Zakaria NH. Linear and non-linear navigations of learning content. IJCISIM [Internet]. 2015 [cited 2017 Dec 15];7:22-31. Available from: http://www.mirlabs.net/ijcisim/regular_papers_2015/IJCISIM_3.pdf
  » http://www.mirlabs.net/ijcisim/regular_papers_2015/IJCISIM_3.pdf
• ⁴¹
  Milic NM, Trajkovic GZ, Bukumiric ZM, Cirkovic A, Nikolic IM, Milin JS, et al. Improving education in medical statistics: implementing a blended learning model in the existing curriculum. PLoS ONE. 2016;11(2):1-10. doi: 10.1371/journal.pone.0148882
  » https://doi.org/10.1371/journal.pone.0148882
• ⁴²
  Weiner DK, Morone NE, Spallek H, Karp JF, Schneider M, Washburn C, et al. E-learning module on chronic low back pain in older adults: evidence of effect on medical student objective structured clinical examination performance. J Am Geriatr Soc. 2014;62(1):1161-67. doi: 10.1111/jgs.12871
  » https://doi.org/10.1111/jgs.12871
• ⁴³
  Prado C, Santiago LC, Silva JAM, Pereira IM, Leonello VM, Otrenti E, et al. Virtual learning environment in nursing education: an experience report. Rev Bras Enferm. 2012;65(5):862-6. doi: 10.1590/S0034-71672012000500022
  » https://doi.org/10.1590/S0034-71672012000500022
• ⁴⁴
  Castro FSF, Dias DMV, Higarashi IH, Scochi CGS, Fonseca LMM. Evaluation of digital educational student technology interaction in neonatal nursing. Rev Esc Enferm USP. 2015;49(1):114-121. doi:10.1590/S0080-623420150000100015
  » https://doi.org/10.1590/S0080-623420150000100015
• ⁴⁵
  Fonseca LMM, Del'angelo N, Góes FSN, Castro FSF, Ponce De Leon CGSM, Scochi CGS. Clinical evaluation of preterm newborns: nursing students opinion about an educational software. Cienc Enferm. 2012;18(2): 83-91. doi: 10.4067/S0717-95532012000200009
  » https://doi.org/10.4067/S0717-95532012000200009
• ⁴⁶
  Fonseca LMM, Aredes NDA, Leite AM, Santos CB, Lima RAG, Scochi CGS. Evaluation of an educational technology regarding clinical evaluation of preterm newborns. Rev Latino-Am Enfermagem. 2013;21(1):1-8. doi: 10.1590/S0104-11692013000100011
  » https://doi.org/10.1590/S0104-11692013000100011
• ⁴⁷
  Croxton RA. The Role of Interactivity in Student Satisfaction and Persistence in Online Learning. J O Learn Teach [Internet]. 2014 [cited 2017 Dec 15];10(2):314-25. Available from: http://jolt.merlot.org/vol10no2/croxton_0614.pdf
  » http://jolt.merlot.org/vol10no2/croxton_0614.pdf
• ⁴⁸
  Margaryan A, Bianco M, Littlejohn A. Instructional quality of Massive Open Online Courses (MOOCs). Comput Educ. 2015;80:77-83. doi: 10.1016/j.compedu.2014.08.0050360-1315
  » https://doi.org/10.1016/j.compedu.2014.08.0050360-1315

Edited by

EDITOR IN CHIEF: Antonio José de Almeida Filho

ASSOCIATE EDITOR: Italo Rodolfo Silva

Publication Dates

• Publication in this collection
  18 May 2020
• Date of issue
  2020

History

• Received
  21 Sept 2018
• Accepted
  24 July 2019