Expert assessment of the “ Neonatal Pain Assessment Program ” online course

Rev Bras Enferm. 2020;73(4): e20180392 http://dx.doi.org/10.1590/0034-7167-2018-0392 7 of ABSTRACT Objectives: to assess layout quality, visual identity and content of the “Neonatal Pain Assessment Program” (Programa de Avaliação da Dor Neonatal) online course. Methods: a descriptive exploratory study. The course was assessed by 24 experts in pain, neonatology and education. A form containing 20 questions on technical, interface and educational aspects was used. For each item, the score ranged from 0 to 1. Mean scores above 0.7 were considered indicative of high quality of the items. Descriptive statistics were used for data analysis. Results: navigation, clarity, ease of localization, content relevance, contextualization, content correction, multiple windows, ease of use, ease of return, ergonomics, esthetics, special brands, audiovisual resources, information and portability were assessed. All aspects obtained a mean ≥ 0.70 and no changes were required. Conclusions: the Neonatal Pain Assessment Program is considered as quality educational technology and promising strategy for health education. Descriptors: Distance Education; Educational Technology; Infant; Neonatal Nursing; Pain Measurement.


INTRODUCTION
Neonatal pain is a complex, subjective and multidimensional phenomenon making its assessment a challenge, which is compounded by the impossibility of verbal reporting in this age group (1) . Thus, behavioral, physiological indicators, endocrine-metabolic changes, contextual and neurophysiological components are important for the multidimensional neonatal pain assessment. To this end, specific pain assessment tools for the Newborn (NB) have been constructed and validated based on these indicators (2)(3) .
Currently, the literature offers a wide range of tools for neonatal pain assessment, although none of these tools is considered ideal, since they consider subjective indicators for their assessment, such as behavioral and contextual parameters (4) . However, some of these have adequately explored and established psychometric properties, and have been widely used in clinical practice and research. For this reason, they are recommended by agreement and guidelines (2,(5)(6) .
Although some health professionals, mainly nursing, recognize the occurrence of pain in the neonatal population, their assessment is still performed empirically in the care settings, especially by isolated assessment of physiological and behavioral parameters, predominantly crying (7)(8)(9) . The lack of specificity in the assessment of pain in NB reinforces the importance of adopting systematic methods to identify pain in this population (10) . To do this, training and qualification of health teams are necessary, in order to offer knowledge and develop skills in assessment performance, depending on the variety of tools available and the different scoring methods (4,11) .
Moreover, Information and Communication Technologies (ICTs) in education are a useful alternative for the teaching of students and training of health professionals, since they facilitate articulation between theory, clinical practice and research. Thus, it enables the development of a critical-reflexive behavior (12)(13) . ICTs offer conveniences such as temporal flexibility (14) , greater ease of interaction (15) , focused and self-directed study, and active learning (16)(17)(18) . In addition, they allow interactivity and inclusion of audiovisual resources (audios, images, videos, animations, quizzes, texts, among others) as a form of educational support (19) . It is also pointed out that the use of technology in education has demonstrated significant cognitive gain for users, as observed in traditional teaching (20)(21) .
Evidence shows the increasing use of ICTs in student education and training of health professionals, although online teaching in the context of neonatology is still incipient and requires investments for the use of new educational strategies (22)(23) .
Considering the challenges related to neonatal pain assessment in clinical practice, as well as the advantages related to the use of ICTs in professional training process, an online course was developed, called Neonatal Pain Assessment Program (PAD-Neo -Programa de Avaliação da Dor Neonatal). The course aims to contribute to the teaching of health students and improve the professionals' knowledge on pain assessment and measurement, neonatal pain recognition skills and, finally, assist in the selection and use of pain assessment tools in clinical practice.
In its initial version, developed between 2009 and 2010, PAD-Neo was hosted on the Modular Object-Oriented Dynamic Learning Environment (Moodle ® ) platform and organized into eight modules. This version was assessed from the users' perspective, which included professionals and undergraduate and graduate students in health. They were satisfied or very satisfied with aspects such as time, content organization and applicability, Virtual Learning Environment (VLE) and audiovisual resources. Additionally, a significant difference was observed between the number of hits observed in the post-test (16, ± 2.41 mean) compared to the pre-test (13 ± 1.88 mean), p = 0.006 (24) .
Due to the need for revision and updating, the course remained hosted on the Moodle ® platform, and its content was extended from eight to ten weeks of activities. The main modification involved neonatal pain assessment tools, with prioritized internationally recommended scales, recently revised and adequately translated into Portuguese (Brazil). The Neonatal Facial Coding System (NFCS) (25)(26) , which was validated and translated in Brazil as Sistema de Codificação da Atividade Facial Neonatal (27) and the Neonatal Infant Pain Scale (NIPS) (28) , translated as Escala de Dor no Recém-Nascido (29) . Premature Infant Pain Profile (PIPP) (30) has been replaced by its upgraded version, the Premature Infant Pain Profile -Revised (PIPP-R) (31) was translated as Perfil de Dor no Recém-Nascido Pré-Termo -Revisado (32) . There was also a need to replace the Crying Requires oxygen for saturation above 95%, Increased vital signs, Expression, Sleepless (CRIES) (33) by the Behavioral Indicators of Infant Pain(BIIP) (34) , validated for Portuguese as Indicadores Comportamentais de Dor no Recém-Nascido (35) , since CRIES only assesses neonatal pain in the postoperative period. The other tools can be used for this purpose. Finally, the scale assessing the prolonged pain Échelle Douleur Inconfort Nouveau-Né (EDIN) (36) , translated in Brazil as Escala de Dor e Desconforto do Recém-Nascido (37) .
Audiovisual resources such as interactive videotapes, texts for reading, forums, exercises using images and videos and links to websites made up the resources developed for the course initial version. Use and satisfaction of this first version were assessed by users, through pre-test and post-test, and satisfaction questionnaire, respectively.In spite of the positive results obtained from the assessment of the initial version of the PAD-Neo, a course quality assessment was considered necessary and pertinent from the perspective of experts in the areas of interest as part of the process of improving the use of ICTs in health education.

OBJECTIVES
To assess layout quality, visual identity and content of the PAD-Neo online course.

METHODS
This is an exploratory descriptive study that set out to describe the process for assessing the quality of layout, visual identity and content of the PAD-Neo online course.

Ethical aspects
The research project was assessed and approved on February

Place of study and period
The present study was conducted between July 2015 and July 2017, and data were collected between July and December 2016 at Universidade de São Paulo's School of Nursing.

Population and sample
An expert committee was composed of professionals working in neonatal and/or pediatric pain, neonatal care and education (with emphasis on the development and assessment of digital educational resources), considering eight experts per area (38) . The selection criteria of pain experts were: to hold a Doctor's degree and to have research projects and/or Master's or Doctorate guidelines related to pain in neonatology and/or pediatrics. For neonatology professionals inclusion, it was adopted as a criterion neonatal care performance for a time equal to or greater than three years and/or hold Master or Doctor title in neonatology. Finally, there was inclusion of experts in educational technology higher or who held Doctor's degree and/ or research projects, Master's or Doctorate guidelines in the area of development and assessment of digital educational resources.
The non-random selection of convenience was selected through non-random selection of contacts and partnerships with leaders and members of research groups registered in the CNPq Research Groups directory and linked to Universidade de São Paulo' School of Nursing.

Data collection
Experts were invited by electronic means. Upon accepting to participate as an evaluator, a new message was sent containing the course's electronic address, instructions for visitor access and password, as well as the electronic form developed in Google Docs ® to record the profile characterization data of evaluator and course quality assessment. The online course assessment tool was developed based on a tool proposed by the Central Coordination of Distance Education (CCEAD -Coordenação Central de Educação a Distância, PUCRIO) (39) .
The assessed aspects included: free navigation (control of the sequence of use by the user); information clarity (clarity of information and emphasis on one concept at a time); information localization (features that make it easier to locate); content pertinence (logical presentation and intuitive navigation); contextualization (consistency and appropriateness of content); content correction (scientific rigor); multiple windows (overlapping windows); ease of learning in interaction (understanding of interaction); use efficiency (information and commands for the use); ease of return (return to previous page); ergonomics (maintenance of interaction and standardization of presentation); esthetics (appropriate interface standards); special marks (use of marks facilitating recognition); audiovisual resources (adequate variety and quality); references (presentation of reliable sources); interactivity (interaction with the course); error management (error reduction or prevention); users help (information and help resources); information quality (information compatible with the public) and portability (functionality on different devices and browsers) (39) . There is a section for observations and suggestions from evaluators.
There were six scoring options for each item (0, 0.25, 0.5, 0.75, 1, and not applicable). The score 0 represented the worst assessment and 1, the best assessment of the item. In order to establish the standard of online course quality assessment, subsidies of the standard of quality and performance of technologies, ABNT (The Brazilian Association of Technical Standards -Associação Brasileira de Normas Técnicas -is the body responsible for technical standardization in Brazil, providing inputs to the Brazilian technological development) ISO/IEC 14598-6 (38) . This standard also determines that the expected mean for the assessed characteristics is greater than or equal to 70%. Thus, values were calculated for each item, and those with an overall mean lower than 0.70 would be modified by the research team, according to expert assessment and suggestion (38) .

Analysis of results procedures
The data obtained electronically through the Google Docs® form was transported to Microsoft Excel for Windows® for analysis. Descriptive statistics was used, where arithmetic mean and relative frequency of variables of interest were calculated. The results were presented in a descriptive way.

RESULTS
Twenty-four experts, all of them female, participated in the study. The mean age was 43 years, with the majority of the participants being graduated in nursing (18 -75%). Of these, four were considered as pain experts, seven as experts in neonatology and seven as experts in education. The committee was also comprised of four psychologists working in the area of pain, a physiotherapist working in neonatal care and a participant with a degree in instructional design in education.
Two professionals with PhD (8.3%), fifteen professionals with doctor degree (62.5%), six with master's degree (25%) and one instructional design expert (4.2%) were included. As to position and function performed, ten evaluators are university professors (41.6%). Five (20.8%) experts work at neonatal or joint care units as nurses. Four work in the area of education (16.6%). Two are graduate students (8,3%). One expert is an instructional designer (4.2%), one is a people management nurse (4.2%) and one is a supervisor of the physiotherapy service (4.2%). Regarding course quality assessment, Table 1 shows the means obtained by the overall areas of expertise.
All aspects obtained a mean above 0.70 in the committee overallassessment, which constitutes adequate quality of the PAD-Neo (Figures 1 and 2). The four aspects assessed by education experts, whose means were below 0.70 (audiovisual resources, interactivity, error management and help to users), were reviewed in order to adapt them to the requirements.
Space for suggestions and observations on the form was used by 17 experts who expressed satisfaction with presentation and organization of the online course and made suggestions for course improvement:

DISCUSSION
The increasing production of evidence on the use of technologies in the teaching and training of students and health professionals, especially nursing professionals (20)(21)(22) . Studies in this subject are essential to give validity and quality to educational proposals, such as the PAD-Neo, which, through assessment, offers subsidies to develop and improve the teaching-learning process in the virtual environment (20)(21)(22) . The linear format of the course was considered by some experts as a limitation to the user's free navigation, mainly by experts in education (0.81 mean). The literature shows that nonlinear navigation can affect user control, however, there is no significant influence on user learning (40) . It is worth noting that the first modules of the course cover basic themes (Pain: definitions and basic concepts of physiology, Components of pain assessment in NB, and Principles of measurement and pain assessment in NB). Linearity was necessary in these steps for the user's adequate understanding of the content. On the other hand, modules referring to neonatal pain assessment tools have independent content and, for this reason, will be offered in the future in an open way so that the user can develop their activities according to their preference, thus accepting experts' suggestion.
Information clarity was assessed positively by all experts. The language used in an online course should be clear, objective and accessible in order to promote the understanding and learning process of the Distance Education user (EAD -Educação a Distância) (41) .
In order to promote user comprehension, a virtual environment should allow self-centered learning, favoring autonomy and independent navigation (42) . The use of resources to locate information helps to understand the functioning of the course, from the tutorials and menus. For this reason, special identification marks were developed for activities such as  classes, exercises, forum, among others. The main highlights and orientations of each unit are arranged from a short document guide in Portable Document Format (PDF) format, as well as the interactive classes developed with Articulate Storyline ® Software, among others. Regarding content pertinence, this criterion refers to the logical and clear presentation of the information, and to the navigation as it is intuitive to the user (39) . Adding to this, contextualization, when assessing adequacy and consistency in content, as well as scientific rigor, assesses correctness of content. From the initial version, the PAD-Neo was developed on the basis of highquality scientific evidence (24) . For the current version, the material content was updated by the research team, in order to include new contents and references, which is reflected in the positive assessment made by experts in these three aspects.
However, the number of articles in Portuguese available as a complementary reading resource in the PAD-Neo was an observation pointed out by two experts. Furthermore, in spite of the scarcity of scientific articles pertinent to the course theme and published in Portuguese, efforts were made to follow the suggestions and expand the references made available in the course.
Regarding the use of multiple windows, this feature was considered by experts as preferably the user's individual, as it may facilitate navigation or confuse. Some of the features featured in PAD-Neo are viewed from new window openings because it is hosted on Moodle ® . On the other hand, hosting PAD-Neo on the Moodle ® platform may have favored the positive assessment of the ease of learning in the interaction. It is considered that this platform allows a greater participation and interaction of users with the virtual learning environment (43) .
Ergonomics assesses uniformity in the presentation of online course content and resources. Its assessment is fundamental in order to improve technical and usability aspects that involve the teaching-learning process (44) . In this way, adapting the online course to ergonomic criteria favors the understanding and use by users (44) . Overall, experts pointed out that the course presents an organization favorable to the understanding of the user, configuration of the fonts that facilitate readability and that the icons, screens and resources are presented in a uniform way.
Evaluators considered "esthetic" and "use of special brands" as high quality. Visual information influences learning, since the organization of the interface can either encourage the individual, favor teaching or be a distracting resource (45) . As part of the course update, a new visual identity was developed for PAD-Neo. According to experts, the selected colors and font facilitated content presentation and readability.
As to audiovisual resources and interactivity, four specialists judged the variety of resources as limited. These resources are considered as fundamental in the interaction between the user and the virtual environment, since its use benefits the learning, making the process more interesting and pleasurable (46) . In addition, interactivity in online courses is characterized as an important tool for the active participation of the user, guaranteeing their satisfaction and avoiding their discontinuity (47) . By offering videotapes, interactive lessons, exercises based on videos and photos, and discussion forums, it is believed that the PAD-Neo presents a considerable variety of audiovisual resources. Due to the overall mean attributed by "audiovisual resources" experts, no changes were made to this item.
Regarding errors management, along the course, icons and guidelines indicative of the following activities and/or steps are available, in order to minimize navigational errors. As for the following item, referring to user help, experts suggested creating a user doubts forum, which was included on the main page.
As to information quality, experts assessed the item in a positive way, since the references, workload and vocabulary used were considered adequate and pertinent to the intended target audience. Despite the success of experts accessing the course from different browsers, some of them reported difficulties in watching the course and its content on mobile devices, such as mobile phones and tablets. This is considered an important limitation and will be met in future course updates.
PAD-Neo can be considered as an educational tool of adequate quality, considering the rigor with which it was developed and assessed. The vast majority of ICTs are developed and implemented without assessment. However, expert assessment with specific competences is determinant and necessary for the quality of the proposed teaching (48) . It is also worth noting the pioneering and innovative character of the PAD-Neo in the Brazilian educational setting, given the scarcity of studies on the development and assessment of online courses in neonatal care, including in relation to pain (23) .
In addition, studies on the course's clinical practice of health professionals repercussions, such as the development of institutional protocols for neonatal pain assessment and increase of records of pain assessment in medical records are necessary.

Study limitations
As a limitation of this study, some of the scores attributed below 0.50 were not adequately justified by experts, according to guidelines previously highlighted in the assessmenttool. This made it difficult to assess the course quality accurately and prevented the team from making improvements in order to adapt the items to the proposed criteria. It is also observed that the tool used in the present study may have contributed to failure and/or difficulty of interpretation of some items, thus generating meaning ambiguity of descriptions.

Contributions to Health
It is expected that the availability of the PAD-Neo for undergraduate and graduate students in health, through its classroom applicability, extension courses, and improvement and professional training in continuing and permanent education, contributes to disseminate knowledge about neonatal pain.

CONCLUSIONS
This study assessed the quality of PAD-Neo from the perspective of experts from different areas that present interface in relation to the subject addressed. PAD-Neo is considered as an educational technology of adequate quality and multi-applicability contributing to the teaching of undergraduate and graduate students in health and qualification of health professionals. Expert assessment contributed to the early identification and adjustment of failures that could negatively influence the user experience. The quality and innovative character of PAD-Neo make this initiative promising in the Brazilian setting.