Instrument for evaluating child hearing health services : construction and validity *

www.ee.usp.br/reeusp Rev Esc Enferm USP · 2018;52:e03357 * Extracted from the dissertation: “Práticas dos enfermeiros na atenção à saúde auditiva infantil”, Programa de Pós-graduação em Enfermagem, Universidade Federal de Pernambuco, 2014. 1 Universidade Federal de Pernambuco, Centro de Ciências da Saúde, Programa de PósGraduação em Enfermagem, Recife, PE, Brazil. 2 Universidade Federal de Pernambuco, Centro de Ciências da Saúde, Departamento de Fonoaudiologia, Recife, PE, Brazil. 3 Universidade Federal de Pernambuco, Centro de Ciências da Saúde, Departamento de Enfermagem, Recife, PE, Brazil. ABSTRACT Objective: To validate a matrix of indicators to assess the structure and process of child hearing health services. Method: A study of semantic and content validation with specialists in the area of Nursing and Speech therapy. The instrument contained 20 indicators with scores to be analyzed by experts. The condition for the validity suitability of each indicator and scores were an Item-level Content Validity Index (I-CVI) of 0.80 and a Scale-level CVI (S-CVI) of 0.80. Results: Twenty-two (22) specialists participated in the study, with 59% being nurses and 41% speech therapists, of which 32% had specializations, 45% had a Master’s degree, 18% had a Doctorate degree and 5% had a Post-doctorate degree. The mean I-CVI and S-CVI of the indicators evaluated as suitable were 0.96, while for the suggested scores the I-CVI was 0.80 and the S-CVI was 0.82. Conclusion: The matrix of indicators was considered valid for evaluating child hearing health services.


INTRODUCTION
In Brazil, 5% of the general population has hearing loss, which corresponds to approximately 9.7 million people (1) .Hearing losses may be associated with several risk factors in neonates and infants such as family history, congenital infections, postnatal infections, craniofacial anomalies, head trauma, hyperbilirubinemia with exchange transfusion, ototoxic medications, low birth weight, prematurity, length of stay in Neonatal Intensive Care Units (NICU), mechanical ventilation, suspected family developmental, hearing, speech or language delay, among others (2)(3)(4) .
The National Care Policy for Hearing Health (Política Nacional de Atenção à Saúde Auditiva) aims at successful interventions through actions of health promotion, prevention, treatment and rehabilitation of diseases/injuries, determining the technical quality required for good performance at all levels of health care through the interdisciplinary team and intersectoral actions (5)(6) .
In the scope of Primary Care and as participants in the Family Health Strategy team (FHS), nurses perform the systematic follow-up of children (0 to 5 years old) during routine consultations by assessing their growth and development, vaccination, breastfeeding, complementary feeding and providing guidance to mothers or caregivers on the prevention of accidents, individual/environmental hygiene and early identification of diseases for appropriate intervention (7) .
Child development milestones are evaluated during childcare consultations, and behavioral measures are adopted in case of possible changes, which should be based on the diagnostic impression of probable developmental delay, development alert, normal development with risk factors or normal development, as described in the monitoring instrument for development of the Child Health Handbook (8)(9)(10)(11)(12) .
Childcare follow-ups allow nurses to identify the absence of milestones related to children's hearing health early on.Proper practice by this professional in childcare may reduce the effects of hearing loss.Normative evaluation in the health service enables investigating how primary care nurses act in their attention to children's hearing health, offering subsidies for planning and managing health services (13) .
A normative evaluation consists in judging an intervention based on criteria and norms, comparing the employed resources and the organization (structure), the services produced (process) and the obtained results (14) .
In order to evaluate the structure and process of a child hearing health service, encompassing nursing practice, it is necessary to validate the proposed instrument in order to increase the reliability of the results, since the content validity evidences the specialized judges' measurement of the instrument's representativeness for the study population and its correspondence with the proposed objectives in the research (15)(16) .Thus, the objective of this study was to validate a matrix of indicators to evaluate the structure and process in child hearing health services.

METHOD
A methodological study carried out for semantic and content validation of a matrix of indicators to evaluate the structure and process in child hearing health services.
Validity is composed of a parameter that is congruent with the measured characteristic of the objects (17) .The semantic validation verifies the experts' understanding of the instrument items and the possible needs for modifications, seeking to increase understandability (16) .Content validation is the proportion of specialists who rank each item according to its relevance or suitability (15) .
The sample was intentional, consisting of 22 specialists graduated in Nursing or Speech therapy, with at least one post-graduation (specialization, Master's, Doctorate and/ or Post-doctorate degree), working in the area of childcare, primary care or neonatal hearing screening.The sample size calculation was based on the formula that considers the final proportion of specialists related to a dichotomous variable with the maximum acceptable difference of this proportion.A minimum proportion of 85% agreement between the experts and a 15% difference in this agreement was adopted.The formula used was: n =Zα 2 .P.(1-P) /d 2 , considering a 95% confidence level, resulting in 22 experts (18) .
An invitation letter was sent to the e-mail of 26 experts, explaining the purpose of validating the matrix of indicators, and 22 of these confirmed their participation (13 nurses and 9 speech therapists).The face-to-face interview was performed upon prior scheduling with each specialist to sign the clear and Informed Consent Form (ICF) and an analysis of the evaluation form, which occurred between July and August 2013.After consolidating the corrections/ suggestions resulting from the first analysis of experts in the matrix of indicators, the corrected instrument was sent to the e-mails of each specialist for a second evaluation, which resulted in no further suggestions regarding the content and semantics of aforementioned instrument.
The matrix of indicators originated from a logical model developed by the researchers based on the following references: National Care Policy on Child Hearing Health (Política Nacional de Atenção à Saúde Auditiva Infantil), report of the Joint Committee on Infant Hearing ( JCIH), National Policy for Basic Care (Política Nacional de Atenção Básica), Handbook of Integrated Care for Prevalent Childhood Illnesses (Manual de Atenção Integrada às Doenças Prevalentes na Infância -AIDPI) and Child Health Handbook (3,5,(9)(10)19) .
The matrix contemplates three dimensions: 1) Physical structure and necessary materials; 2) Technical and scientific suitability; and 3) Professional training.The established criteria correspond to attributes to measure the components of a service interconnected to indicators, which reach an expected score when met (14) .First, 20 indicators were created by the researchers, 9 for the physical structure dimension and 11 for the other two dimensions.The scores initially suggested by the researchers were distributed as 40 points for physical structure and necessary materials (4.4 points per criterion), and 60 points for technical-scientific suitability and professional training (3.1 points per criterion), totaling 100 points.
For the technical-scientific suitability dimension in the sub-dimension of monitoring hearing development milestones and conduct, one of the indicators corresponds to four clinical cases, and 5.0 points were suggested for the correct conduct in each clinical case, totaling 20.0 points.Clinical cases were constructed by the researchers according to the diagnostic impression of probable developmental delay, development alert, normal development with risk factors, or normal development, according to the instrument for monitoring development of the Child Health Handbook (8) .The four suggested clinical cases are described below: Case 1 -A 2-month-old child has a family history of deafness and did not perform the Neonatal Hearing Screening.On the current evaluation, the child does not react to the sound stimulus performed about 30 cm from the ear.The conduct will be forwarding him/her for evaluation with other professionals and providing guidance to the caregiver.
Case 2 -A 6-month-old child presenting a previous history of meningitis at 5 months performed the Neonatal Hearing Screening in the first month of life, with no alterations in the results.On the current evaluation, the child reacts to the soft sound stimulus by turning its head toward the sound.The conduct will be providing guidance to the caregiver on the warning signs and reassessing the child in 30 days.
Case 3 -An 8-month-old child has no risk factors for hearing health, as he/she performed the Neonatal Hearing Screening without any changes in the result.On the current evaluation, the child reacts to the conversation by saying "da da ... da da ...".The conduct will be to guide the caregiver to (Continue) stimulating the child and to schedule the next routine appointment.
Case 4 -A 1-month-old child with a birth weight of 2,300 g remained hospitalized for 15 days in the NICU shortly after birth due to respiratory failure and seizure episodes.The Neonatal Hearing Screening was performed after hos-pital discharge, presenting alterations in its result.In the present evaluation, the child does not show any response to the sound stimulus about 30 cm from the ear.The conduct will be directing the child to a neuropsychomotor evaluation and to provide guidance to the caregiver.
Each item in the indicator matrix was evaluated for the appropriate nomenclature, clarity, objectivity and applicability according to the Likert scale, adapted as: unsuitable, not very suitable, suitable and very suitable.The conditions for validating each indicator and the suggested scores were given when the mean of the Item-level Content Validity Index (I-CVI) was greater than or equal to 0.80, and the Scalelevel Content Validity Index (S-CVI) was 0.80, in order to establish excellence in content validity (20) .Modifications were made to content, the text and the grammar when the indicator was considered unsuitable by the experts, and the scores were redistributed between the criteria.
The study was approved by the Ethics and Research Committee of the Health Sciences Center of UFPE, opinion number 511.566 (01/15/2014), in compliance with the norms established by Resolution 466/2012 of the National Health Council -Ministry of Health (21) .

RESULTS
Of the 22 specialists, 59% were nurses and 41% were speech therapists, 32% had specific training in child health, 45% had Master's degrees, 18% had a PhD and 5% had a post-doctorate degree.The indicators classified as partially or completely suitable obtained a mean for both I-CVI and S-CVI equal to 0.96, and in the scores classified as fairly or completely suitable, the mean of the I-CVI was 0.80, and for the S-CVI it was 0.82 (Table 1).The S-CVI scores were considered valid based on the recommendations of the literature as this index was greater than 0.80 (20) .According to the experts' evaluation, an indicator of the technical-scientific suitability dimension in the sub-dimension of educational practices was excluded (Continuous education on child hearing health for the health team -Carrying out at least one continuous educational activity for the Family Health Team), and two more indicators were added in the sub-dimension monitoring the development of hearing and conduct development milestones (Registering in the chart/medical record/Child Health Handbook on the hearing milestones -Identifying at least one chart, medical record or Child Health Handbook with the records of hearing milestones; Using percussion instruments (rattles, bells, etc.) and/or other strategies (clapping hands, snapping fingers, etc.) to emit sounds -Verifying the use of percussion instruments (rattles, bells, etc.) and/or strategies (clapping hands, snapping fingers, etc.) to evaluate hearing development milestones), which resulted in 21 indicators.Chart 1 shows the matrix with the validated indicators and scores to be applied in evaluating the structure and process in child hearing health services, also taking into account the practice of Family Health Strategy nurses.

DISCUSSION
Six of the nine indicators for the physical structure/necessary materials dimension reached an I-CVI > 90%.Of the three indicators with an I-CVI < 90%, the term examination table was included in the indicator of beds or stretchers, and the term Child Health Handbook was removed from the registration form/child's medical record/Child Health Handbook, since it is described as one of the criteria related to the nine indicators.This last indicator raised questions regarding the importance of recording the information on printed forms requested during the child care consultation, thereby leading to introduction of a new indicator in the technical-scientific suitability dimension, sub-dimension of monitoring the hearing development milestones and conduct to evaluate the hearing development milestones recorded in the forms/printouts (registration form, medical record or Child Health Handbook) (Chart 1).
The indicator percussion instruments (bells or rattles) and/or strategies used to evaluate hearing and language milestones had a disagreement of 14% among the experts on the term "strategies", due to the fact that this term cannot be included in the physical structure, but rather in the evaluation process; therefore it was removed from the physical structure and inserted into the process as a new indicator entitled "Using percussion instruments (rattles, bells, etc.) and/or (other) strategies (clapping hands, snapping fingers, etc.) to emit sounds in the sub-dimension of monitoring hearing development milestones and conduct (Chart 1).These modifications had the purpose of increasing the comprehensibility of the instrument for evaluating the structure and the process in child hearing health services, considering nurses' practice in childcare.The construction of indicators interconnected to well-established criteria in a normative evaluation aims to offer reliability to the evaluation process, thus enabling the results of the analyzed aspects to show the reality of the evaluated service/program (14) .In the Family Health Strategy context, a normative evaluation of the physical structure and material resources of prenatal care carried out in seven Family Health Units (FHU) in the city of Rio de Janeiro demonstrated the consonance of the indicators and adopted criteria to support professional practice in its totality (22) .

(Continuation)
With regard to the physical structure/necessary materials dimension, only the indicator related to the availability of chairs for accommodating professionals, patients and companions reached 90% agreement among the experts; while an 86% agreement was found for the indicators regarding the room/space for consultations and beds or stretchers, followed by 81% for the availability of space suitable for educational activities, Child Health Handbook and Informative materials.
A greater disagreement between the experts was observed for the indicator instruments used to emit sounds and for the registration form, medical records and the Child Health Handbook of 28% and 23%, respectively.The disagreement in relation to the instruments used to emit sounds may be due to the fact that the experts consider this criterion specific for child hearing health, and the possibility that these instruments are not available among the basic materials used in the FHU, as shown by a normative evaluation in Brazilian states that did not adopt the existence of this indicator in its criteria among the basic materials for performing the activities in the FHS (23) .
Eight of the 11 indicators in the dimension of technical-scientific suitability and professional training reached an I-CVI ≥ 95% agreement among the experts.Only one indicator, which addresses the identification of the most favorable age for the initial evaluation (NHS), hearing loss diagnosis and intervention/rehabilitation, of the sub-dimension for monitoring the hearing development milestones reached an I-CVI of 86%.This indicator presented disagreement (among the experts) regarding the most favorable/ probable age due to the term "age" corresponding to an exact measurement which may lead the nurse to a misunderstanding in the evaluation, with recommendations to replace the term by "most favorable moment".
The four clinical cases evaluated separately obtained 100% agreement among the experts.It is worth pointing out that some changes in grammar have been made to make them clearer, more objective, applicable and appropriate.The four clinical cases validated according to the diagnostic impressions of the child development milestones evaluation can be applied in other settings to evaluate the practice of primary care nurses with the aim of highlighting the difficulties and to support possible improvements within the scope of child hearing health care.Application of the clinical cases in other studies will enable an understanding of the effectiveness of nurses' practice in evaluating child development milestones related to hearing health in order to make it a consolidated practice in nurses' performance.
Validating indicators and clinical cases for evaluating the structure and the process including nurses' practice in child hearing health services is important, as the FHS acts as the entry point of the health system in Primary Care through actions of health promotion and prevention of diseases/injuries during the childhood period, favoring implementation of the National Policy on Hearing Health Care (Política Nacional de Atenção à Saúde Auditiva) to improve the diagnostic and therapeutic assistance of neonates and infants with hearing loss (5,(24)(25) .
In the sub-dimension educational practices, the indicator continuous education in child hearing health for the health team was classified as not very suitable or unsuitable by 32% of the experts, as the matrix of indicators was directed towards nurses rather than the health team.However, this indicator was disregarded, as nurses continuous education is contemplated by the indicators described in the professional training sub-dimension.
The continuous education indicator demonstrates nurses' knowledge about child hearing health acquired during their initial graduation, post-graduation and training, and it is directly associated with the performance of educational practices during their professional actuation.Not only nurses but all FHS professionals who act as health educators should be responsible for the systematization, development and monitoring of the educational process that values human beings in its biopsychosocial aspects, thereby seeking better results according to the reality of the population served at the FHU (26)(27) .
Regarding the scores of the technical-scientific suitability dimension, a 90% agreement was achieved regarding the suggested clinical cases, reinforcing the importance of a higher score for this indicator in relation to the others in this dimension.The conduct of clinical cases is a relevant indicator for evaluating the structure and the process in child hearing health services, within the framework of the Family Health Strategy nurses' practice.
In relation to the indicator matrix scores, in the sub-dimension management of risk factors for hearing loss, the indicators for identification of risk factors for hearing loss and the association of these factors with child development milestones reached an agreement of 86%, emphasizing the need for nurses to be aware of some of the risk factors for hearing loss described by the Joint (3) , since they can be identified in monitoring hearing and language development in childcare consultations.
In relation to developmental milestones, 23% of the experts disagreed on the score, arguing that it should be higher than the other evaluated indicators.Also, 28% of the experts disagreed for the initial evaluation criteria (NHS), trained professionals, diagnosis, hearing loss intervention/rehabilitation represented in two indicators (excluding the trained professional for the NHS) for not considering the necessity of the suggested scoring in this criterion, and in the indicator score for the performance of educational practices in child hearing health in the community they can be very specific for evaluating structure and process, considering the practice of the FHS nurses.It is known that performing health education is part of the attributions of primary health care nurses, which can be directed individually or collectively to develop the responsibility of family members on child health, and consequently to promote improvement in prognostics of child hearing health (28) .
In the professional training dimension, disagreement by the experts on the scores was 28% for participation in courses related to the child hearing health, and 32% for specific courses and training.These percentages may be due to the Instrument for evaluating child hearing health services: construction and validity Rev Esc Enferm USP • 2018;52:e03357 experts questioning the Nursing training offered by various educational institutions and the availability of courses/training offered by the health secretariats, which may be incipient regarding child hearing health.
The suggested score could be lower in relation to the reality experienced by FHS nurses, considering that the educational practices are based on knowledge acquired during professional training, and in the routines and demands of the FHU to meet the objectives of the National Primary Health Care Policy (25)(26) .One of the roles of health management is to offer courses and training for professionals in order to improve the care quality provided to children's health, especially with regard to child hearing health, in order to reinforce the recommendations of the National Policy on Hearing Health Care (5) .
Another aspect to be considered was the indicator of nurses' participation in continuous education for the health team, with 32% disagreement among experts due to the questions raised in the indicators of the professional training dimension.Therefore, this indicator has been removed from the technical-scientific suitability dimension since it covers the continuous education of the health team, and the purpose of the instrument is to evaluate the structure and the process in child hearing health services within the framework of nursing practice.

CONCLUSION
Validation of the matrix of indicators for evaluating the structure and the process, including nurses' practice in child hearing health services was fundamental to increase the understanding of this professional category during data collection and to facilitate the evaluation process.The experts' opinions and suggestions were analyzed according to the I-CVI values, and most of the indicators obtained an I-CVI agreement ≥ 80%.A higher disagreement among the experts was observed in relation to the suggested scores, since the values for each indicator were analyzed in relation to the evaluated aspect.The scores remained uniformly distributed among each aspect to be evaluated, and in agreement with the S-CVI values suggested by the literature.
The matrix to be analyzed initially consisted of 20 indicators and 21 indicators were validated at the end, in addition to the four clinical cases.Only two indicators were added and one was disregarded after the experts' analysis.The four clinical cases can be applied with other nurses to evaluate the effectiveness of this professional category's practice in evaluating child development milestones related to hearing health.Thus, the matrix of indicators was considered valid for evaluating child hearing health services.Furthermore, the validation of this matrix of indicators for evaluating the structure and the process, taking into consideration nurses' practice in child hearing health services will enable this tool to be applied in other studies.

Aspect: Physical structure Dimension: Physical structure and necessary materials Considered criteria Indicators I-CVI (Indicators) Suggested scores I-CVI (Scores) Physical structure
Registration form/child's medical record/Child Health Handbook with the milestones Availability of registration form/medical record/Child Health Handbook with evaluation of hearing and language milestones 0.86 4.4 0.77 Informative materials Availability of informative materials for educational activities on children's hearing health in the community and health staff (folders, booklets, brochures, etc.) Instrument for evaluating child hearing health services: construction and validity Rev Esc Enferm USP • 2018;52:e03357 Chart 1 -Matrix of validated indicators for evaluating the structure and process in child hearing health services, considering nurses' practice -Recife, PE, 2013.