Validation of an indicator matrix for the assessment of the Neonatal Hearing Screening Program

ABSTRACT Purpose: to validate an indicator matrix to assess the Neonatal Hearing Screening Program (NHSP). Methods: methodology development research. A total of 13 speech-language-hearing therapists with a specialization in audiology and/or at least three-year experience in neonatal hearing screening participated in the validation process. Quantitative and qualitative data were collected to develop the indicator matrix, which was then submitted to the validation process. The results of the specialists’ evaluation, in this stage, were quantitatively analyzed with the item content validation index (I-CVI) and scale content validation index (S-CVI). Results: regarding the indicators classified as quite or fully adequate, the mean I-CVI was the same as the mean S-CVI (0.95), evidencing excellence in their content validity. Concerning the scores classified as quite or fully adequate, the I-CVI mean was also identical to that of S-CVI (0.83), thus, reaching a consensus. Conclusion: this matrix with 33 indicators that had their content validated with consensus, will consistently contribute to assessing NHS services in Brazil.


INTRODUCTION
In the field of health assessment, indicator matrices are tools that facilitate the organization, design, and analysis of programs. They are made up of criteria and indicators that help analyze and interpret information, besides making it easier to organize and plan data collection, and develop assessment instruments, analysis techniques, and result presentation 1 .
The purpose of the Neonatal Hearing Screening Program (NHSP) is the timely detection of hearing loss (HL) in babies, from their first 24 hours of life up to three months old. It is part of a set of actions that must be performed aiming at the comprehensive hearing health care in childhood: screening, monitoring, and following up the hearing and language development, diagnosis, and (re)habilitation 2 .
The NHSP must be an integral part of the Health Care Network for People with a Disability and the mother/child follow-up actions, and be coordinated with primary health care to ensure the monitoring and follow-up of hearing and language development, and the adherence to referrals to the specialized services 2,3 .
By employing a validated instrument, it is possible to know better the evidence of the contribution of NHS to the timely diagnosis and intervention in hearing. Also, considering the local difficulties to receive speechlanguage-hearing health care, future negotiations with the government can be made to reorganize and improve the NHSP. Hence, the actions can be coordinated in a network, permeated by a flexible service organization, respect to the users, and interprofessional work 4 .
In the literature, various articles are available reporting the results obtained with the NHSP in screening services of different countries [5][6][7][8][9] . However, as far as Brazil is concerned, the publications that were found only demonstrated such results at a local level, not denoting the nationwide situation [10][11][12][13] .
Thus, the indicator matrix can become a research instrument, and particularly an assessment instrument.
For this to happen, the indicators contained in the matrix must be validated by a group of specialists 13 .
It is through such a validation process that this instrument will be able to generate valid and reliable results, making it widely useful in the clinical practice and research in the field of public health. Moreover, it will be an aid in other fields, including the assessment of speech-language-hearing services 14 .
This paper aimed to develop and validate the NHSP Indicator Matrix in the hearing health care network.

METHODS
The research was approved by the Research Ethics Committee (REC) of the Universidade Federal de Pernambuco/Department of Health Sciences -UFPE, PE, Brazil, under evaluation report number: 2.695.541.
This study was designed as a methodological development research whose purpose was to develop an instrument to assess the NHSP.
The study sample comprised 13 speech-languagehearing therapists that contributed to the consensus group to validate the indicator matrix. The specialists were selected based on the following criteria: being either speech-language-hearing therapists or otorhinolaryngologists specialized in audiology and/or experienced in coordinating/performing NHS for at least three years.
Quantitative and qualitative data were collected do develop the indicator matrix.
The study was conducted in two different stages: 1 st ) development of the NHSP indicator matrix in the hearing health network; 2 nd ) validation of the indicator matrix to assess the NHSP.
In the first stage, the NHSP indicator matrix was developed based on the NHSP logical model proposed and validated by Pimentel, Figueiredo, and Lima 15 in a previous study. The logical model is summarized in Figure 1.
Henceforth, the indicator matrix was developed, encompassing the aspects of Structure and Processes, and containing criteria, indicators, dimensions, subdimensions, and the expected score ( Figure 2).
In the second stage of the research, the indicator matrix was assessed in consultation with professionals experienced in the field of NHS.
In this study, 26 specialists (23 speech-languagehearing therapists and three otorhinolaryngologists) were contacted, using the snowball sampling technique 16 -in which a specialist indicates others that meet the profile necessary to the research. They worked in the Brazilian states of Alagoas, Amazonas, Paraíba, Pernambuco, São Paulo, Sergipe, and in the Federal District, and their titles included, at least, specialization/ residency, with experience in implementing, coordinating, and/or performing NHS 17 .
According to the literature [18][19][20] , the participants of the research, besides being experts or having knowledge of the phenomenon, must be available and motivated to participate in the different stages of the study. In their selection process, they were contacted via phone call and sent an invitation letter, justifying the research, and clarifying the importance of their participation.
When contacted via phone call, only two specialists did not confirm their participation -one otorhinolaryngologist and one speech-language-hearing therapist. After the other ones expressed their willingness to participate in the research and confirmed their electronic address for further communication throughout the data collection process, they were sent an e-mail with the invitation letter presenting the research, the informed consent form (ICF) for formal authorization, an online form with a questionnaire for a brief characterization of these specialists, and a Microsoft Excel spreadsheet with the NHSP indicator matrix for them to evaluate and fill out.
After the e-mails had been sent with the said material, 17 specialists (65.3%) answered the online form. Of these, four answered only the form; since they did not return the indicator matrix, they were excluded from the research. Those who effectively participated in the research were 13 specialists (50% of all those invited).
The filled-out forms were analyzed, and the data contained in the indicator matrix were processed according to the validation criteria described below.   The professionals were asked to evaluate the suggested score for each indicator and classify it as: (1) Inadequate, when they strongly disagreed; (2) Little adequate, when they disagreed; (3) Quite adequate, when they agreed; and (4) Fully adequate, when they strongly agreed.
The specialists were also instructed to suggest a new score for each indicator in case they did not agree with the one proposed by the researcher.
The results of the specialists' evaluation were qualitatively (with the analysis of the comments registered by them) and quantitatively analyzed (with the item content validation index [I-CVI] and scale content validation index [S-CVI] 18-20 ), as described below: Number of specialists that classified the item as 3 or 4 Total number of specialists The individual item content validity is the proportion of specialists that classified the item according to its relevance or suitability 19 .
Hence, on a scale in which 1 represents an inadequate item and 4, a fully adequate item, the real CVI is the proportion of items the specialists classified as 3 or 4 18 .
The condition required to validate each item of the matrix was that its I-CVI value be equal to or above 0.80, categorizing it as adequate. In case the I-CVI were considered inadequate, i.e., obtaining a value below 0.80, the item would have to be either eliminated or (if any of the specialists had suggested alterations to adequate the item) maintained, though modified.
After calculating the I-CVI for each item, the S-CVI was calculated, following this formula:

Number of items classified as 3 or 4 Total number of indicators
The S-CVI must be equal to or above 0.90 to characterize excellence in content validity 19,20 .
After the specialists' evaluation, the answers were tabulated and separated by indicators and suggested scores. Hence, the I-CVI was calculated, and so was the S-CVI afterward.
The specialists judged each indicator regarding nomenclature suitability, clarity, objectivity, and applicability. These items were presented as options in the assessment matrix sent to each one of them.
Then, they were invited to evaluate whether the indicator in question was appropriate to assess the NHSP, based on the selected aspects.
To this end, a Likert-like scale [18][19][20] was used, categorizing the indicator evaluation as: (1) Inadequate, when the four aspects (nomenclature suitability, clarity, objectivity, and applicability of the indicator) were absent; (2) Little adequate, when only one aspect was present; (3) Quite adequate, when two or three aspects were present; and (4) Fully adequate, when the four aspects were present.
The score distribution was also evaluated by the specialists during the validation process. It was proposed that its measurement be based on the distribution between Structure and Process. The total score given to Structure was 30 points, while to Process, it was 70 points, totaling 100 points. Those of Process were subdivided into dimensions, as follows: hearing health education (10 points), neonatal hearing screening (50 points), and administration (10 points).
In principle, this individual indicator score was evenly distributed by the researcher and demonstrates the degree of importance each aspect (Structure and Process) has within the NHSP. This distribution was predetermined by the specialists that participated in the validation of the NHSP logical model, in the first stage of the research.
The score to be analyzed by the specialists was evenly distributed into the following dimensions: facilities, material resources, and human resources (2 points/indicator); hearing health education (2 points/ indicator); neonatal hearing screening -which was given a higher score (5 points/indicator); and administration (2.5 points/indicator).
It should be highlighted that the researcher chose to evenly distribute these points so the judges' analysis would not be positively or negatively influenced.

RESULTS
The indicator matrix that resulted from the validation process is shown in Figure 2.
14. Available speech-language-hearing therapists and/or physicians (neonatologists or otorhinolaryngologists) with a specialization or proven experience in Audiology to perform, coordinate, and administer the NHSP actions.

SUBDIMENSION: INSTRUCTIVE ACTIONS
Disseminating information on the importance of immunization, prenatal follow-up, RIHL, and NHS to early detect hearing loss (HL).
17. Periodically disseminate information on the importance of immunization, prenatal follow-up, RIHL, and NHS to early detect hearing loss (HL). Promote dialogue circles about the importance of the NHS with health professionals that work at the maternity hospitals, outpatient centers, primary health care, family health strategy, and extended family health care centers.
18. Periodically promote dialogue circles with health professionals.

SUBDIMENSION: TEST
Performing the otoacoustic emissions examination (OAE) in newborns after the first 24 hours of life and before hospital discharge. 22. Identification of the babies that undergo the NHS with OAE examination. Performing the a-BAEP examination. 23. Identification of babies that undergo a-BAEP examination in the NHSP.

SUBDIMENSION: RETEST
Repeating the OAE examination up to 30 days after the test.
24. Periodicity of the retest. Immediately carrying out the a-BAEP in babies that "failed" again in the OAE retest.
25. Periodicity of a-BAEP examination in babies that "failed" the retest. 6 6.0 points = if a-BAEP is immediately carried out in babies that "failed" the retest.
3.0 points = if a-BAEP is not immediately carried out in babies that "failed" the retest, but they are referred for it in specialized services. 0 points = if a-BAEP is not immediately carried out in babies that "failed" the retest, neither are they referred for it in specialized services.

PROCESS NEONATAL HEARING SCREENING
Immediate referral of babies that will need a diagnosis. 26. Immediate referral for a diagnostic assessment in the specialized hearing services of babies that did not obtain a satisfactory response with a-BAEP and have suspicion for hearing loss. 5 5.0 points = if parents are instructed and the baby is immediately referred for diagnosis in specialized services. 0 points = if parents are not instructed, neither is the baby immediately referred for diagnosis in specialized services.
Referral for auditory monitoring. 27. Instructing the parents and providing referrals for auditory monitoring of all the babies that underwent the NHS but had RIHL and of those that failed the OAE record but had satisfactory results in the a-BAEP. NHS results presented to the parents/ guardians, administrators, and other health professionals.
28. Test and retest results presented with: (1) support to the parents when the result is told and presented in print; (2) requirement that parents sign a statement that they received the referrals, results, and instructions; (3) record of the results in the medical record; (4) record in the Personal Child Health Record; (5) record in the computerized database; (6) quarterly and annual reports sent to the coordination of the program and the health professionals. Record in a database with the name of the mothers, address and phone number of those whose newborn or infant failed the test and need to retest, and the ones who passed the NHS but have RIHL and need additional monitoring in specialized services.
29. The computerized database has a list to control the mothers whose babies were referred for the retest, that need auditory monitoring, or that need a diagnosis.

ADMINISTRATION
Coordination of health services with partnerships involving various local institutions and entities to develop hearing health promotion actions, including partnerships with centers specialized in diagnosis and rehabilitation.
30. Coordination between health services, including partnerships with centers specialized in diagnosis and rehabilitation.

2.5
2.5 points = if health services are coordinated, including partnerships with centers specialized in diagnosis and rehabilitation. 0 points = if health services are not coordinated.
Encouragement of partnerships with social work teams located where the NHSP is conducted, to ensure the referral of children that were designated for retest and diagnosis in specialized centers.
31. Partnerships with social work teams. 32. Monthly follow-up of the results and tracking of cases that were lost or that did not finish all the necessary stages of retest or diagnosis. Ensure that the equipment is working, calibrated, and maintained.
33. Annual calibration and preventive maintenance of the equipment.  Altogether, the validation process of the indicator matrix counted with the participation of three administrators (indicated by coordinators), four research professors, and six specialists. Among the administrators, one of the founders of the NHS in Brazil contributed to the research. She coordinated and actively participated in the implementation of the universal neonatal hearing screening (UNHS) in the country.
Of the professionals that answered the online form, 12 (92.3%) were females, and one (7.7%), male. Their mean age was 41 years (ranging from 34 to 57 years), and the time since they graduated ranged from 11 to 34 years. Of the specialists, 92.3% were audiologists that worked directly with performing the NHS in one or more settings: public maternity hospitals, private maternity hospitals, community health centers, teaching clinics, and/or private clinics.
Regarding the titles of the professionals that answered the online form, two (15.4%) had a postdoctoral degree; four (30.8%) had a doctor's degree; two (15.4%) had a master's degree; and five (38.5%) had a specialization or residency in Audiology.
The mean time working directly with NHS was nine years (the shortest time of experience was three years, and the longest, 28 years). The specialists' suggestions regarding the content, writing, and grammar of the criteria and indicators were analyzed and accepted.
Concerning the scores classified as either quite or fully adequate, the mean values of I-CVI and S-CVI were also equivalent: 0.83 (Table 2), thus having reached a consensus.
The analysis of the results revealed that two indicators had the lowest agreement rate, with I-CVI of 0.84 and 0.77, respectively. They were indicators number 2 and 12 ("room for educational activities" and "computer with Internet access to be used by the NHSP").
A total of 18 indicators achieved the maximum score in the I-CVI (1.0).
After data analysis, it was observed that the specialists disagreed regarding the initial score distribution in 14 out of the 33 indicators assessed. This datum can be verified in Table 2, in the individual item I-CVI analysis, in which the scores classified as either quite or fully adequate are marked with an "X".

DISCUSSION
In the present research, when analyzing the Structure aspect, it was observed that two indicators had the lowest agreement rate, with I-CVI of 0.84 and 0.77: "room for educational activities" and "computer with Internet access to be used by the NHSP", respectively. Although the "room for educational activities" had obtained an acceptable grade in the I-CVI (0.84) and therefore could not be dismissed, it was pointed out by two specialists as little adequate and inadequate. Some justifications were presented regarding this classification. One of the specialists reported the lack of structure in the health centers, which often do not even have an adequate room to perform the NHS, let alone having a room for educational activities. Another one classified the item as quite adequate but pointed out that dialogue circles in meeting rooms or lounges (with other health professionals), and in waiting rooms (with pregnant women in their prenatal care) are strategies that are already used in some services.
Even though "computer with Internet access to be used by the NHSP" obtained an I-CVI of 0.77, i.e., the lowest agreement of the Structure indicators, a contrast that called the attention was that "computerized database to register and follow up the NHS results" reached maximum agreement (1.0), given the need for a computer to register the results in a database.
In 2004, Durante et al. 21 already mentioned the importance of using a computerized database to register the NHS results. It was suggested then that information that ensured the quality of the UNHS program be included in the database, periodically registering and evaluating them. A justification found for the low score given to the item may have been the terminology used: "computer with Internet access" -perhaps using the Internet to develop a database was considered irrelevant.
Nevertheless, the literature recommends that the hearing screening results be registered in a digital database that makes it possible to control the information on the results and quality of the implemented NHSP 22 .
The National Hearing Health Care Policy (PNASA, its Portuguese acronym) 23 also recommends that these data be registered. Hence, with internet access, they could be included in the existing information systems of the Sistema Único de Saúde (SUS, the Brazilian public health care system) and provide to the administrators the means to plan, regulate, control, assess, and disseminate the information. There is not yet a national data system to register the NHS results, one that could centralize information on the services. Hence, surveying epidemiological data (such as the incidence of hearing loss, national coverage, and so on) is hindered 24 .
In Process, all the indicators in "hearing health education" were considered necessary and obtained an acceptable degree of agreement. Since 38.5% of the specialists were university professors/researchers, this datum may point to a paradigm shift in the speechlanguage-hearing therapist's training, diminishing the importance given to disease and specialized training, and enhancing the importance of educational and preventive actions. Thus, health promotion is established as the central guideline for every single health practice in the various social contexts 25 .
The other indicators in "neonatal hearing screening" and "administration" obtained a high agreement degree in the individual I-CVI, ranging from 0.92 to 1.0.
Regarding the scores, all the 14 indicators that obtained an I-CVI lower than 0.80 were given adjustment suggestions by at least six specialists, based on the greater or lesser relevance it would have in an actual assessment process. For instance, in Structure, specialists mentioned that items such as "desk to help when registering the NHS" and "sink/toilette where the NHS is performed" deserved a quite lower score than "equipment for physiological and electrophysiological hearing assessment", which is essential to perform the NHS.
The participants' analysis of the indicator matrix revealed that most of the indicators met the objectives of the assessment, as no major changes were suggested regarding the content of the indicators in the matrix. The total I-CVI and S-CVI achieved scores higher than 0.80, which ensured its content validity.
It is expected that the development and validation of an assessment instrument will trigger the creation of the National Neonatal Hearing Screening Program, along with new governmental policies and a nationwide database to input and analyze epidemiological data on the prevalence of hearing losses in the country, besides identifying the real contribution the NHSP has brought to society at large 26 .

CONCLUSION
The purpose of developing and validating the NHSP indicator matrix was achieved. Using it in the NHS services will consistently contribute to the periodic assessments and monitoring of NHS results in the country, as well as its efficacy and effectiveness in the services where the program is implemented.