Validation of an indicator matrix for the assessment of the Neonatal Hearing Screening Program

ABSTRACT

Purpose:

to validate an indicator matrix to assess the Neonatal Hearing Screening Program (NHSP).

Methods:

methodology development research. A total of 13 speech-language-hearing therapists with a specialization in audiology and/or at least three-year experience in neonatal hearing screening participated in the validation process. Quantitative and qualitative data were collected to develop the indicator matrix, which was then submitted to the validation process. The results of the specialists’ evaluation, in this stage, were quantitatively analyzed with the item content validation index (I-CVI) and scale content validation index (S-CVI).

Results:

regarding the indicators classified as quite or fully adequate, the mean I-CVI was the same as the mean S-CVI (0.95), evidencing excellence in their content validity. Concerning the scores classified as quite or fully adequate, the I-CVI mean was also identical to that of S-CVI (0.83), thus, reaching a consensus.

Conclusion:

this matrix with 33 indicators that had their content validated with consensus, will consistently contribute to assessing NHS services in Brazil.

Keywords:
Validation Studies; Neonatal Screening; Hearing

RESUMO

Objetivo:

validar uma matriz de indicadores para avaliação do Programa de Triagem Auditiva Neonatal (PTAN).

Métodos:

pesquisa de desenvolvimento metodológico. Participaram do processo de validação 13 fonoaudiólogos com especialização em audiologia e/ou experiência em triagem auditiva neonatal por um período igual ou superior a 3 anos. Foram coletados dados quantitativos e qualitativos para elaboração da matriz de indicadores que, em seguida, foi submetida ao processo de validação. Os resultados da avaliação dos especialistas, nesta etapa, foram analisados quantitativamente por meio do Índice de Validação de Conteúdo por Item (I-IVC) e do Índice de Validação de Conteúdo por Escala (S-IVC).

Resultados:

quanto aos indicadores classificados como bastante ou totalmente adequados, a média do I-IVC foi igual a do S-IVC (0,95), o que evidenciou a excelência na validade de seu conteúdo. Em relação às Pontuações classificadas como bastante ou totalmente adequadas, a média do I-IVC também foi idêntica a do S-IVC (0,83), tendo sido considerada a obtenção do consenso.

Conclusão:

um conjunto de 33 indicadores foram validados no aspecto do conteúdo com a obtenção do consenso, cuja utilização poderá contribuir consistentemente para a avaliação dos serviços de TAN existentes no Brasil.

Descritores:
Estudos de Validação; Triagem Neonatal; Audição

Introduction

In the field of health assessment, indicator matrices are tools that facilitate the organization, design, and analysis of programs. They are made up of criteria and indicators that help analyze and interpret information, besides making it easier to organize and plan data collection, and develop assessment instruments, analysis techniques, and result presentation¹.

The purpose of the Neonatal Hearing Screening Program (NHSP) is the timely detection of hearing loss (HL) in babies, from their first 24 hours of life up to three months old. It is part of a set of actions that must be performed aiming at the comprehensive hearing health care in childhood: screening, monitoring, and following up the hearing and language development, diagnosis, and (re)habilitation².

The NHSP must be an integral part of the Health Care Network for People with a Disability and the mother/child follow-up actions, and be coordinated with primary health care to ensure the monitoring and follow-up of hearing and language development, and the adherence to referrals to the specialized services^2,3.

By employing a validated instrument, it is possible to know better the evidence of the contribution of NHS to the timely diagnosis and intervention in hearing. Also, considering the local difficulties to receive speech-language-hearing health care, future negotiations with the government can be made to reorganize and improve the NHSP. Hence, the actions can be coordinated in a network, permeated by a flexible service organization, respect to the users, and interprofessional work⁴.

In the literature, various articles are available reporting the results obtained with the NHSP in screening services of different countries^5-9. However, as far as Brazil is concerned, the publications that were found only demonstrated such results at a local level, not denoting the nationwide situation^10-13.

Thus, the indicator matrix can become a research instrument, and particularly an assessment instrument. For this to happen, the indicators contained in the matrix must be validated by a group of specialists¹³.

It is through such a validation process that this instrument will be able to generate valid and reliable results, making it widely useful in the clinical practice and research in the field of public health. Moreover, it will be an aid in other fields, including the assessment of speech-language-hearing services¹⁴.

This paper aimed to develop and validate the NHSP Indicator Matrix in the hearing health care network.

Methods

The research was approved by the Research Ethics Committee (REC) of the Universidade Federal de Pernambuco/Department of Health Sciences - UFPE, PE, Brazil, under evaluation report number: 2.695.541.

This study was designed as a methodological development research whose purpose was to develop an instrument to assess the NHSP.

The study sample comprised 13 speech-language-hearing therapists that contributed to the consensus group to validate the indicator matrix. The specialists were selected based on the following criteria: being either speech-language-hearing therapists or otorhinolaryngologists specialized in audiology and/or experienced in coordinating/performing NHS for at least three years.

Quantitative and qualitative data were collected do develop the indicator matrix.

The study was conducted in two different stages: 1^st) development of the NHSP indicator matrix in the hearing health network; 2^nd) validation of the indicator matrix to assess the NHSP.

In the first stage, the NHSP indicator matrix was developed based on the NHSP logical model proposed and validated by Pimentel, Figueiredo, and Lima¹⁵ in a previous study. The logical model is summarized in Figure 1.

Figure 1:
Summarized logical model of the Neonatal Hearing Screening Program

Henceforth, the indicator matrix was developed, encompassing the aspects of Structure and Processes, and containing criteria, indicators, dimensions, subdimensions, and the expected score (Figure 2).

In the second stage of the research, the indicator matrix was assessed in consultation with professionals experienced in the field of NHS.

In this study, 26 specialists (23 speech-language-hearing therapists and three otorhinolaryngologists) were contacted, using the snowball sampling technique¹⁶ - in which a specialist indicates others that meet the profile necessary to the research. They worked in the Brazilian states of Alagoas, Amazonas, Paraíba, Pernambuco, São Paulo, Sergipe, and in the Federal District, and their titles included, at least, specialization/residency, with experience in implementing, coordinating, and/or performing NHS¹⁷.

According to the literature^18-20, the participants of the research, besides being experts or having knowledge of the phenomenon, must be available and motivated to participate in the different stages of the study. In their selection process, they were contacted via phone call and sent an invitation letter, justifying the research, and clarifying the importance of their participation.

When contacted via phone call, only two specialists did not confirm their participation - one otorhinolaryngologist and one speech-language-hearing therapist. After the other ones expressed their willingness to participate in the research and confirmed their electronic address for further communication throughout the data collection process, they were sent an e-mail with the invitation letter presenting the research, the informed consent form (ICF) for formal authorization, an online form with a questionnaire for a brief characterization of these specialists, and a Microsoft Excel spreadsheet with the NHSP indicator matrix for them to evaluate and fill out.

After the e-mails had been sent with the said material, 17 specialists (65.3%) answered the online form. Of these, four answered only the form; since they did not return the indicator matrix, they were excluded from the research. Those who effectively participated in the research were 13 specialists (50% of all those invited).

The filled-out forms were analyzed, and the data contained in the indicator matrix were processed according to the validation criteria described below.

The specialists judged each indicator regarding nomenclature suitability, clarity, objectivity, and applicability. These items were presented as options in the assessment matrix sent to each one of them.

Then, they were invited to evaluate whether the indicator in question was appropriate to assess the NHSP, based on the selected aspects.

To this end, a Likert-like scale^18-20 was used, categorizing the indicator evaluation as: (1) Inadequate, when the four aspects (nomenclature suitability, clarity, objectivity, and applicability of the indicator) were absent; (2) Little adequate, when only one aspect was present; (3) Quite adequate, when two or three aspects were present; and (4) Fully adequate, when the four aspects were present.

The score distribution was also evaluated by the specialists during the validation process. It was proposed that its measurement be based on the distribution between Structure and Process. The total score given to Structure was 30 points, while to Process, it was 70 points, totaling 100 points. Those of Process were subdivided into dimensions, as follows: hearing health education (10 points), neonatal hearing screening (50 points), and administration (10 points).

In principle, this individual indicator score was evenly distributed by the researcher and demonstrates the degree of importance each aspect (Structure and Process) has within the NHSP. This distribution was predetermined by the specialists that participated in the validation of the NHSP logical model, in the first stage of the research.

The score to be analyzed by the specialists was evenly distributed into the following dimensions: facilities, material resources, and human resources (2 points/indicator); hearing health education (2 points/indicator); neonatal hearing screening - which was given a higher score (5 points/indicator); and administration (2.5 points/indicator).

It should be highlighted that the researcher chose to evenly distribute these points so the judges’ analysis would not be positively or negatively influenced.

The professionals were asked to evaluate the suggested score for each indicator and classify it as: (1) Inadequate, when they strongly disagreed; (2) Little adequate, when they disagreed; (3) Quite adequate, when they agreed; and (4) Fully adequate, when they strongly agreed.

The specialists were also instructed to suggest a new score for each indicator in case they did not agree with the one proposed by the researcher.

The results of the specialists’ evaluation were qualitatively (with the analysis of the comments registered by them) and quantitatively analyzed (with the item content validation index [I-CVI] and scale content validation index [S-CVI]^18-20), as described below:

The individual item content validity is the proportion of specialists that classified the item according to its relevance or suitability¹⁹.

Hence, on a scale in which 1 represents an inadequate item and 4, a fully adequate item, the real CVI is the proportion of items the specialists classified as 3 or 4¹⁸.

The condition required to validate each item of the matrix was that its I-CVI value be equal to or above 0.80, categorizing it as adequate. In case the I-CVI were considered inadequate, i.e., obtaining a value below 0.80, the item would have to be either eliminated or (if any of the specialists had suggested alterations to adequate the item) maintained, though modified.

After calculating the I-CVI for each item, the S-CVI was calculated, following this formula:

The S-CVI must be equal to or above 0.90 to characterize excellence in content validity^19,20.

After the specialists’ evaluation, the answers were tabulated and separated by indicators and suggested scores. Hence, the I-CVI was calculated, and so was the S-CVI afterward.

Results

The indicator matrix that resulted from the validation process is shown in Figure 2.

Figure 2:
Indicator Matrix for the assessment of the Neonatal Hearing Screening Program

Altogether, the validation process of the indicator matrix counted with the participation of three administrators (indicated by coordinators), four research professors, and six specialists. Among the administrators, one of the founders of the NHS in Brazil contributed to the research. She coordinated and actively participated in the implementation of the universal neonatal hearing screening (UNHS) in the country.

Of the professionals that answered the online form, 12 (92.3%) were females, and one (7.7%), male. Their mean age was 41 years (ranging from 34 to 57 years), and the time since they graduated ranged from 11 to 34 years. Of the specialists, 92.3% were audiologists that worked directly with performing the NHS in one or more settings: public maternity hospitals, private maternity hospitals, community health centers, teaching clinics, and/or private clinics.

Regarding the titles of the professionals that answered the online form, two (15.4%) had a postdoctoral degree; four (30.8%) had a doctor’s degree; two (15.4%) had a master’s degree; and five (38.5%) had a specialization or residency in Audiology.

The mean time working directly with NHS was nine years (the shortest time of experience was three years, and the longest, 28 years).

In Table 1, the indicators classified as either quite or fully adequate are marked with an “X”. The mean I-CVI value was the same as that of the S-CVI (0.95), evidencing excellence in content validity¹⁹.

Table 1:
Indicators classified as quite or fully adequate, 2019

Indicators	Specialists													Consensus no.1	I-CVI2
	1	2	3	4	5	6	7	8	9	10	11	12	13
1	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
2	X	-	X	X	X	X	X	X	X	X	X	X	-	11	0.84
3	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
4	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
5	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
6	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
7	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
8	-	X	X	X	X	X	X	X	X	X	X	X	X	12	0.92
9	X	X	X	X	X	X	X	X	X	-	X	X	X	12	0.92
10	X	X	X	-	X	X	X	X	X	X	X	X	X	12	0.92
11	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
12	X	X	X	-	X	X	X	-	X	X	X	-	X	10	0.77
13	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
14	X	X	X	X	X	X	X	X	X	X	X	-	X	12	0.92
15	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
16	X	X	X	X	X	X	-	X	X	X	X	X	-	11	0.84
17	X	X	X	X	X	X	X	X	X	X	X	X	-	12	0.92
18	X	X	X	X	X	X	X	X	X	X	X	-	X	12	0.92
19	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
20	X	X	X	X	X	X	X	X	X	X	X	X	-	12	0.92
21	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
22	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
23	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
24	X	X	X	X	X	X	X	X	X	-	X	X	X	12	0.92
25	X	X	X	X	-	X	X	X	X	X	X	X	X	12	0.92
26	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
27	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
28	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
29	X	X	X	X	X	X	X	X	X	X	X	X	-	12	0.92
30	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
31	X	X	X	X	X	-	X	X	X	X	X	X	X	12	0.92
32	X	X	X	X	-	X	X	X	X	X	X	X	X	12	0.92
33	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
Prop3	0.97	0.97	1	0.94	0.94	0.97	0.97	0.97	1	0.94	1	0.90	0.85	S-CVI4 0.95	I-CVI 0.95

• Source: Authors of the research (2019). Captions: ¹ Number of agreement between the specialists. ² Item Content Validity Index. ³ Proportion considered adequate. ⁴ Scale Content Validation Index.

The specialists’ suggestions regarding the content, writing, and grammar of the criteria and indicators were analyzed and accepted.

Concerning the scores classified as either quite or fully adequate, the mean values of I-CVI and S-CVI were also equivalent: 0.83 (Table 2), thus having reached a consensus.

Table 2:
Scores classified as quite or fully adequate, 2019 (Conclusion)

Indicators	Specialists													Consensus no.1	I-CVI2
	1	2	3	4	5	6	7	8	9	10	11	12	13
1	X	X	-	X	X	X	X	X	X	X	X	X	-	11	0.84
2	-	X	-	X	X	X	X	X	X	X	X	X	-	10	0.77
3	-	X	-	X	X	X	X	X	X	X	X	X	-	10	0.77
4	-	X	-	X	X	X	X	X	X	X	X	X	-	10	0.77
5	-	X	-	X	X	X	X	X	X	X	X	X	X	11	0.84
6	X	X	-	X	X	X	X	X	X	X	X	X	-	11	0.84
7	X	X	-	X	X	X	X	-	X	X	X	X	X	12	0.92
8	-	X	-	X	X	X	X	X	X	X	X	X	-	10	0.77
9	X	X	-	X	X	X	X	-	X	-	X	X	-	9	0.75
10	-	X	-	-	X	X	X	X	X	X	X	-	X	9	0.66
11	X	X	-	-	X	X	X	-	X	X	X	X	-	9	0.75
12	X	X	-	-	X	X	X	-	X	X	X	-	X	9	0.66
13	X	X	X	X	X	X	X	-	X	X	X	-	X	11	0.84
14	X	X	X	X	X	X	X	X	X	X	X	-	-	11	0.84
15	X	X	X	X	X	X	X	X	X	X	X	-	X	12	0.92
16	X	X	X	X	X	X	X	-	X	X	X	-	-	10	0.77
17		X	X	X	X	X	X	X	X	X	X	-	-	10	0.77
18		X	X	X	X	X	X	X	X	X	X	-	-	10	0.77
19	-	X	X	X	X	X	X	X	X	X	X	-	-	10	0.77
20	X	X	X	X	X	X	X	-	X	X	X	-	-	10	0.77
21	-	X	X	X	X	X	X	X	X	X	X	X	X	12	0.92
22	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
23	-	X	X	X	X	X	X	X	X	X	X	X	X	12	0.92
24	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
25	X	X	X	X	-	X	X	X	X	X	X	X	X	12	0.92
26	X	X	X	X	X	X	X	X	X	X	X	-	X	12	0.92
27	X	X	X	X	X	X	X	X	X	X	X	-	X	12	0.92
28	X	X	X	X	X	X	X	X	X	X	X	X	X	13	1
29	X	X	X	X	X	X	X	X	X	X	X	X	-	12	0.92
30	-	X	X	X	X	-	X	X	X	X	X	-	X	10	0.77
31	X	X	X	X	X	-	X	X	X	X	X	-	X	11	0.84
32	X	X	X	X	-	X	X	X	X	X	X	-	X	11	0.84
33	-	X	X	X	X	X	X	-	X	X	X	X	X	11	0.84
Prop3	0.60	1	0.63	0.90	0.94	0.94	1	0.75	1	0.97	1	0.54	0.54	S-CVI4 0.83	Total I-CVI 0.83

• Source: Authors of the research (2019). Captions: ¹ Number of agreement between the specialists. ² Item Content Validity Index. ³ Proportion considered adequate. ⁴ Scale Content Validation Index.

The analysis of the results revealed that two indicators had the lowest agreement rate, with I-CVI of 0.84 and 0.77, respectively. They were indicators number 2 and 12 (“room for educational activities” and “computer with Internet access to be used by the NHSP”).

A total of 18 indicators achieved the maximum score in the I-CVI (1.0).

After data analysis, it was observed that the specialists disagreed regarding the initial score distribution in 14 out of the 33 indicators assessed. This datum can be verified in Table 2, in the individual item I-CVI analysis, in which the scores classified as either quite or fully adequate are marked with an “X”.

Discussion

In the present research, when analyzing the Structure aspect, it was observed that two indicators had the lowest agreement rate, with I-CVI of 0.84 and 0.77: “room for educational activities” and “computer with Internet access to be used by the NHSP”, respectively. Although the “room for educational activities” had obtained an acceptable grade in the I-CVI (0.84) and therefore could not be dismissed, it was pointed out by two specialists as little adequate and inadequate. Some justifications were presented regarding this classification. One of the specialists reported the lack of structure in the health centers, which often do not even have an adequate room to perform the NHS, let alone having a room for educational activities. Another one classified the item as quite adequate but pointed out that dialogue circles in meeting rooms or lounges (with other health professionals), and in waiting rooms (with pregnant women in their prenatal care) are strategies that are already used in some services.

Even though “computer with Internet access to be used by the NHSP” obtained an I-CVI of 0.77, i.e., the lowest agreement of the Structure indicators, a contrast that called the attention was that “computerized database to register and follow up the NHS results” reached maximum agreement (1.0), given the need for a computer to register the results in a database.

In 2004, Durante et al.²¹ already mentioned the importance of using a computerized database to register the NHS results. It was suggested then that information that ensured the quality of the UNHS program be included in the database, periodically registering and evaluating them. A justification found for the low score given to the item may have been the terminology used: “computer with Internet access” - perhaps using the Internet to develop a database was considered irrelevant.

Nevertheless, the literature recommends that the hearing screening results be registered in a digital database that makes it possible to control the information on the results and quality of the implemented NHSP²².

The National Hearing Health Care Policy (PNASA, its Portuguese acronym)²³ also recommends that these data be registered. Hence, with internet access, they could be included in the existing information systems of the Sistema Único de Saúde (SUS, the Brazilian public health care system) and provide to the administrators the means to plan, regulate, control, assess, and disseminate the information. There is not yet a national data system to register the NHS results, one that could centralize information on the services. Hence, surveying epidemiological data (such as the incidence of hearing loss, national coverage, and so on) is hindered²⁴.

In Process, all the indicators in “hearing health education” were considered necessary and obtained an acceptable degree of agreement. Since 38.5% of the specialists were university professors/researchers, this datum may point to a paradigm shift in the speech-language-hearing therapist’s training, diminishing the importance given to disease and specialized training, and enhancing the importance of educational and preventive actions. Thus, health promotion is established as the central guideline for every single health practice in the various social contexts²⁵.

The other indicators in “neonatal hearing screening” and “administration” obtained a high agreement degree in the individual I-CVI, ranging from 0.92 to 1.0.

Regarding the scores, all the 14 indicators that obtained an I-CVI lower than 0.80 were given adjustment suggestions by at least six specialists, based on the greater or lesser relevance it would have in an actual assessment process. For instance, in Structure, specialists mentioned that items such as “desk to help when registering the NHS” and “sink/toilette where the NHS is performed” deserved a quite lower score than “equipment for physiological and electrophysiological hearing assessment”, which is essential to perform the NHS.

The participants’ analysis of the indicator matrix revealed that most of the indicators met the objectives of the assessment, as no major changes were suggested regarding the content of the indicators in the matrix. The total I-CVI and S-CVI achieved scores higher than 0.80, which ensured its content validity.

It is expected that the development and validation of an assessment instrument will trigger the creation of the National Neonatal Hearing Screening Program, along with new governmental policies and a nationwide database to input and analyze epidemiological data on the prevalence of hearing losses in the country, besides identifying the real contribution the NHSP has brought to society at large²⁶.

Conclusion

The purpose of developing and validating the NHSP indicator matrix was achieved. Using it in the NHS services will consistently contribute to the periodic assessments and monitoring of NHS results in the country, as well as its efficacy and effectiveness in the services where the program is implemented.

REFERENCES

• ¹
  Alves CKA, Natal S, Felisberto E, Samico I. Interpretação e análise de informações: o uso de matrizes, critérios, indicadores e padrões. In: Samico I, Felisberto E, Figueiró AC, Frias PG (orgs). Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: Medbook, 2010. p. 89-107.
• ²
  Brasil, Ministério da Saúde. Diretrizes de Atenção da Triagem Auditiva Neonatal Secretaria de Atenção à Saúde. Departamento de Ações Programáticas Estratégicas e Departamento de Atenção Especializada. Brasília: Ministério da Saúde, 2012a. Disponível em: http://bvsms.saude.gov.br/bvs/publicacoes/diretrizes_atencao_triagem_auditiva_neonatal.pdf. Acesso em: 19 set. de 2016.
• ³
  Sabbag JC, Lacerda ABM. Neonatal Hearing Screening in primary health care and family health care. CoDAS. 2017;29(4):e20160102.
• ⁴
  Bezerra TCA, Falcão MLP, Goes PSA, Felisberto E. Avaliação de programas de formação profissional em saúde: construção e validação de indicadores. Trab. Educ. Saúde. 2016;14(2):445-72.
• ⁵
  Sedano MC, Martín UAS, Rahal EM. Realidad nacional de los programas de detección auditiva temprana con miras a la cobertura universal. Rev. Otorrinolaringol. Cir. Cabeza Cuello. 2018;78(1):9-14.
• ⁶
  Bouillot L, Vercherat M, Durand C. Implementing universal newborn hearing screening in the French Rhône-Alpes region. State of affairs in 2016 and the 1st half of 2017. Int J Pediatr Otorhinolaryngol. 2019;117:30-6.
• ⁷
  Wenjin W, Xiangrong T, Yun L, Jingrong L, Jianyong C, Xueling W et al. Neonatal hearing screening in remote areas of China: a comparison between rural and urban populations. J Int Med Res. 2018;46(2):637-51.
• ⁸
  Fort M. Newborn Hearing Screening: Making a Difference. NCMJ. 2017; 78(2): 96-100.
• ⁹
  Greczka G, Zych M, Szyfter W, Wróbel M. Analysis of the changes in the Polish Universal Neonatal Hearing Screening Program over 15 years of activity. Otolaryngol Pol. 2018;72(2):13-20.
• ¹⁰
  Dantas MBS, Anjos CAL, Camboim ED, Pimentel MCR. Resultados de um programa de triagem auditiva neonatal em Maceió. Rev Bras Otorrinolaringol. 2009;75(1):58-63.
• ¹¹
  Rodrigues PAL, Carvalho TSF, Lauris JRP, Schochat E. Resultados de um Programa de Triagem Auditiva Neonatal em Cuiabá - Mato Grosso. Rev Soc Bras Fonoaudiol. 2011;16(4):454-8.
• ¹²
  Canabarro MS, Machado NM, Fossa V, Weiss MS, Mitre EI. Programa de triagem auditiva neonatal: resultados de um Hospital Universitário de Porto Alegre. Rev HCPA. 2012;32(1):30-4.
• ¹³
  Sgorla JB, Ferreira MIDC. Characterization of newborn hearing screening programs of maternity units located in the city of João Pessoa, PB, Brazil. Distúrb. Comum. 2014;26(3):559-68.
• ¹⁴
  Almeida MHM, Spínola AWP, Lancman S. Técnica Delphi: validação de um instrumento para uso do terapeuta ocupacional em gerontologia. Rev. Ter. Ocup. Univ. 2009;20(1):49-58.
• ¹⁵
  Pimentel MCR, Figueiredo N, Lima MLLT. Development and validation of the Logical Model of the Neonatal Hearing Screening Program. Rev. CEFAC. 2020;22(4):e14019.
• ¹⁶
  Vinuto J. A amostragem em bola de neve na pesquisa qualitativa: um debate em aberto. Temáticas. 2014;22(44):203-20.
• ¹⁷
  Azevedo SB, Lima MLLT, Griz SMS, Leal LP. Instrument for evaluating child hearing health services: construction and validity. Rev Esc Enferm USP. 2018;52:e03357.
• ¹⁸
  Lynn MR. Determination and quantification of content validity. Nursing research. 1986;35(6):382-6.
• ¹⁹
  Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007;30(4):459-67.
• ²⁰
  Polit DF, Beck CT. The content validity index: are you sure you know what's being reported? Critique and recommendations. Res Nurs Health. 2006;29(5):489-97.
• ²¹
  Durante AS, Carvallo RMM, Costa MTZ, Cianciarullo MA, Voegels RL, Takahashi GM et al. Newborn Hearing Screening Program: implementing model international. Int. Arco. Otorhinolaryngol. [Periódico na Internet]. 2004;8(1). [Acesso em: 28 maio 2018]. Disponível em: http://arquivosdeorl.org.br/additional/acervo_port.asp?Id=263
  » http://arquivosdeorl.org.br/additional/acervo_port.asp?Id=263
• ²²
  Ribeiro FM, Chapchap MJ, Lewis DR. Indicadores de risco para a deficiência auditiva no contexto atual da TANU. In: Boéchat EM, Menezes PL, Couto CM, Frizzo ACF, Scharlach RC, Anastasio ART (orgs). Tratado de Audiologia. Rio de Janeiro: Guanabara koogan, 2015. 2 ed. p. 381-5.
• ²³
  BRASIL. Ministério da Saúde. Portaria GM/MS nº 2.073/ 2004. Institui a Política Nacional de Atenção à Saúde Auditiva. Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2004/prt2073_28_09_2004.html Acesso em: 07 out. 2016.
  » http://bvsms.saude.gov.br/bvs/saudelegis/gm/2004/prt2073_28_09_2004.html
• ²⁴
  Colella-Santos MF, Souza GL, Hein TAD. Triagem Auditiva Neonatal em UTI. In: Boéchat EM, Menezes PL, Couto CM, Frizzo ACF, Scharlach RC, Anastasio ART (orgs). Tratado de Audiologia. Rio de Janeiro: Guanabara Koogan, 2015. 2 ed. P.386-94.
• ²⁵
  Zanin LE, Albuquerque IMN, Carneiro MSM, Melo DH. Evaluation of speech-language pathology care in the family health strategy from user perspective. CoDAS. 2017;29(6):e20160192.
• ²⁶
  Leite SS, Afio ACE, Carvalho LV, Silva JM, Almeida PC, Pagliuca LMF. Construction and validation of an Educational Content Validation Instrument in Health. Rev Bras Enferm [periódico na internet]. 2018 [acesso em 21 julho 2019];71(4):1635-41. Disponível em: https://www.scielo.br/pdf/reben/v71s4/0034-7167-reben-71-s4-1635.pdf
  » https://www.scielo.br/pdf/reben/v71s4/0034-7167-reben-71-s4-1635.pdf

• This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

Publication Dates

• Publication in this collection
  11 Nov 2020
• Date of issue
  2020

History

• Received
  21 Aug 2020
• Accepted
  23 Sept 2020