Content validation of an instrument to measure listening effort

ABSTRACT

Purpose:

To validate the content of an instrument to measure listening effort for hearing-impaired individuals.

Method:

This is a validation study, developed in two stages, which the Stage 1 is the planning and development of the first version of the instrument, and Stage 2 the investigation of the evidences of validity based on the content and development of the final version of the instrument to measure listening effort. Ten professionals with expertise in the field of audiology, with more than five years of clinical experience participated in this study. The instrument to be validated was composed of three parts: I - “speech perception of logatomes and listening effort”; II - “listening effort and working memory” and; III - “speech perception of meaningless sentences and working memory” and they were presented monoaurally, in quiet and in the signal-to-noise ratios + 5dB, 0dB and -5dB. It was conducted a descriptive analysis regarding the suggestions of the committee judge audiologists and the analysis of the individual and scale content validity index.

Results:

The results showed that parts I and III which constitute the proposed instrument reached a scale content validity index above 0.78, which means that the presented items did not need modification in their construct.

Conclusion:

The evidences of validity studied allowed relevant modifications and made this instrument adequate to its construct.

Keywords
Hearing; Listening Effort; Auditory Perception; Auditory Tests; Memory; Cognition

RESUMO

Objetivo:

Validar o conteúdo de um instrumento para mensuração do esforço auditivo para indivíduos com perda auditiva.

Método:

Trata-se de um estudo de validação, desenvolvido em duas fases, sendo a fase 1 o planejamento e desenvolvimento da primeira versão do instrumento e a fase 2 a investigação das evidências de validade baseadas no conteúdo do instrumento e desenvolvimento da versão final para mensuração de esforço auditivo. Participaram dez profissionais com expertise na área audiológica, com mais de cinco anos de experiência. O instrumento a ser validado foi composto por três partes: I - “percepção de fala de logatomas e esforço auditivo”; II - “esforço auditivo e memória operacional”; e III - “percepção de sentenças sem sentido e memória operacional”, apresentadas de forma monoaural no silêncio e nas relações sinal-ruído +5dB, 0dB e -5dB. Foi realizada a análise descritiva das sugestões do comitê de fonoaudiólogos e do índice de validade de conteúdo individual e total.

Resultados:

Os resultados mostraram que as partes I e III do instrumento proposto atingiram o índice de validade de conteúdo total acima de 0,78, ou seja, os itens apresentados não necessitaram de modificações em seu constructo.

Conclusão:

As evidências de validade estudadas permitiram relevantes modificações e tornaram esse instrumento adequado ao seu constructo.

Descritores
Audição; Esforço Auditivo; Percepção Auditiva; Testes Auditivos; Memória; Cognição

INTRODUCTION

Currently, audiology and cognitive psychology fields have endeavored to define the term “listening effort” and find the most accurate, reliable method, whose applicability is feasible for clinical audiology. Audiologists and cognitive psychologists have studied cognitive theories related to mental resources involved in measuring listening effort, especially in behavioral evaluation⁽¹⁾.

One of the definitions adopted for listening effort refers to the amount of cognitive resources needed for recognizing acoustic signals, especially speech^(2,3). Other authors defined the listening effort as a deliberate allocation of mental resources needed to overcome obstacles in the pursuit of objectives when performing a listening task⁽⁴⁾. However, the agreement regarding the best definition of this auditory parameter has not yet been scientifically agreed. Researchers⁽⁵⁾ state that listening effort can be measured through three main methodological approaches, which are categorized into subjective, behavioral⁽⁶⁾ and psychophysiological⁽⁷⁾ measurements.

Behavioral measurements are described in the literature as a dual task paradigm. This paradigm refers to the accomplishment of two tasks, a primary and a secondary one, performed simultaneously. The literature⁽⁸⁾ states that to measure listening effort, the primary task will involve speech perception tasks/tests, such as word and/or sentence recognition, in quiet and with manipulation of the signal-to noise ratio. However, the secondary task can refer to activities/tests of working memory or visual pattern recognition.

Some authors^(8,9) reported that the alteration in performing the secondary task at different levels of difficulties of the primary task reflects a change in the cognitive resources for speech processing, that is, listening effort. This interpretation assumes that performance in both the primary and secondary tasks requires the allocation of some common cognitive resources to each task. As cognitive resources are limited, greater listening effort and, consequently, greater demand for cognitive resources for the speech perception task will be dispensed.

Over the years and the beginning of the aging process, researchers have demonstrated the existence of a strong relationship between age and decline in working memory, and greater listening effort is needed to understand speech^(10,11). According to some researchers⁽¹²⁾, modern models that conceptualize working memory define it as a flexible, even though limited, resource that is implemented regarding the quality of coded representations, that is, the better the quality of auditory representations, the greater the possibility of coding the acoustic signal.

The hypothesis of studies that measure listening effort using a behavioral measurement with dual task paradigms whose secondary task involves the working memory capacity in different signal-to-noise ratios, is that listeners with less working memory capacity are more susceptible to effects of background noise and, consequently, put greater listening effort when performing the speech perception task.

Currently, the availability of instruments and/or questionnaires that can be applied both in the scientific area and in clinical practice has been increasing in the health area, but most of these procedures have not been properly validated, especially regarding their measurement properties⁽¹³⁾. An instrument considered valid is one that evaluates exactly what it proposes to measure⁽¹⁴⁾, and is related to a specific research question and to a particular population⁽¹⁵⁾.

The literature⁽¹⁶⁾ describes three main types of validation, which are content validation, criterion validation and construct validation. Content validity aims to assess the relevance and representativeness of each element present in an instrument of a specific construct⁽¹⁷⁾.

Considering that at the national level there is no a validated behavioral measurement to estimate listening effort, and that this parameter has been investigated indirectly through subjective (questionnaires or scales) and/or objective approaches (electrophysiological responses, heart rate, pupillometry, among others), it is hypothesized for this study that the content validation of an instrument for listening effort measurement will have great relevance and usefulness for the beginning of a complete validation process, which also involves the criterion and construct validation, and seeks a sensitive instrument to predict behavioral performance in measuring this parameter. Thus, the aim of this study was to validate the content of an instrument to measure listening effort for individuals with hearing loss.

The relevance of the investigation of listening effort is related to the complaints reported by patients with hearing loss, hearing aids users or not users, associated with reports of fatigue in speech comprehension situations, mainly in noise, or when the message is not familiar. Thus, the relevance of validating the content of an instrument for listening effort measurement is justified due to its importance for the adaptation of a measurement instrument. It is worth emphasizing that only the hearing threshold measurement is not a good predictor in cases in which patients have difficulty understanding speech, since it portrays the auditory sensitivity and not listening effort.

Moreover, this content validation may be beneficial for professionals and researchers working with auditory training programs, since it may be used as an instrument for pre and post-training assessment of individuals who have undergone sessions for the improvement of auditory abilities.

METHOD

This study was submitted to the Research Ethics Committee of the Faculty of Philosophy and Sciences of the Universidade Estadual Paulista “Júlio de Mesquita Filho” (UNESP) - Marilia and under protocol number 2.179.639. The place of study development was the Centro de Estudos de Educação e da Saúde - CER II of the Faculty of Philosophy and Sciences of the Universidade Estadual Paulista, Campus of Marília, São Paulo - Brazil. Participants signed the Informed Consent Form (ICF) with the explanation of procedures that would be performed before beginning data collection.

This is a validation study, described in two stages, with stage 1 being planning and development of the first version of the instrument for measuring listening effort of the individuals with hearing loss; stage 2, investigation of the evidence of validity based on the content, that is, permanence or exclusion of items from the instrument after analysis and agreement of the judge audiologists and development of the final version of the proposed instrument.

STAGE 1: Planning and development of the first version

Stage 1 consisted of planning and developing the first version of the instrument. Previously the elaboration and development of this version, questions related to planning, such as the population of interest and global domain, the theoretical definition of clinical evaluation of listening effort, were considered.

For the literature review, databases as US National Library of Medicine National Institutes of Health (PUBMED), Latin American and Caribbean Health Sciences Literature (Literatura Latino-Americana e do Caribe em Ciências da Saúde (LILACS)), Scientific Eletronic Library Online (SCIELO) and Cochrane Library were reviewed, with selection of English and Portuguese languages, without limit on the publication. The keywords and descriptors used in English were hearing loss, listening effort, perceptual effort, ease of listening, speech perception, working memory, assessment and, in Portuguese, perda auditiva, esforço auditivo, esforço de escuta, esforço perceptivo, facilidade em ouvir, percepção de fala, memória operacional and avaliação in different combinations, aiming to find the largest number of studies. Subsequently, original studies were selected that used behavioral measurements to measure listening effort and working memory in the population with hearing loss, as well as existing instruments, whether validated or not, and the selection of the items and score system of instruments was also carried out.

- Part I: “speech perception of logatomes and listening effort” which aimed to evaluate the perception of consonants of the Brazilian Portuguese when they were isolated between vowels with the same amount of acoustic energy, forming logatomes.

This part consisted of two word lists, in which most of them have no meaning (logatomes). These lists were composed of words whose consonants are isolated by the vowel “A”, such as “ANHA”, “ALA”, “ARA”, among others. The objective of this part, moreover the speech perception task, was to verify if the participant produced the acoustic signal received in a reliable way or as a real word, performing auditory closure and making use of the contextual clue, such as the production of the logatome “ ALA ”, like the real word “FALA”, due to the manipulation of different signal-to noise ratios.

- Part II: “listening effort and working memory: set of real words” aimed at evaluating listening effort performed by the participant when performing a memorizing task and recalling real words, that is, words that have meaning, derived from the logatomes from the first part of the instrument, at different levels of signal-to noise ratio. This part consisted of four sets

of real words, in which each set has three series of words (Set I: composed of three series of two words each; Set II: composed of three series of three words each; Set III: composed of three series of four words each; Set IV: composed of three series of five words each). Participants were instructed to listen to each series of words and, subsequently, to remember and repeat the first word heard in each series.

Considering the complexity of the task, subsequent sets were only presented by memorizing and producing the first word of each series. As far as the participants answer correctly the memorized words, the other sets were presented.

- Part III: “speech perception of meaningless sentences and working memory” aimed to measure listening effort needed in a speech perception task of long duration stimuli. At this stage, the participant should repeat each sentence heard and, then remember the last word of each sentence. This part consisted of five meaningless sentences to avoid guessing by the participants. The last word of each sentence was derived from the logatomes in Part I of the instrument. In this part of the test, the more sentences the participants produced correctly, the better their speech perception skills were, and the more words they remembered, the better their working memory capacity and the less the amount of listening effort required.

In order to ensure uniformity and avoid bias regarding the differences related to the speech emission characteristics of each applicator, the logatomes, real words and meaningless sentences were recorded. The recording of the speech stimuli of the instrument was made by a female speaker of Brazilian Portuguese, in an acoustically treated room. This recording was conducted with the speaker seated on a chair, using a Sennheiser microphone (model E855) and a digital recorder from MARANTZ (model PMD660, configured for single-channel recording, with a sampling rate of 44 kHz and 16 bits of resolution). The microphone was positioned at 45 degrees and a distance of 10 cm in front of the speaker’s mouth.

For testing, the speech stimuli of the instrument were played monaurally, in the following listening situations: quiet and with backnoise type White Noise in the signal-to-noise ratios + 5dB, 0dB and -5dB in relation to the participant’s Maximum Comfort Level. Moreover, an instructional guide for the application of each part of the instrument was elaborated to facilitate the application process of the instrument for the judge audiologists.

STAGE 2. Investigation of the evidence of validity based on the content of the instrument for measuring listening effort

Stage 2 consisted of investigating the content validity of the instrument. The literature⁽¹⁸⁾ recommends a minimum number of five and a maximum of ten people participating in this process.

According to the criteria suggested by the international literature¹⁹, for the content validation a committee of Brazilian specialists was formed, composed of ten specialized audiologists, who developed research in the area, and with proven clinical experience, that is, who worked in the area for five years or

more. They filled out and sent their informed consent terms, questionnaires of academic and professional characterization and analysis of the instrument based on the guidelines received.

Audiologists were invited to participate in the committee of judges by email. After acceptance, all material was sent by mail. All judge audiologists received an envelope that contained the following documents/materials: an explanatory letter of presentation of the study and its objective, stating the reason for choosing the judge, mentioned in the previous paragraph and the relevance of the concepts involved and the instrument as a whole^(18,19); informed consent form for the judge audiologist; informed consent form for the participant with hearing loss; a questionnaire for academic and professional characterization of the audiologist; the instrument for measuring listening effort with its answer sheet; an explanatory video of the instrument with a duration of 10 minutes, containing guidelines about the population profile to be assessed and how the instrument was developed, the objective and application method of the instrument parts, the reason for choosing each speech stimulus (logatomes, real words and meaningless sentences) and also about the different signal-to noise relations and how to manipulate them and, finally, how to fill out the answer sheet of the instrument; a CD or pen-drive with the audio recording of the instrument to measure listening effort; and an instruction guideline also containing explanations about each part of the instrument and how to properly complete the answer sheet.

The judges filled out and sent the consent terms, questionnaires of academic and professional characterization and analyses of the instrument based on the guidelines provided. The characterization of the audiologist professionals who are judges according to the variables gender, age, length of professional experience in the area and degree is shown in table 1.

Table 1
Characterization of the judges participating in the study

Nº	Gender	Age (years)	Professional performance time (years)	Degree
1	F	24	5	E
2	F	25	5	E
3	F	39	17	M
4	F	31	8	E
5	F	50	20	E
6	F	34	13	D
7	F	33	7	E
8	F	36	5	E
9	F	25	5	E
10	F	50	20	E
Mean	-	41,66	12,66	-
Median	-	33,50	7,50	-
SD	-	8,98	6,07	-

• Caption: SD = Standard Deviation, S = Specialization, M = Master, P = PhD

The mean age of the judges was 41.6 years and all participants had a degree. Of these, 80% had specialization, 10% were “masters” and 10% were “doctors”.

Initially, the judges were instructed to evaluate the whole instrument, determining its scope. The judges were also instructed to analyze each part of the instrument individually as adequate or inadequate, verifying its clarity and pertinence. Regarding clarity, it was requested to verify the structure and the writing of items, if they had been written in an understandable way and if they adequately expressed what should be measured. Regarding pertinence, it was requested to verify the items in relation to the concepts involved and whether they were relevant to the proposed objective.

If the judges deemed it inappropriate, they were instructed to justify and suggest changes to the item in the space indicated. At the end of the protocol, there was a space for the judges write comments and add the items they deemed relevant and which had not been included in the instrument.

Analysis of data

A qualitative analysis was conducted, based on the suggestions and comments made available by the judges regarding the content of the instrument. Moreover, in order to determine acceptance and agreement of the questions by the judges, the Individual Content Validity Index (I-CVI) and the Scale Content Validity Index (S-CVI) were calculated.

This measurement calculates the proportion or percentage of judges who are in agreement on certain aspects of the instrument and its items, allowing individual and global analysis of the instrument⁽²⁰⁾, and uses a Likert scale with a score from 1 to 4. To assess the relevance/representativeness, responses may include: 1 = not relevant or not representative, 2 = item needs major revision to be representative, 3 = item needs minor revision to be representative, 4 = relevant or representative item¹⁸.

The score of this index was calculated through the sum of agreement of the items marked as “3” or “4” by the judge audiologists¹⁹, with the items scored as “1” or “2” being revised or eliminated. As described in the international literature^(18,20), when the committee of judges has six or more specialists, the recommended values for I-CVI and S-CVI should not be less than 0.78. After this analysis, the indexes with values below 0.78, like all suggestions recommended by the judges were analyzed by the authors of the instrument and, after consensus, changes were made to the final version of the instrument.

RESULTS

Table 2 shows results of the analysis from audiologist judges for the three parts of the instrument (parts I, II and III).

Table 2
Content validation indexes of the instrument to measure listening effort for individuals with hearing loss (parts I, II and III)

Part I	Number of judges who considered the item appropriate	I-CVI	Part II	Number of judges who considered the item appropriate	I-CVI	Part III	Number of judges who considered the item appropriate	I-CVI
List 1	10	1.00	Set 1	7	0.70*	Score - 20%	9	0.90
List 2	10	1.00	Set 2	6	0.60*	Score - 40%	9	0.90
Score - 10-15 points	10	1.00	Set 3	6	0.60*	Score - 60%	9	0.90
Score - 15-25 points	10	1.00	Set 4	6	0.60*	Score - 80%	9	0.90
Score - 25-35 points	10	1.00	Score - 25%	5	0.50*	Score - 100%	9	0.90
Score - 35-40 points	10	1.00	Score - 50%	5	0.50*	Quiet	10	1.00
Quiet	10	1.00	Score - 75%	5	0.50*	SNR +5dB	9	0.90
SNR +5dB	9	0.90	Score 100%	5	0.50*	SNR 0dB	10	1.00
SNR 0dB	10	1.00	Quiet	5	0.50*	SNR -5dB	10	1.00
SNR -5dB	6	0.60*	SNR +5dB	5	0.50*
			SNR 0dB	5	0.50*
			SNR -5dB	4	0.40*
	S-CVI	0.95		S-CVI	0.53*		S-CVI	0.93

• Caption: SNR = signal-to-noise ratio, dB = decibels, I-CVI = Content Validity Index for Items, S-CVI = Content Validity index for the Scale
• ^*
  Items with I-CVI e/ou S-CVI less than 0.78

Regarding the results showed in table 2, it was found that most of the items analyzed by the judge audiologists reached the I-CVI higher than 0.78, and that the S-CVI of this stage was 0.95. Only the item referring to the SNR (-5dB) obtained an I-CVI score of 0.60, that is, a score less than 0.78 and, it was opted to remove this listening condition from Part I of the instrument.

With respect of Part II of the instrument to measure listening effort for individuals with hearing loss, it was observed that all I-CVI values did not reach the minimum score of 0.78, requiring reconstruction in the presentation form of this part of the instrument measure listening effort. Moreover, it was also observed that the S-CVI of this stage was 0.53.

Regarding to Part III of the instrument to measure listening effort for individuals with hearing loss, it was found that the I-CVI values reached the minimum score of 0.78 and that the S-CVI of this stage was 0.93.

In chart 1, suggestions offered by the judges are shown for reformulation and/ou improvement of the validated items.

Chart 1
Suggestions for structural alterations proposed by the judges about the items analyzed during content validation process

Instrument	Item before analysis	Item after analysis
Part I	(10-15 points) - Modify "minimum listening effort"	(10-15 points) - Modify to "no listening effort"
	Keep SNR -5dB	Exclude SNR -5dB
Part II	25% - Minimum listening effort and higher degree of working memory ability - (memorization of the first words of each series of Set I); 50% - Small listening effort and preserved degree of working memory ability - (memorization of the first words of each series of Set II); 75% - Medium listening effort and moderate degree of working memory ability - (memorization of the first words of each series of Set III); 100% - Maximum listening effort and severe degree of working memory ability - (memorization of the first words of each series of Set IV).	0% - Não foi possível mensurar o esforço auditivo; 25% - Esforço auditivo máximo e habilidade de memória operacional de grau grave (memorização das primeiras palavras de cada série do Conjunto I); 50% - Esforço auditivo médio e habilidade de memória operacional de grau moderado (memorização das primeiras palavras de cada série dos Conjuntos I e II); 75% - Esforço auditivo pequeno e habilidade de memória operacional de grau preservado (memorização das primeiras palavras de cada série dos Conjuntos I, II e III); e 100 % - Esforço auditivo mínimo e habilidade de memória operacional de grau superior (memorização das primeiras palavras de cada série do Conjuntos I, II, III e IV).
	______________________	Use of visual support to assist in evoking the response, for example a poster or cards containing the words from the sets.
	Keep SNR -5dB.	Exclude SNRs +5dB and -5dB.
Part III	20% - Minimum listening effort and higher degree of working memory ability; 40% - Small listening effort and preserved degree of working memory ability; 60% - Medium listening effort and moderate degree of working memory ability; 80% - Large listening effort and severe degree of working memory ability; 100% - Maximum listening effort and very severe degree of working memory ability.	0% - It was not possible to measure listening effort (absence of correct answers); 20% - Maximum listening effort and very severe degree of working memory ability; 40% - Large listening effort and severe degree of working memory ability; 60% - Medium listening effort and moderate degree of working memory ability; 80% - Small listening effort and preserved degree of working memory ability; 100% - Minimum listening effort and higher degree of working memory ability.
	______________________	Use of visual support to assist in evoking the response, for example a poster or cards containing the words from the sets.
	Keep SNR -5dB.	Exclude SNR +5dB and -5dB
	______________________	Add the order of words that patients should memorize and leave a space for evaluators to write down the sequence in which the patients memorized the words.

• Caption: dB = Decibel, SNR = Signal-to-Noise Ratio

Modifications showed in chart 1 were made regarding the judge audiologists’ suggestions with respect to the improvement of content of the instrument, and not due to the agreement between them regarding validity of the items.

Chart 2 shows the final version of the instrument and its answer sheet, where all suggestions proposed by the judge audiologists were accepted after analysis of the instrument by the audiologists committee.

Chart 2
Final version of the instrument and its answer sheet elaborated after content validation

PART I - SPEECH PERCEPTION OF LOGATOMES AND LISTENING EFFORT
		List 1		Intensity of consonant (dB)			List 2		Intensity of consonant (dB)			Frequency of consonant (Hz)
1		AMA		35			ANA		35			250
2		ALA		40			ANHA		40			250
3		ABA		25			APA		25			500
4		ALHA		35			ARA		35			750
5		ARRA		25			AKA		30			1500
6		AKA		30			AGA		25			1500
7		AJA		25			ACHA		25			2500
8		ADA		25			ATA		25			4000
9		AZA		20			ASSA		20			4000
10		AVA		15			AFA		15			6000
PART II - LISTENING EFFORT AND WORKING MEMORY: SET OF REAL WORDS
SET I
CAMA BALA					JANA MANHA					CHAMA CANA
SET II
FALA CALHA LAMA					BANHA CARA TAPA					PALHA CAPA CHAPA
SET III
FARRA JACA TALHA SALA					FACA JOGA FAIXA FRONHA					JARRA TAXA FALHA DAMA
SET IV
CADA PARA SOFA TAÇA CASA					FAÇA BATA DADA VAZA BRAVA					NADA PLAZA PATA LAÇA TRUFA
PART III - SPEECH PERCEPTION OF MEANINGLESS SENTENCES AND WORKING MEMORY
1. A Flor azul da mulher estava dentro da dama
2. O cachorro do quintal costuma brincar na bala.
3. O menino bebeu tudo daquela farra.
4. As crianças comeram tanto até ficarem cheias de taça.
5. A cor da minha blusa é rosa igual minha manhã.
PART I - SPEECH PERCEPTION OF LOGATOMES AND LISTENING EFFORT
LIST 1 (RE)		Omission		Correct answer		Negative substitution		Positive substitution		Score Signal-to-noise ratio
										Quiet		+5dB		0dB
AMA		1		2		3		4
ALA		1		2		3		4
ABA		1		2		3		4
ALHA		1		2		3		4
ARRA		1		2		3		4
AKA		1		2		3		4
AJA		1		2		3		4
ADA		1		2		3		4
AZA		1		2		3		4
AVA		1		2		3		4
Total List 1
LIST 2 (LE)		Omission		Correct answer		Negative substitution		Positive substitution		Signal-to-noise ratio
										Quiet		+5dB		0dB
ANA		1		2		3		4
ANHA		1		2		3		4
APA		1		2		3		4
ARA		1		2		3		4
AKA		1		2		3		4
AGA		1		2		3		4
ACHA		1		2		3		4
ATA		1		2		3		4
ASSA		1		2		3		4
AFA		1		2		3		4
Total List 2
SCORE PART I: SPEECH PERCEPTION OF LOGATOMES AND LISTENING EFFORT
10-15 points			15-25 points				25-35 points				35-40 points
No listening effort			Minimum listening effort				Medium listening effort				Maximal listening effort
PARTE II: ESFORÇO AUDITIVO E MEMÓRIA OPERACIONAL: CONJUNTO DE PALAVRAS REAIS
		Set I			Set II			Set III			Set IV
Ear	Signal-to-noise ratio	1	2	3	1	2	3	1	2	3	1	2	3	Correct answers (%)
RE	Quiet
RE	+5
RE	0
LE	Quiet
LE	+5
LE	0
Score: 0% - It was not possible to measure listening effort (absence of correct answers); 25% - Maximum listening effort and severe degree of working memory ability - (memorization of the first words of each series of Set I, equivalent to one set); 50% - Medium listening effort and moderate degree of working memory ability - (memorization of the first words of each series of Sets I and II, equivalent to two sets); 75% - Small listening effort and preserved degree of working memory ability - (memorization of the first words of each series of Sets I, II and III, equivalent to three sets); 100% - Minimum listening effort and higher degree of working memory ability - (memorization of the first words of each series of Sets I, II, III and IV, equivalent to four sets);
PART III: PERCEPTION OF MEANINGLESS SENTENCES AND WORKING MEMORY
		Sentences					Words
RE/LE	SNR	1	2	3	4	5	1	2	3	4	5	Correct answers (%)
RE	Quiet						DAMA (   )	BALA (   )	FARRA (   )	TAÇA (   )	MANHA (   )
RE	+5						BALA (   )	TAÇA (   )	MANHA (   )	FARRA (   )	DAMA (   )
RE	0						TAÇA (   )	DAMA (   )	BALA (   )	MANHA (   )	FARRA (   )
LE	Quiet						FARRA (   )	MANHA (   )	BALA (   )	DAMA (   )	BALA (   )
LE	+5						DAMA (   )	BALA (   )	FARRA (   )	TAÇA (   )	MANHA (   )
LE	0						BALA (   )	TAÇA (   )	MANHA (   )	FARRA (   )	DAMA (   )
Score: 0% - It was not possible to measure listening effort (absence of correct answers); 20% - Maximum listening effort and very severe degree of working memory ability; 40% - Large listening effort and severe degree of working memory ability; 60% - Medium listening effort and moderate degree of working memory ability; 80% - Small listening effort and preserved degree of working memory ability; 100% - Minimum listening effort and higher degree of working memory ability.

• Caption: RE = Right ear, LE = Left ear, dB = Decibel, SNR = Signal-to-Noise Ratio, Hz = Hertz

DISCUSSION

In recent years, the discussion on conceptual issues of different approaches to measure listening effort and the instruments used in the population with hearing loss, especially at an international level, has gained emphasis. This discussion was structured according to the objective of this study and its hypothesis.

This study aimed to validate the content and the applicability process of an instrument to measure listening effort for individuals with hearing loss.

In order to start the elaboration of the proposed instrument, it was determined the population to be measured, that is, its construct. Thus, the population chosen for validating the instrument was hearing-impaired persons. Since this study is committed to validate a behavioral method of listening effort measurement, the definition of this type of method was discussed, since in the national literature this concept is new and needs to be differentiated from the other approaches that can be used to estimate this effort.

Researchers^8,9 state that the applicability of behavioral measurements in the listening effort estimation considers the occurrence of a decline in cognitive functions regarding to prolonged mental effort, using an auditory dual task paradigm, whose primary task refers to speech perception and secondary tasks to memorization and reaction time tasks of visual response.

The instrument of this study was developed to estimate listening effort needed to understand spoken language and, consequently, to identify aspects that hinder auditory perception in a natural way, that is, without effort. In addition, this instrument was developed in order to enable the measurement of this auditory parameter in a more economical way, a particularity not possible in an objective evaluation, and in a more reliable way, an aspect not ensured by subjective measures, which have been used as complementary tool for the objective measurement of listening effort^(21,22). Therefore, for the proposed instrument to be properly constructed in its construct, the stages for validating of these aspects were followed.

Although current literature points out the need to study the aspects linked to the effort put by individuals with hearing loss in an attempt to understand speech in challenging listening situations, little is known about the validation of the instruments used in this measurement, especially regarding the description of the validity stages conducted for the construction of each instrument.

The elaboration of the first version of the instrument to measure listening effort was carried out after the literature review, and each part of the instrument was composed of speech stimuli described in other instruments previously reported in the literature, such as logatomes, words⁽²³⁾ and meaningless sentences^(24,25).

The judges’ analysis regarding Part I of the instrument showed a good S-CVI, equivalent to 0.95. Despite the S-CVI value of this part of the instrument, the committee of judge audiologists made suggestions regarding the presentation of the logatomes in the SNR -5dB, whose I-CVI value was 0.60, so it was decided to remove this listening condition to that part of the instrument instead of changing its position on the answer sheet. The judges justified that the low score in this item is due to the fact that the SNR -5dB is a situation of arduous listening, in which individuals with hearing loss would probably have difficulty or give up when trying to perform the task.

Authors^(26,27) state that listening effort seems to depend on cognitive processes related to the input of the auditory stimulus, such as listening in noise compared to listening in quiet and, also, the own cognitive functions and individuals’ internal factors. Thus, the presence of noise may have affected the performance of participants with hearing loss and, consequently, influenced the judges’ opinion for the low score of this SNR and their exclusion from the instrument.

With respect to Part II of this instrument, the S-CVI (0.53) was less than the reference value described in the literature for the non-modification and/or exclusion of the instrument items (0.78), requiring modifications as the suggestions and consensus of the committee of judge audiologists.

According to agreement between the judges regarding the I-CVI values, and regarding the aspects suggested by them, all items evaluated in Part II were scored with values ​​below 0.78. In relation to the I-CVI values ​​of the Part II scores, it was decided to make the adjustments mentioned in Chart 2. Regarding the word sets that constituted Part II, it was decided to modify the presentation in which the stimuli were introduced in the instrument, using visual stimuli corresponding to the words that must be memorized in this task during the listening effort measurement, such as a poster or cards containing the words present in the sets, in order to facilitate the evocation of responses.

Regarding the I-CVI values referring to the different SNR showed in Part II, it was again chosen to exclude the measurement in the SNR -5dB, since this was the listening condition reported by the participants with hearing loss as the more arduous to the memorization of words.

Some researchers⁽²⁸⁾ reported in their studies that, when listening to degraded speech signals, normal hearing individuals and those with hearing loss face increased difficulty in processing and memorizing speech signals. In addition, they are less precise in terms of speech perception, because even when speech is understood, words or syllables that are acoustically degraded are more difficult to recall. Thus, the performance of participants with hearing loss may have been a crucial aspect for the judges’ decision regarding the items that should or should not be maintained in the proposed instrument. As the memorization task in noise is complex, most of the judges scored Part II of the instrument with low scores and, consequently, the score and presentation form during the test of that part needed to be modified.

Regarding Part III of the instrument for listening effort measurement for individuals with hearing loss, it was found that the I-ICVI values reached the minimum score of 0.78 and that the S-CVI was 0.93. Thus, the judges agreed that all items in this part were relevant and would not require alterations and/or exclusions. However, some structural suggestions were considered in the judges’ analysis, such as changing the positioning of the SNR -5dB, which due to the long time of application of the instrument, was also excluded from this part.

Some authors⁽²⁹⁾ applied a dual task paradigm, with the repetition of final words from sets of spoken sentences and the coding of final words in memory for later recall. The authors demonstrated that noise impaired word recall in a competitive speech context for young people with normal hearing, particularly for the sentences at the beginning of the lists, but this noise effect was weakened when a noise reduction algorithm was applied. Thus, the results of this study suggested that the presence of noise could impair the transfer of information contained in the speech to long-term storage⁽²⁹⁾.

Moreover to the changes previously mentioned, in order to facilitate the annotation of results by the instrument’s applicators, the judge audiologists suggested the addition of the order of the words that the participants must memorize and parentheses to the answer sheet so that the evaluators record the order in which the participants repeated the words in each SNR. This addition will facilitate the process of filling out the answer sheet, and will also enable future analyzes on the order of recalling and its relationship with listening effort used by individuals with hearing loss.

In one study⁽³⁰⁾, the task of perceiving sentence lists and recalling their words was applied and the participants who underwent this test were elderly listeners with mild to moderate hearing loss. Both groups of participants listened to the lists of 15 sentences that were interrupted at random points and, as a task, they had to recall only the last word of the last three sentences they heard. With the results of this study it was found that although both groups had excellent memorization skills for the last word heard, referring to the last sentence, the recall of the two words that preceded was worse for the group of individuals with hearing loss than for the group of normal hearing participants, regarding that the three words were said at the same level of intensity.

It is worth mentioning the suggestion made by the committee of judge audiologists about the use of the listening condition “quiet” for the training stage of individuals submitted to this instrument. The judges suggested this modification because it is a long test, consisting of three distinct stages, with task change in all three parts and because it requires the use of cognitive resources to achieve a good performance. Thus, it is important that the individuals evaluated know how to answer adequately to the task requested by the evaluator.

The results showed in this study were also analyzed regarding their limitations. The first is related to the search for audiologists who are committed and available to apply the instrument to be validated and contribute to the growth and development of the thematic listening effort in research of the audiology area in the country.

Another limitation and suggestion for future research refers to the improvement and development of instruments related to the area of speech perception and listening effort with áudio recording by both female and male speakers, so that speech stimuli are presented randomly as to the characteristics of speakers’ vocal emission. In addition, an explanatory training stage is suggested regarding the stages of the instrument, in which the listening condition in “quiet” can be used as the participants’ training.

The tool to measure listening effort in the population with hearing loss has an innovative character in the national scope, since it included the stages of validity needed for the elaboration of a measurement instrument. However, for this instrument to be applied to the hearing-impaired population and to provide the most effective values for measuring listening effort, further studies in this population should be conducted to validate the evidence based on the response processes, criterion and construct. However, there is the need to continue the validation process to analyze other evidence of validity.

CONCLUSION

The instrument to measure listening effort for individuals with hearing loss was elaborated and validated in terms of its content. The analysis of the content by judge audiologits with expertise in the audiological field enabled the modification of questions and exclusion one of the signal-to noise ratios, as well as structural additions in the parts of the validated instrument.

• Study conducted at Departamento de Fonoaudiologia, Faculdade de Filosofia e Ciências, Universidade Estadual Paulista “Júlio de Mesquita Filho” - UNESP - Marília (SP), Brasil.
• Financial support: Nothing to declare.

REFERÊNCIAS

• ¹
  Wingfield A. The evolution of models of working memoryand cognitive resources. Ear Hear, 2016; 37(1): 35S-43S. https://doi.org/10.1097/AUD.0000000000000310 PMid:27355768.
  » https://doi.org/10.1097/AUD.0000000000000310
• ²
  Fraser S, Gagné JP, Alepins M, Dubois P. Evaluating the effort expended to understand speech in noise using a dual-task paradigm: The effects of providing visual speech cues. J Speech Lang Hear Res, 2010; 53(1): 18-33. https://doi.org/10.1044/1092-4388(2009/08-0140) PMid:19635945
  » https://doi.org/10.1044/1092-4388(2009/08-0140)
• ³
  Picou EM, Ricketts TA, Hornsby BW. How hearing aids, background noise, and visual cues influence objective listening effort. Ear Hear, 2013; 34(5): 52-64. https://doi.org/10.1097/AUD.0b013e31827f0431 PMid:23416751
  » https://doi.org/10.1097/AUD.0b013e31827f0431
• ⁴
  Pichora- Fuller MK, Kramer SE, Eckert MA, Edwards B, Hornsby BW, Humes LE, et al. Hearing impairment and cognitive energy: The Framework for Understanding Effortful Listening (FUEL). Ear Hear. 2016; 37: 5S-27S (citar como 4). https://doi.org/10.1097/AUD.0000000000000312 PMid:27355771
  » https://doi.org/10.1097/AUD.0000000000000312
• ⁵
  McGarrigle R, Munro KJ, Dawes P, Stewart AJ, Moore DR, Barry JG et al. Listening effort and fatigue: What exactly are we measuring? A British Society of Audiology Cognition in Hearing Special Interest Group 'white paper'. Int J Audiol, 2014; 53(7): 433-40. https://doi.org/10.3109/14992027.2014.890296 PMid:24673660
  » https://doi.org/10.3109/14992027.2014.890296
• ⁶
  Pals C, Sarampalis A, Başkent D. Listening effort with cochlear implant simulations. J Speech Lang Hear Res, 2013; 56(4): 1075-84. https://doi.org/10.1044/1092-4388(2012/12-0074) PMid:23275424
  » https://doi.org/10.1044/1092-4388(2012/12-0074)
• ⁷
  Koelewijn T, Zekveld AA, Festen JM, Kramer SE. Pupil dilation uncovers extra listening effort in the presence of a single-talker masker. Ear Hear, 2012; 33(2): 291-300. https://doi.org/10.1097/AUD.0b013e3182310019 PMid:21921797
  » https://doi.org/10.1097/AUD.0b013e3182310019
• ⁸
  Anderson-Gosselin PA, Gagné JP. Use of a Dual-Task Paradigm to Measure Listening Effort. Can J Speech Lang Pathol Audiol, 2010; 34(1): 43-51. https://doi.org/10.1590/2317-1782/20192018181
  » https://doi.org/10.1590/2317-1782/20192018181
• ⁹
  Wu YH, Aksan N, Rizzo M, Stangl E, Zhang X, Bentler R. Measuring listening effort: Driving simulator vs. simple dual-task paradigm. Ear Hear, 2014; 35(6): 623-32. https://doi.org/10.1097/AUD.0000000000000079 PMCid:PMC4208979. PMid:25083599
  » https://doi.org/10.1097/AUD.0000000000000079
• ¹⁰
  Rönnberg J. Lunner T, Zekveld A, Sörqvist P, Danielsson H, Lyxell B, et al. The Ease of Language Understanding (ELU) model: theoretical, empirical, and clinical advances. Front Syst Neurosc, 2013; 7(31): 1-17. https://doi.org/10.3389/fnsys.2013.00031 PMCid: PMC3710434. PMid:23874273
  » https://doi.org/10.3389/fnsys.2013.00031
• ¹¹
  Rudner M, Lunner T, Behrens T, Thorén ES, Rönnberg J. Working memory capacity may influence perceived effort during aided speech recognition in noise. J Am Acad Audiol, 2012; 23(8): 577-89. https://doi.org/10.3766/jaaa.23.7.7 PMid:22967733
  » https://doi.org/10.3766/jaaa.23.7.7
• ¹²
  Ma WJ, Husain M, Bays PM. Changing concepts of working memory. Nat neurosci, 2014; 17(3): 347-56. https://doi.org/10.1038/nn.3655
  » https://doi.org/10.1038/nn.3655
• ¹³
  Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciênc. Saude Colet, 2011; 16(7); 3061-68. https://doi.org/10.1590/S1413-81232011000800006
  » https://doi.org/10.1590/S1413-81232011000800006
• ¹⁴
  Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol, 2010; 63(7): 737-45. https://doi.org/10.1016/j.jclinepi.2010.02.006
  » https://doi.org/10.1016/j.jclinepi.2010.02.006
• ¹⁵
  Roach KE. Measurement of health outcomes: reliability, validity and responsiveness. J Prosthet Orthot, 2006; 18(1S): 8-12
• ¹⁶
  Polit DF. Assessing measurement in health: beyond reliability and validity. Int J Nurs Stud, 2015; 52(11): 1746-53. https://doi.org/10.1016/j.ijnurstu.2015.07.002 PMid:26234936
  » https://doi.org/10.1016/j.ijnurstu.2015.07.002
• ¹⁷
  Haynes SN, Richard DCS, Kubany ES. Content validity in psychological assessment: a functional approach to concepts and methods. Psychol Assess, 1995; 7(3): 238-47. https://doi.org/10.1037/1040-3590.7.3.238
  » https://doi.org/10.1037/1040-3590.7.3.238
• ¹⁸
  Lynn MR. Determination and quantification of content validity. Nurs Res, 1986; 35(6): 382-5. https://doi.org/10.1097/00006199-198611000-00017
  » https://doi.org/10.1097/00006199-198611000-00017
• ¹⁹
  Grant JS, Davis LL. Selection and use of content experts for instrument development. Res Nurs Health, 1997; 20(3): 269-74. https://doi.org/10.1002/(sici)1098-240x(199706)20:3<269::aid-nur9>3.0.co;2-g PMid:9179180
  » https://doi.org/10.1002/(sici)1098-240x(199706)20:3<269::aid-nur9>3.0.co;2-g
• ²⁰
  Polit DF, Beck CT. The content validity index: are you sure you know what's being reported? Critique and recommendations. Res Nurs Health, 2006; 29(5). 489-97. https://doi.org/10.1002/nur.20147 PMid:16977646
  » https://doi.org/10.1002/nur.20147
• ²¹
  Mackersie CL, Cones H. Subjective and Psychophysiological Indexes of Listening Effort in a Competing-Talker Task. J Am Acad Audiol, 2011; 22(2): 113-22. https://doi.org/10.3766/jaaa.22.2.6 PMCid: PMC3072569. PMid:21463566
  » https://doi.org/10.3766/jaaa.22.2.6
• ²²
  Koelewijn T, Zekveld AA, Festen JM, Kramer SEl. Pupil dilation uncovers extra listening effort in the presence of a single-talker masker. Ear Hear, 2012; 33(2): 291-300. https://doi.org/10.1097/AUD.0b013e3182310019 PMid:21921797
  » https://doi.org/10.1097/AUD.0b013e3182310019
• ²³
  Giangiacomo MCPB, Navas ALGP. A influência da memória operacional nas habilidades de compreensão de leitura em escolares de 4ª série. Rev Soc Bras Fonoaudiol, 2008; 13(1): 69-74. https://doi.org/10.1590/S1516-80342008000100012
  » https://doi.org/10.1590/S1516-80342008000100012
• ²⁴
  Ng HNE, Rudner M, Lunner T, Pederson MS, Ronnberg J. Effects of noise and working memory capacity on memory processing of speech for hearing-aid users. Int J Audiol, 2013; 52(7): 433-41. https://doi.org/10.3109/14992027.2013.776181 PMid:23550584
  » https://doi.org/10.3109/14992027.2013.776181
• ²⁵
  Rönnberg N, Stenfelt S, Rudner M. Testing listening effort for speech comprehension using the individuals' cognitive spare capacity. Audiol Res, 2011; 1(1): 1-8. https://doi.org/10.4081/audiores.2011.e22 PMid:26557307
  » https://doi.org/10.4081/audiores.2011.e22
• ²⁶
  Brannstrom KJ, Karlsson E, Waechter S, Kastberg T. Listening Effort: Order Effects and Core Executive Functions. J Am Acad Audiol, 2017; 29(8): 1-13. https://doi.org/10.3766/jaaa.17024 PMid:30222543
  » https://doi.org/10.3766/jaaa.17024
• ²⁷
  Peelle JE. Listening Effort: How the Cognitive Consequences of Acoustic Challenge Are Reflected in Brain and Behavior. Ear Hear, 2018; 39(2): 204-14. PMid:28938250. PMCID: PMC5821557. https://doi.org/10.1097/AUD.0000000000000494
  » https://doi.org/10.1097/AUD.0000000000000494
• ²⁸
  Heinrich A, Schneider BA, Craik FI. Investigating the influence of continuous babble on auditory short-term memory performance. Q J Exp Psychol, 2008; 61(5): 735-51. https://doi.org/10.1080/17470210701402372 PMid:17853231
  » https://doi.org/10.1080/17470210701402372
• ²⁹
  Sarampalis A, Kalluri S, Edwards B, Hafter E. Objective measures of listening effort: effects of background noise and noise reduction. J Speech Lang Hear Res, 2009; 52(5): 1230-40. https://doi.org/10.1044/1092-4388(2009/08-0111) PMid:19380604
  » https://doi.org/10.1044/1092-4388(2009/08-0111)
• ³⁰
  McCoy SL, Tun PA, Cox LC, Colangelo M, Stewart RA, Wingfield A. Q J Exp Psychol, 2005; 58(1): 22-33. https://doi.org/10.1080/02724980443000151 PMid:15881289
  » https://doi.org/10.1080/02724980443000151

Publication Dates

• Publication in this collection
  12 Oct 2020
• Date of issue
  2020

History

• Received
  17 Nov 2018
• Accepted
  30 Sept 2019