Content validation of an instrument to measure listening effort.

PURPOSE
To validate the content of an instrument to measure listening effort for hearing-impaired individuals.


METHOD
This is a validation study, developed in two stages, which the Stage 1 is the planning and development of the first version of the instrument, and Stage 2 the investigation of the evidences of validity based on the content and development of the final version of the instrument to measure listening effort. Ten professionals with expertise in the field of audiology, with more than five years of clinical experience participated in this study. The instrument to be validated was composed of three parts: I - "speech perception of logatomes and listening effort"; II - "listening effort and working memory" and; III - "speech perception of meaningless sentences and working memory" and they were presented monoaurally, in quiet and in the signal-to-noise ratios + 5dB, 0dB and -5dB. It was conducted a descriptive analysis regarding the suggestions of the committee judge audiologists and the analysis of the individual and scale content validity index.


RESULTS
The results showed that parts I and III which constitute the proposed instrument reached a scale content validity index above 0.78, which means that the presented items did not need modification in their construct.


CONCLUSION
The evidences of validity studied allowed relevant modifications and made this instrument adequate to its construct.


INTRODUCTION
Currently, audiology and cognitive psychology fields have endeavored to define the term "listening effort" and find the most accurate, reliable method, whose applicability is feasible for clinical audiology. Audiologists and cognitive psychologists have studied cognitive theories related to mental resources involved in measuring listening effort, especially in behavioral evaluation (1) .
One of the definitions adopted for listening effort refers to the amount of cognitive resources needed for recognizing acoustic signals, especially speech (2,3) . Other authors defined the listening effort as a deliberate allocation of mental resources needed to overcome obstacles in the pursuit of objectives when performing a listening task (4) . However, the agreement regarding the best definition of this auditory parameter has not yet been scientifically agreed. Researchers (5) state that listening effort can be measured through three main methodological approaches, which are categorized into subjective, behavioral (6) and psychophysiological (7) measurements.
Behavioral measurements are described in the literature as a dual task paradigm. This paradigm refers to the accomplishment of two tasks, a primary and a secondary one, performed simultaneously. The literature (8) states that to measure listening effort, the primary task will involve speech perception tasks/ tests, such as word and/or sentence recognition, in quiet and with manipulation of the signal-to noise ratio. However, the secondary task can refer to activities/tests of working memory or visual pattern recognition.
Some authors (8,9) reported that the alteration in performing the secondary task at different levels of difficulties of the primary task reflects a change in the cognitive resources for speech processing, that is, listening effort. This interpretation assumes that performance in both the primary and secondary tasks requires the allocation of some common cognitive resources to each task. As cognitive resources are limited, greater listening effort and, consequently, greater demand for cognitive resources for the speech perception task will be dispensed.
Over the years and the beginning of the aging process, researchers have demonstrated the existence of a strong relationship between age and decline in working memory, and greater listening effort is needed to understand speech (10,11) . According to some researchers (12) , modern models that conceptualize working memory define it as a flexible, even though limited, resource that is implemented regarding the quality of coded representations, that is, the better the quality of auditory representations, the greater the possibility of coding the acoustic signal.
The hypothesis of studies that measure listening effort using a behavioral measurement with dual task paradigms whose secondary task involves the working memory capacity in different signal-to-noise ratios, is that listeners with less working memory capacity are more susceptible to effects of background noise and, consequently, put greater listening effort when performing the speech perception task.
Currently, the availability of instruments and/or questionnaires that can be applied both in the scientific area and in clinical practice has been increasing in the health area, but most of these procedures have not been properly validated, especially regarding their measurement properties (13) . An instrument considered valid is one that evaluates exactly what it proposes to measure (14) , and is related to a specific research question and to a particular population (15) .
The literature (16) describes three main types of validation, which are content validation, criterion validation and construct validation. Content validity aims to assess the relevance and representativeness of each element present in an instrument of a specific construct (17) .
Considering that at the national level there is no a validated behavioral measurement to estimate listening effort, and that this parameter has been investigated indirectly through subjective (questionnaires or scales) and/or objective approaches (electrophysiological responses, heart rate, pupillometry, among others), it is hypothesized for this study that the content validation of an instrument for listening effort measurement will have great relevance and usefulness for the beginning of a complete validation process, which also involves the criterion and construct validation, and seeks a sensitive instrument to predict behavioral performance in measuring this parameter. Thus, the aim of this study was to validate the content of an instrument to measure listening effort for individuals with hearing loss.
The relevance of the investigation of listening effort is related to the complaints reported by patients with hearing loss, hearing aids users or not users, associated with reports of fatigue in speech comprehension situations, mainly in noise, or when the message is not familiar. Thus, the relevance of validating the content of an instrument for listening effort measurement is justified due to its importance for the adaptation of a measurement instrument. It is worth emphasizing that only the hearing threshold measurement is not a good predictor in cases in which patients have difficulty understanding speech, since it portrays the auditory sensitivity and not listening effort.
Moreover, this content validation may be beneficial for professionals and researchers working with auditory training programs, since it may be used as an instrument for pre and post-training assessment of individuals who have undergone sessions for the improvement of auditory abilities. This is a validation study, described in two stages, with stage 1 being planning and development of the first version of the instrument for measuring listening effort of the individuals with hearing loss; stage 2, investigation of the evidence of validity based on the content, that is, permanence or exclusion of items from the instrument after analysis and agreement of the judge audiologists and development of the final version of the proposed instrument. For the literature review, databases as US National Library of Medicine National Institutes of Health (PUBMED), Latin American and Caribbean Health Sciences Literature (Literatura Latino-Americana e do Caribe em Ciências da Saúde (LILACS)), Scientific Eletronic Library Online (SCIELO) and Cochrane Library were reviewed, with selection of English and Portuguese languages, without limit on the publication. The keywords and descriptors used in English were hearing loss, listening effort, perceptual effort, ease of listening, speech perception, working memory, assessment and, in Portuguese, perda auditiva, esforço auditivo, esforço de escuta, esforço perceptivo, facilidade em ouvir, percepção de fala, memória operacional and avaliação in different combinations, aiming to find the largest number of studies. Subsequently, original studies were selected that used behavioral measurements to measure listening effort and working memory in the population with hearing loss, as well as existing instruments, whether validated or not, and the selection of the items and score system of instruments was also carried out.

METHOD
-Part I: "speech perception of logatomes and listening effort" which aimed to evaluate the perception of consonants of the Brazilian Portuguese when they were isolated between vowels with the same amount of acoustic energy, forming logatomes.
This part consisted of two word lists, in which most of them have no meaning (logatomes). These lists were composed of words whose consonants are isolated by the vowel "A", such as "ANHA", "ALA", "ARA", among others. The objective of this part, moreover the speech perception task, was to verify if the participant produced the acoustic signal received in a reliable way or as a real word, performing auditory closure and making use of the contextual clue, such as the production of the logatome " ALA ", like the real word "FALA", due to the manipulation of different signal-to noise ratios.
-Part II: "listening effort and working memory: set of real words" aimed at evaluating listening effort performed by the participant when performing a memorizing task and recalling real words, that is, words that have meaning, derived from the logatomes from the first part of the instrument, at different levels of signal-to noise ratio. This part consisted of four sets of real words, in which each set has three series of words (Set I: composed of three series of two words each; Set II: composed of three series of three words each; Set III: composed of three series of four words each; Set IV: composed of three series of five words each). Participants were instructed to listen to each series of words and, subsequently, to remember and repeat the first word heard in each series.
Considering the complexity of the task, subsequent sets were only presented by memorizing and producing the first word of each series. As far as the participants answer correctly the memorized words, the other sets were presented.
-Part III: "speech perception of meaningless sentences and working memory" aimed to measure listening effort needed in a speech perception task of long duration stimuli. At this stage, the participant should repeat each sentence heard and, then remember the last word of each sentence. This part consisted of five meaningless sentences to avoid guessing by the participants. The last word of each sentence was derived from the logatomes in Part I of the instrument. In this part of the test, the more sentences the participants produced correctly, the better their speech perception skills were, and the more words they remembered, the better their working memory capacity and the less the amount of listening effort required.
In order to ensure uniformity and avoid bias regarding the differences related to the speech emission characteristics of each applicator, the logatomes, real words and meaningless sentences were recorded. The recording of the speech stimuli of the instrument was made by a female speaker of Brazilian Portuguese, in an acoustically treated room. This recording was conducted with the speaker seated on a chair, using a Sennheiser microphone (model E855) and a digital recorder from MARANTZ (model PMD660, configured for single-channel recording, with a sampling rate of 44 kHz and 16 bits of resolution). The microphone was positioned at 45 degrees and a distance of 10 cm in front of the speaker's mouth.
For testing, the speech stimuli of the instrument were played monaurally, in the following listening situations: quiet and with backnoise type White Noise in the signal-to-noise ratios + 5dB, 0dB and -5dB in relation to the participant's Maximum Comfort Level. Moreover, an instructional guide for the application of each part of the instrument was elaborated to facilitate the application process of the instrument for the judge audiologists.

STAGE 2. Investigation of the evidence of validity based on the content of the instrument for measuring listening effort
Stage 2 consisted of investigating the content validity of the instrument. The literature (18) recommends a minimum number of five and a maximum of ten people participating in this process.
According to the criteria suggested by the international literature 19 , for the content validation a committee of Brazilian specialists was formed, composed of ten specialized audiologists, who developed research in the area, and with proven clinical experience, that is, who worked in the area for five years or more. They filled out and sent their informed consent terms, questionnaires of academic and professional characterization and analysis of the instrument based on the guidelines received.
Audiologists were invited to participate in the committee of judges by email. After acceptance, all material was sent by mail. All judge audiologists received an envelope that contained the following documents/materials: an explanatory letter of presentation of the study and its objective, stating the reason for choosing the judge, mentioned in the previous paragraph and the relevance of the concepts involved and the instrument as a whole (18,19) ; informed consent form for the judge audiologist; informed consent form for the participant with hearing loss; a questionnaire for academic and professional characterization of the audiologist; the instrument for measuring listening effort with its answer sheet; an explanatory video of the instrument with a duration of 10 minutes, containing guidelines about the population profile to be assessed and how the instrument was developed, the objective and application method of the instrument parts, the reason for choosing each speech stimulus (logatomes, real words and meaningless sentences) and also about the different signal-to noise relations and how to manipulate them and, finally, how to fill out the answer sheet of the instrument; a CD or pen-drive with the audio recording of the instrument to measure listening effort; and an instruction guideline also containing explanations about each part of the instrument and how to properly complete the answer sheet.
The judges filled out and sent the consent terms, questionnaires of academic and professional characterization and analyses of the instrument based on the guidelines provided. The characterization of the audiologist professionals who are judges according to the variables gender, age, length of professional experience in the area and degree is shown in table 1. The mean age of the judges was 41.6 years and all participants had a degree. Of these, 80% had specialization, 10% were "masters" and 10% were "doctors".
Initially, the judges were instructed to evaluate the whole instrument, determining its scope. The judges were also instructed to analyze each part of the instrument individually as adequate or inadequate, verifying its clarity and pertinence. Regarding clarity, it was requested to verify the structure and the writing of items, if they had been written in an understandable way and if they adequately expressed what should be measured. Regarding pertinence, it was requested to verify the items in relation to the concepts involved and whether they were relevant to the proposed objective.
If the judges deemed it inappropriate, they were instructed to justify and suggest changes to the item in the space indicated. At the end of the protocol, there was a space for the judges write comments and add the items they deemed relevant and which had not been included in the instrument.

Analysis of data
A qualitative analysis was conducted, based on the suggestions and comments made available by the judges regarding the content of the instrument. Moreover, in order to determine acceptance and agreement of the questions by the judges, the Individual Content Validity Index (I-CVI) and the Scale Content Validity Index (S-CVI) were calculated.
This measurement calculates the proportion or percentage of judges who are in agreement on certain aspects of the instrument and its items, allowing individual and global analysis of the instrument (20) , and uses a Likert scale with a score from 1 to 4. To assess the relevance/representativeness, responses may include: 1 = not relevant or not representative, 2 = item needs major revision to be representative, 3 = item needs minor revision to be representative, 4 = relevant or representative item 18 .
The score of this index was calculated through the sum of agreement of the items marked as "3" or "4" by the judge audiologists 19 , with the items scored as "1" or "2" being revised or eliminated. As described in the international literature (18,20) , when the committee of judges has six or more specialists, the recommended values for I-CVI and S-CVI should not be less than 0.78. After this analysis, the indexes with values below 0.78, like all suggestions recommended by the judges were analyzed by the authors of the instrument and, after consensus, changes were made to the final version of the instrument. Table 2 shows results of the analysis from audiologist judges for the three parts of the instrument (parts I, II and III).

RESULTS
Regarding the results showed in table 2, it was found that most of the items analyzed by the judge audiologists reached the I-CVI higher than 0.78, and that the S-CVI of this stage was 0.95. Only the item referring to the SNR (-5dB) obtained an I-CVI score of 0.60, that is, a score less than 0.78 and, it was opted to remove this listening condition from Part I of the instrument. In chart 1, suggestions offered by the judges are shown for reformulation and/ou improvement of the validated items.
Modifications showed in chart 1 were made regarding the judge audiologists' suggestions with respect to the improvement of content of the instrument, and not due to the agreement between them regarding validity of the items.
Chart 2 shows the final version of the instrument and its answer sheet, where all suggestions proposed by the judge audiologists were accepted after analysis of the instrument by the audiologists committee.

______________________
Use of visual support to assist in evoking the response, for example a poster or cards containing the words from the sets.

Part III
20% -Minimum listening effort and higher degree of working memory ability; 40% -Small listening effort and preserved degree of working memory ability; 60% -Medium listening effort and moderate degree of working memory ability; 80% -Large listening effort and severe degree of working memory ability; 100% -Maximum listening effort and very severe degree of working memory ability. 0% -It was not possible to measure listening effort (absence of correct answers); 20% -Maximum listening effort and very severe degree of working memory ability; 40% -Large listening effort and severe degree of working memory ability; 60% -Medium listening effort and moderate degree of working memory ability; 80% -Small listening effort and preserved degree of working memory ability; 100% -Minimum listening effort and higher degree of working memory ability.

______________________
Use of visual support to assist in evoking the response, for example a poster or cards containing the words from the sets.
Keep SNR -5dB. Exclude SNR +5dB and -5dB ______________________ Add the order of words that patients should memorize and leave a space for evaluators to write down the sequence in which the patients memorized the words.
Caption: dB = Decibel, SNR = Signal-to-Noise Ratio Score: 0% -It was not possible to measure listening effort (absence of correct answers); 25% -Maximum listening effort and severe degree of working memory ability -(memorization of the first words of each series of Set I, equivalent to one set); 50% -Medium listening effort and moderate degree of working memory ability -(memorization of the first words of each series of Sets I and II, equivalent to two sets); 75% -Small listening effort and preserved degree of working memory ability -(memorization of the first words of each series of Sets I, II and III, equivalent to three sets); 100% -Minimum listening effort and higher degree of working memory ability -(memorization of the first words of each series of Sets I, II, III and IV, equivalent to four sets); RE/LE  SNR  1  2  3  4  5  1  2  3  4  5 Correct answers (%)

DISCUSSION
In recent years, the discussion on conceptual issues of different approaches to measure listening effort and the instruments used in the population with hearing loss, especially at an international level, has gained emphasis. This discussion was structured according to the objective of this study and its hypothesis.
This study aimed to validate the content and the applicability process of an instrument to measure listening effort for individuals with hearing loss.
In order to start the elaboration of the proposed instrument, it was determined the population to be measured, that is, its construct. Thus, the population chosen for validating the instrument was hearing-impaired persons. Since this study is committed to validate a behavioral method of listening effort measurement, the definition of this type of method was discussed, since in the national literature this concept is new and needs to be differentiated from the other approaches that can be used to estimate this effort.
Researchers 8,9 state that the applicability of behavioral measurements in the listening effort estimation considers the occurrence of a decline in cognitive functions regarding to prolonged mental effort, using an auditory dual task paradigm, whose primary task refers to speech perception and secondary tasks to memorization and reaction time tasks of visual response.
The instrument of this study was developed to estimate listening effort needed to understand spoken language and, consequently, to identify aspects that hinder auditory perception in a natural way, that is, without effort. In addition, this instrument was developed in order to enable the measurement of this auditory parameter in a more economical way, a particularity not possible in an objective evaluation, and in a more reliable way, an aspect not ensured by subjective measures, which have been used as complementary tool for the objective measurement of listening effort (21,22) . Therefore, for the proposed instrument to be properly constructed in its construct, the stages for validating of these aspects were followed.
Although current literature points out the need to study the aspects linked to the effort put by individuals with hearing loss in an attempt to understand speech in challenging listening situations, little is known about the validation of the instruments used in this measurement, especially regarding the description of the validity stages conducted for the construction of each instrument.
The elaboration of the first version of the instrument to measure listening effort was carried out after the literature review, and each part of the instrument was composed of speech stimuli described in other instruments previously reported in the literature, such as logatomes, words (23) and meaningless sentences (24,25) .
The judges' analysis regarding Part I of the instrument showed a good S-CVI, equivalent to 0.95. Despite the S-CVI value of this part of the instrument, the committee of judge audiologists made suggestions regarding the presentation of the logatomes in the SNR -5dB, whose I-CVI value was 0.60, so it was decided to remove this listening condition to that part of the instrument instead of changing its position on the answer sheet. The judges justified that the low score in this item is due to the fact that the SNR -5dB is a situation of arduous listening, in which individuals with hearing loss would probably have difficulty or give up when trying to perform the task.
Authors (26,27) state that listening effort seems to depend on cognitive processes related to the input of the auditory stimulus, such as listening in noise compared to listening in quiet and, also, the own cognitive functions and individuals' internal factors. Thus, the presence of noise may have affected the performance of participants with hearing loss and, consequently, influenced the judges' opinion for the low score of this SNR and their exclusion from the instrument.
With respect to Part II of this instrument, the S-CVI (0.53) was less than the reference value described in the literature for the non-modification and/or exclusion of the instrument items (0.78), requiring modifications as the suggestions and consensus of the committee of judge audiologists.
According to agreement between the judges regarding the I-CVI values, and regarding the aspects suggested by them, all items evaluated in Part II were scored with values below 0.78. In relation to the I-CVI values of the Part II scores, it was decided to make the adjustments mentioned in Chart 2. Regarding the word sets that constituted Part II, it was decided to modify the presentation in which the stimuli were introduced in the instrument, using visual stimuli corresponding to the words that must be memorized in this task during the listening effort measurement, such as a poster or cards containing the words present in the sets, in order to facilitate the evocation of responses.
Regarding the I-CVI values referring to the different SNR showed in Part II, it was again chosen to exclude the measurement in the SNR -5dB, since this was the listening condition reported by the participants with hearing loss as the more arduous to the memorization of words.
Some researchers (28) reported in their studies that, when listening to degraded speech signals, normal hearing individuals and those with hearing loss face increased difficulty in processing and memorizing speech signals. In addition, they are less precise in terms of speech perception, because even when speech is understood, words or syllables that are acoustically degraded are more difficult to recall. Thus, the performance of participants with hearing loss may have been a crucial aspect for the judges' decision regarding the items that should or should not be maintained in the proposed instrument. As the memorization task in noise is complex, most of the judges scored Part II of the instrument with low scores and, consequently, the score and presentation form during the test of that part needed to be modified.
Regarding Part III of the instrument for listening effort measurement for individuals with hearing loss, it was found that the I-ICVI values reached the minimum score of 0.78 and that the S-CVI was 0.93. Thus, the judges agreed that all items in this part were relevant and would not require alterations and/or exclusions. However, some structural suggestions were considered in the judges' analysis, such as changing the positioning of the SNR -5dB, which due to the long time of application of the instrument, was also excluded from this part.
Some authors (29) applied a dual task paradigm, with the repetition of final words from sets of spoken sentences and the coding of final words in memory for later recall. The authors demonstrated that noise impaired word recall in a competitive speech context for young people with normal hearing, particularly for the sentences at the beginning of the lists, but this noise effect was weakened when a noise reduction algorithm was applied. Thus, the results of this study suggested that the presence of noise could impair the transfer of information contained in the speech to long-term storage (29) .
Moreover to the changes previously mentioned, in order to facilitate the annotation of results by the instrument's applicators, the judge audiologists suggested the addition of the order of the words that the participants must memorize and parentheses to the answer sheet so that the evaluators record the order in which the participants repeated the words in each SNR. This addition will facilitate the process of filling out the answer sheet, and will also enable future analyzes on the order of recalling and its relationship with listening effort used by individuals with hearing loss.
In one study (30) , the task of perceiving sentence lists and recalling their words was applied and the participants who underwent this test were elderly listeners with mild to moderate hearing loss. Both groups of participants listened to the lists of 15 sentences that were interrupted at random points and, as a task, they had to recall only the last word of the last three sentences they heard. With the results of this study it was found that although both groups had excellent memorization skills for the last word heard, referring to the last sentence, the recall of the two words that preceded was worse for the group of individuals with hearing loss than for the group of normal hearing participants, regarding that the three words were said at the same level of intensity.
It is worth mentioning the suggestion made by the committee of judge audiologists about the use of the listening condition "quiet" for the training stage of individuals submitted to this instrument. The judges suggested this modification because it is a long test, consisting of three distinct stages, with task change in all three parts and because it requires the use of cognitive resources to achieve a good performance. Thus, it is important that the individuals evaluated know how to answer adequately to the task requested by the evaluator.
The results showed in this study were also analyzed regarding their limitations. The first is related to the search for audiologists who are committed and available to apply the instrument to be validated and contribute to the growth and development of the thematic listening effort in research of the audiology area in the country.
Another limitation and suggestion for future research refers to the improvement and development of instruments related to the area of speech perception and listening effort with áudio recording by both female and male speakers, so that speech stimuli are presented randomly as to the characteristics of speakers' vocal emission. In addition, an explanatory training stage is suggested regarding the stages of the instrument, in which the listening condition in "quiet" can be used as the participants' training.
The tool to measure listening effort in the population with hearing loss has an innovative character in the national scope, since it included the stages of validity needed for the elaboration of a measurement instrument. However, for this instrument to be applied to the hearing-impaired population and to provide the most effective values for measuring listening effort, further studies in this population should be conducted to validate the evidence based on the response processes, criterion and construct. However, there is the need to continue the validation process to analyze other evidence of validity.

CONCLUSION
The instrument to measure listening effort for individuals with hearing loss was elaborated and validated in terms of its content. The analysis of the content by judge audiologits with expertise in the audiological field enabled the modification of