Influence of speech sample on perceptual rating of hypernasality

ABSTRACT

Purpose

To investigate the influence of speech sample of spontaneous conversation or sentences repetition on intra and inter-rater hypernasality reliability.

Methods

One hundred and twenty audio recorded speech samples (60 containing spontaneous conversation and 60 containing repeated sentences) of individuals with repaired cleft palate±lip, both genders, aged between 6 and 52 years old (mean=21±10) were selected and edited. Three experienced speech and language pathologists rated hypernasality according to their own criteria using 4-point scale: 1=absence of hypernasality, 2=mild hypernasality, 3=moderate hypernasality and 4=severe hypernasality, first in spontaneous speech samples and 30 days after, in sentences repetition samples. Intra- and inter-rater agreements were calculated for both speech samples and were statistically compared by the Z test at a significance level of 5%.

Results

Comparison of intra-rater agreements between both speech samples showed an increase of the coefficients obtained in the analysis of sentences repetition compared to those obtained in spontaneous conversation. Comparison between inter-rater agreement showed no significant difference among the three raters for the two speech samples.

Conclusion

Sentences repetition improved intra-raters reliability of perceptual judgment of hypernasality. However, the speech sample had no influence on reliability among different raters.

Keywords:
Cleft Palate; Velopharyngeal Insufficiency; Speech; Speech Disorders; Speech Perception

RESUMO

Objetivo

Investigar a influência do tipo de amostra de fala, conversa espontânea ou repetição de sentenças, sobre o índice de concordância intra e interavaliadores obtido na classificação perceptiva da hipernasalidade.

Métodos

Foram selecionadas e editadas 120 amostras de fala gravadas em áudio (60 contendo trechos de conversa espontânea e 60 contendo repetição de sentenças) de indivíduos com fissura de palato±lábio reparada, de ambos os sexos, com idade entre 6 e 52 anos (média=21±10 anos). Três fonoaudiólogas experientes, utilizando seus critérios internos, classificaram a hipernasalidade em escala de 4 pontos: 1=ausente, 2=leve, 3=moderada e 4=grave, primeiramente na amostra de conversa espontânea e, 30 dias depois, na repetição de sentenças. Os índices de concordância intra e interavaliadores foram estabelecidos para ambos os tipos de amostra de fala e comparados entre si por meio do Teste Z com nível de significância de 5%.

Resultados

A comparação dos índices de concordância intra-avaliadores entre os dois tipos de amostra de fala mostrou aumento dos coeficientes obtidos na análise da repetição de sentenças em relação aos obtidos na conversa espontânea, já a comparação entre os índices de concordância interavaliadores não mostrou diferença significante entre as três avaliadoras para os dois tipos de amostras de fala.

Conclusão

A repetição de sentenças favoreceu a confiabilidade do julgamento perceptivo da hipernasalidade de um mesmo avaliador, visto que a concordância intra-avaliadores na análise desta amostra de fala foi maior. No entanto, o tipo de amostra de fala não influenciou a concordância entre diferentes avaliadores.

Descritores:
Fissura Palatina; Insuficiência Velofaríngea; Fala; Percepção da Fala; Distúrbios da Fala

INTRODUCTION

Individuals with cleft palate are at high risk of developing speech disorders such as hypernasality, nasal air emission, low intraoral pressure and compensatory articulations resulting from velopharyngeal dysfunction (VPD). Clinically, hypernasality is the most evident symptom of VPD in these individuals^(1,2).

The assessment of velopharyngeal function is a difficult process due to its complexity and dynamic nature. Therefore, many authors have proposed different ways to categorize the speech of individuals with cleft palate in an attempt to establish an universal standard allowing multicentric studies^(2-5). Although instrumental evaluation, as videofluoroscopy, nasopharyngoscopy, nasometry and pressure-flow technique are essential for the diagnostic and management of the VPD, the identification of speech symptoms is mainly performed by the auditory perceptual assessment, that is considered “gold standard” in assessing individuals with cleft palate and the main indicator of the clinical significance of these symptoms^(1,6,7). However, due to its subjectivity, the evaluation may involves errors and variations, even when done by experienced professionals. The literature recommends that the perceptual assessment must be based on audio and/or video recorder in order to present results as an agreement between more than one evaluator regarding the judgment as to the presence and severity of the speech symptoms^(2,5,6,8-10). Among the factors that can affect perceptual judgment of hypernasality, the type of speech sample remains one of the most relevant. Some authors believe that hypernasality is identified only during spontaneous conversation or is considered to be more severe in this type of sample^(11,12). With an increase of the spontaneous speech, due to additional requirements, such as muscle fatigue of the velopharyngeal structures, the hypernasality becomes more noticeable⁽¹²⁾. This means that one individual may have different degrees of hypernasality depending on the speech sample being analyzed, suggesting that the results of different raters is only comparable when using the same speech sample. This fact has led many researchers to propose the standardization of the speech characteristics that should be added in the perceptual assessment of individuals with cleft palate in order to minimize the influence of various factors on the assessment of hypernasality and improve the reliability of this method^(4,9,13,14).

This study aimed to investigate the influence of the speech sample - spontaneous conversation or repeated sentences - on the perceptual judgment of hypernasality in individuals with repaired cleft palate. Ultimately, the study aimed to investigate which speech sample makes the hypernasality judgment the most reliable with regard to intra- and inter-raters agreement.

METHODS

Speech samples

This study was approved by the Human Research Ethics Committee of the Institution (nº 1.008.414). The study included 120 audio recorded speech samples (60 containing spontaneous conversation and 60 containing repeated sentences), from 60 patients with repaired cleft palate associated or not with cleft lip, of both genders, aged 6 to 52 years (mean 21 ± 10 years old), presenting or not VPD (i.e. two speech samples from each individual were analyzed).

Samples containing spontaneous conversation were obtained from personal answers to general questions adapted to the age of each individual, in order to obtain a speech sample long enough to allow the perceptual analysis of hypernasality. Samples with repeated sentences were composed of 11 standard sentences containing exclusively oral sounds. All samples were selected from digital audio recordings routinely performed in treated soundproof room and stored in the database of the Institution. Consent for data usage was obtained from all patients or their guardians, upon registration in the hospital. It was included only recordings with a good audio quality and with no noise that could compromise the analysis. However, samples containing other speech symptoms such as nasal air issuing audible compensatory articulations, nasal snoring and dysphonia were not excluded.

Procedures

The recordings were retrieved from the database, saved in MP3 and edited excluding the participation of the professional party's record of speech and standardizing the recording time in format at least 15 seconds and a maximum of 34 seconds. After editing, the speech samples were numbered and randomly copied onto two compact discs (CD), one containing samples of spontaneous conversation and the other containing the sentences of repeat samples. In order to analyze the intra evaluators concordance index, 30% of the samples were duplicated, randomized and included in CDs, care is taking to repeated samples were not included in the same CD in order to avoid being identified.

Perceptual analysis of hypernasality

Hypernasality was judged by three experienced speech therapists experienced in the perceptual assessment of individuals with cleft palate rated hypernasality in two stages. At first, the raters analyzed samples containing spontaneous conversation and after one month, the samples from the same patients containing sentences were analyzed. Although they are different speech samples, we settled this time interval between the two stages in order to avoid the patient’s identification. On both analysis, the evaluators ranked hypernasality according to their own criteria (internal standard) using the following 4-point scale: 1 = absence of hypernasality (normal resonance), 2 = mild hypernasality, 3 = moderate hypernasality and 4 = severe hypernasality. As recommended, analysis were made individually using stereo headphones available for the study. Raters were allowed to listen to the recordings as many times as necessary.

Data analysis

Hypernasality was expressed as a score, according to the 4 point scale. Intra- and inter-rater agreements were established for the two types of speech samples: spontaneous conversation and repeated sentences using the Kappa coefficient considering the following strength of agreement: <0 = no agreement; 0-0.19 = poor agreement; 0.20-0.39 = fair agreement; 0.40-0.59 = moderate agreement; 0.60-0.79 = substantial agreement; 0.80 to 1.00 = almost perfect agreement⁽¹⁵⁾. The intra-rater agreement coefficient was established based on the repeated analysis of 30% of the total samples (36 samples, with 18 containing spontaneous conversation and 18 containing repeated sentences). A comparison of the intra- e inter-rater agreement coefficients obtained in each step was analyzed using the Z test. Values of p<0.05 were accepted as statistically significant.

RESULTS

Intra-raters agreement

The intra-rater agreement of the degree of hypernasality obtained in the analysis of repeated sentences was significantly higher than that observed in the samples containing spontaneous conversation, as shown in Table 1. For rater 1, the Kappa coefficient significantly increased from 0.45 (moderate) to 1.00 (almost perfect), for spontaneous conversation and repeated sentences, respectively (p <0.001). For rater 2, the Kappa coefficient also increased from 0.60 to 0.74 for spontaneous conversation and repeated sentences, respectively, both interpreted as substantial, but with no significant difference (p = 0.590). As for the rater 3, there was a significant increase of the Kappa coefficient from 0.44 (moderate) to 0.92 (almost perfect) for spontaneous conversation and repeated sentences, respectively (p = 0.006).

Table 1
Statistical comparison between the intra-rater concordance indexes in perceptual analysis of hypernasality of both speech samples (spontaneous conversation and repeated sentences): percentage of concordance (%), kappa coefficient and its interpretation

Intra-rater concordance
Raters	Spontaneous conversation			Sentence repetition
	Concordance %	Kappa coefficient	Interpretation	Concordance %	Kappa coefficient	Interpretation
1	61	0.45	Moderate	100	1.00*	Almost perfect	p<0.001
2	72	0.60	Substantial	83	0.74	Substantial	p=0.590
3	61	0.44	Moderate	94	0.92*	Almost perfect	p=0.006

• Caption: *Spontaneous conversation vs Sentence repetition – Z test

Inter-rater agreement

The inter-rater agreement for both speech samples (Table 2) were 0.40 for spontaneous conversation and 0.38 for repeated sentences, indicating moderate and regular agreement, respectively. Data analysis showed no difference between the coefficients of the two stages (p = 0.970).

Table 2
Statistical comparison between the raters concordance indexes in perceptual analysis of hypernasality of both speech samples (spontaneous conversation and repeated sentences): percentage of concordance (%), kappa coefficient and its interpretation

Inter-rater concordance
Raters	Spontaneous conversation			Sentence repetition
	Concordance %	Kappa coefficient	Interpretation	Concordance %	Kappa coefficient	Interpretation
1 and 2	53	0.37	Regular	60	0.43	Moderate		p=0.628
1 and 3	62	0.48	Moderate	58	0.42	Moderate		p=0.633
2 and 3	52	0.34	Regular	50	0.31	Regular		p=0.876
1, 2 and 3	37	0.40	Moderate	37	0.38	Regular		p=0.970

When analyzed separately, the agreement between each two raters the results showed an increase of the Kappa coefficient from 0.37 (spontaneous conversation) interpreted as regular to 0.43 (repeated sentences) interpreted as moderate between raters 1 and 2, with no significant difference (p = 0.628). For raters 1 and 3, there was a slight reduction of the Kappa coefficient from 0.48 in spontaneous conversation to 0.42 in repeated sentences, both interpreted as moderate and with no significant difference (p = 0.663). The comparison between raters 2 and 3 also showed a slight reduction of the Kappa coefficient, from 0.34 for spontaneous conversation and 0.31 to repeated sentences, both interpreted as regular, and this difference was not significant (p = 0.876).

DISCUSSION

In the present study, the comparison of intra-raters agreement coefficient between the two stages showed better agreement obtained in the repeated sentences than the spontaneous conversation for the three evaluators. Statistically significant difference were verified for two of them. One can speculate that the perceptual judgment of hypernasality in spontaneous conversation is harder to analyze due to the influence of several factors, such as context, rhythm of speech, pitch and compensatory articulation. According to the literature, in the presence of other speech symptoms, it is difficult for the rater to isolate hypernasality, often leading to the ratings as more severe^(4,5,10-12). In addition, some authors believe that there isn’t always a clear distinction between the passive errors, such as hypernasality, and the compensatory articulations⁽²⁾. Based on previous analysis of the speech samples of this study, it was found that approximately 50% (30/60) of the spontaneous speech recordings in the present study presented other passive errors, such as nasal air emission and low intraoral air pressure, 35% (21/60) presented compensatory articulation and 7% (4/60) presented vocal symptoms. Repeated sentences recordings presented 30% (18/60) of other passive errors and compensatory articulation and 7% (4/60) of vocal symptoms. The fact that the samples with repeated sentences presented lower proportion of coexisting speech symptoms may have favored and thus made reliable the judgment of hypernasality in this sample.

Significant intra-raters agreement using repeated sentences and standardized words were shown in previous studies of the Institution, which ranged from substantial to almost perfect^(16,17), moderate to almost perfect^(18,19) and regular to almost perfect⁽²⁰⁾. Others studies present percentage of intra-rater agreement above 80%^(21-23). A similar result was found comparing nasalance scores with the results of perceptual speech assessment (spontaneous conversation and repeated sentences). The authors showed in the intra-rater analysis of experienced listeners, percentages of agreement ranging from 62.5% to 100% for spontaneous speech and 75% to 100% for repeated sentences⁽²⁴⁾.

It is also known that the speech material and the elicitation technique may influence the speech intelligibility score obtained from the perceptual assessment of speech and significant differences may exist between the production of a word obtained from the repetition of sentences or from the spontaneous conversation⁽¹⁴⁾. It can be speculated, then, that the elicitation of the speech sample using repetition has facilitated the identification of the hypernasality. In the case of repeated sentences, the individual that is being evaluated has a tendency to reproduce the speech similarly to the evaluator, thus performing a better control of the rhythm of speech and articulation in order to produce the correct sounds, which does not occur in the spontaneous conversation⁽⁸⁾.

Although some authors⁽¹¹⁾ advocate that spontaneous conversation is an important tool for perceptual speech assessment since it reflects the individual's daily life, the use of sentences repetition facilitates the perceptual analysis of speech once it consists in a kind of speech sample more accurate. By proposing universal parameters for documentation of speech in individuals with cleft palate, experts recommend the use of repeated sentences and single words for the purpose of perceptual judgment of hypernasality as they are comparable even between different languages with similar phonetic context⁽⁴⁾. These same authors also suggest that spontaneous speech is used for rating other characteristics than the degree of hypernasality, for example, voice disorders and acceptability and speech intelligibility.

This study also showed no significant difference between the repeated sentences and spontaneous conversation. It suggests that although the repeated sentences samples somehow favor the consistency of the judgments of the same rater, this effect is not enough to increase the agreement between the different raters. These results confirm what is already well established in the literature, i.e., that achieving a high level of agreement between different raters in the hypernasality judgment, using their own internal standards, is difficult due to its perceptual nature, characterized as a sensation and considered the most difficult to obtain high reliability^(10,25). This is because the internal standards differ between raters. Researches report that the judgments of speech symptoms made by different raters are not comparable and that experience in the assessment of individuals with cleft palate does not guarantee a high level of concordance⁽²⁵⁾. Inter-rater agreement coefficients similar to those found in this study have been verified by authors for both types of speech sample, which ranged from moderate to substantial^(13,26), moderate^(3,9,17,24), regular to moderate⁽¹⁸⁾ and regular⁽²⁰⁾.

It is noteworthy that no other study in the literature, to date, compared the ratings of hypernasality degree in different types of speech sample for the same individual. These findings are important to show that regardless of speech samples produced by the same individual (spontaneous conversation or repeated sentences), the inter-rater agreement coefficients remain fair, meaning that the type of speech sample does not improve the reliability of the judgment between different raters. This result may be explained by the type of scale used to classify hypernasality. As in most studies in the literature, the present study used an ordinal scale, which has been the most widely used both in research and in clinical practice^(27,28). However, due to the psychophysical nature of nasality, high agreement among different raters have been difficult to achieve using this method⁽²⁹⁾. This is because the scale divides the different categories of speech symptom without to quantify the magnitude of the difference between each category and listeners tend to subdivide, especially the lower end of the scale into smaller intervals⁽³⁰⁾. Thus, it is possible that this type of scale is not an effective method for hypernasality ratings, even for experienced evaluators.

Finally, the results of this study reinforce the need to adopt the constant practice of listener’s auditory training in research centers for individuals with cleft palate in order to standardize the assessment criteria and calibrate professionals in an attempt to obtain reliable and comparable results with regard to the perceptual assessment of speech symptoms.

CONCLUSION

Sentences repetition improved the intra-rater reliability of perceptual judgment of hypernasality, as the agreement in this speech samples analysis was higher. However, the speech sample had no influence on reliability among different raters.

ACKNOWLEDGEMENTS

To the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior and to Fundação de Amparo à Pesquisa do Estado de São Paulo (Process 2013/14769-4) for the financial assistance to the implementation of this study.

• Study carried out at Laboratório de Fisiologia do Hospital de Reabilitação de Anomalias Craniofaciais, Universidade de São Paulo – HRAC-USP, Bauru (SP), Brazil.
• Financial support: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) and Fundação de Amparo à Pesquisa do Estado de São Paulo - FAPESP - (Processo 2013/14769-4).

REFERÊNCIAS

• ¹
  Trindade IEK, Genaro KF, Yamashita RP, Miguel HC, Fukushiro AP. Proposta de classificação da função velofaríngea na avaliação perceptivo-auditiva da fala. Pro Fono. 2005;17(2):259-62. http://dx.doi.org/10.1590/S0104-56872005000200015 PMid:16909536.
  » http://dx.doi.org/10.1590/S0104-56872005000200015
• ²
  Britton L, Albery L, Bowden M, Harding-Bell A, Phippen G, Sell D. A cross-sectional cohort study of speech in five-years-olds with cleft palate ± lip to support development of national audit standards. Cleft Palate Craniofac J. 2014;51(4):431-51. http://dx.doi.org/10.1597/13-121 PMid:24635034.
  » http://dx.doi.org/10.1597/13-121
• ³
  John A, Sell D, Sweeney T, Harding-Bell A, Williams A. The cleft audit protocol for speech-augmented: a validated and reliable measure for auditing. Cleft Palate Craniofac J. 2006;43(3):272-88. http://dx.doi.org/10.1597/04-141R.1 PMid:16681400.
  » http://dx.doi.org/10.1597/04-141R.1
• ⁴
  Henningsson G, Kuehn DP, Sell D, Sweeney T, Trost-Cardamone JE, Whitehill TL, et al. Universal Parameters for reporting speech outcomes in individuals with cleft palate. Cleft Palate Craniofac J. 2008;45(1):1-17. http://dx.doi.org/10.1597/06-086.1 PMid:18215095.
  » http://dx.doi.org/10.1597/06-086.1
• ⁵
  Lohmander A, Willadsen E, Persson C, Henningsson G, Bowden M, Hutters B. Methodology for speech assessment in the scandcleft project – An international randomized clinical trial on palatal surgery: experiences from a pilot study. Cleft Palate Craniofac J. 2009;46(4):347-62. http://dx.doi.org/10.1597/08-039.1 PMid:19642772.
  » http://dx.doi.org/10.1597/08-039.1
• ⁶
  Lohmander A, Olsson M. Methodology for perceptual assessment of speech in patients with cleft palate: A critical review of literature. Cleft Palate Craniofac J. 2004;41(1):64-70. http://dx.doi.org/10.1597/02-136 PMid:14697067.
  » http://dx.doi.org/10.1597/02-136
• ⁷
  Genaro KF, Yamashita RP, Trindade IEK. Avaliação clínica e instrumental na fissura labiopalatina. In: Fernandes FDM, Mendes BCA, Navas ALPGP, organizadores. Tratado de fonoaudiologia. São Paulo: Roca; 2010. p. 488-503.
• ⁸
  Sell D. Issues in a perceptual speech analysis in cleft palate and related disorders: a review. Int J Lang Commun Disord. 2005;40(2):103-21. http://dx.doi.org/10.1080/13682820400016522 PMid:16101269.
  » http://dx.doi.org/10.1080/13682820400016522
• ⁹
  Brunnegård K, Lohmander A. A cross-sectional study in 10-year-old children with cleft palate: results and issues of rater reliability. Cleft Palate Craniofac J. 2007;44(1):33-44. http://dx.doi.org/10.1597/05-164 PMid:17214536.
  » http://dx.doi.org/10.1597/05-164
• ¹⁰
  Lee A, Whitehill TL, Ciocca V. Effect of listener training on the perceptual judgement of hypernasality. Clin Linguist Phon. 2009;23(5):319-34. http://dx.doi.org/10.1080/02699200802688596 PMid:19399664.
  » http://dx.doi.org/10.1080/02699200802688596
• ¹¹
  Kuehn D, Moller KT. Speech and language issues in the cleft palate population: the state of the art. Cleft Palate Craniofac J. 2000;37(4):348-1-348-35.
• ¹²
  Kummer AW. Speech evaluation for patients with cleft palate. Clin Plast Surg. 2014;41(2):241-51. http://dx.doi.org/10.1016/j.cps.2013.12.004 PMid:24607192.
  » http://dx.doi.org/10.1016/j.cps.2013.12.004
• ¹³
  Sweeney T, Sell D. Relationship between perceptual ratings of nasality and nasometry in children/adolescents with cleft palate and/or velopharyngeal dysfunction. Int J Lang Commun Disord. 2008;43(3):265-82. http://dx.doi.org/10.1080/13682820701438177 PMid:17852526.
  » http://dx.doi.org/10.1080/13682820701438177
• ¹⁴
  Johannisson TB, Lohmander A, Persson C. Assessing intelligibility by single words, sentences and spontaneous speech: a methodological study of the speech production of 10-year-olds. Logoped Phoniatr Vocol. 2014;39(4):159-68. http://dx.doi.org/10.3109/14015439.2013.820487 PMid:23906041.
  » http://dx.doi.org/10.3109/14015439.2013.820487
• ¹⁵
  Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74. http://dx.doi.org/10.2307/2529310 PMid:843571.
  » http://dx.doi.org/10.2307/2529310
• ¹⁶
  Brandão GR, Souza Freitas JA, Genaro KF, Yamashita RP, Fukushiro AP, Lauris JR. Speech outcomes and velopharyngeal function after surgical treatment of velopharyngeal insufficiency in individuals with signs of velocardiofacial syndrome. J Craniofac Surg. 2011;22(5):1736-42. http://dx.doi.org/10.1097/SCS.0b013e31822e624f PMid:21959422.
  » http://dx.doi.org/10.1097/SCS.0b013e31822e624f
• ¹⁷
  Scarmagnani RH, Barbosa DA, Fukushiro AP, Salgado MH, Trindade IET, Yamashita RP. Relationship between velopharyngeal closure, hypernasality, nasal air emission and nasal rustle in subjects with repaires cleft palate. CoDAS. 2015;27(3):267-72. PMID: 26222944.
• ¹⁸
  Barbosa DA. Resultados de fala e de função velofaríngea do retalho faríngeo e da veloplastia intravelar na correção da insuficiência velofaríngea: estudo comparativo [dissertação]. Bauru (SP): Universidade de São Paulo; 2011. 129 p.
• ¹⁹
  Oliveira ACASF, Scarmagnani RH, Fukushiro AP, Yamashita RP. The influence of listener training on the perceptual assessment of hypernasality. CoDAS. 2016;28(2):141-148. [Portuguese]. http://dx.doi.org/10.1590/2317-1782/20162015163
  » http://dx.doi.org/10.1590/2317-1782/20162015163
• ²⁰
  Ferlin F. Influência das consoantes de alta e baixa pressão intraoral sobre a nasalidade e nasalância da fala [dissertação]. Bauru (SP): Universidade de São Paulo; 2014. 82 p.
• ²¹
  Persson C, Elander A, Lohmander-Agerskov A, Söderpalm E. Speech outcomes in isolated cleft palate: impact of cleft extent and additional malformations. Cleft Palate Craniofac J. 2002;39(4):397-408. http://dx.doi.org/10.1597/1545-1569(2002)039<0397:SOIICP>2.0.CO;2 PMid:12071788.
  » http://dx.doi.org/10.1597/1545-1569(2002)039<0397:SOIICP>2.0.CO;2
• ²²
  Persson C, Lohmander A, Elander A. Speech in children with an islated cleft palate: a longitudinal perspective. Cleft Palate Craniofac J. 2006;43(3):295-309. http://dx.doi.org/10.1597/04-0711.1 PMid:16681402.
  » http://dx.doi.org/10.1597/04-0711.1
• ²³
  Lohmander A, Friede H, Lilja J. Long-term, longitudinal follow-up of individuals with unilateral cleft lip and palate after the Gothenburg primary early veloplasty and delayed hard palate closure protocol: speech outcome. Cleft Palate Craniofac J. 2012;49(6):657-71. http://dx.doi.org/10.1597/11-085 PMid:22364610.
  » http://dx.doi.org/10.1597/11-085
• ²⁴
  Brunnegård K, Lohmander A, van Doorn J. Comparison between perceptual assessments of nasality and nasalance scores. Int J Lang Commun Disord. 2012;47(5):556-66. http://dx.doi.org/10.1111/j.1460-6984.2012.00165.x PMid:22938066.
  » http://dx.doi.org/10.1111/j.1460-6984.2012.00165.x
• ²⁵
  Keuning KH, Wieneke GH, Dejonckere PH. The intrajudge reliability of the perceptual rating of cleft palate speech before and after pharyngeal flap surgery: the effect of judges and speech samples. Cleft Palate Craniofac J. 1999;36(4):328-33. http://dx.doi.org/10.1597/1545-1569(1999)036<0328:TIROTP>2.3.CO;2 PMid:10426599.
  » http://dx.doi.org/10.1597/1545-1569(1999)036<0328:TIROTP>2.3.CO;2
• ²⁶
  Brunnegård K. Evaluation of nasal speech: a study of assessments by speech-language pathologists, untrained listeners and nasometry [tese]. Umeå: Umeå University; 2008. 55 p
• ²⁷
  Brancamp TU, Lewis KE, Watterson T. The relationship between nasalance scores and nasality ratings obtained with equal appearing interval and direct magnitude estimation scaling methods. Cleft Palate Craniofac J. 2010;47(6):631-7. http://dx.doi.org/10.1597/09-106 PMid:20500059.
  » http://dx.doi.org/10.1597/09-106
• ²⁸
  Bressmann T, Sell D. Plus Ça Change: selected papers on speech research from 1964 issue of the Cleft Palate Journal. Cleft Palate Craniofac J. 2014;51(2):124-8. http://dx.doi.org/10.1597/13-310 PMid:24446923.
  » http://dx.doi.org/10.1597/13-310
• ²⁹
  Zraick RI, Liss JM. A comparison of equal-appearing interval scaling and direct magnitude estimation of nasal voice quality. J Speech Lang Hear Res. 2000;43(4):979-88. http://dx.doi.org/10.1044/jslhr.4304.979 PMid:11386483.
  » http://dx.doi.org/10.1044/jslhr.4304.979
• ³⁰
  Stevens SS. Psychophysics: introduction to its perceptual, neural and social prospects. New York: Wiley; 1975.

Publication Dates

• Publication in this collection
  7 July 2016
• Date of issue
  May-Jun 2016

History

• Received
  05 Aug 2015
• Accepted
  17 Sept 2015