Instrument to evaluate the perception of minimal contrasts in Chilean Sign Language – reliability evidence

ABSTRACT

Purpose

Obtain evidence of the test reliability to evaluate the perception of minimum contrasts in Chilean Sign Language (LSCh).

Methods

Ten deaf children and adolescents aged between 7 and 14 years participated in this study. They were evaluated with the test of perception of minimal contrasts in LSCh. The test was reapplied 11 and 14 days after the first application (test-retest reliability). Spearman's Rho correlation was performed. During the first application, authorization was requested from the parents of the children and adolescents to record the responses of the participants so that another evaluator could re-score the protocols, in order to obtain inter-rater reliability. First-order agreement coefficient (AC1) Gwet’s was used for data analysis.

Results

Test-retest obtained a strong and significant correlation (Rho= 0.741; p=0.014). The concordance values obtained inter-rater vary between 0.962 and 1 (p<0.001), indicating that the test presents almost perfect concordance.

Conclusion

The minimum pairs perception test in LSCh presents satisfactory test-retest and inter-rater reliability.

Keywords:
Evaluation; Sign Language; Reliability; Child; Deafness

RESUMEN

Objetivo

Obtener evidencias de confiabilidad de la prueba para evaluar la percepción de los contrastes mínimos en Lengua de Señas Chilena (LSCh).

Método

Participaron 10 niños y adolescentes Sordos con edades entre los 7 y 14 años, que fueron evaluados con la prueba de percepción de los contrastes mínimos en LSCh. En un primer momento se les aplicó la prueba, y entre 11 y 14 días después se les reaplicó nuevamente (confiabilidad test - retest). Para analizar los datos, fue realizada la correlación Rho de Spearman. Durante la primera aplicación se solicitó autorización a los padres de los niños y adolescentes para grabar las respuestas de los participantes para que otro evaluador pudiese repuntuar los protocolos, con el fin de obtener la confiabilidad interevaluador. Para el análisis de los datos se utilizó el cálculo estadístico first-order agreement coefficient (AC1) de Gwet.

Resultados

En la confiabilidad test - retest se obtuvo una correlación fuerte y significativa (Rho= 0,741; p=0,014). En la confiabilidad interevaluador, los valores de concordancia obtenidos varían entre 0,962 a 1 (p<0,001), indicando que la prueba presenta concordancia casi perfecta.

Conclusión

La prueba de percepción de pares de mínimos en LSCh presenta confiabilidad test - retest e interevaluador satisfactoria.

Descriptor
es: Evaluación; Lengua de Señas; Confiabilidad; Sordera; Niño

INTRODUCTION

Deaf people are a heterogeneous population(¹). Both children and adolescents have variability in sign language proficiency(²). Sometimes, sign language is not the first language to which the child is exposed, which represents a great challenge when it comes to evaluation. Approximately, 90-95% of deaf children have hearing parents who do not know sign language, which often delays the child's linguistic development since the priority is the acquisition of speech(³). On the other hand, deaf children of deaf parents have contact with sign language from birth and therefore a greater possibility of acquiring sign language from an early age.

In recent years, there has been great interest by researchers in the creation of instruments that measure the development of visual language in deaf children(⁴). The evaluation of a child's language requires adequate measures that are appropriate to their age, language, and culture(⁵). There are few valid and reliable evaluations available for deaf children, especially those that evaluate language among deaf children in their first 5 years of life(⁶,⁷). In addition, a small number of evaluations are listed as available to measure the developmental skills of sign language in preschool children(⁴). The above contrasts with the growing interest in the evaluation of language disorders in deaf children who use sign language as their first language(⁴,⁸). Many language assessment instruments developed for deaf people are adaptations of instruments made to assess language in hearing people. In this way, they do not reflect the real needs of the deaf person(⁷). It is possible to identify some reasons for the scarcity of these instruments, such as the heterogeneity of the population and sign language, and the low access to native deaf people(⁹).

Currently, in Chilean reality, evaluation instruments built for hearing people are used to evaluate deaf people. For this reason, the construction of appropriate tests is required to evaluate language in the sign language (visuospatial) modality.

There are tests with psychometric properties but they are in other languages not in Spanish, for example: the British Sign Language Vocabulary Tasks(¹⁰); the American Sign Language Sentence Repetition Test(¹¹); The Visual Communication and Sign Language Checklist(¹²). Also, in Brazilian Sign Language (LIBRAS), an instrument was developed to evaluate the mastery of minimal contrasts(¹³) which serves to know the performance of deaf children in the recognition and perception of minimal contrast pairs in LIBRAS. Furthermore, the development of the perception of minimal contrasts in LIBRAS was studied in hearing children of deaf parents(¹⁴).

The evaluation of the minimum contrasts allows us to know the perception of children and adolescents of the different training parameters of the language. Furthermore, it allows us to verify some aspects of the development of these parameters in children(¹³), which can influence communication as a whole. It is important to time exposure to sign language since it makes sense in the evaluation of the perception of minimal contrasts in LIBRAS. The longer the time of exposure to sign language, the better the performance in the perception of minimum contrasts(¹⁴).

The minimum contrasts perception evaluation test was adapted to Chilean Sign Language (LSCh)(¹⁵) and is based on Vargas et al.(¹³). It consists of 24 items distributed as follows: 7 for Handshape, 5 for location, 8 for movement and 4 for orientation. Among the 24 pairs, 2 items were selected as examples (January-February and young-suffer). The evaluation went through a content validation process in 7 stages: 1. Collection of minimal contrast pairs in LSCh; 2. Analysis by specialist judges (two native deaf people and two hearing people who are LSCh interpreters); 3. Minimum contrast pair design; 4. Analysis of non-specialist judges (six deaf participants between 7 and 14 years old); 5. Test preparation; 6. Video recording; 7. Pilot Study. These stages allowed us to present satisfactory content validity.

Despite being a substantial advance and becoming the first test to evaluate minimal contrasts in Chilean sign language and having good validity indices, more reliable psychometric studies are needed. From that, this research aims to obtain evidence of the reliability of the minimum contrasts’ perception test in LSCh.

METHOD

This study is linked to a research project duly registered and approved by the Research Ethics Committee of the Federal University of Santa María, RS, Brazil, page number 3,022,041. Authorization was requested from those responsible for the children and adolescents who participated in the research following the standards determined by the National Health Council in its resolution 510/16 through the signing of the Informed Consent Form (ICF).

Participants

Ten deaf children and adolescents who attend a special school for deaf people and who do not have any other diagnosis participated in this study. There were four boys and six girls between 7 and 14 years old at the time of the evaluation (M=11.6 years). The participants were recruited by non-probabilistic convenience sampling, all of whom were from the city of Talca, Chile. The inclusion criteria of this research for children and adolescents were being an LSCh user, between 6 and 18 years old, and having a diagnosis of severe or profound bilateral hearing loss. The exclusion criteria were having some other type of observable deficit (neurological injury, syndromes, uncorrected visual deficit). Data were collected through an interview which was applied to the parents or caregivers of children and adolescents participating in this study. The interview asked about the age of exposure to sign language by children and adolescents, and the sample average of 5 years. Finally, two speech-language therapist evaluators, LSCh interpreters recognized by the Ministry of Education, were recruited for inter-rater analysis.

Test-retest reliability

To establish test-retest reliability, the minimum contrasts perception test in LSCh was fully applied to 10 deaf children and adolescents at two different times. The time between the test and the retest varied between 11 and 14 days.

The evaluator in charge showed the booklet containing the images of the minimal contrast pairs to deaf children and adolescents. Additionally, he showed them a video where the instructions in LSCh were delivered by an interpreter. The deaf child or adolescent were instructed to watch the video, understand the instructions given, and answer which is the pair of minimum contrast that the interpreter represented, indicating the chosen alternative by pointing his or her finger in the booklet. Correct answers were assigned a score of 1 and incorrect answers received a score of 0.

For data analysis, Spearman's Rho correlation coefficient was used, correlating the total score of the test with the score of the retest. Values between 0.31 and 0.5 were considered weak; between 0.51 and 0.7 moderate; between 0.71 and 0.9 strong, and greater than 0.9 very strong(¹⁶).

Interrater reliability

During the first application of the minimum contrast perception test in LSCh to the 10 deaf children and adolescents, we requested permission from the parents to record the procedure. We used a professional camera, and the responses of the 10 deaf children and adolescents were recorded. The moment in which they wrote down the answer on the paper sheet was recorded and after a month, another evaluator reanalyzed the videos and rescored the protocols. To verify the agreement between the judges, we considered the statistical calculation first-order agreement coefficient (AC1) of Gwet(¹⁷).

RESULTS

Table 1 shows the descriptive data obtained in the test-retest analysis considering the training parameter of the items and the correlation test.

Table 1
Test–retest correlation analysis

Item	Training parameter	Test M(SD)	Retest M(SD)	Rho	p-value
Cheese – celebrate	Handshape/6	5.6 (0.70)	5.5 (0.71)	0.741	0.014
Wednesday – Friday
Turtle – Snail
Cow - Bull
Tree – December
Duck – Chicken
Notebook – Book	Movement/7	5.2 (1.40)	6.6 (0.70)
Bee – Fly
Backpack – Jacket
Pencil – Gun
Llama – Giraffe
Near - far
Walk – Jump
Example – sign language	Orientation/4	3.4 (0.97)	3.7 (0.48)
Thursday – Play
How? - Busy
Positive – Negative
Nice – Son	location/5	3.7 (1.25)	4.6 (0.70)
Wash – Caress
Butter – Paint
Orange - fruit
Yellow-green

• Caption: M = Media; SD = Standart desviation.

The results of the test-retest reliability and the correlation coefficient were strong, positive, and significant (Table 1). A better score is observed for children in the retest. Table 2 shows the inter-rater reliability results for each training parameter.

Table 2
Interrater reliability results

Training parameter	Gwet's AC1	p-value
Handshape/6	0.962	<0.001
Movement/7	0.977	<0.001
Orientation/ 4	0.966	<0.001
Location/5	1	<0.001

Table 2 shows that the agreement values obtained vary between 0.962 to 1 (p<0.001), indicating that the test has almost perfect agreement.

DISCUSSION

The evaluation of language in the visuospatial modality (sign language) has become a challenge for researchers, evaluators, and clinicians(¹). Few tests have psychometric properties (reliable and valid) to evaluate deaf children who communicate through sign language(⁷). From that, the objective of this research was to obtain evidence of the reliability of the minimum contrast perception test in the LSCh.

In this investigation, satisfactory reliability results were observed for the instrument to Evaluate the Perception of Minimum Contrasts in Chilean Sign Language. Reliability is one of the main quality criteria of an instrument since it means that it is capable of reproducing a result stably over time or from different examiners(¹⁸,¹⁹). Also, the instrument managed to go through a rigorous adaptation and validation process, where linguistic and cultural characteristics and differences were considered for adaptation to the LSCh(²⁰).

The test-retest reliability showed a strong association between the results of the test and those of the retest, which demonstrates the temporal stability of the test(¹⁹,²¹). Better performance of children is observed in the retests. Such a result may be related to learning/memory since the retest was carried out in a short period (¹⁸), or even to the acquisition of the training parameter. The inter-rater analysis (Table 2) shows almost perfect agreement, indicating precision when it involves different examiners of the same subject.

The results obtained in this research are a contribution to the Chilean deaf community, as it becomes the first instrument to evaluate minimal contrasts in visuospatial modality (sign language) with validity and reliability. However, more studies of construct validity, criterion validity, sensitivity, and specificity are necessary.

Understanding how deaf children and adolescents perceive minimal contrasts can help with aspects such as the clinician's understanding of how they are acquiring, understanding, and executing signs. The perception of minimal contrasts that change the composition of a sign is important for the acquisition of sign language(¹³). Therefore, the instrument investigated here is of utmost importance for people who research this population.

As a limitation of this research, we observed that the low number of participants was because it was carried out in the context of the COVID-19 pandemic, and the sanitary conditions in Chile at that time were at critical levels. This is why we recommend that future studies expand the sample of both deaf children and adolescents and judges.

CONCLUSION

The test to evaluate the perception of minimal contrasts in LSCh has good test-retest reliability indices and is accurate for analysis involving different evaluators, presenting satisfactory inter-rater reliability. It is a replicable and consistent instrument, that is, reapplication on the same subject produces similar results.

• Study conducted at Universidad de Talca – UTALCA - Talca (VII región del Maule), Chile.
• Financial support: nothing to declare.

REFERENCIAS

• ¹
  Mayer C, Trezek BJ. Literacy outcomes in deaf students with cochlear implants: current state of the knowledge. J Deaf Stud Deaf Educ. 2018;23(1):1-16. http://dx.doi.org/10.1093/deafed/enx043 PMid:29040702.
  » http://dx.doi.org/10.1093/deafed/enx043
• ²
  Mann W, Roy P, Morgan G. Adaptation of a vocabulary test from British Sign Language to American Sign Language. Lang Test. 2016;33(1):3-22. http://dx.doi.org/10.1177/0265532215575627
  » http://dx.doi.org/10.1177/0265532215575627
• ³
  Corina DP, Farnady L, LaMarr T, Pedersen S, Lawyer L, Winsler K, et al. Effects of age on American Sign Language sentence repetition. Psychol Aging. 2020;35(4):529-35. http://dx.doi.org/10.1037/pag0000461 PMid:32271068.
  » http://dx.doi.org/10.1037/pag0000461
• ⁴
  Henner J, Novogrodsky R, Reis J, Hoffmeister R. Recent issues in the use of signed language assessments for diagnosis of language disorders in signing deaf and hard of hearing children. J Deaf Stud Deaf Educ. 2018;23(4):307-16. http://dx.doi.org/10.1093/deafed/eny014 PMid:29767737.
  » http://dx.doi.org/10.1093/deafed/eny014
• ⁵
  Graham PJ, Shuler-Krause E. Building strong foundations for educational achievement: language assessments in early childhood education for deaf and hard of hearing children. Psychol Sch. 2021;57(3):418-25. http://dx.doi.org/10.1002/pits.22333
  » http://dx.doi.org/10.1002/pits.22333
• ⁶
  Allen TE, Morere DA. Psychometric characteristics of the Visual Communication and Sign Language Checklist. J Deaf Stud Deaf Educ. 2022;27(3):297-309. http://dx.doi.org/10.1093/deafed/enac011
  » http://dx.doi.org/10.1093/deafed/enac011
• ⁷
  Haug T, Mann W. Adapting tests of sign language assessment for other sign languages--a review of linguistic, cultural, and psychometric problems. J Deaf Stud Deaf Educ. 2008;13(1):138-47. http://dx.doi.org/10.1093/deafed/enm027 PMid:17569751.
  » http://dx.doi.org/10.1093/deafed/enm027
• ⁸
  Quinto-Pozos D, Singleton JL, Hauser PC. A case of specific language impairment in a deaf signer of American sign language. J Deaf Stud Deaf Educ. 2017;22(2):204-18. http://dx.doi.org/10.1093/deafed/enw074
  » http://dx.doi.org/10.1093/deafed/enw074
• ⁹
  Mann W, Haug T. 5. Mapping out guidelines for the development and use of sign language assessments: some critical issues, comments and suggestions. In: Quinto-Pozos D, editor. Multilingual aspects of signed language communication and disorder. Bristol: Multilingual Matters; 2014. p. 123-39. https://doi.org/10.21832/9781783091317-008
  » https://doi.org/10.21832/9781783091317-008
• ¹⁰
  Mann W, Marshall C. Investigating deaf children’s vocabulary knowledge in British sign language: vocabulary development in sign language. Lang Learn. 2012;62(4):1024-51. http://dx.doi.org/10.1111/j.1467-9922.2011.00670.x [Internet]
  » http://dx.doi.org/10.1111/j.1467-9922.2011.00670.x
• ¹¹
  Supalla T, Hauser PC, Bavelier D. Reproducing American Sign Language sentences: cognitive scaffolding in working memory. Front Psychol. 2014;5:859. http://dx.doi.org/10.3389/fpsyg.2014.00859 PMid:25152744.
  » http://dx.doi.org/10.3389/fpsyg.2014.00859
• ¹²
  Simms L, Baker S, Clark MD. The standardized visual communication and sign language checklist for signing children. Sign Lang Stud. 2013;14(1):101-24. http://dx.doi.org/10.1353/sls.2013.0029
  » http://dx.doi.org/10.1353/sls.2013.0029
• ¹³
  Vargas DZ, Mezzomo CL, Kessler TM. A elaboração de um instrumento para investigar o domínio da percepção dos contrastes mínimos na língua brasileira de sinais. CoDAS. 2017;29(4):e20160234. http://dx.doi.org/10.1590/2317-1782/20172016234 PMid:28746464.
  » http://dx.doi.org/10.1590/2317-1782/20172016234
• ¹⁴
  Vargas DZ, Mezzomo CL, Kessler TM. O desenvolvimento da percepção dos contrastes mínimos na língua brasileira de sinais em um grupo de Codas. Rev CEFAC. 2016;18(4):835-42. http://dx.doi.org/10.1590/1982-021620161842016
  » http://dx.doi.org/10.1590/1982-021620161842016
• ¹⁵
  González CF. Instrumento de evaluación de los contrastes mínimos en lengua de señas - adaptación a la lengua de señas Chilena [tesis]. Santa Maria: Universidade Federal de Santa Maria; 2022.
• ¹⁶
  Mukaka MM. Statistics corner: a guide to appropriate use of correlation coefficient in medical research. Malawi Med J. 2012;24(3):69-71. PMid:23638278.
• ¹⁷
  Gwet K. Handbook of inter-rater reliability. 5th ed. Vol. 1: Analysis of Categorical Ratings. Maryland: AgreeStat Analytics; 2021.
• ¹⁸
  Friberg JC. Considerations for test selection: how do validity and reliability impact diagnostic decisions? Child Lang Teach Ther. 2010;26(1):77-92. http://dx.doi.org/10.1177/0265659009349972
  » http://dx.doi.org/10.1177/0265659009349972
• ¹⁹
  de Souza AC, Alexandre NMC, Guirardello EB, de Souza AC, Alexandre NMC, Guirardello EB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saude. 2017;26(3):649-59. http://dx.doi.org/10.5123/S1679-49742017000300022 PMid:28977189.
  » http://dx.doi.org/10.5123/S1679-49742017000300022
• ²⁰
  Sarıibrahim Astepe B, Köleli I. Translation, cultural adaptation, and validation of Australian pelvic floor questionnaire in a Turkish population. Eur J Obstet Gynecol Reprod Biol. 2019;234:71-4. http://dx.doi.org/10.1016/j.ejogrb.2019.01.004 PMid:30665079.
  » http://dx.doi.org/10.1016/j.ejogrb.2019.01.004
• ²¹
  Fernandez P, Pertega DP. Pruebas diagnósticas. Cad Aten Primaria. 2003;10:120-4.

Publication Dates

• Publication in this collection
  04 Dec 2023
• Date of issue
  2023

History

• Received
  06 July 2022
• Accepted
  02 Aug 2023