Exploratory study on the multimodal analysis of the joint attention

ABSTRACT

The aim of this study was to produce exploratory multimodal analysis on joint attention. We used the ELAN tool: software with resources for temporal and special synchronization for verbal and non-verbal modalities that facilitate the visualization and annotation of interactional contexts. Excerpts of ten minutes of speech and language pathology evaluation of the child with suspected ASD and of another child with typical development matched by age group and gender, were analyzed and transcribed. Spontaneous occurrences of the child's gaze were investigated: a) Towards the interlocutor's eyes. b) Towards toys or play. c) Towards the interlocutor's eyes and toys (Joint Attention). The occurrences of look direction differed between children in a quantitative and qualitative way. The child with suspected ASD did not produce an episode of joint attention, nor did he direct his gaze to the interlocutor. There were 56 occurrences of gazing towards the toys. The typical child produced 18 occurrences of joint attention, and whenever he directed his eyes to the interlocutor's eyes, he did so with the intention of sharing the toy or play. We also observed 37 occurrences of gazing towards the toy or play. From the exploratory analysis produced by ELAN tool, it was possible to observe that there were differences in the number of occurrences and look direction between the evaluated children and to verify that the joint attention was absent in the child with ASD risk.

Keywords:
Autism Spectrum Disorder; Communication; Language; Child; Speech Language Hearing Science

RESUMO

O objetivo deste estudo foi produzir análise multimodal exploratória sobre a atenção compartilhada. Utilizou-se a ferramenta ELAN (EUDICO Language Annotator), software com recursos para sincronização temporal e especiais para modalidades verbal e não verbal, que facilitam a visualização e anotação de contextos interacionais. Foram analisados e transcritos trechos de dez minutos de sessão de avaliação fonoaudiológica de uma criança em investigação para transtorno do espectro do autismo (TEA) e outra, com desenvolvimento típico, pareadas por faixa etária e gênero. Foram investigadas e quantificadas as ocorrências espontâneas de direcionamento do olhar das crianças para: os olhos do interlocutor; os brinquedos e/ou brincadeiras; os olhos do interlocutor – brinquedos (atenção compartilhada). As ocorrências de direcionamento do olhar diferiram entre as crianças de forma quantitativa e qualitativa. A criança com suspeita de TEA não produziu episódio de atenção compartilhada, nem direcionou seu olhar para o interlocutor. Foram 56 ocorrências de direcionamentos de olhar apenas para os brinquedos. A criança em desenvolvimento típico produziu 18 ocorrências de atenção compartilhada, sendo que, em todas as vezes que direcionou o olhar para os olhos do interlocutor, o fez para compartilhar o brinquedo ou brincadeira. Observou-se, ainda, 37 ocorrências de direcionamento do olhar para o brinquedo. A partir da análise exploratória produzida pela ferramenta ELAN, foi possível observar que houve diferença em número de ocorrência e trajetória do olhar entre as duas crianças e verificar que a atenção compartilhada esteve ausente na criança com risco para TEA.

Palavras-chave:
Transtorno Autístico; Comunicação; Linguagem; Criança; Fonoaudiologia

INTRODUCTION

Joint attention is the shared focus of two people on the same object or event, to share a common experience. It is observed when a person directs their gaze to another individual, looks at or points to an object or event, and then turns their gaze, again, back to the individual⁽¹⁾. Joint attention difficulties are among the strongest predictors of developmental disabilities, especially in regard to Autism Spectrum Disorder (ASD)^(1-3).

There is a consensus that the earlier and more timely the intervention, the better its effectiveness. The benefits of an early intervention clearly rely on prompt detection, requiring multidisciplinary teams of professionals that know how to accurately evaluate children and plan appropriate interventions suitable to the child’s individual needs.

In this sense, the diagnosis and subsequent investigation of the ASD warning signs are essential steps, especially in the very early childhood stages. Reduced eye contact, lack of eye gaze, joint attention, and use of gestures are considered not only warning signs, but also potential diagnostic criteria for ASD.

Several studies have been produced for mapping and documenting such disabilities, using the Eye Tracking software ^(4-9). The application of Eye Tracking, in a research context, began two decades ago, with studies on gaze trajectory in adults with high-functioning ASD. More recently, its use has enabled the recording of the decline or absence of discourse markers throughout the development of children diagnosed with ASD during very early childhood^(4-9). Although Eye Tracking is considered to be a notorious tool for ASD screening^(8,9), it is known that its use in public health services in Brazil is still limited.

Nevertheless, ELAN (EUDICO Language Annotator)^(10,11) is a software that has spatio-temporal synchronization and coordination features for different modalities - verbal and non-verbal -, helping with the visualization and annotation of the communication skills triggered in situations of social interaction. It is free software and, albeit requiring some training, it can be a useful resource for multidisciplinary teams in a naturalistic scenario for both evaluation and treatment.

This study aimed to produce an exploratory multimodal data analysis on joint attention.

The hypothesis raised herein is that an exploratory multimodal analysis can support the speech-language assessment, providing consistent data on interactional discourse markers, such as joint attention.

CLINICAL CASE PRESENTATION

This is a case study of a child going through a multidisciplinary diagnostic investigation at the Center for Speech-Language Pathology Language Investigation of Children and Adolescents with Autism Spectrum Disorder - NIFLINC-TEA (an acronym for “Núcleo de Investigação Fonoaudiológica de Linguagem da Criança e Adolescente no Transtorno do Espectro do Autismo”) of the Department of Speech-Language Pathology and Audiology of the Federal University of São Paulo - UNIFESP, due to the child’s risk for ASD, compared with another child with typical development, matched by age group and gender.

The study was approved by the institution’s Research Ethics Committee in the Brazilian CEP-CONEP system, registered under the number: 0715/2019. The parents signed the Informed Consent Form, thus consenting to the research being carried out and its results disseminated.

Child 1 – M.B., 21 months of age, male

The complaint made by the family referred to speech regression.

According to the parents, the production of the first words occurred at 12 months, but around 16 months, they started to notice a decrease and subsequent cessation of the emission of words that were already established in the child's repertoire (names of people and objects, action verbs). At the same time, they noticed a reduction in eye contact, unconscious responses when they called him by name as well as a lack of interest in toys and games.

As stated by the parents, when wanting something that is out of his reach, M.B. does not point, only leads the adult's hand towards the desired object. He recognizes his older sister but shows no interest in interacting or sharing games or toys with her. He has a preference for stacking or lining up objects and, more recently, has been showing interest in numbers and letters.

As for the congenital data, the mother reported that the pregnancy was planned. She underwent prenatal care and at five months of gestation, a clinical manifestation of preeclampsia was identified, requiring a cesarean delivery. The child was born extremely premature, at 27 weeks old, weighing 680 grams. He was admitted to the neonatal Intensive Care Unit (NICU) for 30 days, evolving into a semi-intensive care unit for two months. He had speech-language therapy during the first month of life, for the development of the suckling movements. He used a bottle until 18 months old. Although currently able to chew different food textures, the family mentioned some food selectivity (preference for certain textures and food consistencies).

Regarding the neuropsychomotor development, the mother reported that M. was able to hold up his head at 6 months of age and started walking only at 1 year and 9 months. She did not mention the need for physiotherapy treatment.

There were no reports of cases of mental disorders in the family.

At 18 months, in a pediatric follow-up appointment, the M-CHAT scale (Modified Checklist for Autism in Toddlers)⁽¹²⁾ was applied, resulting in an 18-point score that indicated the need for a multidisciplinary evaluation, due to the high risk of autism spectrum disorder.

In the speech-language pathology evaluation, there were distinct regressions in all behaviors that regulate and sustain interaction and social communication, that is: reduction of eye contact, joint attention, use of facial expressions, and social responsiveness. He had difficulty interacting with the evaluators and engaging with the proposed activities.

Restricted communicative intention as well as the absence of use of indicative and/or representative gestures were observed. His speech was limited to the naming of the letters along with the numbers that were attached to toys and that were read spontaneously.

He was unresponsive to his name, as well as to the formal testing of verbal comprehension of the Language Development Assessment - LDA⁽¹³⁾.

On the subject of playful exploration, the child limited himself to removing the toys one by one from the box; sometimes he lined them up, and sometimes he fixed his attention on the numbers and letters in said toys. He did not play in any imitative, functional, or symbolic way, even with the encouragement and exemplification given by the evaluators.

The presence of stereotypical movements (hand flapping) in moments of joy was noted.

During the application of the Bayley Scales of Infant and Toddler Development⁽¹⁴⁾, the child showed extremely low scores in the linguistic and social-emotional competencies, low adaptive behavior; borderline motor skills, and on average, cognitive control.

The child underwent a multidisciplinary diagnostic investigation and the family participated in a parent orientation program, which focused on the development of interaction and social communication skills. At 3 years of age, the multidisciplinary diagnosis of autism spectrum disorder was confirmed.

Child 2 – X.Y., 21 months of age, male

The mother reported that the pregnancy was planned. She underwent prenatal care and there were no reports of prenatal, perinatal, or postnatal complications. The child was born at term and delivery was normal.

Regarding the neuropsychomotor development, the mother reported that X.Y. was able to hold up his head at 3 months, sit at 6 months, and started walking at 12 months old, the same period in which the emission of the first words was observed. At 18 months the child started juxtaposing words.

There was no mention in the anamnesis concerning cases of autism spectrum disorder or other mental disorders in the family.

During behavioral observation, he showed appropriate social engagement, using the multiple non-verbal behaviors that regulate interaction and social communication. His playful exploration proved to be functional and symbolic.

In the formal testing of verbal production and comprehension, the results obtained in the application of the Language Development Assessment – LDA⁽¹³⁾ were within the expected parameters for his age group.

Both children were not yet enrolled in kindergarten.

For the exploratory analysis of joint attention, we selected ten minutes excerpts of a speech-language evaluation session, in which the children's spontaneous playful exploration of toys was observed. The sessions were recorded with the camcorder positioned in front of the child, with sufficient distance to capture the movements of both the child and the speech-language pathologist (evaluator). Afterward, the video was inputted, analyzed, and transcribed by the ELAN tool^(10,11). This tool was developed by psycholinguists from the Max Planck Institute and has spatio-temporal synchronization and coordination features for different modalities - verbal and non-verbal -, helping with the visualization and annotation of the communication skills triggered in situations of social interaction.

We investigated the number of spontaneous occurrences for each child's gaze towards:

1. a

   the eyes of the interlocutor;

2. b

   the toys and/or games;

3. c

   the eyes of the interlocutor – to the toy – back to the eyes of the interlocutor (joint attention).

Figure 1 shows the comparative analysis of the gaze direction occurrences for both children.

Figure 1
Amount of gaze direction of both children

The gaze direction occurrences differed between the children in a quantitative and qualitative way.

Figure 2 shows an example of an ELAN screenshot regarding the analysis of children at risk for ASD.

Figure 2
ELAN (EUDICO Linguistic Annotator) Screenshot in the Analysis of Child at Risk for Autism Spectrum Disorder

DISCUSSION

Joint attention is the focus of two people on the same event or object. This ability emerges during the first year of life and is considered to be a warning sign for ASD risk when absent or reduced in the child^(1-3).

Recent studies on joint attention have been designed using the Eye Tracking software ^(4-9) in experimental situations. It is acknowledged as a promising ^tool, and it has been used to document the absence of interactional discourse markers in children diagnosed with ASD during very early childhood^(4-6). However, its application is still limited in Brazil.

Furthermore, ELAN^(10,11) is a software that helps with the visualization and annotation of the communication skills triggered in situations of social interaction. It is multimodal, as it analyzes verbal and non-verbal behaviors. It is free software and, albeit requiring some training, it can be a useful resource for multidisciplinary teams in a naturalistic scenario for both evaluation and treatment.

For this study, ELAN^(10,11) was used to transcribe excerpts of the speech-language evaluation sessions with two children of the same age and gender, that showed different developmental processes.

Although the total number of gaze direction occurrences was similar for both children, they differed in their trajectory and purpose.

The typical child produced a total of 55 occurrences, 18 of them with joint attention, given that every time he directed his gaze to the interlocutor's eyes, he did so to share the toy or play. The 37 remaining occurrences of gaze direction were related to toys and/or games.

The child at risk for ASD, on the other hand, did not present episodes of joint attention, nor did it direct its gaze to the interlocutor. All 56 of his occurrences were in the direction of the toys. Therefore, the detailed visualization of the video allowed us to confirm the atypical behavior evidenced throughout the speech-language pathology evaluation process.

Thus, the ELAN tool^(10,11) was able to provide endorsement with its spatio-temporal synchronization and coordination features, proving the atypical trajectory of gaze direction as well as the absence of joint attention in the case of the child at risk for ASD.

FINAL COMMENTS

With the aid of the exploratory analysis developed by the ELAN tool^(10,11), it was possible to observe that there was a difference in the number of occurrences as well as the trajectory of gaze among the evaluated children and to verify that joint attention was absent in the child at risk for ASD.

• Study carried out at Departamento de Fonoaudiologia, Universidade Federal de São Paulo – UNIFESP – São Paulo (SP), Brasil.
• Funding: National Council for Scientific and Technological Development – CNPq, Brasil, Processes n. 421937/2018-1 and n. 405091/2018-4. The Research Support Foundation of the State of São Paulo – FAPESP, Process n.2018/07565-7.

REFERÊNCIAS

• ¹
  Murza KA, Schwartz JB, Hahs-Vaughn DL, Nye C. Joint attention interventions for children with Autism Spectrum Disorder: a systematic review and meta-analysis. Int J Lang Commun Disord. 2016;51(3):236-51. https://dx.doi.org/10.1111/1460-6984.12212. PMid:26952136.
• ²
  Klin A, Klaiman C, Jones W. Reducing age of autism diagnosis: developmental social neuroscience meets public health challenge. Rev Neurol. 2015;60(Suppl 1):S3-11. PMID: 25726820.
• ³
  Tamanaha AC, Chiari BM, Perissinoto J. A eficácia da intervenção terapêutica fonoaudiológica nos Distúrbios do Espectro do Autismo. Rev CEFAC. 2015;17(2):552-8. https://dx.doi.org/10.1590/1982-021620156314
  » https://dx.doi.org/10.1590/1982-021620156314
• ⁴
  Mundy P. A review of joint attention and social-cognitive brain systems in typical development and autism spectrum disorder. Eur J Neurosci. 2018;47(6):497-514. https://dx.doi.org/10.1111/ejn.13720 PMid:28922520.
  » https://dx.doi.org/10.1111/ejn.13720
• ⁵
  Klin A, Schultz S, Jones W. Social visual engagement in infants and toddlers with autism: early developmental transitions and a model of pathogenesis. Neurosci Biobehav Rev. 2015;50:189-203. https://dx.doi.org/10.1016/j.neubiorev.2014.10.006. PMid:25445180.
• ⁶
  Constantino JN, Kennon-McGill S, Weichselbaum C, Marrus N, Haider A, Glowinski AL, et al. Infant viewing of social scenes is under genetic control and atypical in autism. Nature. 2017;547:340-4. https://dx.doi.org/10.1038/nature22999.
• ⁷
  Dindar K, Korkiakangas T, Laitila A, Kärnä E. An interactional “live eye tracking” study in autismo spectrum disorder: combining qualitative and quantitative approaches in the study of gaze. Qual Res Psychol. 2017;14(3):239-65. https://doi.org/10.1080/14780887.2017.1290174
  » https://doi.org/10.1080/14780887.2017.1290174
• ⁸
  Moriuchi JM, Klin A, Jones W. Mechanism of diminished attention to eyes in autism. Am J Psychiatry. 2017;174(1):26-35. https://dx.doi.org/10.1176/appi.ajp.2016.15091222. PMid:27855484.
• ⁹
  Sifre R, Olson L, Gillespie S, Klin A, Jones W, Shultz S. A longitudinal investigation of preferential atention to biological motion in 2 to 24 months old infants. Sci Rep. 2018;8(1):2527. http://dx.doi.org/10.1038/s41598-018-20808-0 PMid:29410484.
  » http://dx.doi.org/10.1038/s41598-018-20808-0
• ¹⁰
  Wittenburg P, Brugman H, Russel A, Klassmann A, Sloetjes H. ELAN: a professional framework for multimodality research. In: Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006); 2006; Genoa, Italy. France: European Language Resources Association (ELRA); 2006. p. 1156-9.
• ¹¹
  Cruz FM, Ostermann AC, Andrade DNP, Frezza M. O trabalho técnico-metodológico e analítico com dados interacionais audiovisuais: a disponibilidade de recursos multimodais nas interações. Delta. 2019;35(4):e2019350404. https://dx.doi.org/10.1590/1678-460X2019350404
  » https://dx.doi.org/10.1590/1678-460X2019350404
• ¹²
  Losapio MF, Ponde MP. Tradução para português da Escala MCHAT para rastreamento precoce do autismo. Rev Psiquiatr Rio Gd Sul. 2008;30(3):221-9. http://dx.doi.org/10.1590/S0101-81082008000400011
  » http://dx.doi.org/10.1590/S0101-81082008000400011
• ¹³
  Menezes MLN. A construção de um instrumento para avaliação do desenvolvimento da linguagem – ADL: idealização, estudo piloto para padronização e validação [tese]. Rio de Janeiro: Fundação Oswaldo Cruz; 2003.
• ¹⁴
  Madaschi V, Mecca TP, Macedo EC, Paula CS. Bayley III Scales of Infant and Toddler Development: transcultural adaptation and psychometric properties. Paidéia. 2016;26(64):189-97. https://doi.org/10.1590/1982-43272664201606
  » https://doi.org/10.1590/1982-43272664201606

Publication Dates

• Publication in this collection
  25 July 2022
• Date of issue
  2022

History

• Received
  23 Nov 2020
• Accepted
  26 May 2022