Analysis of softwares for emotion recognition in children and teenagers with autism spectrum disorder

ABSTRACT

Purpose:

to investigate the use of softwares for emotion recognition in children and teenagers with Autism Spectrum Disorders (ASD).

Methods:

an integrative review of the literature with scientific papers published from 2012 to 2017 indexed in Periódico Capes, Science Direct, and PudMed; combined descriptors: autism AND emotion AND software; autism AND emotion recognition AND software. Inclusion criterion was the use of software related to emotion recognition in children and teenagers with ASD, up to 18 years old. Review papers and those using robots were excluded.

Results:

ten international papers were reviewed. The most used emotional expressions were “happiness”, “fear”, “anger”, “disgust”, “sadness”, and “surprise”. Ten software programs were described: Emotion Recognition Task (1), Cambridge Mindreading Face-Voice Battery for Children (3), Mind Reading (2), Mood Maker (1), Virtual-Reality Emotion Sensitivity Test (1), FaceSay (1), Penn Emotion Recognition (1), FaceMaze Game (1), Computer Emotion Recognition Toolbox (CERT) (1), and Emotiplay (1).

Conclusion:

studies with software programs focused on ASD intervention allow future research efforts in the diagnosis and intervention of this disorder.

Keywords:
Autism; Emotions; Software

Introduction

Autism Spectrum Disorders (ASD) are neurodevelopmental disorders characterized by restricted and repetitive behavioral patterns, impairment of social interaction and verbal and nonverbal communication such as facial expressions, gestures, and eye contact, as well as socio-emotional reciprocity deficits present since the beginning of childhood. Other possible traits are echolalia, impairment of functional language use, aversion to physical contact, and stereotypies, among others¹. Thus, impairments in social interaction and nonverbal communication, as well as facial expressions of emotions, have been included among ASD diagnostic criteria and they are often described as critical for the social difficulties of individuals with ASD².

Ever since Leo Kanner described autism as a biological disorder in 1943, many more cases have been observed and a multitude of theories have been proposed to explain the various manifestations of ASD³. One of such is the Theory of Mind, which is used to describe individuals with ASD who are unable to recognize mental and emotional states in others and themselves^4,5 and therefore struggle to assign meaning, emotions, desires, and intentions with the interlocutors.

Several studies have shown the ability to recognize emotion is compromised in individuals with ASD^2,6-8 and this refers to the ability to identify and recognize the different types of emotions as presented in multiple modalities (i.e., face, body, and voice)⁹. This ability is essential for interpersonal relations and is an important element for developing empathy and, consequently, communication and social interaction skills^6,10.

In this perspective, studies have been proposed for individuals with ASD to recognize their mental states, more specifically in the development and application of computational tools that involve emotion recognition through facial and vocal expressions^8,11-15.

Additionally, computers are created to analyze, build systems, and assist in understanding and preventing unexpected behaviors⁶, since they may be designed according to each person's specific interests and may be used routinely¹⁶. Hence, this tool has become an ideal way to support individuals with ASD in both high and low functioning cases¹³.

In the United States, one in every 59 individuals has ASD¹⁷. In South Korea, 2.64% of school-age individuals have ASD, which amounts to one in every 38 children¹⁸. In Brazil, despite the scarcity of epidemiological studies, 1.5 million individuals are estimated to have ASD¹⁹, although diagnosing them is still difficult due to several factors such as the complexity of the Brazilian public health service, which does not have specialized care centers to meet the demand for referrals of children at risk, thus, generating late identification at around five or six years of age²⁰.

Given that emotional reciprocity is one of the criteria employed by the Diagnostic and Statistical Manual of Mental Disorders (DSM-5)¹ for diagnosing ASD, emotion recognition is one of the elements in the Theory of Mind deemed essential for communication and social interaction and may be explored as a way of identifying signs of ASD.

Studies on tools for evaluation and/or intervention of emotion recognition in children with ASD are still rare in Brazil, there is a need for a review study to identify the world panorama on the subject, so this article is of scientific relevance, since it aims to investigate, in the Brazilian and international literature market, the use of software programs for the recognition of emotions in children and adolescents with autism spectrum disorders.

Methods

This integrative literature review was carried out by researching Brazilian and international scientific papers indexed in Periódico Capes, Science Direct, and PubMed using combined keywords “autism” AND “emotion” AND “software”, as well as “autism” AND “emotion recognition” AND “software”. Descriptors were based on the DeCS database of health sciences descriptors and the search was performed from July 2016 to October 2017.

The inclusion criteria were: (a) papers published in the last six years (2012-2017) involving the development and application of software programs related to emotion recognition in children and teenagers with ASD; and (b) studies with individuals at most 18 years old. Papers were excluded from the review if they (a) used robots as technology, (b) were systematic or integrative reviews, or (c) were not available in full text.

The paper search criteria were initially employed by reading titles and abstracts. Once included, the full text was read to verify the criteria set out in the review. The analysis was performed using an Excel spreadsheet, with the data being tabulated regarding the goals, methods, ASD tests and diagnostic protocols, as well as software programs employed in each study.

Literature Review

Given the descriptor-based selection, 1.074 papers were found, of which 1.053 did not meet the inclusion criteria and were, hence, excluded in the title and abstract reading phase. Of the 21 papers selected to be read in full, only ten met all criteria and were analyzed. Figure 1 illustrates how the paper selection was carried out.

Figure 1:
Flow chart of articles analyzed

Table 1 lists paper titles, publication dates, authors, journals of publication, the databases in which they were found, and the software programs used in the studies.

Table 2 shows the ASD diagnosis criteria used in the studies, as well as sample sizes and age ranges. Six papers performed studies with a typical development control group and an ASD-diagnosed experimental group, four papers only considered individuals with ASD, while one paper connected ASD to another disorder, namely social phobia.

Figure 2 shows the number of software programs and their proposed uses, either in intervention or evaluation. Figure 3 presents the number of reviewed papers in which each emotion is considered. Figure 4 shows the number of papers each software program is studied in.

Table 1:
Sumary of articles that investigated softwares in autism spectrum disorders

TITLE	YEAR	AUTHORS	JOURNAL	DATABASE	SOFTWARES
‘Emotiplay’: a serious game for learning about emotions in children with autism: results of a cross-cultural evaluation	2017	Fridenson-Hayo et al.	Eur Child Adolesc Psychiatry	Periódicos CAPES	Emotiplay and Cambridge Mindreading Face-Voice Battery for Children (CAM-C)
RCT of mind reading as a component of a psychosocial treatment for high-functioning children with ASD	2016	Lopata et al.	Research in Autism Spectrum Disorders	Science Direct	Mind Reading program and Cambridge Mindreading Face-Voice Battery for Children (CAM-C)
The Cambridge Mindreading Face-Voice Battery for Children (CAM-C): complex emotion recognition in children with and without autism spectrum conditions	2015	Golan, Sinai-Gavrilov & Baron-Cohen	Molecular Autism	Periódicos CAPES	Cambridge Mindreading Face-Voice Battery for Children (CAM-C)
A Virtual Joy-Stick Study of Emotional Responses and Social Motivation in Children with Autism Spectrum Disorder	2015	Kim et al.	J Autism Dev Disord	PubMed	Virtual-reality emotion sensitivity test (V-REST)
The effect of ICT on emotional education and development of young children with Autism Spectrum Disorder	2015	Charitaki	Procedia Computer Science	Science Direct	Mood Maker
Reduced Recognition of Dynamic Facial Emotional Expressions and Emotion-Specific Response Bias in Children with an Autism Spectrum Disorder	2015	Evers et al.	Child Psychiatry Hum Dev	Periódicos CAPES	Emotion Recognition Task
Computer-Assisted Face Processing Instruction Improves Emotion Recognition, Mentalizing, and Social Skills in Students with ASD	2015	Rice et al.	J Autism Dev Disord	Periódicos CAPES	FaceSay Program
Evidence for shared deficits in identifying emotions from faces and from voices in autism spectrum disorders and specific language impairment	2015	Taylor et al.	Int J Lang Commun Disord	Periódicos CAPES	Mind Reading Program
Training Facial Expression Production in children on the Autism Spectrum	2014	Gordon et al.	J Autism Dev Disord	PubMed	Computer Expression Recognition Emotion Toolbox (CERT) and FaceMaze.
Facial Emotion Recognition in Children with High Functioning Autism and Children with Social Phobia	2012	Wong et al.	Child Psychiatry Hum Dev	Periódicos CAPES	Penn emotion recognition

Table 2:
Criteria and tools used for the diagnosis of autism spectrum disorders in the studies

Article	Criteria and tools for diagnosis	Sample and age range of Autism Spectrum Disorders
‘Emotiplay’: a serious game for learning about emotions in children with autism: results of a cross-cultural evaluation	Previous diagnosis of ASD given by a Doctor or Psychologist according to DSM-IV criteria or ICD-10. Diagnosis was confirmed with aplication of Autism Diagnostic Observation Schedule (ADOS-2).	15 participants aged between 6 and 9 years
RCT of mind reading as a component of a psychosocial treatment for high-functioning children with ASD	Previous diagnosis of autism, Asperger Syndrome or Invasive developmental disorder, non-specified and confirmed with application of Autism Diagnostic Interview-Revised (ADI-R).	36 participants aged between 7 and 12 years
The Cambridge Mindreading Face-Voice Battery for Children (CAM-C): complex emotion recognition in children with and without autism spectrum conditions	Diagnosis made for a clinic Psychiatrist or Psychologist using DSM-IV-TR and ICD-10 criteria.	30 participants aged between 8 and 12 years
A Virtual Joy-Stick Study of Emotional Responses and Social Motivation in Children with Autism Spectrum Disorder	Previous clinic diagnosis of ASD and confirmed the conditions with reports of parentes applying the Autism Spectrum Screening Questionnaire (ASSQ), the Social Communication Questionnaire (SCQ) and the Social Responsiveness Scale (SRS).	19 participants aged between 8 and 16 years
The effect of ICT on emotional education and development of young children with Autism Spectrum Disorder	Diagnosis criteria was not cited in the study. The children with ASD attended special school programs.	5 participants aged between 9 and 14 years
Reduced Recognition of Dynamic Facial Emotional Expressions and Emotion-Specific Response Bias in Children with an Autism Spectrum Disorder	Diagnosis of multi-professional team or for a child Psychiatrist, according to DSM-IV-TR criteria. Such diagnoses were confirmed using the Autism Diagnostic Observation Scale (ADOS).	50 participants aged between 6 and 14 years
Computer-Assisted Face Processing Instruction Improves Emotion Recognition, Mentalizing, and Social Skills in Students with ASD	Studants from Ventura County School, California, who were chosen by special education services on condition of autism.	31 participants aged between 5 and 11 years
Evidence for shared deficits in identifying emotions from faces and from voices in autism spectrum disorders and specific language impairment	Previous diagnosis of ASD according to DSM-IV criteria and confirmed using Autism Diagnostic Observation Schedule-Generic (ADOS-G).	29 participants aged between 5 and 12 years
Training Facial Expression Production in children on the Autism Spectrum	Diagnosis of ASD with the British Columbia Autism Assessment Network (BCAAN), Autism Diagnostic Observation Schedule (ADOS) and Autism Diagnostic Interview (ADI).	30 participants aged between 6 and 18 years
Facial Emotion Recognition in Children with High Functioning Autism and Children with Social Phobia	Previous diagnosis of ASD using the Autism Disorders Interview-Revised (ADI-R) for a licensed clinic Psychologist.	57 participants aged between 7 and 13 years

Figure 2:
Relation of softwares and proposal of use

Figure 3:
Relation of emotions by article

Figure 4:
Relation between the number of articles and softwares used

Using software programs in evaluation and intervention processes among children with ASD is still rare in Brazil, which is noticeable from the absence of Brazilian literature in this review. Most of the reviewed international studies use such tools to aid in the intervention process and only a few use software programs for evaluation. We also identified that software programs are predominantly used in groups of children with high-functioning autism (HFA)^14,21-29.

Previous diagnosis with validated protocol application appears as diagnostic and inclusion criteria in most studies^{19,21,22,24-27,29}. Among the protocols considered as diagnostic gold standards, the Autism Diagnostic Observation Schedule (ADOS)^21,22,25,26 and Autism Diagnostic Interview (ADI)^22,24,27 were the most employed. Different versions of the Diagnostic and Statistical Manual of Mental Disorders (DSM) and the International Classification of Diseases (ICD) were employed to aid the processes, seeing they are the most used scientific references for diagnosing the disorder^1,30.

We also observed that most studies used a sample size of approximately thirty participants with ASD^{19,22,24,25,27,28} and that only six papers compared the performance with that of groups of typical development individuals^{14,22,25-27,29}. Another relevant aspect is that software programs were used predominantly with individuals in the range of 4 to 18 years of age^14,21-29.

Most of the tests in the reviewed papers consider the recognition of six basic emotions, namely happiness, fear, anger, disgust, sadness, and surprise^{14,21,23,24,29}, varying the intensity of the facial stimuli from neutral to more expressive²⁷.

From this review, one may be familiarized with software programs with evaluation purposes such as Emotion Recognition Task, CAM-C, CERT, Penn Emotion Recognition, and V-REST^14,25,27,29, as well as programs with intervention purposes such as Mood Maker, FaceMaze, Facesay, Emotiplay, and MindReading^21-24. All of the aforementioned software programs are used as aids to the respective processes and have shown good results when applied to individuals with ASD. CAM-C and MindReading are among the most used.

Conclusion

This literature review allowed identifying which software programs are used for children with ASD, mainly for intervention purposes. Ten software programs were identified in the ten papers reviewed which showed effectiveness in evaluation and intervention in the age range 4 to 18 years old. The software programs showed six basic emotions (happiness, fear, anger, disgust, sadness, and surprise) that support the understanding of facial recognition of emotions in children and teenagers with ASD. Diagnosis varied according to the use of protocols and scales considered of excellence for identifying the condition. No Brazilian study was found in this research field which subsidizes future investments in developing tools that can aid and facilitate the diagnosis and intervention of ASD.

Knowledgments

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

References

• ¹
  American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th ed. Arlington: American Psychiatric Association; 2013.
• ²
  Lozier LM, Vanmeter JW, Marsh AA. Impairments in facial affect recognition associated with autism spectrum disorders: a meta-analysis. Dev Psychopathol. 2014;26(4):933-45.
• ³
  Zimmerman AW. Autism: current theories and evidence. 1th ed. Baltimore: Humana press; 2008.
• ⁴
  Baron-Cohen S. The autistic child's theory of mind: a case of specific developmental delay. J Child Psychol Psychiatry. 1989;30(2):285-97.
• ⁵
  Baron-Cohen S. Mindblindness: an essay on autism and theory of mind, ed 1, Cambridge, MIT press, 1995.
• ⁶
  Myles BS. Using assistive technology to teach emotion recognition to students with Asperger syndrome: a pilot study. Inj Manag Rehabil [serial on the Internet]. 2007 may [cited 2018 Aug 10]; 28(3):[about 7 p.]. Available from: http://dx.doi.org/10.1016/B978-0-323-00699-6.10013-9
  » http://dx.doi.org/10.1016/B978-0-323-00699-6.10013-9
• ⁷
  Tell D, Davidson D, Camras LA. Recognition of emotion from facial expressions with direct or averted eye gaze and varying expression intensities in children with autism disorder and typically developing children. Autism Res Treat [serial on the Internet]. 2014 Apr [cited 2018 Aug 10]; 1(2014):[about 11 p.]. Available from: http://www.hindawi.com/journals/aurt/2014/816137/
  » http://www.hindawi.com/journals/aurt/2014/816137/
• ⁸
  Uljarevic M, Hamilton A. Recognition of emotions in autism: a formal meta-analysis. J Autism Dev Disord. 2013;43(7):1517-26.
• ⁹
  Castro VL, Cheng Y, Halberstadt AG, Grühn D. EUReKA! A conceptual model of emotion understanding. Emot Rev. 2016;8(3):258-68.
• ¹⁰
  Giovannelli JL, Strauss MS. The development of emotion recognition in individuals with autism. Child Dev. 2011;80(5):1434-47.
• ¹¹
  Lozano-Martínez J, Ballesta-Pagán J, García SA. Software para enseñar emociones al alumnado con trastorno del espectro autista. Comunicar. 2011;18(36):139-48.
• ¹²
  Lacava PG, Rankin A, Mahlios E, Cook K, Simpson RL. A single case design evaluation of a software and tutor intervention addressing emotion recognition and social interaction in four boys with ASD. Autism. 2010;14(3):161-78.
• ¹³
  Hopkins IM, Gower MW, Perez TA, Smith DS, Amthor FR, Casey Wimsatt F et al. Avatar assistant: Improving social skills in students with an asd through a computer-based intervention. J Autism Dev Disord. 2011;41(11):1543-55.
• ¹⁴
  Golan O, Sinai-Gavrilov Y, Baron-Cohen S. The Cambridge mindreading face-voice battery for children (CAM-C): complex emotion recognition in children with and without autism spectrum conditions. Mol Autism. 2015;6(1):1-9.
• ¹⁵
  Simmons ES, Paul R, Shic F. Brief report: a mobile application to treat prosodic deficits in autism spectrum disorder and other communication impairments: a pilot study. J Autism Dev Disord. 2016;46(1):320-7.
• ¹⁶
  Golan O, Baron-Cohen S. Systemizing empathy: teaching adults with Asperger syndrome or high-functioning autism to recognize complex emotions using interactive multimedia. Dev Psychopathol. [serial on the Internet] 2006 Jun [cited 2018 Aug 10]; 18(2):[about 27 p.]. Available from: http://www.tandfonline.com/doi/abs/10.1080/03098265.2010.548508
  » http://www.tandfonline.com/doi/abs/10.1080/03098265.2010.548508
• ¹⁷
  Baio J, Wiggins L, Christensen DL, Maenner MJ, Daniels J, Warren Z et al. Prevalence of autism spectrum disorder among children aged 8 years - autism and developmental disabilities monitoring network, 11 sites, united states, 2014. MMWR Surveill Summ [serial on the Internet] 2018 Apr [cited 2018 Aug 10]; 67(6):[about 23 p.]. Available from: http://www.cdc.gov/mmwr/volumes/67/ss/ss6706a1.htm?s_cid=ss6706a1_w.
• ¹⁸
  Kim YS, Leventhal BL, Koh Y-J, Fombonne E, Laska E, Lim E-C et al. Prevalence of Autism Spectrum Disorders in a total population sample. Am J Psychiatry [serial on the Internet] 2011 Sep [cited 2018 Aug 10]; 168(9):[about 8 p]. Available from: http://psychiatryonline.org/doi/abs/10.1176/appi.ajp.2011.10101532
  » http://psychiatryonline.org/doi/abs/10.1176/appi.ajp.2011.10101532
• ¹⁹
  Paula CS, Fombonne E, Gadia C, Tuchman R, Rosanoff M. Autism in Brazil - perspectives from science and society. Rev Assoc Med Bras [serial on the Internet] 2011 Jan/Feb [cited 2018 Aug 10]; 57(1):[about 5 p]. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0104423011702792
  » http://linkinghub.elsevier.com/retrieve/pii/S0104423011702792
• ²⁰
  Zorzetto R. O cérebro no autismo: alterações no córtex temporal podem causar prejuízo na percepção de informações importantes para a interação social. Rev FAPESP. 2011;184:16-23.
• ²¹
  Fridenson-Hayo S, Berggren S, Lassalle A, Tal S, Pigat D, Meir-Goren N et al. 'Emotiplay': a serious game for learning about emotions in children with autism: results of a cross-cultural evaluation. Eur Child Adolesc Psychiatry. 2017;26(8):979-92.
• ²²
  Gordon I, Pierce MD, Bartlett MS, Tanaka JW. Training facial expression production in children on the autism spectrum. J Autism Dev Disord. 2014;44(10):2486-98.
• ²³
  Charitaki G. The effect of ICT on emotional education and development of young children with Autism Spectrum Disorder. Procedia Comput Sci. [serial on the Internet] 2015 Oct [cited 2018 Aug 10]; 65(2015):[about 9 p]. Available from: http://dx.doi.org/10.1016/j.procs.2015.09.081
  » http://dx.doi.org/10.1016/j.procs.2015.09.081
• ²⁴
  Lopata C, Thomeer ML, Rodgers JD, Donnelly JP, McDonald CA. RCT of mind reading as a component of a psychosocial treatment for high-functioning children with ASD. Res Autism Spectr Disord. [serial on the Internet] 2016 Oct [cited 2018 Aug 10]; 21(2016):[about 12 p]. Available from: http://dx.doi.org/10.1016/j.rasd.2015.09.003
  » http://dx.doi.org/10.1016/j.rasd.2015.09.003
• ²⁵
  Evers K, Steyaert J, Noens I, Wagemans J. Reduced recognition of dynamic facial emotional expressions and emotion-specific response bias in children with an Autism Spectrum Disorder. J Autism Dev Disord. 2015;45(6):1774-84.
• ²⁶
  Taylor LJ, Maybery MT, Grayndler L, Whitehouse AJO. Evidence for shared deficits in identifying emotions from faces and from voices in Autism Spectrum Disorders and Specific Language Impairment. Int J Lang Commun Disord. 2015;50(4):452-66.
• ²⁷
  Wong N, Beidel DC, Sarver DE, Sims V. Facial emotion recognition in children with high functioning autism and children with social phobia. Child Psychiatry Hum Dev. 2012;43(5):775-94.
• ²⁸
  Rice LM, Wall CA, Fogel A, Shic F. Computer-assisted face processing instruction improves emotion recognition, mentalizing, and social skills in students with ASD. J Autism Dev Disord. 2015;45(7):2176-86.
• ²⁹
  Kim K, Rosenthal MZ, Gwaltney M, Jarrold W, Hatt N, McIntyre N et al. A virtual joy-stick study of emotional responses and social motivation in children with Autism Spectrum Disorder. J Autism Dev Disord. [serial on the Internet] 2015 Dec [cited 2018 Aug 10]; 45(12):[about 9 p]. Available from: http://dx.doi.org/10.1007/s10803-014-2036-7
  » http://dx.doi.org/10.1007/s10803-014-2036-7
• ³⁰
  Organização Mundial da Saúde. CID-10 Classificação Estatística Internacional de Doenças e Problemas Relacionados à Saúde. 10th. São Paulo: Universidade de São Paulo; 1997.

• ERRATUM

  In the article, “Analysis of softwares for emotion recognition in children and teenagers with autism spectrum disorder” with DOI number: 10.1590/1982-0216/201921112318, published in the journal Revista Cefac, 21(1):e12318, in author name:
  Where it was:
  Paula Dornhoffer Paro Costa
  Read:
  Paula Dornhofer Paro Costa

Publication Dates

• Publication in this collection
  11 Feb 2019
• Date of issue
  2019

History

• Received
  01 Sept 2018
• Accepted
  11 Dec 2018