Cover not available

Article published In: Interaction Studies
Vol. 20:2 (2019) ► pp.256–274

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Augmented reality coloring book: An interactive strategy for teaching children with autism to focus on specific nonverbal social cues to promote their social skills

 | National Taipei University of Technology, Taipei Tech
Published online: 7 October 2019
DOI logo with link
https://doi.org/10.1075/is.18004.lee
Abstract
Autism spectrum disorders (ASD) reduce one’s ability to act appropriately in social situations. Increasing evidence indicates that children with ASD might ignore nonverbal social cues that usually aid social interaction because they do not recognize or understand them. We asked children with ASD to color an augmented reality coloring book (ARCB) to teach them how to recognize and understand some specific social signals and to ignore others. ARCB materials teach children to recognize and understand social signals in various ways. They can, for example, view 3D animations of the ARCB materials on a tablet computer. Thus, the ARCB can be used to help children with ASD focus their attention on the meaning and social value of nonverbal behaviors in specific social situations. The ARCB has multiple functions: it extends the social features of the story, and it restricts attention to the most important parts of the videos. Single-subject research with a multiple-baselines across-subject design was used in this study. After five weeks of ARCB training intervention, all 3 participants’ scores rose significantly and dramatically during the intervention phase (mean rate of correct answers improved from 14.24% to 47.33%), and remained significantly higher in the maintenance phase than at baseline. We conclude that coloring pictures of social situations may help children with ASD recognize and better understand these situations.
Article outline
  • Introduction
  • Related works
  • Methods
    • Participants
    • Developing the training materials for the ARCB system
    • Program design
    • Phases, sessions, and experimental conditions
    • Baseline phase
    • Intervention phase
    • Maintenance phase
    • Measurement materials
    • SSTs test
    • Social reciprocity behavior evaluation
    • Data collection and test reliability
  • Results
    • Training effects of the ARCB system
    • Their overall expression performances of social reciprocity skills
    • Feedback from the children, therapist, and parents
  • Discussion
  • Conclusion
  • References
References (46)
References
Attwood, T. (2006). The complete guide to Asperger’s syndrome. Jessica Kingsley Publishers.Google Scholar logo with link to Google Scholar
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Pub. Google Scholar logo with link to Google Scholar
Baker, J. E. (2003a). Social skills picture book: Teaching play, emotion, and communication to children with autism. Arlington, TX: Future Horizons.Google Scholar logo with link to Google Scholar
(2003b). Social skills training. Overland Park, KS: Autism Asperger Publishing.Google Scholar logo with link to Google Scholar
Baird, G., & Norbury, C. F. (2016). Social (pragmatic) communication disorders and autism spectrum disorder. Archives of Disease in Childhood, 101(8), 745–751. Google Scholar logo with link to Google Scholar
Broadhead, P. (2006). Developing an understanding of young children’s learning through play: the place of observation, interaction and reflection. British Educational Research Journal, 32(2), 191–207. Google Scholar logo with link to Google Scholar
Castelli, F. (2005). Understanding emotions from standardized facial expressions in autism and normal development. Autism, 9(4), 428–449. Google Scholar logo with link to Google Scholar
Chen, C. H., Lee, I. J., & Lin, L. Y. (2015). Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. Research in developmental disabilities, 361, 396–403. Google Scholar logo with link to Google Scholar
(2016). Augmented reality-based video-modeling storybook of nonverbal facial cues for children with autism spectrum disorder to improve their perceptions and judgments of facial expressions and emotions. Computers in Human Behavior, 551, 477–485. Google Scholar logo with link to Google Scholar
Clark, A., & Dünser, A. (2012, March). An interactive augmented reality coloring book. In 3D User Interfaces (3DUI), 2012 IEEE Symposium on (pp. 7–10). IEEE. Google Scholar logo with link to Google Scholar
Curry, N. A., & Kasser, T. (2005). Can coloring mandalas reduce anxiety? Art Therapy, 22(2), 81–85. Google Scholar logo with link to Google Scholar
Escobedo, L., Tentori, M., Quintana, E., Favela, J., & Garcia-Rosas, D. (2014). Using augmented reality to help children with autism stay focused. IEEE Pervasive Computing, 13(1), 38–46.Google Scholar logo with link to Google Scholar
Farr, W., Yuill, N., & Hinske, S. (2012). An augmented toy and social interaction in children with autism. International Journal of Arts and Technology, 5(2–4), 104–125. Google Scholar logo with link to Google Scholar
Fodstad, J. C., Matson, J. L., Hess, J., & Neal, D. (2009). Social and communication behaviours in infants and toddlers with autism and pervasive developmental disorder-not otherwise specified. Developmental Neurorehabilitation, 12(3), 152–157. Google Scholar logo with link to Google Scholar
Geschwind, D. H., & Levitt, P. (2007). Autism spectrum disorders: developmental disconnection syndromes. Current opinion in neurobiology, 17(1), 103–111. Google Scholar logo with link to Google Scholar
Gray, C. A. (1998). Social stories and comic strip conversations with students with Asperger syndrome and high-functioning autism. Asperger syndrome or high-functioning autism?, 167–198. Google Scholar logo with link to Google Scholar
Heath, M. A., Sheen, D., Leavy, D., Young, E., & Money, K. (2005). Bibliotherapy: A resource to facilitate emotional healing and growth. School Psychology International, 26(5), 563–580. Google Scholar logo with link to Google Scholar
Hedley, N. R. (2003, August). Empirical evidence for advanced geographic visualization interface use. In International cartographic congress, Durban, South Africa.Google Scholar logo with link to Google Scholar
Hobson, R. P., & Lee, A. (1998). Hello and goodbye: A study of social engagement in autism. Journal of autism and developmental disorders, 28(2), 117–127. Google Scholar logo with link to Google Scholar
Hornecker, E., & Buur, J. (2006, April). Getting a grip on tangible interaction: a framework on physical space and social interaction. In Proceedings of the SIGCHI conference on Human Factors in computing systems (pp. 437–446). ACM. Google Scholar logo with link to Google Scholar
Hoyek, N., Di Rienzo, F., Collet, C., Creveaux, T., & Guillot, A. (2014). Hand mental rotation is not systematically altered by actual body position: Laterality judgment versus same–different comparison tasks. Attention, Perception, & Psychophysics, 76(2), 519–526. Google Scholar logo with link to Google Scholar
Jarrold, C. (2003). A review of research into pretend play in autism. Autism, 7(4), 379–390. Google Scholar logo with link to Google Scholar
Kaufmann, H., & Schmalstieg, D. (2003). Mathematics and geometry education with collaborative augmented reality. Computers & Graphics, 27(3), 339–345. Google Scholar logo with link to Google Scholar
Klatzky, R. L., Wu, B., Shelton, D., & Stetten, G. (2008). Effectiveness of augmented-reality visualization versus cognitive mediation for learning actions in near space. ACM Transactions on Applied Perception (TAP), 5(1), 1. Google Scholar logo with link to Google Scholar
Koegel, L. K., Vernon, T. W., Koegel, R. L., Koegel, B. L., & Paullin, A. W. (2012). Improving social engagement and initiations between children with autism spectrum disorder and their peers in inclusive settings. Journal of Positive Behavior Interventions, 14(4), 220–227. Google Scholar logo with link to Google Scholar
Koldewyn, K., Weigelt, S., Kanwisher, N., & Jiang, Y. (2013). Multiple objects tracking in autism spectrum disorders. Journal of autism and developmental disorders, 43(6), 1394–1405. Google Scholar logo with link to Google Scholar
Krasny, L., Williams, B. J., Provencal, S., & Ozonoff, S. (2003). Social skills interventions for the autism spectrum: essential ingredients and a model curriculum. Child and Adolescent Psychiatric Clinics of North America, 12(1), 107–122. Google Scholar logo with link to Google Scholar
Lee, I. J., Chen, C. H., & Chang, K. P. (2016). Augmented reality technology combined with three-dimensional holography to train the mental rotation ability of older adults. Computers in Human Behavior, 651, 488–500. Google Scholar logo with link to Google Scholar
Lillard, A. S., Lerner, M. D., Hopkins, E. J., Dore, R. A., Smith, E. D., & Palmquist, C. M. (2013). The impact of pretend play on children’s development: A review of the evidence. Psychological bulletin, 139(1), 1. Google Scholar logo with link to Google Scholar
Lu, L., Petersen, F., Lacroix, L., & Rousseau, C. (2010). Stimulating creative play in children with autism through sandplay. The Arts in Psychotherapy, 37(1), 56–64. Google Scholar logo with link to Google Scholar
Marshall, P. (2007, February). Do tangible interfaces enhance learning? In Proceedings of the 1st international conference on tangible and embedded interaction (pp. 163–170). ACM. Google Scholar logo with link to Google Scholar
Martins, M. P., & Harris, S. L. (2006). Teaching children with autism to respond to joint attention initiations. Child & Family Behavior Therapy, 28(1), 51–68. Google Scholar logo with link to Google Scholar
Matson, J. L. (2009). Applied behavior analysis for children with autism spectrum disorders. New York: Springer. Google Scholar logo with link to Google Scholar
Magnenat, S., Ngo, D. T., Zünd, F., Ryffel, M., Noris, G., Rothlin, G., … & Sumner, R. W. (2015). Live texturing of augmented reality characters from colored drawings. IEEE transactions on visualization and computer graphics, 21(11), 1201–1210. Google Scholar logo with link to Google Scholar
McPartland, J. C., Webb, S. J., Keehn, B., & Dawson, G. (2011). Patterns of visual attention to face and objects in autism spectrum disorder. Journal of Autism and Developmental Disorders, 41(2), 148–157. Google Scholar logo with link to Google Scholar
Moser, T., & Martinsen, M. T. (2010). The outdoor environment in Norwegian kindergartens as pedagogical space for toddlers’ play, learning and development. European Early Childhood Education Research Journal, 18(4), 457–471. Google Scholar logo with link to Google Scholar
Mundy, P., Sigman, M., Ungerer, J., & Sherman, T. (1986). Defining the social deficits of autism: the contribution of non-verbal communication measures. Journal of Child Psychology and Psychiatry, 27(5), 657–669. Google Scholar logo with link to Google Scholar
Mynarikova, L. (2012). Art-Based Program for Social and Emotional Development of Children. Online Submission.Google Scholar logo with link to Google Scholar
QuiverVision. (2016). Quiver Augmented Reality used as art therapy for veterans. Retrieved from [URL]
Reynhout, G., & Carter, M. (2006). Social Stories™ for children with disabilities. Journal of autism and developmental disorders, 36(4), 445–469. Google Scholar logo with link to Google Scholar
Ryan, C., & Ni Charragain, C. (2010). Teaching emotion recognition skills to children with autism. Journal of Autism and Developmental Disorders, 40(12), 1505–1511. Google Scholar logo with link to Google Scholar
Shelton, B. E., & Hedley, N. R. (2004). Exploring a cognitive basis for learning spatial relationships with augmented reality. Technology, Instruction, Cognition and Learning, 1(4), 323.Google Scholar logo with link to Google Scholar
Sparrow, S. S. (2011). Vineland adaptive behavior scales. In Encyclopedia of Clinical Neuropsychology (pp. 2618–2621). Springer New York. Google Scholar logo with link to Google Scholar
Wraga, M., Thompson, W. L., Alpert, N. M., & Kosslyn, S. M. (2003). Implicit transfer of motor strategies in mental rotation. Brain and Cognition, 52(2), 135–143. Google Scholar logo with link to Google Scholar
Yuill, N., Strieth, S., Roake, C., Aspden, R., & Todd, B. (2007). Brief Report: Designing a Playground for Children with Autistic Spectrum Disorders––Effects on Playful Peer Interactions. Journal of autism and developmental disorders, 37(6), 1192–1196. Google Scholar logo with link to Google Scholar
Zünd, F., Ryffel, M., Magnenat, S., Marra, A., Nitti, M., Kapadia, M., … & Sumner, R. W. (2015, November). Augmented creativity: bridging the real and virtual worlds to enhance creative play. In SIGGRAPH Asia 2015 Mobile Graphics and Interactive Applications (p. 21). ACM.Google Scholar logo with link to Google Scholar
Cited by (21)

Cited by 21 other publications

Christou, Eirini, Antigoni Parmaxi, Lawrence Farrugia Caruana, Leonard Busuttil, Jacqueline Żammit, Sonia Rodríguez-Cano, Vanesa Delgado-Benito, Laura Alonso-Martínez & Andri Ioannou
2026. Augmented reality for story-related activities in primary and secondary education: a systematic review. Universal Access in the Information Society 25:1 DOI logo
Xue, Yu, Yukang Xue, Mariola Moeyaert & Don McMahon
2026. The effectiveness of augmented reality‐based interventions for individuals with autism and/or intellectual and developmental disabilities: A Bayesian three‐level meta‐analysis of single‐case experimental design data. British Journal of Educational Technology 57:1 ► pp. 55 ff. DOI logo
孙, 雅兰
2026. A Scientometric Review of Social Interaction Research in Autism Spectrum Disorder—Based on CiteSpace Analysis of Web of Science Literature. Advances in Psychology 16:02 ► pp. 454 ff. DOI logo
Lee, I-Jui & Chiu-Mi Chen
2025. Using asymmetric AR and VR technologies combined with peer collaboration and symbolic play to enhance intent understanding and empathy in children with autism. Interactive Learning Environments 33:9 ► pp. 5599 ff. DOI logo
Lee, I-Jui & Yu-Chen Huang
2025. Improving social skills in children with autism spectrum disorder using augmented reality and mixed reality technology combined with concept maps. Universal Access in the Information Society 24:2 ► pp. 1255 ff. DOI logo
Chen, Hao, Chao Liu, Ayuan Zhang, Wen-Qian Lu, Kan Wu & Wen-Ko Chiou
2024. How flow and mindfulness interact with each other in mindfulness-based augmented reality mandala coloring activities. Frontiers in Psychology 14 DOI logo
Cheng, Yuchen & Loukia Bololia
2024. The Effects of Augmented Reality on Social Skills in Children with an Autism Diagnosis: A Preliminary Systematic Review. Journal of Autism and Developmental Disorders 54:4 ► pp. 1317 ff. DOI logo
Lee, I -Jui & Hsiu-Ting Hsu
2024. Applied the augmented reality technology combined with social stories strategies and computational thinking games to improve the social skills of children with ASD. Interactive Learning Environments 32:10 ► pp. 6346 ff. DOI logo
López-Escribano, Carmen & Cristina Orío-Aparicio
2024. Creative arts therapy for autistic children: A systematic review. The Arts in Psychotherapy 91 ► pp. 102224 ff. DOI logo
Neely, Leslie, Amarie Carnett, John Quarles, Se-Woong Park & Michelle Kelly
2024. Behavior Analytic Technologies Mediated via Augmented Reality for Autism: A Systematic Review. Journal of Developmental and Physical Disabilities 36:2 ► pp. 243 ff. DOI logo
Newbutt, Nigel, Noah Glaser, Marc Sonley Francois, Matthew Schmidt & Sue Cobb
2024. How are Autistic People Involved in the Design of Extended Reality Technologies? A Systematic Literature Review. Journal of Autism and Developmental Disorders 54:11 ► pp. 4232 ff. DOI logo
Robinson, Rachael & Karen Nohelty
2024. Interventions to Improve Social and Play Skills for Autistic Children. In Handbook of Early Intervention for Autism Spectrum Disorders [Autism and Child Psychopathology Series, ], ► pp. 379 ff. DOI logo
Bakır, Çiçek Nur, Sharif Omar Abbas, Egemen Sever, Aslıhan Özcan Morey, Herdem Aslan Genç & Tuba Mutluer
2023. Use of augmented reality in mental health-related conditions: A systematic review. DIGITAL HEALTH 9 DOI logo
Gu, Peidi, Xinhao Xu, Xueqin Qian & Tsung-Han Weng
2023. Leveraging Extended Reality for Autistic Individuals: A Scoping Review of Technical Features and Technology Affordances. IEEE Transactions on Learning Technologies 16:1 ► pp. 133 ff. DOI logo
Lian, Xiaojie, Mohd Shahrizal Sunar, Qingqing Lian & Mohd Khalid Mokhtar
2023. Evaluating user interface of a mobile augmented reality coloring application for children with autism: An eye-tracking investigation. International Journal of Human-Computer Studies 178 ► pp. 103085 ff. DOI logo
Köse, Hasan & Nevin Güner-Yildiz
2021. Augmented reality (AR) as a learning material in special needs education. Education and Information Technologies 26:2 ► pp. 1921 ff. DOI logo
Lian, Xiaojie & Mohd Shahrizal Sunar
2021. Mobile Augmented Reality Technologies for Autism Spectrum Disorder Interventions: A Systematic Literature Review. Applied Sciences 11:10 ► pp. 4550 ff. DOI logo
Hsu, Hsiu-Ting & I-Jui Lee
2020. Using Augmented Reality Technology with Serial Learning Framework to Develop a Serial Social Story Situation Board Game System for Children with Autism to Improve Social Situation Understanding and Social Reciprocity Skills. In Universal Access in Human-Computer Interaction. Applications and Practice [Lecture Notes in Computer Science, 12189], ► pp. 3 ff. DOI logo
Huang, Yu-Chen & I-Jui Lee
2020. Using Augmented Reality and Concept Mapping to Improve Ability to Master Social Relationships and Social Reciprocity for Children with Autism Spectrum Disorder. In Universal Access in Human-Computer Interaction. Applications and Practice [Lecture Notes in Computer Science, 12189], ► pp. 19 ff. DOI logo
Lee, I-Jui
2020. How to use the Advantages of AR and VR Technique to Integrate Special Visual Training Strategies in Non-Verbal Communication Skills Training for Children with Autism. In Types of Nonverbal Communication [Working Title], DOI logo
Wang, Wei-Zhen & I-Jui Lee
2020. Social Intervention Strategy of Augmented Reality Combined with Theater-Based Games to Improve the Performance of Autistic Children in Symbolic Play and Social Skills. In HCI International 2020 – Late Breaking Papers: Universal Access and Inclusive Design [Lecture Notes in Computer Science, 12426], ► pp. 401 ff. DOI logo

This list is based on CrossRef data as of 17 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue