Augmentative and Alternative Communication Based on Eye Tracking Supported by Machine Learning

• Grant number: 24/07256-5
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: July 01, 2024
• End date: June 30, 2025
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Principal Investigator: Bruna Carolina Rodrigues da Cunha
• Grantee: Felipe Scholucha Martinez Roldan
• Host Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil

Abstract

Speech disorders affect the ability to articulate words and can be caused by neurological disorders, neuromuscular transmission disorders, or result from brain and neuromuscular injuries. These disorders can affect any individual and severely impair communication, whether congenital or resulting from diseases or accidents. In some severe cases of Cerebral Palsy, Stroke, and Amyotrophic Lateral Sclerosis, for instance, individuals retain only eye movement, which becomes their primary means of communication. To support the communication of people with severe motor limitations, devices and tools for Augmentative and Alternative Communication (AAC) have been developed. However, dedicated hardware devices are rarely accessible to a large part of the population due to their cost and importation difficulties. In this context, eye tracking through mobile device cameras has been investigated. Despite the limitations arising from using common cameras, Machine Learning methods, particularly those based on Deep Learning (DL), have enabled interaction through eye movement. Considering this scenario, this project aims to investigate the application of Deep Learning-based models for eye tracking for typing on virtual keyboards. The goal is to evaluate the performance in use, in terms of words per second, and the potential portability of the models to different devices.