New features for eyeTalk": user interface for writing and communication using eye and facial commands for mobile devices

• Grant number: 18/15122-8
• Support Opportunities: Research Grants - Innovative Research in Small Business - PIPE
• Start date: September 01, 2021
• End date: August 31, 2023
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Principal Investigator: Marcus Vinicius Pontes Lima
• Grantee: Marcus Vinicius Pontes Lima
• Company: Acta Visio Desenvolvimento em Visão Computacional Ltda
• CNAE: Desenvolvimento e licenciamento de programas de computador customizáveis
• City: Campinas
• Pesquisadores principais: Hélio Pedrini ; Vinicius Benites Bastos
• Associated researchers: Alan Ferreira Pinheiro Tavares ; Victor Cesar de Souza
• Associated research grant: 16/15351-1 - Project, development and assembly of a prototype of an eye tracking system embedded in a first person view goggles for drone control customized for paraplegic people, AP.PIPE
• Associated scholarship(s): 21/13527-3 - New features for eyeTalk: user interface for writing and communication using eye and facial commands for mobile devices, BP.PIPE; 21/10977-8 - Face detection and parameters, BP.TT; 21/10028-6 - Eye and facial detection parameters optimization for EyeTalk, BP.PIPE

Abstract

Supplementary and / or alternative communication (SAC) is characterized by the use of gestures, facial and body expressions, graphic symbols, photos, objects, digitized or synthesized voice, which aims to promote face-to-face communication of individuals with difficulties in communication. It is known that according to the 2010 Census, in Brazil there are 23.9% of the Brazilian population with some type of disability, of which 5.1%, 7% and 1.4% have hearing, motor, mental or intellectual, respectively. From this, we observe the importance and necessity in the development of new technologies or adaptations for people with disabilities. The purpose of this project is to integrate new features into the EyeTalk" application for improved communication, improved accessibility, and increased functionality for people who have difficulty communicating or nonverbal. The EyeTalk" application consists of using the front camera of a smartphone or tablet to communicate for a person who has paralysis or difficulty speaking - the world's first communication solution using mobile application eye commands. The technology uses a facial landmark algorithm to detect the user's eye and thereby perform word-typing using a special keyboard. A specific keyboard was developed for the user to perform the selection of the set of letters of the alphabet. Until the present proposal the developed application presents a good accuracy, but factors like the illumination of the environment interfere in its robustness, which is going to be improved in this project. Additionally, the EyeTalk" application will benefit and improve in many ways so that it can enter the market effectively. The implementation of new features and tools will facilitate the user's communication with other people, such as facial expressions, implementation of ready-made phrases, conversion of typed text (male or female), automatic word completion (this test phase will be in English and Portuguese) and development of a universal keyboard. The expansion of this new tool will give the user greater control of his device with the use of eye movement and facial expressions, creating a basis for an interface for total control of the smartphone or tablet by eye and facial commands - technology not yet available in the market. Initially, for eyeTalk", the business is targeted directly to the consumer (B2C) with source of income through application licensing, with monthly, quarterly, half-yearly, and yearly plans. For this proposal, the B2B model will be analyzed, with the licensing or sale of the technology to companies producing smartphones and tablets such as Apple, Samsung, LG, in order to add more value to their products with increased accessibility and functionality. For this phase 2, it is proposed to expand the functionality of EyeTalk " to make it more complete to enter the market and enable the exploration of licensing or sales of technology for large mobile device manufacturers with the aim of adding more value to their products and provide more accessibility for more people. (AU)