Artificial Intelligence Application for Translating Texts into Libras with an Interactive Digital Avatar

• Grant number: 24/22553-6
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: March 01, 2025
• End date: February 28, 2026
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: João Paulo Papa
• Grantee: Pedro Henrique Crespan Ribeiro
• Host Institution: Faculdade de Ciências (FC). Universidade Estadual Paulista (UNESP). Campus de Bauru. Bauru , SP, Brazil
• Associated research grant: 24/01132-2 - Multidisciplinary Center for the Development of Assistive Technology (MCDAT), AP.CCD

Abstract

This project aims to develop an application that promotes communication accessibility for people who are deaf or hard of hearing. The proposal is to use artificial intelligence (AI) technology to convert texts typed by users into a visual translation into Brazilian Sign Language (Libras). The application will include an interactive digital avatar, capable of clearly reproducing the signs corresponding to the text, according to the norms and conventions of Libras.The first step of the project will be to create a benchmark using sentences with ambiguities, multiple meanings for the same word and other linguistic variations. The candidate language models to be used in the project will be tested with the sentences to assess their accuracy in adapting the Portuguese text to the textual structure of Libras, eliminating irrelevant or redundant parts. The results of these tests will provide relevant information to determine the strengths and limitations of each model.The choice of language model will be based on the tests carried out on the candidate models, focusing on their ability to deal with ambiguities and linguistic variations. In addition, the choice will be made taking into account the limitations of the hardware available for the project, with a view to efficiency and feasibility.To train the neural network that will execute the avatar's gestures, a specific database will be chosen, containing a Portuguese-Libras dictionary to enable translation, in which the variety of information will be considered, taking into account the number of signs, so that the application is robust and can be used in different contexts. With this, the neural network will be developed, in which different neural network algorithms will be evaluated to identify the most suitable for the application, so that gestures are reproduced in a fluid and natural way. At the same time, the digital avatar that will represent the signs in Libras will be defined. Fluid movements and precise facial expressions, essential elements for communication in Libras, will be prioritized.Integration between the artificial intelligences involved in the project will also be developed, allowing the text to be translated into signs in a cohesive manner. The neural network will then be trained to ensure the correct execution of the signs from the previously selected database.Finally, an accessible and intuitive interface will be created for the end user, in which minimalism and ease of use will be prioritized. The application will be designed to work in different contexts, such as education, public services and everyday interactions, broadening its social impact.As well as promoting social inclusion, the project aims to reduce the communication barriers faced by the deaf community in Brazil. In line with the principles of universal accessibility and the guidelines of Brazilian legislation, this initiative aims to contribute to the use of digital technologies to democratize communication and expand opportunities for social and professional integration for deaf people.