Multimodal Embedding Space for Text-Guided Data Generation

• Grant number: 23/05939-5
• Support Opportunities: Scholarships in Brazil - Doctorate
• Effective date (Start): October 01, 2023
• Status: Discontinued
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Principal Investigator: Sandra Eliza Fontes de Avila
• Grantee: Diego Alysson Braga Moreira
• Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Associated research grant: 13/08293-7 - CCES - Center for Computational Engineering and Sciences, AP.CEPID
• Associated scholarship(s): 24/07164-3 - Multimodal Representation Space for Text-Guided Data Generation, BE.EP.DR

Abstract

Multidimensional representation spaces by contrastive training involving images and texts are proposed to approach related concepts between modal signals. Some works expand this concept to audio, speech, or ambient sounds by approaching their description. However, so far, no work in the literature relates concepts of audio, image, and text or creates environments with more than two types of data. Among the challenges of multimodality is the language of the texts used to create the learning and training space for the models. Languages with few resources, which include Portuguese, are left on the margins of research and world advancement. More resources and data must be produced for these languages so that the state-of-the-art techniques also reflect the technological production of the speaking countries. This Ph.D. research project proposes the creation of a multimodal space between three or more types of data, bringing together related concepts, with the possibility of adding information from sensors (e.g., accelerometer, gyroscope, and magnetometer). We hope to recover concepts through related data and create a new set of information from modal data through similar concepts. Brazilian Portuguese will be used as the textual language to provide models and data that collaborate with the advancement of learning technologies and natural language processing in Brazil. Among the expected goals are multiple sets of data, with images/texts entirely in Portuguese, created or translated, and competing models with those found in high-resource languages.