Multimodal Representation Space for Text-Guided Data Generation

• Grant number: 24/07164-3
• Support Opportunities: Scholarships abroad - Research Internship - Doctorate
• Start date: August 01, 2024
• End date: January 31, 2025
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Principal Investigator: Sandra Eliza Fontes de Avila
• Grantee: Diego Alysson Braga Moreira
• Supervisor: Carolina Evaristo Scarton
• Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Institution abroad: University of Sheffield, England
• Associated to the scholarship: 23/05939-5 - Multimodal Embedding Space for Text-Guided Data Generation, BP.DR

Abstract

Multidimensional representation spaces for contrastive training involving images and texts are proposed to approximate related concepts between modal signals. Some work extends the same concept to audio, speech, or environmental sounds by approximating their description. However, to date, nowork in the literature relates audio, image, and text concepts or creates environments with more than two types of data, focusing on text and its correlation with other types of data. Furthermore, in data generation, no studies have used multimodal information to generate sensor data and have not related this type of data to those above.One of the challenges of multimodality is the language used to teach the models. Languages with few resources, such as Portuguese, are at the margins of research and global progress. There is a need for more resources and data for these languages so that state-of-the-art techniques can havesatisfactory results for the countries where these languages are spoken.This project proposes to create a multimodal space between three or more types of data, bringing together related concepts between texts, images, audio, and sensors. We hope to be able to retrieve concepts using related data, as well as to create a new set of information from modal data usingsimilar concepts. Brazilian Portuguese will be used as the text language. The intention is to provide models and data to help advance natural language learning and processing technologies in Brazil.The expected goals, which have been partially achieved, include Several datasets with images/texts entirely in Portuguese, created or translated, and models that are competitive with those found in high-resource languages.