Scholarship 24/07164-3 - Aprendizagem profunda, Linguagem natural - BV FAPESP
Advanced search
Start date
Betweenand

Multimodal Representation Space for Text-Guided Data Generation

Grant number: 24/07164-3
Support Opportunities:Scholarships abroad - Research Internship - Doctorate
Start date until: August 01, 2024
End date until: January 31, 2025
Field of knowledge:Physical Sciences and Mathematics - Computer Science - Computer Systems
Principal Investigator:Sandra Eliza Fontes de Avila
Grantee:Diego Alysson Braga Moreira
Supervisor: Carolina Evaristo Scarton
Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
Institution abroad: University of Sheffield, England  
Associated to the scholarship:23/05939-5 - Multimodal Embedding Space for Text-Guided Data Generation, BP.DR

Abstract

Multidimensional representation spaces for contrastive training involving images and texts are proposed to approximate related concepts between modal signals. Some work extends the same concept to audio, speech, or environmental sounds by approximating their description. However, to date, nowork in the literature relates audio, image, and text concepts or creates environments with more than two types of data, focusing on text and its correlation with other types of data. Furthermore, in data generation, no studies have used multimodal information to generate sensor data and have not related this type of data to those above.One of the challenges of multimodality is the language used to teach the models. Languages with few resources, such as Portuguese, are at the margins of research and global progress. There is a need for more resources and data for these languages so that state-of-the-art techniques can havesatisfactory results for the countries where these languages are spoken.This project proposes to create a multimodal space between three or more types of data, bringing together related concepts between texts, images, audio, and sensors. We hope to be able to retrieve concepts using related data, as well as to create a new set of information from modal data usingsimilar concepts. Brazilian Portuguese will be used as the text language. The intention is to provide models and data to help advance natural language learning and processing technologies in Brazil.The expected goals, which have been partially achieved, include Several datasets with images/texts entirely in Portuguese, created or translated, and models that are competitive with those found in high-resource languages.

News published in Agência FAPESP Newsletter about the scholarship:
More itemsLess items
Articles published in other media outlets ( ):
More itemsLess items
VEICULO: TITULO (DATA)
VEICULO: TITULO (DATA)

Please report errors in scientific publications list using this form.