Multilingual and multimodal learning for Brazilian portuguese

• Grant number: 22/04442-7
• Support type: Scholarships abroad - Research Internship - Scientific Initiation
• Effective date (Start): June 06, 2022
• Effective date (End): September 05, 2022
• Field of knowledge: Physical Sciences and Mathematics - Computer Science
• Principal researcher: Helena de Medeiros Caseli
• Grantee: Júlia Yumi Araújo Sato
• Supervisor abroad: Lucia Specia
• Home Institution: Centro de Ciências Exatas e de Tecnologia (CCET). Universidade Federal de São Carlos (UFSCAR). São Carlos , SP, Brazil
• Research place: University of Sheffield, England
• Associated to the scholarship: 20/15995-1 - Multilingual and multimodal learning for Brazilian Portuguese, BP.IC

Abstract

Humans constantly deal with multimodal information, that is, data from different modalities, such as texts and images. In order for machines to process information similarly to humans, they must be able to process multimodal data and understand the joint relationship between these modalities. This multimodal aspect of learning can also be helpful in multilingual applications, that is, applications that involve two or more languages. The goal of this project is to explore strategies that result in a more efficient pre-training in the context of multilingual and multimodal processing, such as analysis of the effect of visual region masking for the masked region classification task, as well as applying the knowledge and results of this research to complement the initial FAPESP project, which emerged as an attempt to tackle the problem of multimodal and multilingual learning involving the Portuguese language, by proposing the adaptation of the VTLM framework to different languages and to more challenging circumstances related to the image-text relationship. We believe that the results achieved will be interesting for the scientific community. Furthermore, this BEPE project aims to enrich the student's training in important areas of computing, deepening her knowledge in natural language processing and machine learning concepts, approaches and techniques. (AU)