Multilingual and multimodal learning for Brazilian Portuguese

• Grant number: 20/15995-1
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Effective date (Start): April 01, 2021
• Effective date (End): December 31, 2022
• Field of knowledge: Physical Sciences and Mathematics - Computer Science
• Principal Investigator: Helena de Medeiros Caseli
• Grantee: Júlia Yumi Araújo Sato
• Host Institution: Centro de Ciências Exatas e de Tecnologia (CCET). Universidade Federal de São Carlos (UFSCAR). São Carlos , SP, Brazil
• Associated scholarship(s): 22/04442-7 - Multilingual and multimodal learning for Brazilian portuguese, BE.EP.IC

Abstract

Humans constantly deal with multimodal information, that is, data sets of different modalities, such as text and image. For machines to process information similarly to humans, they must be able to process multimodal data and understand the joint relationship between these modalities, not just text or image in isolation, for example. This multimodal aspect of learning can be very useful in multilingual applications, that is, applications that involve two or more languages. This project proposes the extension of the VTLM (Visual Translation Language Modelling) framework, an approach recently published by Caglayan et al. (2021). To accomplish this goal, we will use the multimodal and multilingual dataset How2 (SANABRIA et al., 2018) in three parallel streams with aligned English-Portuguese-Visual information and explore more informed masking strategies for visual regions. Therefore, the basis of language in the image regions will be done between source and target languages together for the generation of a multilingual and multimodal model useful for several NLP applications. (AU)