Advanced search
Start date

Multilingual and multimodal learning for Brazilian Portuguese

Grant number: 20/15995-1
Support Opportunities:Scholarships in Brazil - Scientific Initiation
Effective date (Start): April 01, 2021
Effective date (End): December 31, 2022
Field of knowledge:Physical Sciences and Mathematics - Computer Science
Principal Investigator:Helena de Medeiros Caseli
Grantee:Júlia Yumi Araújo Sato
Host Institution: Centro de Ciências Exatas e de Tecnologia (CCET). Universidade Federal de São Carlos (UFSCAR). São Carlos , SP, Brazil
Associated scholarship(s):22/04442-7 - Multilingual and multimodal learning for Brazilian portuguese, BE.EP.IC


Humans constantly deal with multimodal information, that is, data sets of different modalities, such as text and image. For machines to process information similarly to humans, they must be able to process multimodal data and understand the joint relationship between these modalities, not just text or image in isolation, for example. This multimodal aspect of learning can be very useful in multilingual applications, that is, applications that involve two or more languages. This project proposes the extension of the VTLM (Visual Translation Language Modelling) framework, an approach recently published by Caglayan et al. (2021). To accomplish this goal, we will use the multimodal and multilingual dataset How2 (SANABRIA et al., 2018) in three parallel streams with aligned English-Portuguese-Visual information and explore more informed masking strategies for visual regions. Therefore, the basis of language in the image regions will be done between source and target languages together for the generation of a multilingual and multimodal model useful for several NLP applications. (AU)

News published in Agência FAPESP Newsletter about the scholarship:
Articles published in other media outlets (0 total):
More itemsLess items

Please report errors in scientific publications list by writing to: