Transfer Learning in Multimodal Learning For Video Emotion Recognition

• Grant number: 22/14903-1
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: February 01, 2023
• End date: January 31, 2024
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Ricardo Marcondes Marcacini
• Grantee: Gabriel Natal Coutinho
• Host Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil

Abstract

Automatic emotion recognition is a relevant task for many purposes. purposes, such as opinion polls and identification of hate speech. We know that an emotion we feel has an impact on interpersonal relationships and social interactions, so it is possible to use sentimental analysis for the purpose of also improving human-machine interaction. Although the emotion recognition task has often been treated unimodally, identifying emotion depends on multiple stimuli, as in identifying facial expressions and voice sounds. In this context, multimodal learning is the area of science that studies the relationship between data from different modalities. The goal of this scientific initiation project is to investigate transfer of learning to recognize emotions in videos. Regarding the transfer learning, it is worth mentioning that recently different pre-trained models for images, texts and audio have been made available. While most existing methods focus on generating new unimodal or multimodal pre-trained models, which has a high computational cost, in this project we raise the following question: how to take advantage of different pre-trained unimodal models to obtain a new and unified multimodal feature space to recognize awareness of emotions in videos? Answering this question will allow not only to reduce the computational cost of multimodal learning, but advancing in transfer and reuse of pre-trained models on different data sets.