Learning representations through deep generative models on video

• Grant number: 19/07257-3
• Support Opportunities: Regular Research Grants
• Start date: September 01, 2020
• End date: July 31, 2021
• Field of knowledge: Physical Sciences and Mathematics - Computer Science
• Principal Investigator: Gerberth Adín Ramírez Rivera
• Grantee: Gerberth Adín Ramírez Rivera
• Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Associated researchers: Anderson de Rezende Rocha ; Hélio Pedrini ; Ricardo da Silva Torres

Abstract

Automatic media generation (or synthesis) is a field that had an incredible boost in recent years, with the advent of deep generative models. Nowadays, neural networks can create text, images and videos based on class labels or other media. The common task is to generate content. However, we can take advantage of the learned feature representations on these tasks to understand the relevant features and as a source of interpretability. That is, what features are relevant for the creation of different content, and how can we interpret what the models are learning or paying attention too. In this project, we propose to investigate how to learn efficient and rich representations for video data based on deep generative tasks. We focus on two particular problems for learning effective representations. The first one is semantic transfer between data modalities, in particular video and (written) language. And the second one is disentanglement within the same domain, that is, separate different variations and modalities of the data. The separation of semantics (intra and inter domain) will allow us to better understand the type of features that are learned by the different architectures on these tasks. Our objective is to train the deep generative models on different video reconstruction tasks and study their learning capabilities. We will perform experiments on the existing benchmark datasets for the particular problems. (AU)

Scientific publications

(5)

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

SAIRE, DARWIN; RIVERA, ADIN RAMIREZ. Empirical Study of Multi-Task Hourglass Model for Semantic Segmentation Task. IEEE ACCESS, v. 9, p. 80654-80670, 2021. (19/18678-0, 19/07257-3, 17/16597-7)

SANTANDER, MIGUEL RODRIGUEZ; ALBARRACIN, JUAN HERNANDEZ; RIVERA, ADIN RAMIREZ. On the pitfalls of learning with limited data: A facial expression recognition case study. EXPERT SYSTEMS WITH APPLICATIONS, v. 183, NOV 30 2021. (19/07257-3, 16/19947-6, 17/16144-2)

RIVERA, ADIN RAMIREZ; KHAN, ADIL; BEKKOUCH, IMAD EDDINE IBRAHIM; SHEIKH, TAIMOOR SHAKEEL. Anomaly Detection Based on Zero-Shot Outlier Synthesis and Hierarchical Feature Distillation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, v. 33, n. 1, p. 281-291, JAN 2022. (19/07257-3)

ALBARRACIN, JUAN F. HERNANDEZ; RAMIREZ RIVERA, ADIN. Video Reenactment as Inductive Bias for Content-Motion Disentanglement. IEEE Transactions on Image Processing, v. 31, p. 10-pg., 2022-01-01. (17/16144-2, 19/07257-3)

SAIRE, DARWIN; RIVERA, ADIN RAMIREZ. Global and Local Features Through Gaussian Mixture Models on Image Semantic Segmentation. IEEE ACCESS, v. 10, p. 14-pg., 2022-01-01. (19/07257-3, 17/16597-7, 19/18678-0)