Video-to-video dynamics transfer with deep generative models

• Grant number: 17/16144-2
• Support Opportunities: Scholarships in Brazil - Doctorate
• Start date: September 01, 2018
• End date: January 12, 2022
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Principal Investigator: Gerberth Adín Ramírez Rivera
• Grantee: Juan Felipe Hernández Albarracín
• Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil

Abstract

Automatic content generation (or synthesis) is a field that had an incredible boost in recent years, with the advent of deep generative models. Nowadays, neural networks can create text, images and videos based on class labels or other media. Most of the research in this area focus on semantic image edition (e.g., style transfer or object transfiguration) and video prediction. Particularly, in the domain of videos, there is relatively few research focusing on purposes different from generating future frames of an input sequence. Therefore, in this proposal we intend to explore the field of video synthesis by extending existing ideas for the purpose of dynamics transfer, a narrowly explored scenario with potential new applications. We aim to achieve the video generation task by transferring the dynamics of the objects in a video A into the objects of a video B. Our proposed approach consists in training an autoencoder-like architecture that will split the given videos into an appearance-dynamics hyperspace that is used to synthesize the videos. To perform the transfer, we can use a video appearance representation with the dynamics of another video (extracted from the encoder), and then use the decoder to generate the output video. We will perform a video-dynamics cross training approach to learn robust appearance and dynamics spaces. The challenge we face is the lack of existing robust autoencoders for video data, as current generative methods do not achieve fully natural motion in video sequences yet. Thus, our work needs to create robust autoencoders for video generation, while solving the inter-frame motion consistency problem. (AU)

Scientific publications

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

SANTANDER, MIGUEL RODRIGUEZ; ALBARRACIN, JUAN HERNANDEZ; RIVERA, ADIN RAMIREZ. On the pitfalls of learning with limited data: A facial expression recognition case study. EXPERT SYSTEMS WITH APPLICATIONS, v. 183, NOV 30 2021. (19/07257-3, 16/19947-6, 17/16144-2)

ALBARRACIN, JUAN F. HERNANDEZ; RAMIREZ RIVERA, ADIN. Video Reenactment as Inductive Bias for Content-Motion Disentanglement. IEEE Transactions on Image Processing, v. 31, p. 10-pg., 2022-01-01. (17/16144-2, 19/07257-3)

Academic Publications

(References retrieved automatically from State of São Paulo Research Institutions)

ALBARRACÍN, Juan Felipe Hernández. Sobre o dilema espacial que vincula reencenação profunda e aprendizado de representações desemaranhadas em vídeo. 2023. Doctoral Thesis - Universidade Estadual de Campinas (UNICAMP). Instituto de Computação Campinas, SP.