Spatiotemporal representation learning and zero-shot learning using tensor factorization

• Grant number: 17/00728-5
• Support Opportunities: Scholarships abroad - Research Internship - Doctorate
• Start date: April 04, 2017
• End date: March 30, 2018
• Field of knowledge: Physical Sciences and Mathematics - Computer Science
• Principal Investigator: Moacir Antonelli Ponti
• Grantee: Gabriel de Barros Paranhos da Costa
• Supervisor: Timothy M. Hospedales
• Host Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil
• Institution abroad: University of Edinburgh, Scotland
• Associated to the scholarship: 15/05310-3 - Representation Learning of spatio-temporal features from video, BP.DR

Abstract

Representation learning has shown promising results when used to extract relevant features from different image and video datasets, even when the representation model was not trained for that specific dataset or task. Zero-shot Learning aims to allow classifiers to identify classes that are not represented in the training set by incorporating semantic representations into the model. By combining both semantic and visual representations, it is possible to create generic representations that can be used even to classify new examples into classes with no visual examples in the training set. This can be accomplished by using tensor factorisation, which reduces the number of parameters that need to be learnt while sharing information between tasks or domains. During this research internship we intend to use tuples as semantic descriptors that will be embedded into the same parameter space as the video descriptors. By doing so, we can retrieve videos when given a tuple by using a nearest neighbour approach. The internship will take place at the University of Edinburgh under the supervision of Prof. Dr Timothy Hospedales.