Representation Learning of spatio-temporal features from video

• Grant number: 15/05310-3
• Support Opportunities: Scholarships in Brazil - Doctorate
• Start date: July 01, 2015
• End date: January 31, 2019
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Agreement: Coordination of Improvement of Higher Education Personnel (CAPES)
• Principal Investigator: Moacir Antonelli Ponti
• Grantee: Gabriel de Barros Paranhos da Costa
• Host Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil
• Associated research grant: 11/22749-8 - Challenges in exploratory visualization of multidimensional data: paradigms, scalability and applications, AP.TEM
• Associated scholarship(s): 17/00728-5 - Spatiotemporal representation learning and zero-shot learning using tensor factorization, BE.EP.DR

Abstract

The method that is presently considered the state of the art on extraction of spatio-temporal features from videos is based on finding keypoints and dense trajectories to apply classical feature extraction algorithms, like SIFT and HOG. With the recent improvement on the processing capacity of CPUs and GPUs, combined with the increasing avaliability of video and image datasets, representation learning methods, especially deep learning methods, have reached state of the art performance in several areas in artificial inteligence and signal processing. Even with the good results obtained by these methods, they have only been applied a few times to spatio-temporal feature extraction from videos a few times. Research in this area have achieved promissing results, however, most of the methods focus on a single application, restricting the use of the learned features. Also, there are promissing concepts that were used to design hand-crafted features and were not yet tested in a representation learning context. In this project, we propose the development of representation learning algorithms for extraction of spatio-temporal features from videos. We expect that these methods will be able to extract features that are capable of describing events that can't be capture through a single frame, but with the development of a scene. We intend for these features to form a multidimensional time series that encodes spatial information. To evaluate the quality of the representations, visualization and projection techniques will be used to permit the analysis of the feature space. We will also use methods that allow us to visualize extracted features, this way it will be possible to create a connection between features and the events that occur in the videos. (AU)

Scientific publications

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

PONTI, MOACIR A.; DA COSTA, GABRIEL B. PARANHOS; SANTOS, FERNANDO P.; SILVEIRA, KAUE U.. Supervised and unsupervised relevance sampling in handcrafted and deep learning features obtained from image collections. APPLIED SOFT COMPUTING, v. 80, p. 414-424, JUL 2019. (16/16111-4, 13/07375-0, 15/05310-3)

NAZARE, TIAGO S.; PARANHOS DA COSTA, GABRIEL B.; DE MELLO, RODRIGO F.; PONTI, MOACIR A.; IEEE. Color quantization in transfer learning and noisy scenarios: an empirical analysis using convolutional networks. PROCEEDINGS 2018 31ST SIBGRAPI CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), v. N/A, p. 7-pg., 2018-01-01. (15/05310-3, 17/16548-6, 13/07375-0, 15/04883-0, 16/16111-4)

CONTATO, WELINTON A.; NAZARE, TIAGO S.; PARANHOS DA COSTA, GABRIEL B.; PONTI, MOACIR; BATISTA NETO, JOAO E. S.; IEEE. Improving Non-Local Video Denoising with Local Binary Patterns and Image Quantization. 2016 29TH SIBGRAPI CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), v. N/A, p. 8-pg., 2016-01-01. (15/05310-3, 14/21888-2, 15/04883-0)

Academic Publications

(References retrieved automatically from State of São Paulo Research Institutions)

COSTA, Gabriel de Barros Paranhos da. Representation learning of spatio-temporal features from video. 2019. Doctoral Thesis - Universidade de São Paulo (USP). Instituto de Ciências Matemáticas e de Computação (ICMC/SB) São Carlos.