Advanced search
Start date
Betweenand

Convolutional Networks with Attention for Video Color Propagation

Grant number: 22/12294-8
Support Opportunities:Scholarships in Brazil - Doctorate
Start date: April 01, 2023
End date: August 08, 2026
Field of knowledge:Physical Sciences and Mathematics - Computer Science - Computer Systems
Principal Investigator:Hélio Pedrini
Grantee:Leandro Stival
Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
Associated scholarship(s):23/11556-1 - Novel deep learning methods for remote sensing imagery, BE.EP.DR

Abstract

The colorization process consists of reconstituting the colors of images or videos that did not capture this information at the time of obtaining the pixel information. In this sense, the color restoration of images and videos was originally done manually, which required an excess of hours in the coloring process and was something very prone to failure. With the emergence of digital images and videos, the colorization process was computerized, thus streamlining the process, however, the development remained expensive and manual.This way, this project will create a pipeline capable of predicting the color of a pixel in the original video when only a sample color image is available. This type of tool is very important for many practical applications, such as restoring old videos, remote sensing, x-rays, and coloring animations. And since there is still no optimal and defined solution to this problem, this research is more than necessary and current.The prediction method proposed in this project utilizes the latest techniques in machine learning, such as a U-net topology with an encoder and decoder, where the bottleneck will be implemented using a model based on ViT (Visual Transformer) to maintain the temporal context.The main difference between this project and the current works present in the literature goes beyond the ViT + U-net architecture. However, the choice to use keyframes allows for the identification of sudden changes in the scenes. This improves the temporal consistency of the process by individually coloring each scene based on its keyframe.The entire pipeline will be trained on a dataset created from videos from the Videvo website. Many short videos are available on this website without copywriting, which is a common technique among other works in the colorization literature.It is expected, at the end of the project, to obtain a robust model capable of making this type of prediction, as well as to advance the knowledge of ways to create loss functions for colorization methods. These are some of the most challenging aspects of the literature, together with maintaining color saturation, temporal consistency, and choosing the ideal color space for this type of method.

News published in Agência FAPESP Newsletter about the scholarship:
More itemsLess items
Articles published in other media outlets ( ):
More itemsLess items
VEICULO: TITULO (DATA)
VEICULO: TITULO (DATA)

Scientific publications (4)
(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)
STIVAL, LEANDRO; TORRES, RICARDO DA SILVA; PEDRINI, HELIO. Video Colorization Based on a Diffusion Model Implementation. INTELLIGENT SYSTEMS AND APPLICATIONS, VOL 1, INTELLISYS 2024, v. 1065, p. 15-pg., . (23/11556-1, 22/12294-8)
STIVAL, LEANDRO; PEDRINI, HELIO. Survey on Video Colorization: Concepts, Methods and Applications. Journal of Signal Processing Systems for Signal Image and Video Technology, v. 95, n. 6, p. 24-pg., . (22/12294-8)
STIVAL, LEANDRO; TORRES, RICARDO DA SILVA; PEDRINI, HELIO. Semantically-Aware Contrastive Learning for multispectral remote sensing images. ISPRS JOURNAL OF PHOTOGRAMMETRY AND REMOTE SENSING, v. 223, p. 15-pg., . (23/11556-1, 22/12294-8)
STIVAL, LEANDRO; TORRES, RICARDO DA SILVA; PEDRINI, HELIO. Enhancing Video Colorization with Deep Learning: A Comprehensive Analysis of Training Loss Functions. INTELLIGENT SYSTEMS AND APPLICATIONS, VOL 1, INTELLISYS 2024, v. 1065, p. 14-pg., . (23/11556-1, 22/12294-8)