Video understanding through deep learning with minimal human supervision

• Grant number: 23/17577-0
• Support Opportunities: Regular Research Grants
• Start date: September 01, 2024
• End date: August 31, 2026
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Jurandy Gomes de Almeida Junior
• Grantee: Jurandy Gomes de Almeida Junior
• Host Institution: Centro de Ciências em Gestão e Tecnologia (CCGT). Universidade Federal de São Carlos (UFSCAR). Campus de Sorocaba. Sorocaba , SP, Brazil
• City of the host institution: Sorocaba
• Associated researchers: Daniel Carlos Guimarães Pedronette ; João Paulo Papa ; Niculae Sebe ; Roberto Silva Netto ; Tiago Agostinho de Almeida
• Associated research grant(s): 24/22985-3 - AI Based Environment Domain Adaptation for Automotive Safety, AP.R

Abstract

Digital videos have become the medium of choice for a growing number of people communicating via Internet and their mobile devices. Over the past decade, world has witnessed an explosive growth in the amount of video data fostered by astonishing technological developments. In this scenario, there is a growing demand for efficient systems to reduce the work and information overload for people. Making efficient use of video content requires the development of intelligent tools capable to understand videos in a similar way as humans do. This has been the goal of a quickly evolving research area known as video understanding. Last years, thanks to its great learning capacity from data exposure, deep learning has led to remarkable progress in video understanding. Largely, this is due to the availability of large amounts of labeled data that has contributed to the development of models with extraordinary inference capabilities. Acquiring and annotating a desirable amount of data to satisfy these models is often a hard task for many application domains, requiring overwhelming human effort and, sometimes, specific expertise. This reliance on exhaustive labeling is a key limitation to deploying video understanding systems in the real world. Although deep learning models excel in many video understanding tasks, occasionally beating humans, their generalization ability is poor, only performing tasks they are trained for. The ability to adapt to novel scenarios is the hallmark of human intelligence. Motivated by such aspects, this research proposal aims to investigate methods to improve the generalization ability of deep learning, enabling to perform video understanding with minimal human supervision. The main scientific contribution will be a new and more sustainable pipeline for deep learning applications, requiring tailored human knowledge to disambiguate only critical decisions. For this, we intend to advance the state of the art in model generalization, aiming to learn with most informative data and put the human in the loop in a more effective manner. Finally, the results of this research proposal are also meant to enable companies to put AI into their products or production systems reducing the need for experienced operator knowledge for data annotation. (AU)

Scientific publications

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

DA SILVA, PUBLIO ELON CORREA; ALMEIDA, JURANDY. An Edge Computing-Based Solution for Real-Time Leaf Disease Classification Using Thermal Imaging. IEEE Geoscience and Remote Sensing Letters, v. 22, p. 5-pg., 2025-01-01. (23/17577-0)

ALVARENGA E SILVA, LUCAS FERNANDO; DOS SANTOS, SAMUEL FELIPE; SEBE, NICU; ALMEIDA, JURANDY. Beyond the known: Enhancing Open Set Domain Adaptation with unknown exploration. PATTERN RECOGNITION LETTERS, v. 189, p. 8-pg., 2025-02-28. (21/13348-1, 13/08293-7, 20/08770-3, 24/04500-2, 23/17577-0, 19/17874-0, 23/03328-9, 17/25908-6)

SANTOS, SAMUEL FELIPE DOS; BERRIEL, RODRIGO; OLIVEIRA-SANTOS, THIAGO; SEBE, NICU; ALMEIDA, JURANDY. Budget-aware pruning: Handling multiple domains with less parameters. PATTERN RECOGNITION, v. 167, p. 10-pg., 2025-11-01. (24/04500-2, 23/17577-0)