Encontrando evidências visuais da passagem do tempo

Author(s)

Rafael Soares Padilha

• Document type: Doctoral Thesis
• Press: Campinas, SP.
• Institution: Universidade Estadual de Campinas (UNICAMP). Instituto de Computação
• Defense date: 2022-07-29

Examining board members

Anderson de Rezende Rocha; Marjory Cristiany da Costa Abreu; Cristina Nader Vasconcelos; Agma Juci Machado Traina

• Advisor: Anderson de Rezende Rocha; Fernanda Alcântara Andaló

Abstract

Time has always been a mysterious and elusive concept that humans struggle to formally understand and model. When we take a photograph, we visually imprint the temporal context of that moment. From the illumination and color of the scene to how people dress and what the world around us looks like; all such elements carry some degree of temporal information. In this thesis, we investigate how we can visually perceive the passage of time and mine temporal knowledge for forensic tasks. To better understand such concepts, we investigate two temporal inference tasks: temporal metadata verification and chronological sorting of pictures. In the former problem, we propose a high-level deep neural network architecture that combines visual, temporal, and geospatial information to verify the alleged timestamp of a picture. In the latter, we combine a series of binary classifiers with the input of a forensic expert to sort in time images from real-world events. We evaluate our methods in realistic data originating from social media, with several experiments to assess when our techniques work and fail. With the backdrop of forensic application scenarios, we envision accurate techniques that are also transparent and interpretable. With this in mind, we consider the explainability of the proposed techniques, inspecting what decisive visual elements contributed to the algorithmic decisions. The insights gained in this research highlight the importance of combining multiple data modalities (e.g., visual, geographical, and those related to scene appearance), increasing data variability through data augmentation strategies, and keeping the forensic expert in the loop throughout the design and evaluation of forensic methods (AU)

• FAPESP's process: 17/21957-2 - Learning Visual Clues of the Passage of Time
• Grantee: Rafael Soares Padilha
• Support Opportunities: Scholarships in Brazil - Doctorate