Learning Visual Clues of the Passage of Time

• Grant number: 19/15822-2
• Support Opportunities: Scholarships abroad - Research Internship - Doctorate
• Start date: December 01, 2019
• End date: November 30, 2020
• Field of knowledge: Physical Sciences and Mathematics - Computer Science
• Principal Investigator: Anderson de Rezende Rocha
• Grantee: Rafael Soares Padilha
• Supervisor: Nathan Jacobs
• Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Institution abroad: University of Kentucky (UK), United States
• Associated to the scholarship: 17/21957-2 - Learning Visual Clues of the Passage of Time, BP.DR

Abstract

As smartphones became ubiquitous in modern life, people started to register and share moments of their lives in social networks. Modern events, from music concerts to a historical building consumed by fire, generate a massive amount of data, depicting different points of view and moments in time. To understand how the event unfolded, it is essential to organize available footage in time and reconstruct the order of facts. However, data produced in real-world events not always have reliable annotations with respect to time-of-capture, hindering manual reconstruction of its timeline.In this internship project, we propose methods to automatically sort images and videos in time by analyzing their visual content. For this, we will train an event-agnostic model to learn directly from data a representation that emphasizes the passage of time. Then, this representation will be used as input to a model that infers the chronological order between a pair of images. Finally, considering pairwise relations between available data, we will reconstruct the timeline of that event. The proposed methods will be optimized in training setups that consider different levels of annotation with respect to time-of-capture, and will be evaluated with data gathered from real-world events.

Scientific publications

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

PADILHA, RAFAEL; SALEM, TAWFIQ; WORKMAN, SCOTT; ANDALO, FERNANDA A.; ROCHA, ANDERSON; JACOBS, NATHAN. Content-Aware Detection of Temporal Metadata Manipulation. IEEE Transactions on Information Forensics and Security, v. 17, p. 12-pg., 2022-01-01. (17/21957-2, 19/15822-2, 17/12646-3)