DejaVu: social media forensics for interpreting criminal events

• Grant number: 15/19222-9
• Support Opportunities: Scholarships abroad - Research
• Start date: July 01, 2016
• End date: June 30, 2017
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Anderson de Rezende Rocha
• Grantee: Anderson de Rezende Rocha
• Host Investigator: Kevin Wilson Bowyer
• Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Institution abroad: University of Notre Dame, United States

Abstract

With demanding and sophisticated crimes and terrorist threats becoming ever more pervasive, it is paramount to design and develop objective approaches to help us answer the four most important questions in forensics regarding an event: "who", "in what circumstances", "why", and "how". Although aided by surveillance cameras when answering such questions, we often face situations in which cameras are not installed or are insufficient to cover the event. Leveraging on the rise of social media, we can rely on this new channel with orders of magnitude higher in the volume of content available for event understanding and reconstruction. The aim is to complement the content from existing, if any, surveillance cameras at the event location. We could potentially provide a much clearer "bird's eye view" of an event, along with reconstructed scenes from various perspectives of people or objects present as well as mine information related to the event found on social media. In this context, given an event to be investigated, our objective is to gather social media information related in position and time to such an event and mine the collected information to understand what happened, reconstruct the event's timeline, narrow down the search-space of suspects and come up with possible answers to the four aforementioned forensics questions.

Scientific publications

(18)

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

WERNECK, RAFAEL DE OLIVEIRA; DE ALMEIDA, WALDIR RODRIGUES; STEIN, BERNARDO VECCHIA; PAZINATO, DANIEL VATANABE; MENDES JUNIOR, PEDRO RIBEIRO; BIZETTO PENATTI, OTAVIO AUGUSTO; ROCHA, ANDERSON; TORRES, RICARDO DA SILVA. Kuaa: A unified framework for design, deployment, execution, and recommendation of machine learning experiments. FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, v. 78, n. 1, p. 59-76, JAN 2018. (16/18429-1, 15/19222-9, 13/50155-0, 14/12236-1, 13/50169-1)

SEIBEL, JR., HILARIO; GOLDENSTEIN, SIOME; ROCHA, ANDERSON. Eyes on the Target: Super-Resolution and License-Plate Recognition in Low-Quality Surveillance Videos. IEEE ACCESS, v. 5, p. 20020-20035, 2017. (15/19222-9)

ROCHA, ANDERSON; SCHEIRER, WALTER J.; FORSTALL, CHRISTOPHER W.; CAVALCANTE, THIAGO; THEOPHILO, ANTONIO; SHEN, BINGYU; CARVALHO, ARIADNE R. B.; STAMATATOS, EFSTATHIOS. Authorship Attribution for Social Media Forensics. IEEE Transactions on Information Forensics and Security, v. 12, n. 1, p. 5-33, JAN 2017. (15/19222-9)

MOREIRA, DANIEL; AVILA, SANDRA; PEREZ, MAURICIO; MORAES, DANIEL; TESTONI, VANESSA; VALLE, EDUARDO; GOLDENSTEIN, SIOME; ROCHA, ANDERSON. Pornography classification: The hidden clues in video space-time. Forensic Science International, v. 268, p. 46-61, NOV 2016. (15/19222-9)

MARMEROLA, GUILHERME D.; OIKAWA, MARINA A.; DIAS, ZANONI; GOLDENSTEIN, SIOME; ROCHA, ANDERSON. On the Reconstruction of Text Phylogeny Trees: Evaluation and Analysis of Textual Relationships. PLoS One, v. 11, n. 12, DEC 19 2016. (14/19401-8, 15/19222-9, 14/13433-5, 13/08293-7, 14/03535-5)

PEREZ, MAURICIO; AVILA, SANDRA; MOREIRA, DANIEL; MORAES, DANIEL; TESTONI, VANESSA; VALLE, EDUARDO; GOLDENSTEIN, SIOME; ROCHA, ANDERSON. Video pornography detection through deep learning techniques and motion information. Neurocomputing, v. 230, p. 279-293, MAR 22 2017. (15/19222-9)

FERREIRA, ANSELMO; BONDI, LUCA; BAROFFIO, LUCA; BESTAGINI, PAOLO; HUANG, JIWU; DOS SANTOS, JEFERSSON A.; TUBARO, STEFANO; ROCHA, ANDERSON. Data-Driven Feature Characterization Techniques for Laser Printer Attribution. IEEE Transactions on Information Forensics and Security, v. 12, n. 8, p. 1860-1873, AUG 2017. (15/19222-9)

FERREIRA, ALEXANDRE; SANTOS, GEISE; ROCHA, ANDERSON; GOLDENSTEIN, SIOME. User-Centric Coordinates for Applications Leveraging 3-Axis Accelerometer Data. IEEE SENSORS JOURNAL, v. 17, n. 16, p. 5231-5243, AUG 15 2017. (15/19222-9)

BHARATI, A.; MOREIRA, D.; PINTO, A.; BROGAN, J.; BOWYER, K.; FLYNN, P.; SCHEIRER, W.; ROCHA, A.; IEEE. U-PHYLOGENY: UNDIRECTED PROVENANCE GRAPH CONSTRUCTION IN THE WILD. 2017 24TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), v. N/A, p. 5-pg., 2017-01-01. (15/19222-9)

BROGAN, JOEL; BESTAGINI, PAOLO; BHARATI, APARNA; PINTO, ALLAN; MOREIRA, DANIEL; BOWYER, KEVIN; FLYNN, PATRICK; ROCHA, ANDERSON; SCHEIRER, WALTER; IEEE. SPOTTING THE DIFFERENCE: CONTEXT RETRIEVAL AND ANALYSIS FOR IMPROVED FORGERY DETECTION AND LOCALIZATION. 2017 24TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), v. N/A, p. 5-pg., 2017-01-01. (15/19222-9)

PINTO, A.; MOREIRA, D.; BHARATI, A.; BROGAN, J.; BOWYER, K.; FLYNN, P.; SCHEIRER, W.; ROCHA, A.; IEEE. PROVENANCE FILTERING FOR MULTIMEDIA PHYLOGENY. 2017 24TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), v. N/A, p. 5-pg., 2017-01-01. (15/19222-9)

WERNECK, RAFAEL DE OLIVEIRA; DE ALMEIDA, WALDIR RODRIGUES; STEIN, BERNARDO VECCHIA; PAZINATO, DANIEL VATANABE; MENDES JUNIOR, PEDRO RIBEIRO; BIZETTO PENATTI, OTAVIO AUGUSTO; ROCHA, ANDERSON; TORRES, RICARDO DA SILVA. Kuaa: A unified framework for design, deployment, execution, and recommendation of machine learning experiments. FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, v. 78, p. 18-pg., 2018-01-01. (13/50169-1, 13/50155-0, 14/12236-1, 16/18429-1, 15/19222-9)

NETO, LAURINDO BRITTO; GRIJALVA, FELIPE; MARGARETH LIMA MAIKE, VANESSA REGINA; MARTINI, LUIZ CESAR; FLORENCIO, DINEI; CALANI BARANAUSKAS, MARIA CECILIA; ROCHA, ANDERSON; GOLDENSTEIN, SIOME. A Kinect-Based Wearable Face Recognition System to Aid Visually Impaired Users. IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, v. 47, n. 1, p. 52-64, FEB 2017. (12/50468-6, 15/19222-9, 13/21349-1, 14/14630-9)

MOREIRA, DANIEL; AVILA, SANDRA; PEREZ, MAURICIO; MORAES, DANIEL; TESTONI, VANESSA; VALLE, EDUARDO; GOLDENSTEIN, SIOME; ROCHA, ANDERSON; IEEE. Temporal Robust Features for Violence Detection. 2017 IEEE WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION (WACV 2017), v. N/A, p. 9-pg., 2017-01-01. (15/19222-9)

FERREIRA, ANSELMO; FELIPUSSI, SIOVANI C.; ALFARO, CARLOS; FONSECA, PABLO; VARGAS-MUNOZ, JOHN E.; DOS SANTOS, JEFERSSON A.; ROCHA, ANDERSON. Behavior Knowledge Space-Based Fusion for Copy-Move Forgery Detection. IEEE Transactions on Image Processing, v. 25, n. 10, p. 4729-4742, OCT 2016. (15/19222-9)

MENDES JUNIOR, PEDRO R.; DE SOUZA, ROBERTO M.; WERNECK, RAFAEL DE O.; STEIN, BERNARDO V.; PAZINATO, DANIEL V.; DE ALMEIDA, WALDIR R.; PENATTI, OTAVIO A. B.; TORRES, RICARDO DA S.; ROCHA, ANDERSON. Nearest neighbors distance ratio open-set classifier. MACHINE LEARNING, v. 106, n. 3, p. 359-386, MAR 2017. (10/05647-4, 13/50155-0, 15/19222-9, 13/50169-1)

COSTA, FILIPE; OLIVEIRA, ALBERTO; FERRARA, PASQUALE; DIAS, ZANONI; GOLDENSTEIN, SIOME; ROCHA, ANDERSON. New dissimilarity measures for image phylogeny reconstruction. PATTERN ANALYSIS AND APPLICATIONS, v. 20, n. 4, p. 1289-1305, NOV 2017. (13/05815-2, 15/19222-9)

COSTA, FILIPE DE O.; LAMERI, SILVIA; BESTAGINI, PAOLO; DIAS, ZANONI; TUBARO, STEFANO; ROCHA, ANDERSON; IEEE. Hash-Based Frame Selection for Video Phylogeny. 2016 8TH IEEE INTERNATIONAL WORKSHOP ON INFORMATION FORENSICS AND SECURITY (WIFS 2016), v. N/A, p. 6-pg., 2016-01-01. (13/05815-2, 15/19222-9)