Understanding images and deep learning models

• Grant number: 17/25835-9
• Support Opportunities: Research Grants - Research Partnership for Technological Innovation - PITE
• Start date: September 01, 2018
• End date: August 31, 2022
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Agreement: Microsoft Research
• Principal Investigator: Nina Sumiko Tomita Hirata
• Grantee: Nina Sumiko Tomita Hirata
• Host Institution: Instituto de Matemática e Estatística (IME). Universidade de São Paulo (USP). São Paulo , SP, Brazil
• Company: Microsoft Informática Ltda
• City: São Paulo
• Associated researchers: Alexandru-Cristian Telea ; Frank Dennis Julca Aguilar ; Harold Mouchere ; Roberto Hirata Junior
• Associated scholarship(s): 21/02382-4 - Chest CT image analysis using deep learning models: a case study on COVID-19, BP.MS; 20/07089-0 - Visualization in deep learning based image classification tasks, BP.IC; 20/02891-3 - Segmentation of images with fully convolutional networks, BP.IC; 19/17873-3 - Technical support for the project Understanding Images and Deep Learning Models, BP.TT; 18/25671-9 - Astronomical image processing and analysis using deep convolutional models, BP.MS

Abstract

A central goal in the field of Computer Vision is image understanding. In general, appearance cues are used to detect components of interest and then spatial and hierarchical relations among these components are used to "describe" the image content at the semantic level of interest. Current deep models have reached a stage of evolution in which they are able to learn and transfer low level features from one domain to another. However, structural information of images such as spatial and hierarchical relations between constituent components are still explicitly modeled using case specific details. This makes models harder to be understood, useful only for few specific applications, and implications on training data preparation effort is still unclear. The aim of this project is the development of a structure-aware-semantics-unaware deep model, with abilities to learn and encode structural information regardless of the semantic level of image components. This should impact model understandability (as structural information would be more explicitly encoded) and training data requirements (as transfer learning would be possible). Theoretical studies, development of visualization strategies and new deep models, and experimentation with respect to diverse computer vision tasks are planned. (AU)

Scientific publications

(13)

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

ESPADOTO, MATEUS; MARTINS, RAFAEL M.; KERREN, ANDREAS; HIRATA, NINA S. T.; TELEA, ALEXANDRU C.. Toward a Quantitative Survey of Dimension Reduction Techniques. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, v. 27, n. 3, p. 2153-2173, MAR 1 2021. (17/25835-9)

ESPADOTO, MATEUS; RODRIGUES, FRANCISCO C. M.; HIRATA, NINA S. T.; TELEA, ALEXANDRU C.; HURTER, C; PURCHASE, H; BRAZ, J; BOUATOUCH, K. OptMap: Using Dense Maps for Visualizing Multidimensional Optimization Problems. VISAPP: PROCEEDINGS OF THE 16TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS - VOL. 5: VISAPP, v. N/A, p. 10-pg., 2021-01-01. (15/22308-2, 17/25835-9)

HAYASHI, SERGIO Y.; HIRATA, NINA S. T.; IEEE. Understanding attention-based encoder-decoder networks: a case study with chess scoresheet recognition. 2022 26TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), v. N/A, p. 7-pg., 2022-01-01. (15/22308-2, 17/25835-9)

MARTINAZZO, ANA; ESPADOTO, MATEUS; HIRATA, NINA S. T.; FARINELLA, GM; RADEVA, P; BRAZ, J. Deep Learning for Astronomical Object Classification: A Case Study. VISAPP: PROCEEDINGS OF THE 15TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS, VOL 4: VISAPP, v. N/A, p. 9-pg., 2020-01-01. (17/25835-9, 18/25671-9, 15/22308-2)

MONTEIRO SILVA, AUGUSTO CESAR; HIRATA, NINA S. T.; JIANG, XIAOYI; IEEE COMP SOC. Skeletal Similarity based Structural Performance Evaluation for Document Binarization. 2020 17TH INTERNATIONAL CONFERENCE ON FRONTIERS IN HANDWRITING RECOGNITION (ICFHR 2020), v. N/A, p. 6-pg., 2020-01-01. (15/22308-2, 19/07361-5, 18/00477-5, 17/25835-9)

BARRERA, JUNIOR; HASHIMOTO, RONALDO F.; HIRATA, NINA S. T.; HIRATA, R., JR.; REIS, MARCELO S.. From Mathematical Morphology to machine learning of image operators. SAO PAULO JOURNAL OF MATHEMATICAL SCIENCES, v. 16, n. 1, p. 42-pg., 2022-04-21. (15/22308-2, 13/07467-1, 19/21619-5, 17/25835-9)

BLANGER, LEONARDO; HIRATA, NINA S. T.; JIANG, XIAOYI; IEEE COMP SOC. Reducing the need for bounding box annotations in Object Detection using Image Classification data. 2021 34TH SIBGRAPI CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI 2021), v. N/A, p. 8-pg., 2021-01-01. (18/00390-7, 15/22308-2, 17/25835-9, 19/17312-1)

ESPADOTO, MATEUS; HIRATA, NINA; TELEA, ALEXANDRU; HURTER, C; PURCHASE, H; BRAZ, J; BOUATOUCH, K. Self-supervised Dimensionality Reduction with Neural Networks and Pseudo-labeling. VISAPP: PROCEEDINGS OF THE 16TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS - VOL. 5: VISAPP, v. N/A, p. 11-pg., 2021-01-01. (15/22308-2, 17/25835-9)

MARTINAZZO, ANA; ESPADOTO, MATEUS; HIRATA, NINA S. T.; IEEE COMP SOC. Self-supervised Learning for Astronomical Image Classification. 2020 25TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), v. N/A, p. 7-pg., 2021-01-01. (17/25835-9, 18/25671-9)

OLIVEIRA, ARTUR ANDRE A. M.; ESPADOTO, MATEUS; HIRATA JR, ROBERTO; TELEA, ALEXANDRU C.; HURTER, C; PURCHASE, H; BOUATOUCH, K. SDBM: Supervised Decision Boundary Maps for Machine Learning Classifiers. PROCEEDINGS OF THE 17TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS (VISAPP), VOL 4, v. N/A, p. 11-pg., 2022-01-01. (15/22308-2, 17/25835-9)

NAKAZONO, L.; DE OLIVEIRA, C. MENDES; HIRATA, N. S. T.; JERAM, S.; QUEIROZ, C.; EIKENBERRY, STEPHEN S.; GONZALEZ, A. H.; ABRAMO, R.; OVERZIER, R.; ESPADOTO, M.; et al. On the discovery of stars, quasars, and galaxies in the Southern Hemisphere with S-PLUS DR2. Monthly Notices of the Royal Astronomical Society, v. 507, n. 4, p. 5847-5868, NOV 2021. (18/09165-6, 19/06766-1, 19/10923-5, 15/22308-2, 19/26492-3, 18/20977-2, 14/10566-4, 19/01312-2, 18/25671-9, 19/23388-0, 17/25835-9, 15/11442-0, 16/12331-0)

HIRATA, NINA S. T.; PAPAKOSTAS, GEORGE A.. On Machine-Learning Morphological Image Operators. MATHEMATICS, v. 9, n. 16, AUG 2021. (17/25835-9, 15/22308-2)

BLANGER, LEONARDO; HIRATA, NINA S. T.; IEEE. AN EVALUATION OF DEEP LEARNING TECHNIQUES FOR QR CODE DETECTION. 2019 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), v. N/A, p. 5-pg., 2019-01-01. (17/25835-9, 18/00390-7, 15/22308-2)