Understanding images and deep learning models

• Grant number: 17/25835-9
• Support Opportunities: Research Grants - Research Partnership for Technological Innovation - PITE
• Start date: September 01, 2018
• End date: August 31, 2022
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Agreement: Microsoft Research
• Principal Investigator: Nina Sumiko Tomita Hirata
• Grantee: Nina Sumiko Tomita Hirata
• Host Institution: Instituto de Matemática e Estatística (IME). Universidade de São Paulo (USP). São Paulo , SP, Brazil
• City of the host institution: São Paulo
• Company: Microsoft Informática Ltda
• Company city: São Paulo
• Associated researchers: Alexandru-Cristian Telea ; Frank Dennis Julca Aguilar ; Harold Mouchere ; Roberto Hirata Junior
• Associated scholarship(s): 21/02382-4 - Chest CT image analysis using deep learning models: a case study on COVID-19, BP.MS; 20/07089-0 - Visualization in Deep Learning based Image Classification Tasks, BP.IC; 20/02891-3 - Segmentation of images with fully convolutional networks, BP.IC; 19/17873-3 - Technical support for the project Understanding Images and Deep Learning Models, BP.TT; 18/25671-9 - Astronomical image processing and analysis using deep convolutional models, BP.MS

Abstract

A central goal in the field of Computer Vision is image understanding. In general, appearance cues are used to detect components of interest and then spatial and hierarchical relations among these components are used to "describe" the image content at the semantic level of interest. Current deep models have reached a stage of evolution in which they are able to learn and transfer low level features from one domain to another. However, structural information of images such as spatial and hierarchical relations between constituent components are still explicitly modeled using case specific details. This makes models harder to be understood, useful only for few specific applications, and implications on training data preparation effort is still unclear. The aim of this project is the development of a structure-aware-semantics-unaware deep model, with abilities to learn and encode structural information regardless of the semantic level of image components. This should impact model understandability (as structural information would be more explicitly encoded) and training data requirements (as transfer learning would be possible). Theoretical studies, development of visualization strategies and new deep models, and experimentation with respect to diverse computer vision tasks are planned. (AU)

Scientific publications

(16)

(The scientific publications listed on this page originate from the Web of Science or SciELO databases. Their authors have cited FAPESP grant or fellowship project numbers awarded to Principal Investigators or Fellowship Recipients, whether or not they are among the authors. This information is collected automatically and retrieved directly from those bibliometric databases.)

NAKAZONO, L.; DE OLIVEIRA, C. MENDES; HIRATA, N. S. T.; JERAM, S.; QUEIROZ, C.; EIKENBERRY, STEPHEN S.; GONZALEZ, A. H.; ABRAMO, R.; OVERZIER, R.; ESPADOTO, M.; et al. On the discovery of stars, quasars, and galaxies in the Southern Hemisphere with S-PLUS DR2. Monthly Notices of the Royal Astronomical Society, v. 507, n. 4, p. 5847-5868, NOV 2021. (18/09165-6, 19/06766-1, 19/10923-5, 15/22308-2, 19/26492-3, 18/20977-2, 14/10566-4, 19/01312-2, 18/25671-9, 19/23388-0, 17/25835-9, 15/11442-0, 16/12331-0)

OLIVEIRA, ARTUR ANDRE A. M.; ESPADOTO, MATEUS; HIRATA JR, ROBERTO; TELEA, ALEXANDRU C.; HURTER, C; PURCHASE, H; BOUATOUCH, K. SDBM: Supervised Decision Boundary Maps for Machine Learning Classifiers. PROCEEDINGS OF THE 17TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS (VISAPP), VOL 4, v. N/A, p. 11-pg., 2022-01-01. (15/22308-2, 17/25835-9)

ESPADOTO, MATEUS; HIRATA, NINA; TELEA, ALEXANDRU; HURTER, C; PURCHASE, H; BRAZ, J; BOUATOUCH, K. Self-supervised Dimensionality Reduction with Neural Networks and Pseudo-labeling. VISAPP: PROCEEDINGS OF THE 16TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS - VOL. 5: VISAPP, v. N/A, p. 11-pg., 2021-01-01. (15/22308-2, 17/25835-9)

HIRATA, NINA S. T.; PAPAKOSTAS, GEORGE A.. On Machine-Learning Morphological Image Operators. MATHEMATICS, v. 9, n. 16, AUG 2021. (17/25835-9, 15/22308-2)

BLANGER, LEONARDO; HIRATA, NINA S. T.; JIANG, XIAOYI; IEEE COMP SOC. Reducing the need for bounding box annotations in Object Detection using Image Classification data. 2021 34TH SIBGRAPI CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI 2021), v. N/A, p. 8-pg., 2021-01-01. (18/00390-7, 15/22308-2, 17/25835-9, 19/17312-1)

BLANGER, LEONARDO; HIRATA, NINA S. T.; IEEE. AN EVALUATION OF DEEP LEARNING TECHNIQUES FOR QR CODE DETECTION. 2019 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), v. N/A, p. 5-pg., 2019-01-01. (17/25835-9, 18/00390-7, 15/22308-2)

MARTINAZZO, ANA; ESPADOTO, MATEUS; HIRATA, NINA S. T.; IEEE COMP SOC. Self-supervised Learning for Astronomical Image Classification. 2020 25TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), v. N/A, p. 7-pg., 2021-01-01. (17/25835-9, 18/25671-9)

A M OLIVEIRA, ARTUR ANDRE; ESPADOTO, MATEUS; HIRATA, ROBERTO, JR.; T HIRATA, NINA S.; TELEA, ALEXANDRU C.. Improving Self-supervised Dimensionality Reduction: Exploring Hyperparameters and Pseudo-Labeling Strategies. COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS, VISIGRAPP 2021, v. 1691, p. 27-pg., 2023-01-01. (17/25835-9, 15/22308-2, 20/13275-1)

RODRIGUES, NATALIA V. N.; ABRAMO, L. RAUL; HIRATA, NINA S. T.. The information of attribute uncertainties: what convolutional neural networks can learn about errors in input data. MACHINE LEARNING-SCIENCE AND TECHNOLOGY, v. 4, n. 4, p. 25-pg., 2023-12-01. (19/26492-3, 17/25835-9, 15/22308-2)

MONTEIRO SILVA, AUGUSTO CESAR; HIRATA, NINA S. T.; JIANG, XIAOYI; IEEE COMP SOC. Skeletal Similarity based Structural Performance Evaluation for Document Binarization. 2020 17TH INTERNATIONAL CONFERENCE ON FRONTIERS IN HANDWRITING RECOGNITION (ICFHR 2020), v. N/A, p. 6-pg., 2020-01-01. (15/22308-2, 19/07361-5, 18/00477-5, 17/25835-9)

ESPADOTO, MATEUS; RODRIGUES, FRANCISCO C. M.; HIRATA, NINA S. T.; TELEA, ALEXANDRU C.; HURTER, C; PURCHASE, H; BRAZ, J; BOUATOUCH, K. OptMap: Using Dense Maps for Visualizing Multidimensional Optimization Problems. VISAPP: PROCEEDINGS OF THE 16TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS - VOL. 5: VISAPP, v. N/A, p. 10-pg., 2021-01-01. (15/22308-2, 17/25835-9)

ESPADOTO, MATEUS; MARTINS, RAFAEL M.; KERREN, ANDREAS; HIRATA, NINA S. T.; TELEA, ALEXANDRU C.. Toward a Quantitative Survey of Dimension Reduction Techniques. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, v. 27, n. 3, p. 2153-2173, MAR 1 2021. (17/25835-9)

HAYASHI, SERGIO Y.; HIRATA, NINA S. T.; IEEE. Understanding attention-based encoder-decoder networks: a case study with chess scoresheet recognition. 2022 26TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), v. N/A, p. 7-pg., 2022-01-01. (15/22308-2, 17/25835-9)

MARTINAZZO, ANA; ESPADOTO, MATEUS; HIRATA, NINA S. T.; FARINELLA, GM; RADEVA, P; BRAZ, J. Deep Learning for Astronomical Object Classification: A Case Study. VISAPP: PROCEEDINGS OF THE 15TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS, VOL 4: VISAPP, v. N/A, p. 9-pg., 2020-01-01. (17/25835-9, 18/25671-9, 15/22308-2)

ESPADOTO, M.; RODRIGUES, F. C. M.; HIRATA, N. S. T.; HIRATA, R., JR.; TELEA, A. C.. Deep Learning Inverse Multidimensional Projections. EUROVIS WORKSHOP ON VISUAL ANALYTICS, EUROVA 2019, v. N/A, p. 5-pg., 2019-01-01. (17/25835-9)

BARRERA, JUNIOR; HASHIMOTO, RONALDO F.; HIRATA, NINA S. T.; HIRATA, R., JR.; REIS, MARCELO S.. From Mathematical Morphology to machine learning of image operators. SAO PAULO JOURNAL OF MATHEMATICAL SCIENCES, v. 16, n. 1, p. 42-pg., 2022-04-21. (15/22308-2, 13/07467-1, 19/21619-5, 17/25835-9)