Development of methods for image captioning: a framework based on computer vision and natural language processing

• Grant number: 22/12204-9
• Support Opportunities: Scholarships in Brazil - Post-Doctoral
• Start date: December 01, 2022
• End date: March 31, 2023
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Roberto Marcondes Cesar Junior
• Grantee: Rafael Jeferson Pezzuto Damaceno
• Host Institution: Instituto de Matemática e Estatística (IME). Universidade de São Paulo (USP). São Paulo , SP, Brazil
• Associated research grant: 15/22308-2 - Intermediate representations in Computational Science for knowledge discovery, AP.TEM

Abstract

Scene description is a process that aims to associate one or more textual sentences to an image. Concerning urban infrastructure, this is a technique that allows the description of urban scenarios, such as the characterization of sidewalks in terms of size, the identification of objects and the positional relationship they have with each other, among others. In recent years important advances have been achieved in this task through the use of deep neural network techniques. These advances are facilitated by the greater availability of GPUs and large data sets. Furthermore, advances achieved by neural networks have been mainly obtained in individual modalities such as vision, language or sound. In many cases, real-world problems have components that are embedded in more than one modality - as can be the scenarios of Urban Informatics. In this sense, this project aims at developing computational methods to improve the processes of description of urban scenes. As object of study, we intend to explore remote-sensing and street-level images. The project will involve the use of computer vision techniques associated with natural language processing. The main idea is to use the textual language framework to improve the generated descriptions.