Advanced search
Start date
Betweenand

Unified representations considering visual attributes and textual semantics in image recognition tasks

Grant number: 18/23392-5
Support type:Scholarships in Brazil - Scientific Initiation
Effective date (Start): March 01, 2019
Effective date (End): February 29, 2020
Field of knowledge:Physical Sciences and Mathematics - Computer Science
Principal Investigator:Moacir Antonelli Ponti
Grantee:Juliana de Mello Crivelli
Home Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil
Associated research grant:13/07375-0 - CeMEAI - Center for Mathematical Sciences Applied to Industry, AP.CEPID

Abstract

The human cognitive-visual system is capable of abstracting visual concepts from multiple elements in a scene. For instance, it is possible to categorize a photo with people in the theme work or vacation according to visual attributes such as clothing and objects in the scene. From the point of view of computer vision and pattern recognition, these representations could be identified as the same category. Therefore, abstract visual characteristics commonly extracted by computational vision methods are usually insufficient, being necessary complement with semantic information. In this project semantic information complimentary to abstract visual characteristics will be investigated. In particular, characteristics obtained by convolution neural networks will be used as abstract visual representations, complemented by categorical textual information from object recognition methods or annotations. As result, first we aim to understand improvement in representation when characteristics are combined, and in second place how to extend the methods for translating visual characteristics in textual characteristics and vice versa. Possible applications include scene description, visual sub-categories detection, anomaly detection and others.