Advanced search
Start date

Generating image descriptions on news websites

Grant number: 18/11367-6
Support type:Scholarships in Brazil - Scientific Initiation
Effective date (Start): August 01, 2018
Effective date (End): December 31, 2019
Field of knowledge:Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
Principal Investigator:Helena de Medeiros Caseli
Grantee:João Gabriel Melo Barbirato
Home Institution: Centro de Ciências Exatas e de Tecnologia (CCET). Universidade Federal de São Carlos (UFSCAR). São Carlos , SP, Brazil


The amount of content produced on the internet and on social networks increases the demand for automatic systems capable of processing and transforming them into information. The particular interest of this project are news from online newspapers, massively produced and available on the internet. For a better understanding of the news and its images, this project proposes description generation for specific regions of the image based on: (1) the explicitly defined alignment between the text and the image of the news and (2) the semantic information present in the text. To do so, it is intended to use a text-image aligner developed in (VELTRONI, 2018) and Abstract Meaning Representation (AMR) of (BANARESCU et al., 2013). This proposal is linked to the project MMeaning (Regular Aid FAPESP # 2016 / 13002-0) and to the proposal "RSMM: Region-specific multimodal models and their application in language tasks" submitted to Newton Mobility Grants 2018 Round 1, proposed in partnership with Profa. Dr. Lucia Specia of the University of Sheffield.