Generating image descriptions on news websites

• Grant number: 18/11367-6
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: August 01, 2018
• End date: December 31, 2019
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Helena de Medeiros Caseli
• Grantee: João Gabriel Melo Barbirato
• Host Institution: Centro de Ciências Exatas e de Tecnologia (CCET). Universidade Federal de São Carlos (UFSCAR). São Carlos , SP, Brazil

Abstract

The amount of content produced on the internet and on social networks increases the demand for automatic systems capable of processing and transforming them into information. The particular interest of this project are news from online newspapers, massively produced and available on the internet. For a better understanding of the news and its images, this project proposes description generation for specific regions of the image based on: (1) the explicitly defined alignment between the text and the image of the news and (2) the semantic information present in the text. To do so, it is intended to use a text-image aligner developed in (VELTRONI, 2018) and Abstract Meaning Representation (AMR) of (BANARESCU et al., 2013). This proposal is linked to the project MMeaning (Regular Aid FAPESP # 2016 / 13002-0) and to the proposal "RSMM: Region-specific multimodal models and their application in language tasks" submitted to Newton Mobility Grants 2018 Round 1, proposed in partnership with Profa. Dr. Lucia Specia of the University of Sheffield.