Generating natural language descriptions of objects in visual contexts

• Grant number: 13/23169-0
• Support Opportunities: Regular Research Grants
• Start date: June 01, 2014
• End date: May 31, 2016
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Ivandre Paraboni
• Grantee: Ivandre Paraboni
• Host Institution: Escola de Artes, Ciências e Humanidades (EACH). Universidade de São Paulo (USP). São Paulo , SP, Brazil

Abstract

This project is focused on the Natural Language Generation field, addressing the computational problem of describing objects in visual contexts. The project will address tree main computational challenges, which are currently the focus of ongoing research at this institute, namely: (a) the use of relational descriptions; (b) the generation of descriptions with various degrees of under and overspecification; and (c) the issue of human variation in the generation of these expressions.To this end, the project proposes collecting and annotating a large-scale, language-independent computational linguistic resource - a corpus of descriptions in a visual domain - and using this resource to build a computational model that extends our previous work on the issues (a-c) above.The corpus will be made publicly available for reuse for the wide research community (e.g., likewise TUNA, GRE3D7 and other high-impact NLG resources), and the computational model built from the data is expected to further the knowledge on the computational task of generating object descriptions in visual domain for practical applications such as those involving virtual interactive worlds etc. (AU)