Generating audiovisual summaries from literary works using emotion analysis

Author(s)

Milon-Flores, Daniela F. ; Ochoa-Luna, Jose ; Gomez-Nieto, Erick ; IEEE

• Document type: Journal article
• Source: 2019 32ND SIBGRAPI CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI); v. N/A, p. 8-pg., 2019-01-01.

Abstract

Literature work reading is an essential activity for human communication and learning. However, several relevant tasks as selection, filter or analyze in a high number of such works become complex. For dealing with this requirement, several strategies are proposed to rapidly inspect substantial amounts of text, or retrieve information previously read, exploiting graphical, textual or auditory resources. In this paper, we propose a methodology to generate audiovisual summaries by the combination of emotion-based music composition and graph-based animation. We applied natural language processing algorithms for extracting emotions and characters involved in literary work. Then, we use the extracted information to compose a musical piece to accompany the visual narration of the story aiming to convey the extracted emotion. For that, we set important musical features as chord progressions, tempo, scale, and octaves, and we assign a set of suitable instruments. Moreover, we animate a graph to sum up the dialogues between the characters in the literary work. Finally, to assess the quality of our methodology, we conducted two user studies that reveal that our proposal provides a high level of understanding over the content of the literary work besides bringing a pleasant experience to the user. (AU)

• FAPESP's process: 19/10560-0 - Visual Analysis and Engineering of Urban Features for Crime Prediction in São Paulo City
• Grantee: Erick Mauricio Gómez Nieto
• Support Opportunities: Scholarships in Brazil - Post-Doctoral