Timber and Emotion: strategies for the analysis of orchestral sonorities using audio descriptors

• Grant number: 15/25160-6
• Support Opportunities: Scholarships in Brazil - Post-Doctoral
• Start date: April 01, 2016
• End date: March 31, 2018
• Field of knowledge: Linguistics, Literature and Arts - Arts - Music
• Principal Investigator: Marcelo Gomes de Queiroz
• Grantee: Ivan Eiji Yamauchi Simurra
• Host Institution: Instituto de Matemática e Estatística (IME). Universidade de São Paulo (USP). São Paulo , SP, Brazil

Abstract

This project proposes an analytic study of the correlates of affective states of emotions with the timbre characteristics of orchestral instruments. This is a post-doctoral research that builds upon our doctoral studies and extends it, through the development and implementation of a computational tool to analyze the orchestral sonorities from the standpoint of physiological responses associated with affective states of emotion. The research relies mainly on three fundamental aspects: a) the study of audio descriptors such as sound analysis tools; b) conducting experiments to collect objective data of the sounds; and c) the analysis of data collected by the experiments that will serve as reference for the development of models associated with the recognition and induction of emotional responses correlated with musical timbre. In particular, our research will carry out a total of two experiments: the first will collect objective data in the form of audio descriptors from selected sounds, using a computing environment that has been developed in our doctoral research. This environment allows building orchestral configurations from a database of sound recordings of various instrumental techniques. In the second experiment we will correlate the results of the first experiment with semantic descriptions of timbre perception. We will map the timbre characteristics via audio features from the perception of the semantic qualities already used in the psycho-acoustic literature such as `shiny',`empty, `rough', `metallic', `rough', `dense', `soft', among others. At the same time, we will collect the physiological responses of the participants by the use of equipment such as electroencephalography (EEG), heart rate variability (HRV) and galvanic skin conductance response (GSR). At this stage we will associate the semantic descriptions of timbre perception with emotional responses of arousal and valence, from the standpoint of the circumplex model of affective states of emotion proposed by James Russell, and look for statistic correlations from the analysis of the data collected in the two experiments. This is an interdisciplinary research proposal that will be developed at the NuSom -- Sonology Research Center at USP, and the Institute of Mathematics and Statistics of the University of São Paulo.