Analysis of Oral Exams With Speaker Diarization and Speech Emotion Recognition: A Case Study

Author(s)

Beccaro, Wesley ; Ramirez, Miguel Arjona ; Liaw, William ; Guimaraes, Heitor Rodrigues

• Document type: Journal article
• Source: IEEE TRANSACTIONS ON EDUCATION; v. 67, n. 1, p. 13-pg., 2023-10-18.

Abstract

Contribution: During the lockdown period of the COVID-19 pandemic, online oral exams have become a necessary reality. Considering the possibility of carrying out and recording the oral exam sessions, analyses were conducted to assess how the speaking time ratio and emotional states are affected during the exams.Background: Although many studies have indicated that oral exams are an effective tool to evaluate students' conceptual understanding, their use has been limited or disregarded due to concerns about students' anxiety/emotional state, technical failures, and scalability.Research Questions: Two hypotheses were formulated. First, better grades are obtained when students maintain the flow of response, with few pauses/interruptions and hesitations during the oral exam. Second, the prevalence of more positive (i.e., happiness) and neutral emotions points out better grades whereas the prevalence of negative (i.e., sadness and anger) emotions implies worse students' performance.Methodology: Speaker diarization algorithms have been employed to extract speaking ratios and students' speech segments. A transfer learning method for speech emotion recognition (SER) has been used to distinguish students' emotional state in four emotional classes by using a pretrained HuBERT model and an InceptionTime network.Findings: Correlation tests demonstrate that students' speaking ratios are correlated to their grades, that is, the higher the students' speaking ratios are, the better the students' grades tend to be. The SER model indicates a neutral emotional state in most periods of the oral test and sadness in some hesitation moments. However, correlation tests point out that emotional states are not correlated with the students' grades. (AU)

• FAPESP's process: 22/10909-5 - Differentiable learning and processing of temporal, spatial, spectral and time-frequency signal representations
• Grantee: Miguel Arjona Ramírez
• Support Opportunities: Regular Research Grants