Expressive Multimodal TTS for Robots

• Grant number: 25/09875-7
• Support Opportunities: Scholarships abroad - Research Internship - Master's degree
• Start date: October 17, 2025
• End date: March 21, 2026
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Paula Dornhofer Paro Costa
• Grantee: Pedro Rodrigues Corrêa
• Supervisor: Thomas Hueber
• Host Institution: Faculdade de Engenharia Elétrica e de Computação (FEEC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Institution abroad: Université Grenoble Alpes (UGA), France
• Associated to the scholarship: 24/13098-3 - FACE: Facial Animation with Dynamic Contextual Emotions, BP.MS

Abstract

This project aims to enhance human-robot interaction by developing an expressive multimodal text-to-speech (TTS) pipeline capable of synchronizing emotionally nuanced speech with facial animation in robotic platforms. Building upon the ongoing FAPESP Master's research (FACE), we propose adapting and extending the FastLips baseline architecture to explicitly predict facial Action Units (AUs) compatible with the Furhat robot. Verbal emotional tags (Emotags) will be integrated to control the emotional style of synthesized speech and facial expressions. The audiovisual outputs will be evaluated through both online (simulated) and offline (physical robot) user studies, assessing expressiveness, realism, and interaction quality. Additionally, alternative TTS models will be explored to enhance the multimodal generation capabilities. The project leverages collaboration with the CRISSP team at GIPSA-lab, providing an unique opportunity to transition from screen-based avatars to physically embodied social agents, and advancing research in expressive, synchronized human-robot communication.