Audio-Visual Speech Processing by Machine Learning

• Grant number: 18/26455-8
• Support Opportunities: Regular Research Grants
• Field of knowledge: Engineering - Electrical Engineering - Telecommunications
• Principal Investigator: Miguel Arjona Ramírez
• Grantee: Miguel Arjona Ramírez
• Host Institution: Escola Politécnica (EP). Universidade de São Paulo (USP). São Paulo , SP, Brazil
• Associated researchers: Celso Setsuo Kurashima ; Demostenes Zegarra Rodriguez ; Edmilson da Silva Morais ; Emilio Del Moral Hernandez ; Mario Minami ; Renata Lopes Rosa ; Wesley Beccaro
• Associated scholarship(s): 20/13459-5 - Evaluation of voice quality using machine learning and considering wireless networks, BP.TT; 19/10329-6 - Evaluation of voice quality using machine learning and considering wireless networks, BP.TT; 19/10551-0 - Video quality assessment using machine learning in network scenarios, BP.TT

Abstract

This research plan addresses a common basis for a number of areas in signal processing such as speech analysis, speech coding and audio coding, speech recognition and audio feature recognition as well as source separation with regularizations to carry out adjustments suitable to the desired application. Traditionally, speech analysis, in addition to its own importance, also provides signal representations and model parameters that are necessary to the other areas. In this role it is losing appeal with deep learning and parallels are set to be established in order to bring about some interpretation. Beyond usual types of time-frequency decomposition and modification and autoregressive analysis, new algorithms will be explored and proposed based on machine learning and deep learning for enhancement, separation and synthesis of speech and audio signals, partially or totally replacing traditional analysis. Research will focus on generative machines capable of handling video signals and time series as well.Additionally, the parameters and representations of the speech signal will also be used to model and elaborate non-intrusive speech quality metrics; for this purpose, the speech signal is degraded using different communication system parameters. (AU)

Scientific publications

(22)

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

ROSA, RENATA LOPES; DE SILVA, MARIELLE JORDANE; SILVA, DOUGLAS HENRIQUE; AYUB, MUHAMMAD SHOAIB; CARRILLO, DICK; NARDELLI, PEDRO H. J.; RODRIGUEZ, DEMOSTENES ZEGARRA. Event Detection System Based on User Behavior Changes in Online Social Networks: Case of the COVID-19 Pandemic. IEEE ACCESS, v. 8, p. 158806-158825, 2020. (18/26455-8, 15/24496-0)

BARBOSA, RODRIGO CARVALHO; AYUB, MUHAMMAD SHOAIB; ROSA, RENATA LOPES; RODRIGUEZ, DEMOSTENES ZEGARRA; WUTTISITTIKULKIJ, LUNCHAKORN. Lightweight PVIDNet: A Priority Vehicles Detection Network Model Based on Deep Learning for Intelligent Traffic Lights. SENSORS, v. 20, n. 21, NOV 2020. (19/07665-4, 18/26455-8, 18/12579-7)

VIEIRA, SAMUEL TERRA; ROSA, RENATA LOPES; RODRIGUEZ, DEMOSTENES ZEGARRA. A Speech Quality Classifier based on Tree-CNN Algorithm that Considers Network Degradations. JOURNAL OF COMMUNICATIONS SOFTWARE AND SYSTEMS, v. 16, n. 2, p. 180-187, JUN 2020. (15/24496-0, 18/26455-8)

MILITANI, DAVI RIBEIRO; DE MORAES, HERMES PIMENTA; ROSA, RENATA LOPES; WUTTISITTIKULKIJ, LUNCHAKORN; RAMIREZ, MIGUEL ARJONA; RODRIGUEZ, DEMOSTENES ZEGARRA. Enhanced Routing Algorithm Based on Reinforcement Machine Learning-A Case of VoIP Service. SENSORS, v. 21, n. 2, JAN 2021. (19/07665-4, 18/26455-8, 18/12579-7)

ESCOTTA, ALVARO TEIXEIRA; BECCARO, WESLEY; RAMIREZ, MIGUEL ARJONA. Evaluation of 1D and 2D Deep Convolutional Neural Networks for Driving Event Recognition. SENSORS, v. 22, n. 11, p. 21-pg., 2022-06-01. (18/26455-8)

GUIMARAES, HEITOR R.; BECCARO, WESLEY; RAMIREZ, MIGUEL A.; IEEE. OPTIMIZING TIME DOMAIN FULLY CONVOLUTIONAL NETWORKS FOR 3D SPEECH ENHANCEMENT IN A REVERBERANT ENVIRONMENT USING PERCEPTUAL LOSSES. 2021 IEEE 31ST INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING (MLSP), v. N/A, p. 6-pg., 2021-01-01. (19/07665-4, 18/26455-8)

MILITANI, DAVI; BEGAZO, DANTE COAQUIRA; ROSA, RENATA; RODRIGUEZ, DEMOSTENES Z.; BEGUSIC, D; ROZIC, N; RADIC, J; SARIC, M. A Speech Quality Classifier based on Signal Information that Considers Wired and Wireless Degradations. 2019 27TH INTERNATIONAL CONFERENCE ON SOFTWARE, TELECOMMUNICATIONS AND COMPUTER NETWORKS (SOFTCOM), v. N/A, p. 6-pg., 2019-01-01. (15/24496-0, 18/26455-8)

DOS SANTOS, MARCELO RODRIGO; BATISTA, ANDREZA PATRICIA; ROSA, RENATA LOPES; SAADI, MUHAMMAD; MELGAREJO, DICK CARRILLO; RODRIGUEZ, DEMOSTENES ZEGARRA. AsQM: Audio Streaming Quality Metric Based on Network Impairments and User Preferences. IEEE TRANSACTIONS ON CONSUMER ELECTRONICS, v. 69, n. 3, p. 13-pg., 2023-08-01. (18/26455-8)

MENDONCA, ROBSON V.; SILVA, JUAN C.; ROSA, RENATA L.; SAADI, MUHAMMAD; RODRIGUEZ, DEMOSTENES Z.; FAROUK, AHMED. A lightweight intelligent intrusion detection system for industrial internet of things using deep learning algorithm. EXPERT SYSTEMS, DEC 2021. (15/24496-0, 18/26455-8)

HAJAROLASVADI, NOUSHIN; RAMIREZ, MIGUEL ARJONA; BECCARO, WESLEY; DEMIREL, HASAN. Generative Adversarial Networks in Human Emotion Synthesis: A Review. IEEE ACCESS, v. 8, p. 218499-218529, 2020. (19/07665-4, 18/12579-7, 18/26455-8)

RODRIGUEZ, DEMOSTENES Z.; CARRILLO, DICK; RAMIREZ, MIGUEL A.; NARDELLI, PEDRO H. J.; MOELLER, SEBASTIAN. Incorporating Wireless Communication Parameters Into the E-Model Algorithm. IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, v. 29, p. 956-968, 2021. (18/26455-8, 15/24496-0)

RAMIREZ, MIGUEL ARJONA; BECCARO, WESLEY; RODRIGUEZ, DEMOSTENES ZEGARRA; ROSA, RENATA LOPES. Differentiable Measures for Speech Spectral Modeling. IEEE ACCESS, v. 10, p. 10-pg., 2022-01-01. (19/07665-4, 18/26455-8)

TERRA VIEIRA, SAMUEL; LOPES ROSA, RENATA; ZEGARRA RODRIGUEZ, DEMOSTENES; ARJONA RAMIREZ, MIGUEL; SAADI, MUHAMMAD; WUTTISITTIKULKIJ, LUNCHAKORN. Q-Meter: Quality Monitoring System for Telecommunication Services Based on Sentiment Analysis Using Deep Learning. SENSORS, v. 21, n. 5, MAR 2021. (18/26455-8)

RIBEIRO, DAVID AUGUSTO; MELGAREJO, DICK CARRILLO; SAADI, MUHAMMAD; ROSA, RENATA LOPES; RODRIGUEZ, DEMOSTENES ZEGARRA. A novel deep deterministic policy gradient model applied to intelligent transportation system security problems in 5G and 6G network scenarios. PHYSICAL COMMUNICATION, v. 56, p. 10-pg., 2023-02-01. (18/26455-8)

OGOBUCHI, OKEY DANIEL; VIEIRA, SAMUEL TERRA; SAADI, MUHAMMAD; ROSA, RENATA LOPES; RODRIGUEZ, DEMOSTENES ZEGARRA. Intelligent network planning tool for location optimization of unmanned aerial vehicle base stations using geographical images. JOURNAL OF ELECTRONIC IMAGING, v. 31, n. 6, p. 19-pg., 2022-11-01. (18/26455-8)

RIBEIRO, DAVID AUGUSTO; SILVA, JUAN CASAVILCA; LOPES ROSA, RENATA; SAADI, MUHAMMAD; MUMTAZ, SHAHID; WUTTISITTIKULKIJ, LUNCHAKORN; ZEGARRA RODRIGUEZ, DEMOSTENES; AL OTAIBI, SATTAM. Light Field Image Quality Enhancement by a Lightweight Deformable Deep Learning Framework for Intelligent Transportation Systems. ELECTRONICS, v. 10, n. 10, MAY 2021. (18/26455-8)

SILVA, JUAN CASAVILCA; SAADI, MUHAMMAD; WUTTISITTIKULKIJ, LUNCHAKORN; MILITANI, DAVI RIBEIRO; ROSA, RENATA LOPES; RODRIGUEZ, DEMOSTENES ZEGARRA; AL OTAIBI, SATTAM. ight-Field Imaging Reconstruction Using Deep Learning Enabling Intelligent Autonomous Transportation Syste. IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, v. 23, n. 2, MAY 2021. (18/26455-8)

NUNES, RODRIGO DANTAS; ROSA, RENATA LOPES; RODRIGUEZ, DEMOSTENES ZEGARRA. Performance improvement of a non-intrusive voice quality metric in lossy networks. IET COMMUNICATIONS, v. 13, n. 20, p. 3401-3408, DEC 19 2019. (15/24496-0, 18/26455-8)

DA SILVA, MARIELLE JORDANE; MELGAREJO, DICK CARRILLO; ROSA, RENATA LOPES; RODRIGUEZ, DEMOSTENES ZEGARRA. Speech Quality Classifier Model based on DBN that Considers Atmospheric Phenomena. JOURNAL OF COMMUNICATIONS SOFTWARE AND SYSTEMS, v. 16, n. 1, p. 75-84, MAR 2020. (15/24496-0, 18/26455-8)

MILITANI, DAVI; VIEIRA, SAMUEL; VALADAO, EVERTHON; NELES, KATIA; ROSA, RENATA; RODRIGUEZ, DEMOSTENES Z.; BEGUSIC, D; ROZIC, N; RADIC, J; SARIC, M. A Machine Learning Model to Resource Allocation Service for Access Point on Wireless Network. 2019 27TH INTERNATIONAL CONFERENCE ON SOFTWARE, TELECOMMUNICATIONS AND COMPUTER NETWORKS (SOFTCOM), v. N/A, p. 6-pg., 2019-01-01. (15/24496-0, 18/26455-8)

DA SILVA, MARIELLE J.; BEGAZO, DANTE C.; RODRIGUEZ, DEMOSTENES Z.; BEGUSIC, D; ROZIC, N; RADIC, J; SARIC, M. Evaluation of Speech Quality Degradation due to Atmospheric Phenomena. 2019 27TH INTERNATIONAL CONFERENCE ON SOFTWARE, TELECOMMUNICATIONS AND COMPUTER NETWORKS (SOFTCOM), v. N/A, p. 6-pg., 2019-01-01. (15/24496-0, 18/26455-8)

BARBOSA, RODRIGO; OGOBUCHI, OKEY DANIEL; JOY, OMOLE OLUWATOYIN; SAADI, MUHAMMAD; ROSA, RENATA LOPES; AL OTAIBI, SATTAM; RODRIGUEZ, DEMOSTENES ZEGARRA. IoT based real-time traffic monitoring system using images sensors by sparse deep learning algorithm. COMPUTER COMMUNICATIONS, v. 210, p. 10-pg., 2023-10-01. (18/26455-8)