Uncovering Human Multimodal Activity Recognition with a Deep Learning Approach

Autor(es)

Ranieri, Caetano M. ; Vargas, Patricia A. ; Romero, Roseli A. F. ; IEEE

• Tipo de documento: Artigo Científico
• Fonte: 2020 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN); v. N/A, p. 8-pg., 2020-01-01.

Resumo

Recent breakthroughs on deep learning and computer vision have encouraged the use of multimodal human activity recognition aiming at applications in human-robot-interaction. The wide availability of videos at online platforms has made this modality one of the most promising for this task, whereas some researchers have tried to enhance the video data with wearable sensors attached to human subjects. However, temporal information on both video and inertial sensors are still under investigation. Most of the current work focusing on daily activities do not present comparative studies considering different temporal approaches. In this paper, we are proposing a new model build upon a Two-Stream ConvNet for action recognition, enhanced with Long Short-Term Memory (LSTM) and a Temporal Convolution Networks (TCN) to investigate the temporal information on videos and inertial sensors. A feature-level fusion approach prior to temporal modelling is also proposed and evaluated. Experiments have been conducted on the egocentric multimodal dataset and on the UTD-MHAD. LSTM and TCN showed competitive results, with the TCN performing slightly better for most applications. The feature-level fusion approach also performed well on the UTD-MHAD with some overfitting on the egocentric multimodal dataset. Overall the proposed model presented promising results on both datasets compatible with the state-of-the-art, providing insights on the use of deep learning for human-robot-interaction applications. (AU)

• Processo FAPESP: 17/02377-5 - Aprendizado de Máquina e Aplicações para Robótica em Ambientes Inteligentes
• Beneficiário: Caetano Mazzoni Ranieri
• Modalidade de apoio: Bolsas no Brasil - Doutorado
• Processo FAPESP: 13/07375-0 - CeMEAI - Centro de Ciências Matemáticas Aplicadas à Indústria
• Beneficiário: Francisco Louzada Neto
• Modalidade de apoio: Auxílio à Pesquisa - Centros de Pesquisa, Inovação e Difusão - CEPIDs
• Processo FAPESP: 17/01687-0 - Arquitetura e aplicações para robótica em ambientes inteligentes
• Beneficiário: Roseli Aparecida Francelin Romero
• Modalidade de apoio: Auxílio à Pesquisa - Regular
• Processo FAPESP: 18/25902-0 - Aprendizado de máquina para ajudar a encontrar correlatos neurais do Mal de Parkinson
• Beneficiário: Caetano Mazzoni Ranieri
• Modalidade de apoio: Bolsas no Exterior - Estágio de Pesquisa - Doutorado