Advanced search
Start date
Betweenand


Uncovering Human Multimodal Activity Recognition with a Deep Learning Approach

Full text
Author(s):
Ranieri, Caetano M. ; Vargas, Patricia A. ; Romero, Roseli A. F. ; IEEE
Total Authors: 4
Document type: Journal article
Source: 2020 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN); v. N/A, p. 8-pg., 2020-01-01.
Abstract

Recent breakthroughs on deep learning and computer vision have encouraged the use of multimodal human activity recognition aiming at applications in human-robot-interaction. The wide availability of videos at online platforms has made this modality one of the most promising for this task, whereas some researchers have tried to enhance the video data with wearable sensors attached to human subjects. However, temporal information on both video and inertial sensors are still under investigation. Most of the current work focusing on daily activities do not present comparative studies considering different temporal approaches. In this paper, we are proposing a new model build upon a Two-Stream ConvNet for action recognition, enhanced with Long Short-Term Memory (LSTM) and a Temporal Convolution Networks (TCN) to investigate the temporal information on videos and inertial sensors. A feature-level fusion approach prior to temporal modelling is also proposed and evaluated. Experiments have been conducted on the egocentric multimodal dataset and on the UTD-MHAD. LSTM and TCN showed competitive results, with the TCN performing slightly better for most applications. The feature-level fusion approach also performed well on the UTD-MHAD with some overfitting on the egocentric multimodal dataset. Overall the proposed model presented promising results on both datasets compatible with the state-of-the-art, providing insights on the use of deep learning for human-robot-interaction applications. (AU)

FAPESP's process: 17/02377-5 - Machine Learning and Applications for Robotics in Smart Environments
Grantee:Caetano Mazzoni Ranieri
Support Opportunities: Scholarships in Brazil - Doctorate
FAPESP's process: 13/07375-0 - CeMEAI - Center for Mathematical Sciences Applied to Industry
Grantee:José Alberto Cuminato
Support Opportunities: Research Grants - Research, Innovation and Dissemination Centers - RIDC
FAPESP's process: 17/01687-0 - Architecture and applications for robotics in intelligent environments
Grantee:Roseli Aparecida Francelin Romero
Support Opportunities: Regular Research Grants
FAPESP's process: 18/25902-0 - Machine learning for help unveiling neural correlates of Parkinson's Disease
Grantee:Caetano Mazzoni Ranieri
Support Opportunities: Scholarships abroad - Research Internship - Doctorate