Multilingual and multimodal learning for Brazilian portuguese
Multimodal Models for Images and 3D Representations in a Unified Vision and Langua...
Robust feature learning and multi-modality investigations for Synthetic Realities
| Full text | |
| Author(s): |
Ranieri, Caetano M.
;
Vargas, Patricia A.
;
Romero, Roseli A. F.
;
IEEE
Total Authors: 4
|
| Document type: | Journal article |
| Source: | 2020 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN); v. N/A, p. 8-pg., 2020-01-01. |
| Abstract | |
Recent breakthroughs on deep learning and computer vision have encouraged the use of multimodal human activity recognition aiming at applications in human-robot-interaction. The wide availability of videos at online platforms has made this modality one of the most promising for this task, whereas some researchers have tried to enhance the video data with wearable sensors attached to human subjects. However, temporal information on both video and inertial sensors are still under investigation. Most of the current work focusing on daily activities do not present comparative studies considering different temporal approaches. In this paper, we are proposing a new model build upon a Two-Stream ConvNet for action recognition, enhanced with Long Short-Term Memory (LSTM) and a Temporal Convolution Networks (TCN) to investigate the temporal information on videos and inertial sensors. A feature-level fusion approach prior to temporal modelling is also proposed and evaluated. Experiments have been conducted on the egocentric multimodal dataset and on the UTD-MHAD. LSTM and TCN showed competitive results, with the TCN performing slightly better for most applications. The feature-level fusion approach also performed well on the UTD-MHAD with some overfitting on the egocentric multimodal dataset. Overall the proposed model presented promising results on both datasets compatible with the state-of-the-art, providing insights on the use of deep learning for human-robot-interaction applications. (AU) | |
| FAPESP's process: | 17/02377-5 - Machine Learning and Applications for Robotics in Smart Environments |
| Grantee: | Caetano Mazzoni Ranieri |
| Support Opportunities: | Scholarships in Brazil - Doctorate |
| FAPESP's process: | 13/07375-0 - CeMEAI - Center for Mathematical Sciences Applied to Industry |
| Grantee: | Francisco Louzada Neto |
| Support Opportunities: | Research Grants - Research, Innovation and Dissemination Centers - RIDC |
| FAPESP's process: | 17/01687-0 - Architecture and applications for robotics in intelligent environments |
| Grantee: | Roseli Aparecida Francelin Romero |
| Support Opportunities: | Regular Research Grants |
| FAPESP's process: | 18/25902-0 - Machine learning for help unveiling neural correlates of Parkinson's Disease |
| Grantee: | Caetano Mazzoni Ranieri |
| Support Opportunities: | Scholarships abroad - Research Internship - Doctorate |