Weighted voting of multi-stream convolutional neural networks for video-based action recognition using optical flow rhythms

Author(s)

Brito, Andre de Souza ^[1] ; Vieira, Marcelo Bernardes ^[1] ; Villela, Saulo Moraes ^[1] ; Tacon, Hemerson ^[1] ; Chaves, Hugo de Lima ^[1] ; Maia, Helena de Almeida ^[2] ; Concha, Darwin Ttito ^[2] ; Pedrini, Helio ^[2]

Affiliation

^[1] Univ Fed Juiz de Fora, Dept Comp Sci, Juiz De Fora - Brazil

^[2] Univ Estadual Campinas, Inst Comp, Campinas - Brazil

• Document type: Journal article
• Source: JOURNAL OF VISUAL COMMUNICATION AND IMAGE REPRESENTATION; v. 77, MAY 2021.
• Web of Science Citations: 0

Abstract

Two of the most important premises of an ensemble are the diversity of its components and how to combine their votes. In this paper, we propose a multi-stream architecture based on the weighted voting of convolutional neural networks to deal with the problem of recognizing human actions in videos. A major challenge is how to include temporal aspects into this kind of approach. A key step in this direction is the selection of features that characterize the complexity of human actions in time. In this context, we propose a new stream, Optical Flow Rhythm, besides using other streams for diversity. To combine the streams, a voting system based on a new weighted average fusion method is introduced. In this scheme, the weights of classifiers are defined by an optimization process led by a metaheuristic. Experiments conducted on the UCF101 and HMDB51 datasets demonstrate that our method is comparable to state-of-the-art approaches. (AU)

• FAPESP's process: 17/09160-1 - Human Action Recognition in Videos
• Grantee: Helena de Almeida Maia
• Support Opportunities: Scholarships in Brazil - Doctorate
• FAPESP's process: 17/12646-3 - Déjà vu: feature-space-time coherence from heterogeneous data for media integrity analytics and interpretation of events
• Grantee: Anderson de Rezende Rocha
• Support Opportunities: Research Projects - Thematic Grants