Weighted voting of multi-stream convolutional neural networks for video-based action recognition using optical flow rhythms

Autor(es)

Brito, Andre de Souza ^[1] ; Vieira, Marcelo Bernardes ^[1] ; Villela, Saulo Moraes ^[1] ; Tacon, Hemerson ^[1] ; Chaves, Hugo de Lima ^[1] ; Maia, Helena de Almeida ^[2] ; Concha, Darwin Ttito ^[2] ; Pedrini, Helio ^[2]

Afiliação do(s) autor(es)

^[1] Univ Fed Juiz de Fora, Dept Comp Sci, Juiz De Fora - Brazil

^[2] Univ Estadual Campinas, Inst Comp, Campinas - Brazil

• Tipo de documento: Artigo Científico
• Fonte: JOURNAL OF VISUAL COMMUNICATION AND IMAGE REPRESENTATION; v. 77, MAY 2021.
• Citações Web of Science: 0

Resumo

Two of the most important premises of an ensemble are the diversity of its components and how to combine their votes. In this paper, we propose a multi-stream architecture based on the weighted voting of convolutional neural networks to deal with the problem of recognizing human actions in videos. A major challenge is how to include temporal aspects into this kind of approach. A key step in this direction is the selection of features that characterize the complexity of human actions in time. In this context, we propose a new stream, Optical Flow Rhythm, besides using other streams for diversity. To combine the streams, a voting system based on a new weighted average fusion method is introduced. In this scheme, the weights of classifiers are defined by an optimization process led by a metaheuristic. Experiments conducted on the UCF101 and HMDB51 datasets demonstrate that our method is comparable to state-of-the-art approaches. (AU)

• Processo FAPESP: 17/09160-1 - Reconhecimento de Ações Humanas em Vídeos
• Beneficiário: Helena de Almeida Maia
• Modalidade de apoio: Bolsas no Brasil - Doutorado
• Processo FAPESP: 17/12646-3 - Déjà vu: coerência temporal, espacial e de caracterização de dados heterogêneos para análise e interpretação de integridade
• Beneficiário: Anderson de Rezende Rocha
• Modalidade de apoio: Auxílio à Pesquisa - Temático