Improving Pre-Trained Weights through Meta-Heuristics Fine-Tuning

Autor(es)

de Rosa, Gustavo H. ; Roder, Mateus ; Papa, Joao Paulo ; dos Santos, Claudio F. G. ; IEEE

• Tipo de documento: Artigo Científico
• Fonte: 2021 IEEE SYMPOSIUM SERIES ON COMPUTATIONAL INTELLIGENCE (IEEE SSCI 2021); v. N/A, p. 8-pg., 2021-01-01.

Resumo

Machine Learning algorithms have been extensively researched throughout the last decade, leading to unprecedented advances in a broad range of applications, such as image classification and reconstruction, object recognition, and text categorization. Nonetheless, most Machine Learning algorithms are trained via derivative-based optimizers, such as the Stochastic Gradient Descent, leading to possible local optimum entrapments and inhibiting them from achieving proper performances. A bio-inspired alternative to traditional optimization techniques, denoted as meta-heuristic, has received significant attention due to its simplicity and ability to avoid local optimums imprisonment. In this work, we propose to use meta-heuristic techniques to fine-tune pre-trained weights, exploring additional regions of the search space, and improving their effectiveness. The experimental evaluation comprises two classification tasks (image and text) and is assessed under four literature datasets. Experimental results show nature-inspired algorithms' capacity in exploring the neighborhood of pre-trained weights, achieving superior results than their counterpart pre-trained architectures. Additionally, a thorough analysis of distinct architectures, such as Multi-Layer Perceptron and Recurrent Neural Networks, attempts to visualize and provide more precise insights into the most critical weights to be fine-tuned in the learning process. (AU)

• Processo FAPESP: 13/07375-0 - CeMEAI - Centro de Ciências Matemáticas Aplicadas à Indústria
• Beneficiário: Francisco Louzada Neto
• Modalidade de apoio: Auxílio à Pesquisa - Centros de Pesquisa, Inovação e Difusão - CEPIDs
• Processo FAPESP: 19/07665-4 - Centro de Inteligência Artificial
• Beneficiário: Fabio Gagliardi Cozman
• Modalidade de apoio: Auxílio à Pesquisa - Programa eScience e Data Science - Centros de Pesquisa Aplicada
• Processo FAPESP: 14/12236-1 - AnImaLS: Anotação de Imagem em Larga Escala: o que máquinas e especialistas podem aprender interagindo?
• Beneficiário: Alexandre Xavier Falcão
• Modalidade de apoio: Auxílio à Pesquisa - Temático
• Processo FAPESP: 19/02205-5 - Aprendizado adversarial em processamento de linguagem natural
• Beneficiário: Gustavo Henrique de Rosa
• Modalidade de apoio: Bolsas no Brasil - Doutorado
• Processo FAPESP: 20/12101-0 - Suporte para o ambiente computacional e execução de experimentos: aquisição de dados, categorização e manutenção
• Beneficiário: Leandro Aparecido Passos Junior
• Modalidade de apoio: Bolsas no Brasil - Programa Capacitação - Treinamento Técnico