ZeroBERTo: Leveraging Zero-Shot Text Classification by Topic Modeling

Autor(es):Mostrar menos -

Alcoforado, Alexandre ; Ferraz, Thomas Palmeira ; Gerber, Rodrigo ; Bustos, Enzo ; Oliveira, Andre Seidel ; Veloso, Bruno Miguel ; Siqueira, Fabio Levy ; Reali Costa, Anna Helena ; Pinheiro, V ; Gamallo, P ; Amaro, R ; Scarton, C ; Batista, F ; Silva, D ; Magro, C ; Pinto, H

• Tipo de documento: Artigo Científico
• Fonte: COMPUTATIONAL PROCESSING OF THE PORTUGUESE LANGUAGE, PROPOR 2022; v. 13208, p. 12-pg., 2022-01-01.

Resumo

Traditional text classification approaches often require a good amount of labeled data, which is difficult to obtain, especially in restricted domains or less widespread languages. This lack of labeled data has led to the rise of low-resource methods, that assume low data availability in natural language processing. Among them, zero-shot learning stands out, which consists of learning a classifier without any previously labeled data. The best results reported with this approach use language models such as Transformers, but fall into two problems: high execution time and inability to handle long texts as input. This paper proposes a new model, ZeroBERTo, which leverages an unsupervised clustering step to obtain a compressed data representation before the classification task. We show that ZeroBERTo has better performance for long inputs and shorter execution time, outperforming XLM-R by about 12% in the F1 score in the FolhaUOL dataset. (AU)

• Processo FAPESP: 19/07665-4 - Centro de Inteligência Artificial
• Beneficiário: Fabio Gagliardi Cozman
• Modalidade de apoio: Auxílio à Pesquisa - Programa eScience e Data Science - Centros de Pesquisa em Engenharia