An Experimental Design to Evaluate Class Imbalance Treatment Methods

Autor(es)

Batista, Gustavo ; Silva, Diego ; Prati, Ronaldo ; Wani, MA ; Khoshgoftaar, T ; Zhu, X ; Seliya, N

• Tipo de documento: Artigo Científico
• Fonte: 2012 11TH INTERNATIONAL CONFERENCE ON MACHINE LEARNING AND APPLICATIONS (ICMLA 2012), VOL 2; v. N/A, p. 7-pg., 2012-01-01.

Resumo

In the last decade, class imbalance has attracted a huge amount of attention from researchers and practitioners. Class imbalance is ubiquitous in Machine Learning, Data Mining and Pattern Recognition applications; therefore, these research communities have responded to such interest with literally dozens of methods and techniques. Surprisingly, there are still many fundamental open-ended questions such as "Are all learning paradigms equally affected by class imbalance?", "What is the expected performance loss for different imbalance degrees?" and "How much of the performance losses can be recovered by the treatment methods?". In this paper, we propose a simple experimental design to assess the performance of class imbalance treatment methods. This experimental setup uses real data sets with artificially modified class distributions to evaluate classifiers in a wide range of class imbalance. We employ such experimental design in a large-scale experimental evaluation with twenty-two data sets and seven learning algorithms from different paradigms. Our results indicate that the expected performance loss, as a percentage of the performance obtained with the balanced distribution, is quite modest (below 5%) for the most balanced distributions up to 10% of minority examples. However, the loss tends to increase quickly for higher degrees of class imbalance, reaching 20% for 1% of minority class examples. Support Vector Machine is the classifier paradigm that is less affected by class imbalance, being almost insensitive to all but the most imbalanced distributions. Finally, we show that the sampling algorithms only partially recover the performance losses. On average, typically about 30% or less of the performance that was lost due to class imbalance was recovered by random over sampling and SMOTE. (AU)

• Processo FAPESP: 12/07295-3 - Invariância à complexidade em classificação, agrupamento e descoberta de motifs em séries temporais
• Beneficiário: Gustavo Enrique de Almeida Prado Alves Batista
• Modalidade de apoio: Auxílio à Pesquisa - Regular
• Processo FAPESP: 11/04054-2 - Classificação de séries temporais por similaridade e extração de atributos intrínsecos com aplicação na identificação de espécie de insetos
• Beneficiário: Diego Furtado Silva
• Modalidade de apoio: Bolsas no Brasil - Mestrado