Evaluating Stream Classifiers with Delayed Labels Information

Autor(es)

Souza, Vinicius M. A. ; da Silva, Tiago Pinho ; Batista, Gustavo E. A. P. A. ; IEEE

• Tipo de documento: Artigo Científico
• Fonte: 2018 7TH BRAZILIAN CONFERENCE ON INTELLIGENT SYSTEMS (BRACIS); v. N/A, p. 6-pg., 2018-01-01.

Resumo

In general, data stream classifiers consider that the actual label of every unlabeled instance is available immediately after it issues a classification. The immediate availability of class labels allows the supervised monitoring of the data distribution and the error rate to verify whether the current classifier is outdated. Further, if a change is detected, the classifier has access to all recent labeled data to update the model. However, this assumption is very optimistic for most (if not all) applications. Given the costs and labor involved to obtain labels, failures in data acquisition or restrictions of the classification problem, a more reasonable assumption would be to consider the delayed availability of class labels. In this paper, we experimentally analyze the impact of latency on the performance of stream classifiers and call the attention of the community for the need to consider this critical variable in the evaluation process. We also make suggestions to avoid possible biased conclusions due to ignoring the delayed nature of stream problems. These are relevant contributions since few studies consider this variable in new algorithms proposals. Also, we propose a new evaluation measure (Kappa-Latency) that takes into account the arrival delay of actual labels to evaluate and compare a set of classifiers. (AU)

• Processo FAPESP: 16/04986-6 - Armadilhas e sensores inteligentes: uma abordagem inovadora para controle de insetos peste e vetores de doenças
• Beneficiário: Gustavo Enrique de Almeida Prado Alves Batista
• Modalidade de apoio: Auxílio à Pesquisa - Programa eScience e Data Science - Regular
• Processo FAPESP: 18/05859-3 - Armadilhas e sensores inteligentes: uma abordagem inovadora para controle de insetos peste e vetores de doenças
• Beneficiário: Vinícius Mourão Alves de Souza
• Modalidade de apoio: Bolsas no Brasil - Pós-Doutorado