Assessing Adversarial Effects of Noise in Missing Data Imputation

Author(s)

Mangussi, Arthur Dantas ; Pereira, Ricardo Cardoso ; Abreu, Pedro Henriques ; Lorena, Ana Carolina

• Document type: Journal article
• Source: INTELLIGENT SYSTEMS, BRACIS 2024, PT I; v. 15412, p. 15-pg., 2025-01-01.

Abstract

In real-world scenarios, a wide variety of datasets contain inconsistencies. One example of such inconsistency is missing data (MD), which refers to the absence of information in one or more variables. Missing imputation strategies emerged as a possible solution for addressing this problem, which can replace the missing values based on mean, median, or Machine Learning (ML) techniques. The performance of such strategies depends on multiple factors. One factor that influences the missing value imputation (MVI) methods is the presence of noisy instances, described as anything that obscures the relationship between the features of an instance and its class, having an adversarial effect. However, the interaction between MD and noisy instances has received little attention in the literature. This work fills this gap by investigating missing and noisy data interplay. Our experimental setup begins with generating missingness under the Missing Not at Random (MNAR) mechanism in a multivariate scenario and performing imputation using seven state-of-the-art MVI methods. Our methodology involves applying a noise filter before performing the imputation task and evaluating the quality of the imputation directly. Additionally, we measure the classification performance with the new estimates. This approach is applied to both synthetic data and 11 real-world datasets. The effects of noise filtering before imputation are evaluated. The results show that noise preprocessing before the imputation task improves the imputation quality and the classification performance for imputed datasets. (AU)

• FAPESP's process: 21/06870-3 - Beyond algorithm selection: meta-learning for data and algorithm analysis and understanding
• Grantee: Ana Carolina Lorena
• Support Opportunities: Research Grants - Young Investigators Grants - Phase 2
• FAPESP's process: 23/13688-2 - Um modelo Autoencoder para lidar com dados ausentes e com ruído
• Grantee: Arthur Dantas Mangussi
• Support Opportunities: Scholarships abroad - Research Internship - Master's degree