Data Augmentation through Distilled Samples and Detection of Critical Learning Periods

• Grant number: 24/17684-4
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: December 01, 2024
• End date: November 30, 2025
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Principal Investigator: Artur Jordão Lima Correia
• Grantee: Ian Guarim Pons
• Host Institution: Escola Politécnica (EP). Universidade de São Paulo (USP). São Paulo , SP, Brazil
• Associated research grant: 23/11163-0 - DeepPruning: Efficient Neural Networks by Exploring Pruning Techniques., AP.R

Abstract

The secret ingredient behind the unprecedented results of modern models lies in the quality and quantity of training data. This ingredient, however, imposes several computational challenges that have motivated efforts toward data distillation methods. This family of techniques produces a synthetic dataset, much smaller than the original, capable of achieving results equivalent to the initial set after training. Regardless of the data used, practical scenarios require models capable of generalizing patterns seen during training for decision-making in critical environments. Thus, ensuring their reliability in Out-of-Distribution scenarios (i.e., adverse to training data) is a central challenge in the field. Regularization techniques represent the main alternative to this problem, imposing constraints during the learning process. Among the most promising forms of regularization, data augmentation stands out. In this direction, in addition to developing new techniques, recent studies seek to understand so-called critical periods: early phases of training where regularization promotes a positive impact on the generalization of deep models. This research project proposes a new data augmentation method that leverages highly discriminative information contained in images from data distillation methods. An inherent disadvantage of data augmentation lies in the increased training time and, consequently, the amount of CO2 emitted and energy consumed; these factors also motivate the use of distilled data. Thus, it is proposed to counterbalance the inherent computational increase by applying data augmentation only during critical periods, instead of throughout the entire learning process. For this purpose, the project proposes to systematically identify these periods, which remains an open problem since, although the literature acknowledges their existence, no methods yet identify when these periods emerge. Among the expected contributions, we highlight greater model generalization without affecting training costs and fine-tuning of modern models.