Sthocastic gradient methods that are less sensitive to hyperparameters

• Grant number: 25/26474-6
• Support Opportunities: Scholarships in Brazil - Master
• Start date: February 01, 2026
• End date: July 31, 2027
• Field of knowledge: Physical Sciences and Mathematics - Mathematics - Applied Mathematics
• Principal Investigator: Paulo José da Silva e Silva
• Grantee: Samara Viriato Vilar Dias
• Host Institution: Instituto de Matemática, Estatística e Computação Científica (IMECC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Associated research grant: 23/08706-1 - Numerical optimization, AP.TEM

Abstract

This master's project investigates the use of stochastic gradient methods in training deep neural networks. These models, with billions of parameters, rely on first-order stochastic optimization methods such as Adam, which are highly sensitive to hyperparameter choices, particularly the learning rate. The work aims to study and implement stochastic gradient methods with adaptive step sizes, as developed by researchers like Aaron Defazio, to achieve greater robustness and efficiency in optimization. (AU)