Study of the String-Averaging algorithmic topology for Stochastic Gradient methods with applications to supervised machine learning in deep neural networks

• Grant number: 23/14437-3
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: December 01, 2023
• End date: November 30, 2024
• Field of knowledge: Physical Sciences and Mathematics - Mathematics - Applied Mathematics
• Principal Investigator: Elias Salomão Helou Neto
• Grantee: Arthur Queiroz Moura
• Host Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil

Abstract

The most successful stochastic gradient methods in deep learning have an incremental characteristic and are executed on multiple Graphics Processing Units (GPUs) in large training data sets. It is expected to divide the large group of training data into mini-batches, each containing a subset of this data, to keep the processing of each iteration of the method computationally viable (called ``epoch'' in machine learning jargon). The stochasticity of the techniques comes from the randomized way the mini-batches are selected from the total set of training data. In the present project, we propose the theoretical and practical study of using stochastic gradient methods in the context of deep learning of an alternative algorithmic topology to the one commonly used to distribute computation between GPUs, called string-averaging. The fellow will focus on theoretical development, studying mainly the demonstrations of convergence for the proposed method, and will maintain less contact with the practical application in machine learning as two other students will be in charge of most of the experimental tasks.