Accelerating deep neural network parameter servers with high performance computing techniques

• Grant number: 19/04519-7
• Support type: Scholarships in Brazil - Scientific Initiation
• Effective date (Start): May 01, 2019
• Effective date (End): August 31, 2019
• Field of knowledge: Physical Sciences and Mathematics - Computer Science
• Principal Investigator: Paulo Sérgio Lopes de Souza
• Grantee: Matheus Henrique Junqueira Saldanha
• Home Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil
• Associated research grant: 13/07375-0 - CeMEAI - Center for Mathematical Sciences Applied to Industry, AP.CEPID

Abstract

Machine Learning (ML) has gathered a lot of interest from industry and academia as it increases the range of tasks that can be performed by computers: speech recognition, sentiment analysis, cancer detection, and much more. Neural Networks (NN) are a particular branch of ML that has achieved great popularity after winning recent well known image recognition contests; however, they can take days or weeks to be trained, which directly impacts productivity of interested parties. This project aims to accelerate Parameter Servers (PS), a common distributed algorithm for increasing the scale of NN applications. This will be initially done by applying High Performance Computing (HPC) techniques on a PS using only one level of parallelism, the minibatch level, and later by also exploiting parallelism on gradient calculation. Both investigation lines will be supported by design (e.g. PCAM analysis) and modelling (e.g. computational models) techniques, programming models (e.g. MPI, CUDA), theoretical results (e.g. Amdahl's Law) and metrics (e.g. speedup, depth-complexity) from HPC. By accelerating NNs we expect to increase the pace at which research in the area is carried, as well as ease the industrial process of designing NN models and deploying them within applications or services.