Advanced search
Start date

Accelerating deep neural network parameter servers with high performance computing techniques

Grant number: 19/04519-7
Support type:Scholarships in Brazil - Scientific Initiation
Effective date (Start): May 01, 2019
Effective date (End): August 31, 2019
Field of knowledge:Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
Principal researcher:Paulo Sérgio Lopes de Souza
Grantee:Matheus Henrique Junqueira Saldanha
Home Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil
Associated research grant:13/07375-0 - CeMEAI - Center for Mathematical Sciences Applied to Industry, AP.CEPID


Machine Learning (ML) has gathered a lot of interest from industry and academia as it increases the range of tasks that can be performed by computers: speech recognition, sentiment analysis, cancer detection, and much more. Neural Networks (NN) are a particular branch of ML that has achieved great popularity after winning recent well known image recognition contests; however, they can take days or weeks to be trained, which directly impacts productivity of interested parties. This project aims to accelerate Parameter Servers (PS), a common distributed algorithm for increasing the scale of NN applications. This will be initially done by applying High Performance Computing (HPC) techniques on a PS using only one level of parallelism, the minibatch level, and later by also exploiting parallelism on gradient calculation. Both investigation lines will be supported by design (e.g. PCAM analysis) and modelling (e.g. computational models) techniques, programming models (e.g. MPI, CUDA), theoretical results (e.g. Amdahl's Law) and metrics (e.g. speedup, depth-complexity) from HPC. By accelerating NNs we expect to increase the pace at which research in the area is carried, as well as ease the industrial process of designing NN models and deploying them within applications or services.