AI Training Series - Scaling Deep Learning - From Single GPU to Clusters

Contents

This 3-days block course combines lectures about Fundamentals of Deep Learning, Data Parallelism and Model Parallelism. 

The lectures are interleaved with many demos and hands-on sessions using Jupyter Notebooks.

The course is co-organised by LRZ and NVIDIA Deep Learning Institute (DLI).  All instructors are NVIDIA certified University Ambassadors.

Day 1: Fundamentals of Deep Learning

Businesses worldwide are using artificial intelligence to solve their greatest challenges. Healthcare professionals use AI to enable more accurate, faster diagnoses in patients. Retail businesses use it to offer personalised customer shopping experiences. Automakers use it to make personal vehicles, shared mobility, and delivery services safer and more efficient. Deep learning is a powerful AI approach that uses multi-layered artificial neural networks to deliver state-of-the-art accuracy in tasks such as object detection, speech recognition, and language translation. Using deep learning, computers can learn and recognise patterns from data that are considered too complex or subtle for expert-written software.

In this lecture, you’ll learn how deep learning works through hands-on exercises in computer vision and natural language processing. You’ll train deep learning models from scratch, learning tools and tricks to achieve highly accurate results. You’ll also learn to leverage freely available, state-of-the-art pre-trained models to save time and get your deep learning application up and running quickly.

By participating in this lecture, you will:

• Learn the fundamental techniques and tools required to train a deep learning model
• Gain experience with common deep learning data types and model architectures
• Enhance datasets through data augmentation to improve model accuracy
• Leverage transfer learning between models to achieve efficient results with less data and computation
• Build confidence to take on your own project with a modern deep learning framework

Day 2: Data Parallelism

Modern deep learning challenges leverage increasingly larger datasets and more complex models. As a result, significant computational power is required to train models effectively and efficiently. Learning to distribute data across multiple GPUs during deep learning model training makes possible an incredible wealth of new applications utilizing deep learning.

Additionally, the effective use of systems with multiple GPUs reduces training time, allowing for faster application development and much faster iteration cycles. Teams who are able to perform training using multiple GPUs will have an edge, building models trained on more data in shorter periods of time and with greater engineer productivity.

This lecture teaches you techniques for data-parallel deep learning training on multiple GPUs to shorten the training time required for data-intensive applications. Working with deep learning tools, frameworks, and workflows to perform neural network training, you’ll learn how to decrease model training time by distributing data to multiple GPUs, while retaining the accuracy of training on a single GPU.

By participating in this lecture, you’ll:

• Understand how data parallel deep learning training is performed using multiple GPUs
• Achieve maximum throughput when training, for the best use of multiple GPUs
• Distribute training to multiple GPUs using Pytorch Distributed Data Parallel
• Understand and utilize algorithmic considerations specific to multi-GPU training performance and accuracy

Day 3: Model Parallelism

Large language models (LLMs) and deep neural networks (DNNs), whether applied to natural language processing (e.g., GPT-3), computer vision (e.g., huge Vision Transformers), or speech AI (e.g., Wave2Vec 2), have certain properties that set them apart from their smaller counterparts. As LLMs and DNNs become larger and are trained on progressively larger datasets, they can adapt to new tasks with just a handful of training examples, accelerating the route toward general artificial intelligence. Training models that contain tens to hundreds of billions of parameters on vast datasets isn’t trivial and requires a unique combination of AI, high-performance computing (HPC), and systems knowledge. The goal of this course is to demonstrate how to train the largest of neural networks and deploy them to production.

• Scale training and deployment of LLMs and neural networks across multiple nodes.
• Use techniques such as activation checkpointing, gradient accumulation, and various forms of model parallelism to overcome the challenges associated with large-model memory footprint.
• Capture and understand training performance characteristics to optimize model architecture.
• Deploy very large multi-GPU, multi-node models to production using NVIDIA Triton™ Inference Server.

Important information

After you are accepted, please create an account under https://learn.nvidia.com/join.

Ensure your laptop / PC will run smoothly by going to http://websocketstest.com/ Make sure that WebSockets work for you by seeing under Environment, WebSockets is supported and Data Receive, Send and Echo Test all check Yes under WebSockets (Port 80).If there are issues with WebSockets, try updating your browser.

NVIDIA Deep Learning Institute

The NVIDIA Deep Learning Institute delivers hands-on training for developers, data scientists, and engineers. The program is designed to help you get started with training, optimising, and deploying neural networks to solve real-world problems across diverse industries such as self-driving cars, healthcare, online services, and robotics.

Prerequisites

• Day 1: An understanding of fundamental programming concepts in Python 3, such as functions, loops, dictionaries, and arrays; familiarity with Pandas data structures; and an understanding of how to compute a regression line.
• Day 2: Experience with deep learning training using Python.
• Day 3: Good understanding of PyTorch, good understanding of deep learning and data parallel training concepts, practice with natural language processing are useful, but optional

Hands-On

The lectures are interleaved with many hands-on sessions using Jupyter Notebooks. The exercises will be done on a fully configured GPU-accelerated workstation in the cloud.

Language

English