A step-by-step guide to building a complete ML workflow with PyTorch.

Getting-Started

An introduction to building a complete ML workflow with PyTorch. Follows the PyTorch Beginner Series on YouTube.

Getting-Started

This tutorial introduces the fundamental concepts of PyTorch through self-contained examples.

Getting-Started

Use torch.nn to create and train a neural network.

Getting-Started

Learn to use TensorBoard to visualize data and model training.

Interpretability,Getting-Started,TensorBoard

A guide on best practices to copy data from CPU to GPU.

Getting-Started

Optimize DataLoader configuration with num_workers, pin_memory, persistent_workers for maximum training throughput.

Getting-Started,Best-Practice

Learn the subtleties of requires_grad, retain_grad, leaf, and non-leaf tensors

Getting-Started

Visualize the gradient flow of a network.

Getting-Started

Finetune a pre-trained Mask R-CNN model.

Image/Video

Train a convolutional neural network for image classification using transfer learning.

Image/Video

Train a convolutional neural network for image classification using transfer learning.

Image/Video

Train a generative adversarial network (GAN) to generate new celebrities.

Image/Video

Learn how to augment your network using a visual attention mechanism.

Image/Video

Learn how to train semi-supervised learning algorithms (on custom data) using USB and PyTorch.

Image/Video

Pre-train a transformer language model across multiple GPUs using PyTorch and Ray Train.

Text,Best-Practice,Ray-Distributed,Parallel-and-Distributed-Training

Learn to load data with torchaudio.

Audio

Learn to resample audio waveforms using torchaudio.

Audio

Learn to apply data augmentations using torchaudio.

Audio

Learn to extract features using torchaudio.

Audio

Learn to augment features using torchaudio.

Audio

Learn to use torchaudio datasets.

Audio

Learn how to use torchaudio's pretrained models for building a speech recognition application.

Audio

Learn how to correctly format an audio dataset and then train/test an audio classifier network on the dataset.

Audio

Learn how to use torchaudio's pretrained models for building a text-to-speech application.

Audio

Learn how to use torchaudio's Wav2Vec2 pretrained models for aligning text to speech

Audio

Build and train a basic character-level RNN to classify word from scratch without the use of torchtext. First in a series of three tutorials.

NLP

After using character-level RNN to classify names, learn how to generate names from languages. Second in a series of three tutorials.

NLP

This is the third and final tutorial on doing “NLP From Scratch”, where we write our own classes and functions to preprocess the data to do our NLP modeling tasks.

NLP

Build a image classifier model in PyTorch and convert it to ONNX before deploying it with ONNX Runtime.

Production,ONNX,Backends

Demonstrate end-to-end how to address unsupported operators in ONNX.

Production,ONNX,Backends

Demonstrate how to handle control flow logic while exporting a PyTorch model to ONNX.

Production,ONNX,Backends

Learn how to use PyTorch to train a Deep Q Learning (DQN) agent on the CartPole-v0 task from the OpenAI Gym.

Reinforcement-Learning

Learn how to use PyTorch and TorchRL to train a Proximal Policy Optimization agent on the Inverted Pendulum task from Gym.

Reinforcement-Learning

Use PyTorch to train a Double Q-learning agent to play Mario.

Reinforcement-Learning

Use TorchRL to train recurrent policies

Reinforcement-Learning

Use TorchRL to code a DDPG Loss

Reinforcement-Learning

Use TorchRL to code a Pendulum

Reinforcement-Learning

Deploy and scale a PyTorch model with Ray Serve.

Production,Best-Practice,Ray-Distributed,Ecosystem

Learn how to profile a PyTorch application

Profiling

Introduction to Holistic Trace Analysis

Profiling

Trace Diff using Holistic Trace Analysis

Profiling

Learn how to use the Mosaic memory profiler to visualize GPU memory usage and identify memory optimization opportunities in PyTorch models.

Model-Optimization,Best-Practice,Profiling

Build a simple FX interpreter to record the runtime of op, module, and function calls and report statistics

FX

Get an overview of Channels Last memory format and understand how it is used to order NCHW tensors in memory preserving dimensions.

Memory-Format,Best-Practice,Frontend-APIs

Walk through an end-to-end example of training a model with the C++ frontend by training a DCGAN – a kind of generative model – to generate images of MNIST digits.

Frontend-APIs,C++

This is the landing page for all things related to custom operators in PyTorch.

Extending-PyTorch,Frontend-APIs,C++,CUDA

Create Custom Operators in Python. Useful for black-boxing a Python function for use with torch.compile.

Extending-PyTorch,Frontend-APIs,C++,CUDA

Learn how to use compiled autograd to capture a larger backward graph.

Model-Optimization,CUDA

How to extend PyTorch with custom C++ and CUDA operators.

Extending-PyTorch,Frontend-APIs,C++,CUDA

The autograd package helps build flexible and dynamic neural netorks. In this tutorial, explore several examples of doing autograd in PyTorch C++ frontend

Frontend-APIs,C++

The dispatcher is an internal component of PyTorch which is responsible for figuring out what code should actually get run when you call a function like torch::add.

Extending-PyTorch,Frontend-APIs,C++

Learn how to extend the dispatcher to add a new device living outside of the pytorch/pytorch repo and maintain it to keep in sync with native PyTorch devices.

Extending-PyTorch,Frontend-APIs,C++

Learn how to integrate a new backend living outside of the pytorch/pytorch repo and maintain it to keep in sync with the native PyTorch backend.

Extending-PyTorch,Frontend-APIs,C++

Learn how to write a custom autograd Function that supports double backward.

Extending-PyTorch,Frontend-APIs

Learn how to create a custom autograd Function that fuses batch norm into a convolution to improve memory usage.

Extending-PyTorch,Frontend-APIs

Learn how to use forward-mode automatic differentiation.

Frontend-APIs

Learn how to compute advanced autodiff quantities using torch.func

Frontend-APIs

Learn how to ensemble models using torch.vmap

Frontend-APIs

Learn how to compute per-sample-gradients using torch.func

Frontend-APIs

Learn how to compute neural tangent kernels using torch.func

Frontend-APIs

Learn how to use the PyTorch Profiler to benchmark your module's performance.

Model-Optimization,Best-Practice,Profiling

Learn how to annotate CUDA graph kernels for semantic profiling traces with custom visualization lanes.

Model-Optimization,Best-Practice,Profiling,CUDA

Learn how to use Ray Tune to find the best performing set of hyperparameters for your model.

Model-Optimization,Best-Practice,Ray-Distributed,Parallel-and-Distributed-Training

Learn how to use torch.nn.utils.parametrize to put constraints on your parameters (e.g. make them orthogonal, symmetric positive definite, low-rank...)

Model-Optimization,Best-Practice

Learn how to use torch.nn.utils.prune to sparsify your neural networks, and how to extend it to implement your own custom pruning technique.

Model-Optimization,Best-Practice

Learn a memory-saving technique through fusing the optimizer step into the backward pass using memory snapshots.

Model-Optimization,Best-Practice,CUDA,Frontend-APIs

Train BERT, prune it to be 2:4 sparse, and then accelerate it to achieve 2x inference speedups with semi-structured sparsity and torch.compile.

Text,Model-Optimization

Learn how to use Ax to search over architectures find optimal tradeoffs between accuracy and latency.

Model-Optimization,Best-Practice,Ax,TorchX

Speed up your models with minimal code changes using torch.compile, the latest PyTorch compiler solution.

Model-Optimization

An example of applying torch.compile to a real model, demonstrating speedups.

Model-Optimization

Build a simple pattern matcher pass that fuses batch norm into convolution to improve performance during inference.

Model-Optimization

Learn the usage, debugging and performance profiling for ``torch.compile`` with Inductor CPU backend.

Model-Optimization

This tutorial explores the new torch.nn.functional.scaled_dot_product_attention and how it can be used to construct Transformer components.

Model-Optimization,Attention,Transformer

Learn how to improve the accuracy of lightweight models using more powerful models as teachers.

Model-Optimization,Image/Video

This tutorial goes over recommended best practices for implementing Transformers with native PyTorch.

Transformer

Briefly go over all concepts and features in the distributed package. Use this document to find the distributed training technology that can best serve your application.

Parallel-and-Distributed-Training

This series of video tutorials walks you through distributed training in PyTorch via DDP.

Parallel-and-Distributed-Training

Learn how to implement model parallel, a distributed training technique which splits a single model onto different GPUs, rather than replicating the entire model on each GPU

Parallel-and-Distributed-Training

Learn the basics of when to use distributed data paralle versus data parallel and work through an example to set it up.

Parallel-and-Distributed-Training

Set up the distributed package of PyTorch, use the different communication strategies, and go over some the internals of the package.

Parallel-and-Distributed-Training

Learn how to train large models with Tensor Parallel package.

Parallel-and-Distributed-Training

Extend ProcessGroup with custom collective communication implementations.

Parallel-and-Distributed-Training

Learn how to build distributed training using the torch.distributed.rpc package.

Parallel-and-Distributed-Training

Walk through a through a simple example of implementing a parameter server using PyTorch’s Distributed RPC framework.

Parallel-and-Distributed-Training

Demonstrate how to implement pipeline parallelism using torch.distributed.pipelining

Parallel-and-Distributed-Training

Learn how to use rpc.functions.async_execution to implement batch RPC

Parallel-and-Distributed-Training

Walk through a through a simple example of how to combine distributed data parallelism with distributed model parallelism.

Parallel-and-Distributed-Training

Learn how to train models with Fully Sharded Data Parallel (fully_shard) package.

Parallel-and-Distributed-Training

TCPStore now uses a new server backend for faster connection and better scalability.

Parallel-and-Distributed-Training

Learn how to spin up distributed applications using Monarch's singler controller model

Parallel-and-Distributed-Training

Learn how to use Monarch's actor framework with TorchTitan to simplify large-scale distributed training across SLURM clusters.

Parallel-and-Distributed-Training

Diagnose hangs using the TorchComms Flight Recorder and Debug Server periodic dumps.

Parallel-and-Distributed-Training,Debugging

Learn about how to use ExecuTorch, a unified ML stack for lowering PyTorch models to edge devices.

Edge

Learn how to load and execute an ExecuTorch model in C++

Edge

Explore how to use the ExecuTorch SDK to profile, debug, and visualize ExecuTorch models

Edge

Explore how to set up the ExecuTorch iOS Demo App, which uses the MobileNet v3 model to process live camera images leveraging three different backends: XNNPACK, Core ML, and Metal Performance Shaders (MPS).

Edge

Learn how to set up the ExecuTorch Android Demo App for image segmentation tasks using the DeepLab v3 model and XNNPACK FP32 backend.

Edge

Learn to accelerate your program using ExecuTorch by applying delegates through three methods: lowering the whole module, composing it with another module, and partitioning parts of a module.

Edge

TorchRec is a PyTorch domain library built to provide common sparsity & parallelism primitives needed for large-scale recommender systems.

TorchRec,Recommender

This tutorial covers the sharding schemes of embedding tables by using EmbeddingPlanner and DistributedModelParallel API.

TorchRec,Recommender