Cloud TPUs are powerful hardware for building large models & have enabled many research successes. We enable scaling up PyTorch models to 10B+ parameters on Cloud TPU with a new Fully Sharded Data Parallel (FSDP) interface in PyTorch/XLA.

Interested in joining us at the PyTorch Conference on Dec. 2nd, 2022 in New Orleans? Apply now for an in-person spot at our flagship event: pytorchconference22.splashthat.com

Do you need fast and memory-efficient transformers which are easy to install?
I'm happy to share that xFormers now ships precompiled conda packages for PyTorch 1.12.1 and CUDA 11.3/11.6 (Linux-only for now). github.com/facebookresear…

Interested in becoming a PyTorch contributor? 🇺🇦Viacheslav Kovalevskyi🇺🇦 walks you through the process, starting with why, sharing the pre-requirements, outlining ways to find issues to work on, and more. Learn about this process: bit.ly/3EzpWxd

In the final video of this DDP series, we put it all together and train a GPT-like model across multiple GPUs and nodes.

Suraj Subramanian walks you through best practices and structuring your project for fault-tolerant distributed training: bit.ly/3ytkGHC

Want to understand where your PyTorch program is allocating GPU memory, or why it ran out?

We recently added an experimental API for visualizing allocated GPU memory in PyTorch using flamegraphs. Check it out in newer nightly builds. zdevito.github.io/2022/08/16/mem…

Maxed out all GPUs on your machine but need more? Add more machines to the mix!

In DDP video 5, we extend our multi-GPU training script to running on multiple nodes, with almost no code changes!

Suraj Subramanian shows you how to run it on a cluster via SLURM: bit.ly/3MeQwh0

Some ML jobs fail due to code bugs, others due to (cloud) matrix glitches. Only need to restart the job? torchrun is your friend.

Watch Suraj Subramanian explain how training jobs can be fault-tolerant w/ torchrun & watch your script auto bounce back to life. bit.ly/3RJ7chY

Watch Soumith Chintala and Yann LeCun discuss accomplishments made in PyTorch and #AI . What are your favorite recent breakthroughs in machine learning #ML ? Share below, and watch Soumith and Yann’s full conversation here: bit.ly/3D62ZkM

Using multiple GPUs to train your model can greatly reduce training time!

In video 3 of the DDP series, we migrate a single-GPU training job to run on 4 GPUs, while Suraj Subramanian explains distributed training concepts in PyTorch code.

Watch the tutorial: bit.ly/3EiApgw

If your training jobs are using DataParallel, consider using DDP!

Watch the video to learn what DDP is doing under all those function calls, and how it can help more than using DP for multi-GPU training

bit.ly/3SSqDG3

Learn about distributed training in PyTorch. In this tutorial, Suraj Subramanian walks you through training your models on a single GPU -> multiple GPUs -> training LMs on multiple machines in less than an hour.

Watch the tutorial: bit.ly/3Sxq2K7

PyTorch Contributors and Maintainers are hosting a Bay Area Meetup on Oct. 12th at 6pm PT. Join to connect with our community and learn about becoming a contributor. Hosted by 🇺🇦Viacheslav Kovalevskyi🇺🇦. RSVP: bit.ly/3y8NqFK

Stable Diffusion in the diffusers library became x3 times faster thanks to a set of optimizations tips, some of which require minimal code changes, making it the fastest implementation of Stable Diffusion (afaik)! What are these optimizations?
A thread 🧵 (1/n)

Abhi Venigalla (aveni.eth) and I turned Andrej Karpathy’s minGPT into a GPT-3 quality model with 30 billion parameters—projected to cost only $450k to train. The code to do so is public: it's easily readable and can be launched on however many GPUs you want.

Here’s how:

We have exciting news! In our latest and greatest LLM blog, we show how MosaicML Cloud can help you train LLMs from 1B - 70B parameters, and for the first time, publish transparent times + costs for doing so. It's a lot cheaper than you think! (1/9)

mosaicml.com/blog/gpt-3-qua…