December 16, 2022
Efficient Large-Scale Training with Pytorch FSDP and AWS
Cutting-edge AI models are becoming extremely large. The cost and overhead of training these models is increasing rapidly, and involves large amounts of engineering and guesswork to find the right training regime. FSDP reduces these costs significantly by enabling you to train much larger models with the same amount of resources. FSDP lowers the memory footprint on your GPUs, and is usable via a lightweight configuration that requires substantially less effort, typically with just a few lines...
December 15, 2022
Scaling PyTorch FSDP for Training Foundation Models on IBM Cloud
Large model training using a cloud native approach is of growing interest for many enterprises given the emergence and success of foundation models. Some AI practitioners may assume that the only way they can achieve high GPU utilization for distributed training jobs is to run them on HPC systems, such as those inter-connected with Infiniband and may not consider Ethernet connected systems. We demonstrate how the latest distributed training technique, Fully Sharded Data Parallel (FSDP) from P...
December 02, 2022
Get Started with PyTorch 2.0 Summary and Overview
Introducing PyTorch 2.0, our first steps toward the next generation 2-series release of PyTorch. Over the last few years we have innovated and iterated from PyTorch 1.0 to the most recent 1.13 and moved to the newly formed PyTorch Foundation, part of the Linux Foundation.
December 02, 2022
Accelerating Hugging Face and TIMM models with PyTorch 2.0
torch.compile() makes it easy to experiment with different compiler backends to make PyTorch code faster with a single line decorator torch.compile(). It works either directly over an nn.Module as a drop-in replacement for torch.jit.script() but without requiring you to make any source code changes. We expect this one line code change to provide you with between 30%-2x training time speedups on the vast majority of models that you’re already running.
November 28, 2022
Optimizing Production PyTorch Models’ Performance with Graph Transformations
1. Introduction
November 21, 2022
Scaling Multimodal Foundation Models in TorchMultimodal with Pytorch Distributed
Introduction
