A Brief Guide to Multi-GPU Training with PyTorch
1. Two Parallelization Models in Torch 1.1 DataParallel DataParallel is a data parallel method provided by PyTorch for model training on multiple GPUs within a single machine. It achieves parallel computation by splitting the input data into several sub-parts (mini-batches) and distributing these sub-parts to different GPUs. During the forward pass, the input data is … Read more