Data Parallelism

Practical Guide to PyTorch Training Acceleration Techniques

by
Practical Guide to PyTorch Training Acceleration Techniques

Introduction This article discusses how to apply torch for mixed precision computation, data parallelism, and distributed computing. Author丨Not Important What Name @ Zhihu Link丨https://zhuanlan.zhihu.com/p/360697168 Due to recent programs requiring high speed, I wanted to get results quickly, so I specifically learned about mixed precision computation and parallel operations. Since there are already many articles explaining … Read more

PyTorch Training Acceleration Techniques

by
PyTorch Training Acceleration Techniques

MLNLP community is a well-known machine learning and natural language processing community both domestically and internationally, covering NLP master’s and doctoral students, university teachers, and corporate researchers. The vision of the community is to promote communication and progress between the academic and industrial sectors of natural language processing and machine learning, especially for beginners. Reprinted … Read more

A Brief Guide to Multi-GPU Training with PyTorch

by
A Brief Guide to Multi-GPU Training with PyTorch

1. Two Parallelization Models in Torch 1.1 DataParallel DataParallel is a data parallel method provided by PyTorch for model training on multiple GPUs within a single machine. It achieves parallel computation by splitting the input data into several sub-parts (mini-batches) and distributing these sub-parts to different GPUs. During the forward pass, the input data is … Read more

Understanding Distributed Logic of Large Models

by

MLNLP community is a well-known machine learning and natural language processing community in China and abroad, covering NLP master’s and doctoral students, university teachers, and corporate researchers. The community’s vision is to promote communication and progress between the academic and industrial circles of natural language processing and machine learning domestically and internationally, especially for beginners. … Read more