Concept of Tensor Model Parallelism

• Authors: Kichang Yang, Kevin Ko, Minho Ryu

Tensor Model Parallelism makes it possible to train larger models by partitioning the parameter tensors into multiple dimensions. We support 1D, 2D, 2.5D, and 3D tensor partitioning algorithms which make tensor parallel training more efficient.

Tensor Parallel Algorithms

• 1D parallel algorithm (same as Megatron-LM)

• 2D parallel (SUMMA) algorithm

• 2.5D parallel (SUMMA-2.5) algorithm

• 3D parallel Algorithm

References

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

• An Efficient 2D Method for Training Super-Large Deep Learning Models

• 2.5-dimensional distributed model training

• Maximizing Parallelism in Distributed Training for Huge Neural Networks