Megatron-LM

Megatron-LM is a GPU-optimized library designed for scalable training of transformer models, providing both a reference implementation and composable building blocks through Megatron Core. Its notable features include advanced parallelism strategies (TP, PP, DP, EP, CP), support for mixed precision training, and pre-configured scripts for rapid experimentation, making it suitable for researchers and machine learning engineers developing custom training frameworks. Additionally, Megatron Bridge facilitates bidirectional checkpoint conversion with Hugging Face models.