摩尔线程开源 Megatron-MUSA-Titan 聚焦万卡级集群的万亿参数 MoE 模型训练Moore Threads Open-Sources Megatron-MUSA-Titan Focusing on Trillion-Parameter MoE Model Training for 10,000-GPU Clusters


AI free token/AI免费积分
摩尔线程开源 Megatron-MUSA-Titan 聚焦万卡级集群的万亿参数 MoE 模型训练
感谢IT之家网友 有鲫雪狐 的线索投递!

IT之家 10 月 28 日消息,摩尔线程今日宣布开源大模型训练基础设施项目 Megatron-MUSA-Titan。该项目基于摩尔线程全能力 GPU,聚焦万卡级集群的万亿参数 MoE 模型训练需要,覆盖预训练与强化学习两大情形,为开发者给出面向 MUSA 平台的训练适配与平台优化能力。

据介绍,Megatron-MUSA-Titan 已在摩尔线程万卡级集群的万亿参数 MoE 模型预训练中持续应用,经过长时间训练实践检验,达成高 MFU(模型算力利用率)与长稳训练,同时兼容与 slime、miles 强化学习框架协同,结合 Megatron 训练与 SGLang 样本生成,构建贯通万亿参数 MoE 预训练与强化学习的训练基础。

IT之家注意到,Megatron-MUSA-Titan 兼容 blockwise FP8 训练,将摩尔线程全能力 GPU 的低精度计算能力引入训练流程,提高大规模训练的计算效率。还可结合 DeepEP 与 ACE 异步通信引擎,加速 MoE 中的 Token 分发与结果合并,并达成专家并行(EP)与计算的通算重叠,让数据交换与计算任务协同执行,提高端到端训练效率。

参考:

https://github.com/MooreThreads/Megatron-MUSA-Titan

本文来源:https://www.ithome.com/1/019/131.htm
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。

Thanks to IT之家 reader 有鲫雪狐 for the tip!

IT之家, October 28 — Moore Threads today announced the open-sourcing of its large-model training infrastructure project, Megatron-MUSA-Titan. Built on Moore Threads’ full-capability GPUs, the project targets the training needs of trillion-parameter MoE models on 10,000-GPU clusters, covering both pre-training and reinforcement learning scenarios, and providing developers with training adaptation and platform optimization capabilities for the MUSA platform.

According to the company, Megatron-MUSA-Titan has been continuously used in the pre-training of trillion-parameter MoE models on Moore Threads’ 10,000-GPU clusters. Validated through long-running training practice, it delivers high MFU (Model FLOPs Utilization) and long-term training stability. It is also compatible with and supports collaboration with the slime and miles reinforcement learning frameworks; combined with Megatron training and SGLang sample generation, it builds a training foundation spanning both trillion-parameter MoE pre-training and reinforcement learning.

IT之家 notes that Megatron-MUSA-Titan supports blockwise FP8 training, bringing the low-precision computing capabilities of Moore Threads’ full-capability GPUs into the training pipeline to improve computational efficiency for large-scale training. It can also work with the DeepEP and ACE asynchronous communication engines to accelerate token dispatch and result combination in MoE, while achieving overlap between expert parallelism (EP) and computation, enabling data exchange and compute tasks to be executed in coordination and improving end-to-end training efficiency.

Reference:

https://github.com/MooreThreads/Megatron-MUSA-Titan

Moore Threads Open-Sources Megatron-MUSA-Titan Focusing on Trillion-Parameter MoE Model Training for 10,000-GPU Clusters

暂无评论,快来发表第一条评论吧!No comments yet. Be the first!

📮 需求咨询📮 Inquiry