论前沿训练基础模式.pptx

1、On Training Foundation Models on Frontier,Sajal DashAnalytics and AI Methods at Scale(AAIMS)NCCS,ORNLdashsornl.gov,Outline,Distributed training of LLMsPreparing a training software stack for FrontierFinding best distributed training strategiesTraining a Trillion parameter model on Frontier(for a few

2、 iterations)Training“Sparsely Activated”models,Transformer Model Architecture,BERT,GPT,Inside a Transformer Layer,Model Parallelism:Why and How,Models(or Memory needed for their training)are too big to fit in a single GPUSo,we need to break the model into piecesWhats broken needs to be rebuilt from

3、the piecesIts like Kintsugi,but with a more practical concern such as price of gold(comm.Latency),Kintsugi,Tensor Parallelism(TP=4),Lessons Learnt From Tensor Parallelism,Pipeline Parallelism(PP=2),Pipeline Parallelism(Gpipe),Pipeline Bubble vs#Microbatches,Increasing the#Microbatches will reduce th

4、e bubbleBut that will result in large global batch size,hurting the convergence,3D Parallelism using Megatron-DeepSpeed,Megatron-DeepSpeed is a state-of-the-art training framework developed by Microsoft and NVIDIAIt supports 3D+parallelism(Tensor,Pipeline,Data,Sharded Data)We ported this Framework t

5、o Frontier through hipify and combining with ROCM specific packageshttps:/,3D Parallelism,A Combination of Tensor,Pipeline,and Data ParallelismDetermine how many GPUs(world-size)you need to fit the modelFactorize world-size into TP(tensor parallel size)and PP(pipeline parallel size),Hybrid(TP=4,PP=2

6、),Node 1,Node 2,Best practices with parallelism paradigms,Tensor Parallelism Keep it within the node(TP 8)Pipeline Parallelism Use large number of micro-batches(But that can increase the global batch-size)Data ParallelismCant use too much data parallelism.A la