vescale-a-pytorch-native-llm-training-framework-vescaledaeppytorch-chang-llmxiao-zhen-hongyu-zhu-bytedance-1.pdf

1、A PyTorchA PyTorch-Native AutoNative Auto-Parallel Framework Parallel Framework for for Ease of UseEase of UseveScale TeamByteDance2024-8-22About MeReceived my PhD degree from University of Toronto(advisor:Gennady Pekhimenko)Joined ByteDance AML group in 2022.3Currently working on LLM training frame

2、worksHongyu ZhuAgendaWhy veScaleWhat is veScalePreliminary Results of veScaleFuture of veScaleMany weeks to write one modelWhy veScaleGradBuffer DefragAllReduce Overlapnn.LinearColumnParallelLinearIntertwined BugsHeavy Maintainance EffortA PyTorch-Native Auto-Parallel Framework for Ease of UseGradBu

3、ffer DefragAllReduce Overlapnn.LinearNo IntertwiningLight Maintainance EffortAgendaWhy veScaleWhat is veScalePreliminary Results of veScaleFuture of veScalenn.Linear,nn.Embedding,.PP=8,DP=4,TP=2,PPSplit=fc1,fc3,.,TPShard=fc1.weight:Shard(),.convert torch.Tensorsave&loadWhat is veScaleAgendaWhy veSca

4、leWhat is veScalePreliminary Results of veScaleFuture of veScaleSimple API of nD Parallel Training(WIP)Preliminary Results of veScaleZero Code Change of ModelZero Code Change of Training LoopnD Parallel Training in 5 LoCPreliminary Results of veScaleBitwise Correctness of 4D Parallel TrainingnanoGPT

5、nanoGPT TrainingPreliminary Results of veScaleBitwise Correctness of 4D Parallel TrainingMixtralLLama2End2End MFU SpeedupPyTorch TPveScale TP1x1.21.4xPreliminary Results of veScaleDecent Performance of TP(WIP)Mixtral MoEEnd2End MFU SpeedupPyTorch TPveScale TP1x1.21.3xLLAMA2AgendaWhy veScaleWhat is v

6、eScalePreliminary Results of veScaleFuture of veScaleveScaleSimple APIBitwise CorrectnessDecent PerformanceOpen Source Community:EveryoneEnd Goal:supportImpact!“An Ambitious Work!”-Llama Training Lead-PyTorch Training Lead“A Promising Work!”-AWS AI Lab-Octol A