6375 - Challenges in implementing AI-ML training job recovery from GPUAccelerator data poisoning events.pdf

1、Anil AgrawalDavid XiaoChallenges in implementing AI/ML training job recovery from GPU/Accelerator data poisoning eventsChallenges in implementing AI/ML training job recovery from GPU/Accelerator data poisoning eventsAnil AgrawalDavid XiaoARTIFICIAL INTELLIGENCE(AI)AI/ML Training Job interruptions-a

2、challengeAL/ML Training Cluster OverviewData Flow with HBM Uncorrected errorOpportunities to prevent Job interruptionsCall to ActionAgendaObserved16K GPU Cluster54 day training period419 job interruptions1 Job interruption every 3 hrsAI/ML Training Job interruptions-a challengeProjection128K GPU Clu

3、ster1 Job interruption every 23 minsReference:https:/ to reduce the Job Interruption Rate?AI/ML Training Cluster 30K ft viewAI/ML Training Cluster-OverviewReference:How Meta trains large language models at scaleGrand Teton Training System-OverviewReference: Teton Training Platform-ArchitectureOAM:OC

4、P Accelerator ModuleOAM:OCP Accelerator ModuleAccelerator/GPUOCP Accelerator Module(OAM)OAM:OCP Accelerator ModuleProcessing EngineProcessing EngineCacheMemory ControllerHigh Bandwidth Memory(HBM)High speed IOSoftware StackReference:OCP Accelerator Module(OAM)Base Specification r2.0 v1.01.Demand Rea

5、d Request2.Uncorrected Error(UCE)Detected in the HBM3.Poisoned Data Returned to the PE.Error is Contained.4.Job is killed resulting in the interruption.5.Offline memory page.Reset Device.HBM row-remap.6.Prevents any subsequent job interruptions.Data Transaction Flow with HBM UCEAccelerator/GPUProces

6、sing EngineProcessing EngineCacheMemory ControllerHigh Bandwidth Memory(HBM)High speed IOSoftware Stack within the OAMRead RequestUCEAny RAS Solution to further reduce the Job Interruption?Poisoned Data Returned123Poison Recovery Example-Host CPU SystemReferences:1.Linux/mm/memory-failure.c2.“RECOVE