simplify-ai-infrastructure-with-kubernetes-operators-zhi-kubernetes-operatorshu-ai27dya-shi-ganeshkumar-ashokavardhanan-microsoft-tariq-ibrahim-us-nvidia.pptx

1、Simplify AI Infrastructurewith Kubernetes Operators,Ganeshkumar Ashokavardhanan,MicrosoftTariq Ibrahim,NVIDIA,About us,Ganeshkumar AshokavardhananMicrosoft:Azure Kubernetes Service,Tariq IbrahimNVIDIA:Cloud Native Technologies,Outline,Why Kubernetes(k8s)for AI/ML workloadsLayers of ML deployment sta

2、ckGPU management:Overview and challengesIntro to Kubernetes OperatorsWhy use Operators for GPU managementGPU operator design and detailsML application layer:LLM fine-tuning demo:application-level operator+GPU operator,Why K8s for AI/ML?,Scalability:Scale resources up and down on-demandFault toleranc

3、e:High availability design.Self-Healing capabilitiesExtensibility:Extend the K8s API through Custom Resource Definitions(CRD)Ecosystem:A rich(and growing)set of applications and operators for AI/ML use-cases,Layers of ML Deployment Stack,GPU Management in Kubernetes,Enabling GPU Support in Kubernete

4、s Today,V100,T4,Install the GPU Device Driver,Enabling GPU Support in Kubernetes Today,V100,T4,Install the NVIDIA Container Toolkit Configure container runtimes to access GPUs from within containers,Container Runtime,Enabling GPU Support in Kubernetes Today,V100,T4,Deploy the Device Plugin Advertise

5、 GPUs to the kubelet,Container Runtime,Enabling GPU Support in Kubernetes Today,V100,T4,Deploy the Device Plugin Advertise GPUs to the kubelet,Container Runtime,$kubectl describe node Name:Labels:Capacity:cpu:40 ephemeral-storage:922257324Ki 8 memory:65487504KiAllocatable:cpu:40 ephemeral-storage:84

6、9952348392 8 memory:65385104Ki,Enabling GPU Support in Kubernetes Today,V100,T4,Host-level Components nvidia-container-toolkit nvidia-gpu-driver,Machine Image,Step 1:Use Machine Images or Provisioning Scripts to install host level components,GPU Machines,Enabling GPU Support in Kubernetes Today,V100