unlocking-llm-performance-with-ebpf-optimizing-training-and-inference-pipelines-chuan-hui-ebpfji-xi-llmxia-daep-xiao-zhen-relia-fa-qiu-yang-xiang-yunshan-networks-inc.pdf

1、Unlocking LLM Performance with :Optimizing Training and Inference Pipelines向阳，云杉网络Outline1.1.背景背景：训练和推理的效率挑战2.2.现状现状：传统解决方案和工具的问题3.3.方法方法：eBPF 构建零侵扰可观测性4.4.实践实践：PyTorch 全栈剖析和追踪 LLM 训练开销大、效率低GPT-4GPT-4Llama-3.1Llama-3.1Size1.8T405BGPU25K A10016K H100Days9010054MFU32%36%38%43%Everything We Know About

2、GPT-4-Klu.aiGPT4-All Details LeakedThe Llama 3 Herd of Models训练时间长：数月GPU 数量多：数万GPU 利用率低：40%GPU 年化故障高：6%11%模型参数大：万亿148/54*365/16384=6%(148+72+19+17+6+6)/54*365/16384=11%故障并不是训练低效的唯一原因内核计算显存拷贝网络传输 Yanjie Gao(Microsoft Research)et al,ACM ICSE 2024,An Empirical Study on Low GPU Utilization of Deep Learn

3、ing Jobs.Yanjie Gao(Microsoft Research)et al,ACM ICSE 2023,An Empirical Study on Quality Issues of Deep Learning Platform.如何知晓你的训练任务是否存在这些问题？LLM 推理开销大、时延高LlamaLlama8B8B70B70B405B405BFP3236GB267GB1.48TB1.48TBFP1620GB135GB758GBINT812GB70GB382GBINT48GB37GB193GBLLM Memory RequirementsLLM Inference Perfo

4、rmance Engineering:Best Practices80GB:1 GPU640GB:1 Node x 8 GPU1.28TB：2 Node x 8 GPUTime To First Token(TTFT)Time Per Output Token(TPOT)Model Bandwidth Utilization(MBU)GPU 并非越少越好GPU 越少，则每个 GPU 需要加载更多的模型参数。GPU 并非越多越好GPU 越多，则通信越复杂，内存碎块越多。没有银弹，可观测性是优化的前提。排查 LLM 推理显存消耗的挑战Python/C+PyTorchvLLMBiz从推理应用到在线

5、LLM 推理服务APIGWAuthServiceRAGVectorDBToolsCacheGatewayLLM-1LLM-2vLLMPyTorchSGLang.LLMAgentMemCacheDiskCache云-在线推理服务是一个复杂的分布式服务TTFT、TPOT、时延、吞吐端-自动驾驶、具身智能（ROS2）的端到端低时延和高稳定性要求Robot Operating System 2(ROS2)-Based Frameworks for Increasing Robot Autonomy:A SurveyPromptsSignalsPerception-Planning-ControlBEV

6、-OCC-Transformer从大模型到小模型：消费级 GPU、CPU 协同Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs,SIGCOMM 2024.AI 训练和推理的可观测性需求GPU计算HBM显存COMM通信SpanSpanSpanSpanSpan训练&推理在线、端侧推理服务cudaMemcpyop(N,x,y)cudaSynccudaMemcpyHBMGPUHBMHostDevncclCOMMtGatewayAuth3rdROS2 NodesGateway.Outlin