4324 - 了解如何使用 llm-d 实现分布式 LLM 推理.pdf

1、IBM TechXchange 2025 conference IBM TechXchange|2025 IBM Corporation1#IBMTechXchangellm-dLLM inference goes distributedCarlos CostaDistinguished Engineer,IBM ResearchIBM TechXchange|2025 IBM CorporationScaling Inference,Not Your Ops Team2IBM TechXchange|2025 IBM CorporationThe cost of operating an A

2、I platform A critical pain point across the industrya constraint optimization problem,balancing GPU hours,model size,and context length against business value per tokenLatencyTime to take to generate a responseThroughputHow many users or agents can run in parallelCost$per output token,amortized acro

3、ss GPUsAccuracy/Quality How close the output is to desired accuracyEfficiencyGPU/CPU utilization vs.wasteEnergy efficiencyJoules per token3IBM TechXchange|2025 IBM Corporation4Distributed inference is essential for cost-effective GenAI at scale,but introduces unique operationalization challengesLLM

4、inference workloads with variable,resource-heavy and hardware-affinity nature of requestsEnsuring SLO(throughput,TTFT,latency)while optimizing resource utilization and reducing operational complexityLeveraging and managing heterogenous hardware for better cost-efficiencyDistributed KV cache manageme

5、nt as key part in inference efficiencyBatch,interactive,multi-turn and agentic patternsAMultiple small models(e.g.granite-8b,mistral-7b,)Single large model(lama3-70b,llama4-Maverick)Single ultra large(MoE)model(e.g.DeekSeek R1-V3,Kimi K2,)BCGPUABCGPUGPUDENode AGPUA(shard 1)GPUGPUNode AA(shard 2)GPUE

6、(shard 1)GPUGPUNode AE(shard 2)GPUE(shard 9)GPUGPUNode BE(shard 10)E(shard 8)E(shard 16)Netw.The industry-wide challenge of scaling inference4IBM TechXchange|2025 IBM CorporationRequests with significant variance in resource utilizationRouting to specific replicas with cached prior computation can a