创新洞察：优化推理时代的人工智能堆栈.pdf

1、Optimizing the AI Stack for the Age of InferencingMaximizing Infrastructure Utility for Scalable AISpeaker:Yujing Qian,VP of Engineering,GMI Cloud World Summit AI,April 16,2025What to Expect:Inference Optimization Key Strategies Hardware AccelerationP/D disaggregationPerformant inference engineDistr

2、ibuted KV CacheWhy Inference Optimization Matters TodayBusiness ValueInference drives real-world applicationsCost ControlOptimization essential for maximizing ROIAdaptationEcosystems must evolve with technologyEfficiencyBalancing performance with resource usageThe Cost of Inefficient Inference30 xPe

3、rformance GapDue to bottlenecks in model execution71%Cost ReductionPossible with proper optimization13 xLatency ImpactDue to challenges balancing latency and throughputFast&Flexible auto-scalingGlobal cluster coverageFast scaling ability for viral applicationsUsers are scattered across different reg

4、ion,need multi-cluster setupNeed High efficiency with latest GPUsStability and security is a growing concern for AICompliance(e.g.GDPR)Core need12345Efficient inference clusterHigh reliability and securityKey of AI inference:efficiency,scalability and stabilityScalable Inference System Best Practice

5、sPerformant inference engineMinimum core component,squeeze every bit of performance out of your hardwareWorkload OptimizationIntelligent distribution across hardware types and workload typesGlobal Autoscaling Scale out our workload to hybrid cloud and multi-clusterAlways on the lookout for Best hard

6、ware*data provided by NVIDIA DeepSeek-FP4 Hardware upgrades is one the easiest way to improve performanceB 200 is more than 25x performant than H200.Latest hardware hosts latest model betterThroughput tested on single 8 x H100、H200、B200 server nodePrefill/Decode DisaggregationRate limitAPI GatewayUs