尹良升-SGLang 推理引擎高效的开源部署方案.pdf

1、演讲人：尹良升010203040506SGLang Milestones and Features OverviewSpeculative Decoding and Constrained Decoding in SGLangEfficient Design and Implementation of PD DisaggregationLarge-scale EP Support for DeepSeek Blog ReproductionHierarchical Caching Design in SGLang The Ecosystem of SGLangpreencoded.png SG

2、Lang is a fast-serving engine for LLMs and VLMs.Among fully open-source LLM Inference Engines,SGLang currently achieves state-of-the-art(SOTA)performance,and It is the first open-source implementation to nearly match the throughput reported in the official DeepSeek blog at a large scale.Meanwhile,it

3、s elegant,lightweight,and customizable design has attracted wide adoption from academics,big tech companies,and startups.(XAI,NVIDIA,AMD,Baseten,Microsoft,Linkedin,etc.)In on-policy RLHF,inference engines are crucial for efficient policy model execution,and SGLang excels as a high-performance soluti

4、on.01SGLang Milestones and Features OverviewSGLang Milestones and Features 2023/12-2024/02:Initial Motivation,Structured LM Programming,Prefix Caching,and Constrained Decoding 2024/07:Leading Performance among inference engines on Llama3 2024/09:v0.3 Release,7x Faster DeepSeek MLA,1.5x Faster pile,M

5、ulti-Image/Video LLaVA-OneVision 2024/12:v0.4 Release:Zero-Overhead Batch Scheduler,Cache-Aware DP Router,X-Grammar Integration,The Firstto Serve DeepSeek V3.2025/01:SGLang provides day-one support for DeepSeek V3/R1 models on NVIDIA and AMD GPUs with DeepSeek-specific optimizations.(10+companies!)2

6、025/05Firstopen-sourceimplementationofDeepSeekV3/R1expertparallelismwithprefill-decodedisaggregation.Achieves 52.3K in-tok/s,22.3K out-tok/s on 96 GPUs5faster than vanilla TP.SGLang has seen extensive adoption and serves as the dominant inference engine for AMD and the default inferenceengine for xA