keynote-running-llms-in-the-cloud-xi-daepnanoai-chan-bo-yun-nf-miley-fu-developer-advocate-second-state.pdf

1、Running LLMs in the Cloud Miley Fu,WasmEdgeGitHub/Twitter:mileyfuhttps:/ Calling in Open Source TechnologyEmbed LLM into your container apphttps:/ run-rm-p 8080:8080-name api-server secondstate/llama-3-8b-nomic-1.5:latest ctx size-4096Video demo hereKey featuresTightly coupled LLM and applicationMat

2、ches prompts,quantization&runtime with the exact version of LLMThe container app always works regardless LLM upgrade cyclesLightweightOnly 5GB as opposed to 10GB PyTorch appPortableThe same binary app inside the container works on multiple CPUs and GPUsDevelop on Mac and deploy on NvidiaEasy to embe

3、d into Rust/JS/Python appsWorks with existing container tools,such as K8sLlamaEdgeReal-world use cases1.Personal LLMs:Gaia Network,users run personal LLMs with embedded knowledge base.2.AI OS:Open interpreter as a local LLM provider;51.5k stars3.Finance:Financial analytics bot.4.Hardware:Robot voice

4、 control 5.Education:UC Berkley TA6.Game:Open-source game engine Cocos AI,use Wasm to run AI models that enhance gameplay experiences enabled by NPC.Use a Gaia Net imageLlamaEdge:easy LLM deployment+inferenceSingle cross-platform binary(Automagically take advantage of local hardware accelerators)Com

5、pile and test apps on one machine(e.g.,Mac)and deploy it to another cloud server(e.g.,Nvidia CUDA 12)The app can be moved around and deploy to new hardware by K8sPackage the Wasm app into Docker image as an embedded AI/LLM service.Only 1/100 the size of a Python runtime.WasmEdgehttps:/ living knowle

6、dge serverhttps:/ AI EcosystemCNCF+LF AI&Data Survey How your company uses GenAI toolsDifferent use casesModalities/models in useChallenges to adoptionThe role of open source in adoption decisionsCalling for contributorsYouTubeStay in Touch