SK海力士人工智能专用计算内存解决方案：从AiM设备到异构AiMX-xPU系统实现全面的LLM推理.pdf

1、SK hynix AI-Specific Computing Memory Solution:From AiM device to Heterogeneous AiMX-xPU System for Comprehensive LLM InferenceGuhyun Kim1,Jinkwon Kim1,Nahsung Kim1,Woojae Shin1,Jongsoon Won1,Hyunha Joo1,Haerang Choi1,Byeongju An1,Gyeongcheol Shin1,DayeonYun1,Jeongbin Kim1,Changhyun Kim1,Ilkon Kim1,

2、Jaehan Park1,Yosub Song1,Byeongsu Yang1,Hyeongdeok Lee1,Seungyeong Park1,Wonjun Lee1,Seonghun Kim1,Yonghoon Park1,Yousub Jung1,Gi-Ho Park2,and Euicheol Lim11SK hynix inc,2Sejong UniversityRecap Accelerator-in-Memory(AiM)&AiMXSystem Extensions of AiMX Card for DatacenterAiM&AiMX for On-device AIDesig

3、n Choices for Future AiM/AiMXConclusionRecap Accelerator-in-Memory(AiM)and AiMX SK hynix Inc.This material is proprietary of SK hynix Inc.and subject to change without notice.3Is LLM Sustainable?Too Expensive Operating ExpenditureYearsCost SK hynix Inc.This material is proprietary of SK hynix Inc.an

4、d subject to change without notice.4Large Language Model Memory Bound(*)Assumptions:batch1 inference during output token generation phaseMatrix-Vector Multiplication(GEMV)Mainly Consists of Matrix-Vector Multiplications(or GEMV)Multi-headAttention(MHA)Fully-connected(FC)LayersTransformer Architectur

5、eRatiox NGEMV:Memory BW-Bound with Low Arithmetic IntensityLLM ArchitectureMatrix-Vector Multiplication+=+yOperational IntensityPerformanceFCMHAMemory-boundCompute-boundCNN SK hynix Inc.This material is proprietary of SK hynix Inc.and subject to change without notice.5Accelerator-in-Memory:“True All

6、-Bank Parallelism”SK hynixs First GDDR6-based Processing-in-Memory Product SampleGDDR6-AiM Die Photograph(External)Bandwidth*32 GB/sOperating Speed1 GHzCompute Throughput*512 GFLOPSInternal Bandwidth*512 GB/sNumeric PrecisionBF16(*)S.Lee et al.,ISSCC22(*)Defined as a peak during burst operations(*)X