1、Native Sparse Attention:Hardware-Aligned and NativelyTrainable Sparse AttentionJingyang Yuan1,2,Huazuo Gao1,Damai Dai1,Junyu Luo2,Liang Zhao1,Zhengyan Zhang1,Zhenda Xie1,Y.X.Wei1,Lean Wang1,Zhiping Xiao3,Yuqing Wang1,Chong Ruan1,Ming Zhang2,Wenfeng Liang1,Wangding Zeng11DeepSeek-AI2Key Laboratory fo
2、r Multimedia Information Processing,Peking University,PKU-Anker LLM Lab3University of Washingtonyuanjy,mzhang_,zengwangding,AbstractLong-context modeling is crucial for next-generation language models,yet the high compu-tational cost of standard attention mechanisms poses significant computational c
3、hallenges.Sparse attention offers a promising direction for improving efficiency while maintaining modelcapabilities.We present NSA,aNatively trainableSparseAttention mechanism that integratesalgorithmic innovations with hardware-aligned optimizations to achieve efficient long-contextmodeling.NSA em
4、ploys a dynamic hierarchical sparse strategy,combining coarse-grainedtoken compression with fine-grained token selection to preserve both global context awarenessand local precision.Our approach advances sparse attention design with two key innovations:(1)We achieve substantial speedups through arit
5、hmetic intensity-balanced algorithm design,with implementation optimizations for modern hardware.(2)We enable end-to-end training,reducing pretraining computation without sacrificing model performance.As shown in Figure 1,experiments show the model pretrained with NSA maintains or exceeds Full Atten
6、tion modelsacross general benchmarks,long-context tasks,and instruction-based reasoning.Meanwhile,NSA achieves substantial speedups over Full Attention on 64k-length sequences across decod-ing,forward propagation,and backward propagation,validating its efficiency throughout themodel lifecycle.1.Intr
0731-84720580
商务合作：really158d
友链申请 (QQ)：1737380874
研搜
微信扫码登录
手机快捷登录
账号登录
