Deepseek：2025原生稀疏注意力：硬件对齐且可原生训练的稀疏注意力机制技术报告（英文版）（24页）.pdf

1、Native Sparse Attention:Hardware-Aligned and NativelyTrainable Sparse AttentionJingyang Yuan1,2,Huazuo Gao1,Damai Dai1,Junyu Luo2,Liang Zhao1,Zhengyan Zhang1,Zhenda Xie1,Y.X.Wei1,Lean Wang1,Zhiping Xiao3,Yuqing Wang1,Chong Ruan1,Ming Zhang2,Wenfeng Liang1,Wangding Zeng11DeepSeek-AI2Key Laboratory fo

2、r Multimedia Information Processing,Peking University,PKU-Anker LLM Lab3University of Washingtonyuanjy,mzhang_,zengwangding,AbstractLong-context modeling is crucial for next-generation language models,yet the high compu-tational cost of standard attention mechanisms poses significant computational c

3、hallenges.Sparse attention offers a promising direction for improving efficiency while maintaining modelcapabilities.We present NSA,aNatively trainableSparseAttention mechanism that integratesalgorithmic innovations with hardware-aligned optimizations to achieve efficient long-contextmodeling.NSA em

4、ploys a dynamic hierarchical sparse strategy,combining coarse-grainedtoken compression with fine-grained token selection to preserve both global context awarenessand local precision.Our approach advances sparse attention design with two key innovations:(1)We achieve substantial speedups through arit

5、hmetic intensity-balanced algorithm design,with implementation optimizations for modern hardware.(2)We enable end-to-end training,reducing pretraining computation without sacrificing model performance.As shown in Figure 1,experiments show the model pretrained with NSA maintains or exceeds Full Atten

6、tion modelsacross general benchmarks,long-context tasks,and instruction-based reasoning.Meanwhile,NSA achieves substantial speedups over Full Attention on 64k-length sequences across decod-ing,forward propagation,and backward propagation,validating its efficiency throughout themodel lifecycle.1.Intr