1、ERNIE 5.0 Technical ReportERNIE Team,BAbstractIn this report,we introduce ERNIE 5.0,a natively autoregressive foundation modeldesinged for unified multimodal understanding and generation across text,image,video,and audio.All modalities are trained from scratch under a unified next-group-of-tokens pr
2、ediction objective,based on an ultra-sparse mixture-of-experts(MoE)architecture with modality-agnostic expert routing.To address practical challengesin large-scale deployment under diverse resource constraints,ERNIE 5.0 adopts anovel elastic training paradigm.Within a single pre-training run,the mod
3、el learnsa family of sub-models with varying depths,expert capacities,and routing sparsity,enabling flexible trade-offs among performance,model size,and inference latencyin memory-or time-constrained scenarios.Moreover,we systematically address thechallenges of scaling reinforcement learning to unif
4、ied foundation models,therebyguaranteeing efficient and stable post-training under ultra-sparse MoE architecturesand diverse multimodal settings.Extensive experiments demonstrate that ERNIE 5.0achieves strong and balanced performance across multiple modalities.To the bestof our knowledge,among publi
5、cly disclosed models,ERNIE 5.0 represents the firstproduction-scale realization of a trillion-parameter unified autoregressive model thatsupports both multimodal understanding and generation.To facilitate further research,we present detailed visualizations of modality-agnostic expert routing in the
6、unifiedmodel,alongside comprehensive empirical analysis of elastic training,aiming to offerprofound insights to the community.Feb 4,20261Contents1Introduction32Architecture42.1Unified Autoregressive Backbone with Ultra-Sparse Mixture-of-Experts.42.2Visual Modeling.52.2.1Vision Tokenization.52.2.2Vis
0731-84720580
商务合作：really158d
友链申请 (QQ)：1737380874
研搜
微信扫码登录
手机快捷登录
账号登录
