百度：2026文心5.0技术报告（英文版）（36页）.pdf

1、ERNIE 5.0 Technical ReportERNIE Team,BAbstractIn this report,we introduce ERNIE 5.0,a natively autoregressive foundation modeldesinged for unified multimodal understanding and generation across text,image,video,and audio.All modalities are trained from scratch under a unified next-group-of-tokens pr

2、ediction objective,based on an ultra-sparse mixture-of-experts(MoE)architecture with modality-agnostic expert routing.To address practical challengesin large-scale deployment under diverse resource constraints,ERNIE 5.0 adopts anovel elastic training paradigm.Within a single pre-training run,the mod

3、el learnsa family of sub-models with varying depths,expert capacities,and routing sparsity,enabling flexible trade-offs among performance,model size,and inference latencyin memory-or time-constrained scenarios.Moreover,we systematically address thechallenges of scaling reinforcement learning to unif

4、ied foundation models,therebyguaranteeing efficient and stable post-training under ultra-sparse MoE architecturesand diverse multimodal settings.Extensive experiments demonstrate that ERNIE 5.0achieves strong and balanced performance across multiple modalities.To the bestof our knowledge,among publi

5、cly disclosed models,ERNIE 5.0 represents the firstproduction-scale realization of a trillion-parameter unified autoregressive model thatsupports both multimodal understanding and generation.To facilitate further research,we present detailed visualizations of modality-agnostic expert routing in the

6、unifiedmodel,alongside comprehensive empirical analysis of elastic training,aiming to offerprofound insights to the community.Feb 4,20261Contents1Introduction32Architecture42.1Unified Autoregressive Backbone with Ultra-Sparse Mixture-of-Experts.42.2Visual Modeling.52.2.1Vision Tokenization.52.2.2Vis