刘群-大语言模型技术现状和发展趋势的思考-掘金.pdf

1、大语言模型技术现状与发展趋势的思考Thoughts on LLM Technology Trends刘群|华为诺亚方舟实验室语音语义首席科学家Content大模型技术概览及总体趋势 大语言模型先天能力发展趋势 大语言模型后天能力发展趋势大语言模型问题、风险和对社会的影响总结业界大模型概览 2023年是大模型爆发的一年 2023年又被称为AGI元年大模型已经深刻影响了AI 大模型还将深刻影响我们的社会decoder-only architecture unsupervised multitask learner in-context learning code pre-training gene

2、rative pre-training scaling the model size exploring scaling limits2022.03capable code model+codeGPT-1 GPT-2 GPT-3 Codex2018.06 2019.02 2020.05 2021.07GPT-42023.03strong reasoning ability multi-modal abilityGPT-3.52022.03ChatGPTcode-davinci-002+instruction text-davinci-002+RLHF text-davinci-003+chat

3、 gpt-3.5-turbo2022.03 2022.09 2023.03instruction following human alignment excellent comprehensive ability2020202320211-45-81-34-67-1011-12T5GPT-32019GPT-NeoX-20BCodeGenOPTMT-NLGT0 9-101-6GPT-4GShardUL2PaLMFlan-T5Flan-PaLMSparrowChatGPTPanGu-GopherGLaMmT5PanGu-PLUGBardLLaMALaMDACPM-2Publicly Availab

4、leCodexErnie 3.0Jurassic-1AnthropicWebGPT Ernie 3.0 TitanNLLBTk-Instruct CoherePythiaVicunaLuminousYaLMHyperCLOVA 11-12InstructGPTFLANYuan 1.0 2022GLMAlexaTMBLOOMmT0 BLOOMZGalatica OPT-IMLWeLMAlphaCodeChinchillaFalconCodeGeeXLLaMABenTsaoBaizeBELLEAlpaca LoraLawyer LLaMA+chatchat datadataInstructBLIP

5、Yulan-ChatZiya+tasktask datadataMultimodal models+tasktask datadata GuanacoKoala +tasktask datadataLLaMAAdapterVicunaAlpacaPandaPandaGPTChinese LLaMA+chatchat datadataCornucopia+tasktask datadataTaoLiChinese AlpacaChatMed+syntheticsynthetic datadataChinese VicunaLinly-Chinese-LLaMAOpen-Chinese-LLaMA

6、+tasktask datadataLAWGPTRLHFRLHFPKU-BeaverChatbridgeOpenFlamingo LLaVAVisionLLMMiniGPT-4Goat+chatchat datadataQiZhenGPTBiLLa+tasktask datadataContinue pre-trainingModel inheritance InstructionData inheritance tuningLawBilingualism EducationMath Finance MedicineParameter-efficient fine-tuningFull par