可预测的扩展和基础设施.pdf

1、ConfidentialPredictable Scaling and InfrastructureTrevor CaiConfidentialPredictable Scaling2Confidential What Collect a dataset of text,code,images,audio,math.Pre-train a model to predict the next word.Post-train it to e.g.follow instructions,be conversational,use tools.How Synchronous SGD of a tran

2、sformer on a large cluster of accelerators.Combine many forms of data and model parallelism.Reinforcement learning from human feedback.3The What and How of ChatGPTConfidential Trained a character-level neural network on product reviews.Observed:there is a neuron encoding sentiment!Results in state-o

3、f-the-art sentiment analysis.4The Sentiment Neuron(2017)Learning to Generate Reviews and Discovering Sentiment,Radford et al(2017)Confidential Next word prediction models the underlying generative process.If the data is the internet,the underlying generative process is the world.Grandiose.But theore

4、tically justified via Solomonoff Inductive Inference.5Prediction is CompressionLearning to Generate Reviews and Discovering Sentiment,Radford et al(2017)ConfidentialReleaseNew BehaviorsGPT-1 June 2018 State-of-the-art language understanding(using task-specific fine tuning).GPT-2 Feb 2019Coherent tex

5、t generation and zero-shot transfer.GPT-3 Mar 2020In-context learning.GPT-4 Mar 2023Actually being useful.6Returns to Scale(2018-2023)Confidential7Scale Works(2023)This example required GPT-4 to:Understand both English and French.Interpret a diagram in context of the text.Solve a physics problem!GPT

6、-4 Technical Report,OpenAI(2023)Confidential8Predictable ScalingGPT-4 Technical Report,OpenAI(2023)Confidential9Predictable Scaling of Practical CapabilitiesGPT-4 Technical Report,OpenAI(2023)Confidential10What Log-Log Plots ObscureHow predictable is language model benchmark performance?,Owen(2024)C