张松昕-Data-centric LLM training.pdf

1、Data-centric LLM trainingSongxin ZhangSUSTechJuly 5th,20241/26Limit of Data Scaling2/26Intelligence emergeing with data scalingBLOOM,June,2022ROOTS is the dataset used by Hugging Face to trainBLOOM.It compiles 0.34T tokens from 498 di?erentdata sources,including classic datasets used in NLPresearch

2、from 2008 to 2021.176B,0.34T tokens0.00850.0660.170.240.340.5GPT-2C4OPTThe PileROOTSGPT-300.10.20.30.40.5tokens3/26Intelligence emergeing with data scalingToday,less than two years later,we are training smaller models thanBLOOM and GPT-3 with more than tokens.2020LLaMA-3 8B is trained on 15T tokens.

3、LLMLlama 3StabilityRedPajamaLaMDAchinchillaGopherMT-NLGgpt-neoxgpt-34/26Large Models are running out of dataThere isnt enough data to train much more capable LLMs.5/26With limited data,how to train large models data-e?ciently?Parametric?t.We?t a parametric modelling of the loss and display contour(l

4、eft)andisoFLOP slices(right).For each isoFLOP slice,we include a corresponding dashed line in the leftplot.In the left plot,we show the e?cient frontier in blue,which is a line in log-log space.Speci?cally,the curve goes through each iso-loss contour at the point with the fewest FLOPs.Weproject the

5、optimal model size given the Gopher FLOP budget to be 40B parameters()Scaling Law()With more data-e?cient training,wecan have better with the same.(N,D)LHo?mann et al.2022Ho?mann et al.2022(N,D)E+(1)LANBDDHo?mann,Jordan,Sebastian Borgeaud,Arthur Mensch,Elena Buchatskaya,Trevor Cai,Eliza Rutherford,D

6、iego de Las Casas,et al.2022.“Training Compute-Optimal Large Language Models.”.Villalobos,Pablo,Anson Ho,Jaime Sevilla,Tamay Besiroglu,Lennart Heim,and Marius Hobbhahn.2024.“Will We Run Out of Data?Limits of LLM Scaling Based on Human-Generated Data.”.https:/arxiv.org/abs/2203.15556https:/arxiv.org/