微软：2021 GPT-3赋能数据标注成本降低研究报告（英文版）（11页）.pdf

1、Want To Reduce Labeling Cost?GPT-3 Can HelpShuohang WangYang LiuYichong XuChenguang ZhuMichael ZengMicrosoft Cognitive Services Research Groupshuowa,yaliu10,yicxu,chezhu,AbstractData annotation is a time-consuming andlabor-intensive process for many NLP tasks.Although there exist various methods to

2、pro-duce pseudo data labels,they are often task-specificandrequireadecentamountoflabeleddata to start with.Recently,the immense lan-guage model GPT-3 with 175 billion param-eters has achieved tremendous improvementacross many few-shot learning tasks.In thispaper,we explore ways to leverage GPT-3 asa

3、 low-cost data labeler to train other models.We find that,to make the downstream modelachieve the same performance on a variety ofNLU and NLG tasks,it costs 50%to 96%less to use labels from GPT-3 than using la-bels from humans.Furthermore,we propose anovel framework of combining pseudo labelsfrom GP

4、T-3 with human labels,which leads toeven better performance with limited labelingbudget.These results present a cost-effectivedata labeling methodology that is generaliz-able to many practical applications.1IntroductionData always plays a crucial role in developing ma-chine learning models.However,c

5、ollecting human-labeled data is a costly and time-consuming pro-cess,especially in multi-task scenarios.With thesuccess of pre-trained models(Zhang et al.,2020;Raffel et al.,2020;Liu et al.,2019;Devlin et al.,2019)on unlabeled data,the performance of mod-els under few-shot and zero-shot settings has

6、 beengreatly enhanced.In particular,the large-scale lan-guage model GPT-3(Brown et al.,2020),with 175billion parameters,is the state-of-the-art few shotlearner on many NLP tasks.However,GPT-3 is constrained on its immensemodel size and requires a large amount of resourceto be deployed for real appli