大型语言模型和人类在环分类法开发.pdf

1、Beyond ChatbotsLarge Language Models&Human in the Loop taxonomy development1Information Architecture-Taxonomy2Information Architecture-TaxonomyOverviewData challengesHow we leveraged GenAILessons we learned3Information Architecture-TaxonomyEAs Data Universe20+Studios100+Active games16m Daily Active

2、UsersCustomer help channels26M cases a year4Information Architecture-TaxonomyFocusing our Data TelescopeSome issues fall into the wrong bucketsSome may not be represented/identifiedToo much ambiguity in like issuesOur categories were too wideWhat if a fan thinks the game crashed when it really disco

3、nnected?Phrasing mattersGame is crashingInformation Architecture-TaxonomyClean boot-5 minutesRe-installation-30min-2hrs+for digitalGame disconnectedRouter/Modem troubleshooting-5-10minContacting your ISP-Between 9am and 6pm all of November5Getting to the pointWhen did it happen ingame?What did you s

4、ee?What have you already done?Support agents probe for infoInformation Architecture-Taxonomy6Its all in the chat logs7Information Architecture-TaxonomyHow did we solve for this?ManualLabelingWe did it the hard wayof chat logs8Information Architecture-TaxonomySo many chat logsManual work takes a long

5、 time4000 of chat logs55+categoriesNeed to be recruited Need to be trained2 years to develop the first taxonomiesRepresents only a small fraction of customer support data9Information Architecture-TaxonomyOur universe is growingWe want to do moreFor more peopleNew games and experiences on the horizon

6、Must process more data10Information Architecture-TaxonomyHow could we scale our data processing?Work with Data Science and AI!Large language models(LLMs)can process thousands of rows in hours instead of years.Went from matching to exceeding previous corpus sizeDo more in less time!What were we able