李乔--钉钉 AI 拟人操作.pdf

1、例：撑海量数据的数据平台与架构 例：茹炳晟例：腾讯Tech Lead，腾讯研究院特约研究员正要求：微软雅：最字号 8号 宋体：最字号 10号 等线：最字号 12号钉钉 AI 拟操作李乔阿巴巴钉钉 级技术专家机器如何操作具VSMachine-orientedHuman-oriented机器狗为什么也 95 式01为什么我们需要 AI 拟操作回到钉钉的场景-需要连接泛的业务系统LLM 普遍具备的基本素养 能听、能说、能写、能画.AI但作为 Agent，还要会操作 OA,ERP,CRM,SCM,MES 报销,请假,审批,差旅 Connectivity and Interoperability回到钉钉

2、的场景-如何连接与操作Graphical User Interface Human-orientedApplication Programming Interface Machine-orientedVSAPI-Oriented 的问题理想 API First现实 对海量历史遗产VSAPI-Oriented vs.GUI-Oriented为系统开发 API，对接 API，去适应机器让 AI 像样使 GUI 机器，去适应VSAI 时代的新“机”界GUIAI AgentUserObservingActingTalkingAI As Users AgentAI As CollaboratorGUIA

3、I AgentUserObservingActingTalking02从强化学习的视经典强化学习范式Reinforcement Learning Imitation LearningExperience03个例：Anya 的常Anya 的常外卖 App 订蛋糕程序 点咖啡评价 App 订餐厅钉钉 App 群消息Web 系统 订会议室硅基版的 Anya 需要什么Environment Mobile Apps,Desktop Apps,Web On Local,On CloudObservation Space Vision-based,Structure-basedAction Space Fi

4、nger-based(Human-like),API-basedExperience From self,From others Language-based,Structure-basedRewards From Meters,From Introspection硅基版的 Anya 做到了什么样Pass Rate 97.5%(Simple Cases)75.2%(Complex Cases)Performance 0.83(Simple Cases)1.91(Complex Cases)Platforms Web Mobile Apps Local and Cloud04Lessons To

5、 Take AwayLessons To Take AwayHow To Measure Can Complete vs.How Soon Can CompleteWhere To Run On Your Phone vs.On“Her Phone”On Local vs.On CloudWhat Is Your Scenario Forms System,Reports System Lessons To Take AwayHow To Observe Vision-based vs.Structure-based Attentions Rule:Clear Weeds,Keep Essen

6、ceHow To Act Finger-based vs.API-based(Scripted)Wait And See vs.Muscle MemoryLessons To Take AwayHow To Interact Dont Block Me(Run In Background)Ask For Helps“Smartly”Experience Matters Collect(From Others,From Self-Play)Recall(RAG vs.Fine-