梁一韬_CraftJarvis-rlchina_watermark.pdf

1、Generalist Agent in an Open WorldTeam CraftJarviscraftjarvis.orgWhy Open WorldsMCU:A TASK-CENTRIC FRAMEWORK FOR OPEN-ENDED AGENT EVALUATION IN MINECRAFT MCUMCU5Minecraft:generalist AI in an open worldTodays embodied AIRestrictive objectivesVery few tasksLimited knowledgeEmbodied AI in an open worldO

2、pen-ended objectivesMassively multitaskWeb-scale knowledge6Challenges in open world environmentsChallenge#1:State distinguishabilityThe intra-task state distribution is highly diverse,while the inter-task state distribution is similar.=Similar inter-task state makes policy hard to learn goal-aware b

3、ehaviors.7Challenges in open world environmentsSetting#1:(red blob)Kill sheep in snowy plains,chop tree in plains.(to mimic the state distribution in Meta-world)Setting#2:(blue blob)Kill sheep and chop tree both in plains.The average success rate of setting#1 is significantly higher than that of set

4、ting#2.=Similar inter-task state makes policy hard to learn goal-aware behaviors.8Challenges in open world environmentsChallenge#2:long-horizon planningOpen worlds have highly abundant object types with complex dependency and relation.As a result,ground-truth plans typically involve a long sequence

5、of sub-goals with strict dependencies.=Planning Success Rate will drops significantly on long-horizon tasks.10Challenges in open world environmentsLanguage Models as Zero-Shot Planners:Extracting Actionable Knowledge for Embodied AgentsDo As I Can,Not As I Say:Grounding Language in Robotic Affordanc

6、e LLM for planning in closed worlds11A Planner+Controller Framework1213chop_treemine_stonehunt_sheepparallel goalsstate?14Initial plan(by Planner).craft(furnace:1,cobblestone:8,crafting_table);.Failure description(by Descriptor)and explanation(by Explainer)Now I locates in Forest