超级智能代理带来灾难性风险：科学家人工智能能否提供更安全的路径.pdf

1、Superintelligent Agents Pose Catastrophic Risks:Can Scientist AI Offer a Safer Path?World Summit AI CanadaApril 16,2025Yoshua BengioWhat happened to me in January 2023We underestimated the acceleration of AI advancesIt would have sounded like science-fiction just a few years earlierFrom rational arg

2、uments to caring for those we loveGoing against my previous beliefs&positions,blinded by my earlier enthusiasm for AINo choice for me:unbearable otherwise.2Benchmark evaluations trends towards AGI3AGI:Artificial General Intelligence Human-level on all cognitive tasksPublicly stated target of DeepMin

3、d,OpenAI and AnthropicEconomic value around 14 trillion$Next step:ASIArtificial Super-IntelligenceSuperior to all humansMain Gaps to AGI4Reasoning:still some incoherences,outstanding progress over past yearPlanning/autonomy/agency:special form of reasoning,worse than humans,but rising exponentially

4、fast(doubling horizon per 7 months)Bodily control/robotics:not necessary to cause major harm(CBRN,persuasion/manipulation,etc),either with malicious goals from humans or from the AI itselfAdvances in abstract reasoningNoteable breakthrough on the Abstract Reasoning Challenge(ARC)5Bengio et al 2025Ex

5、ponential progress on agency 6Extrapolating from this curve human level within 5 yearsFrontier AIs seen trying to escape when told they will be replaced by a new version,copying their weights/code onto the files of the new version,then lying about it78Frontier AI pretending to agree with human train

6、er to avoid changes to its weights that would make it behave against its previous goals later9Frontier AI hacking files containing the game board to cheat,when it knows it would lose against a powerful chess AIAgentic self-preservationShared by all living entitiesResult of evolutionary forcesIn AI,f