大语言模型的价值对齐：多元性、可转移性和可控性.pdf

1、Value Alignment in LLMs:Pluralism,Transferability and Controllability天津大学自然语言处理实验室The Natural Language Processing Laboratory at Tianjin UniversityDeyi XiongTianjin Uhttps:/dyxiong.github.iohttps:/tjunlp-lab.github.ioYSSNLP2024YSSNLP2024YSSNLP2024 YSSNLP2024YSSNLP2024YSSNLP2024What Human Values are?1

2、.Values are beliefs linked inextricably to affect.2.Values refer to desirable goals that motivate action.3.Values transcend specific actions and situations.4.Values serve as standards or criteria.5.Values are ordered by importance relative to one another.6.The relative importance of multiple values

3、guides action.Shalom H.SchwartzYSSNLP2024YSSNLP2024YSSNLP2024 YSSNLP2024YSSNLP2024YSSNLP2024Value AlignmentYSSNLP2024YSSNLP2024YSSNLP2024 YSSNLP2024YSSNLP2024YSSNLP2024Challenges of Value Alignment?o It is usually difficult to understand and define human values or intentions.o There are many differe

4、nt fine-grained dimensions of human values.Do we need to align the specified objective to all these dimensions?o Human values are usually socially and culturally bound.Do we need to align the specified goal to all different cultures and societies or just parts of them?Given the diversity of cultures

5、 and societies,how can we ensure the fairness of value alignment?o As human values/intentions are usually qualitative while a loss or reward to be optimized has to be measurable and computable,how can we bridge the gap between them?This is known as the goal specification problem.o Tianhao Shen,Renre

6、n Jin,Yufei Huang,Chuang Liu,Weilong Dong,Zishan Guo,Xinwei Wu,Yan Liu,Deyi Xiong.Large Language Model Alignment:A Survey.arXiv:2309.15025YSSNLP2024YSSNLP2024YSSNLP2024 YSSNLP2024YSSNLP2024YSSNLP2024Outer vs.Inner AlignmentYSSNLP2024YSSNLP2024YSSNLP2024 YSSNLP2024YSSNLP2024YSSNLP2024Reinforcement Le