3、付海涛-Flink on K8s 在京东的持续优化实践.pdf

1、付海涛/资深技术专家FlinkFlink on K8s on K8s 在京东的持续优化实践在京东的持续优化实践The Practice and Optimization of Flink on K8s in JD 基本介绍基本介绍The IntroductionThe Introduction生产实践生产实践The Practice The Practice#1#2#3#4优化改进优化改进The Optimizations The Optimizations 未来规划未来规划FutureFuture PlanPlan#1#1基本介绍基本介绍The IntroductionWhy Kuberne

2、tes易部署运维easy to setup资源利用better resource utilization隔离安全better resource isolation，security容器化历程The Evolution of Containerization2018年6月20%任务容器化20%jobs running on k8s till Jun.20182018201820192019202120212019年2月计算单元全部容器化All computing units running on k8sbefore Feb.20192021年持续优化实践Continuous optimizati

3、on&practice资源使用Efficient resource sharing混合部署服务，资源共享能力提升节省机器资源30%DevOps效率DevOps efficiency improvement开发、测试、生产一致环境部署和运维自动化能力提升管理和运维成本降低50%业务稳定Full isolation&resilience资源隔离，细粒度权限控制弹性自愈，保障业务稳定Flink on K8s in JD配置调试部署监控日志SQLJARJRC（京东实时计算平台）JDOSJDQKafkaHDFS/OSSMySQLJimDBSourceJDQKafkaHDFS/OSSMySQLESSink

4、物理机+云主机#2 2生产实践生产实践The Practice容器化方案（静态模式）The Standalone K8s K8s客户端JRC平台K8s Master-api server-controller-schedulerK8s DeploymentJobManagerK8s PodK8s DeploymentTaskManagerK8s PodDocker RegistryZK集群Hdfs或oss创建集群提交任务jobmanager-deployment.yamltaskmanager-deployment.yamlHA状态存储方案局限：Limitation资源需要提前分配，无法满足灵

5、活多变的业务需要The resource cannot be allocated based on the resource requirements of the job.容器化方案(弹性模式）The Native K8s在平台进行资源创建&销毁Allocate&free resource by platform支持预分配Support preallocation to be compatible with static mode.兼容原有槽位分配策略compatible with the task slot distributed strategy in standalone k8s mo

6、de.K8s客户端JRC平台K8s Master-api server-controller-schedulerDocker RegistryZK集群HAHdfs或oss状态存储K8s DeploymentFlinkFlink J JobmanagerobmanagerJobMasterDispatcherJDResMngrRest ServerFlink TaskmanagerK8s PodFlink TaskmanagerK8s Pod日志&监控Logs and MetricsNodePodJM or TMMetric Reporter白泽系统JVM指标Flink指标log agentme