scaling-kubernetes-best-practices-for-managing-large-scale-batch-jobs-with-spark-and-argo-workflow-kubernetesdaepfa-lia-mao-lia-zha-huan-sparkreargo-yu-zhuang-liu-jiaxu-alibaba-cloud.pdf

1、Scaling Kubernetes:Best Practices for Managing Large-Scale Batch Jobs with Spark and Argo WorkflowYu Zhuang Alibaba CloudJiaxu Liu Alibaba CloudAbout UsAlibaba Cloud Container Kubernetes Service(ACK)Yu Zhuang:9 years on KubernetesACK ArchitectFocus on offline batch job and multi-clusterJiaxu Liu:7 y

2、ears on KubernetesACK Senior EngineerFocus on observability and large-scale cluster management Agenda1.Why running offline batch job on Kubernetes2.Spark on Kubernetes3.Argo Workflow on Kubernetes4.Best practices of large scale offline batch job on KubernetesCloud Native OS-KubernetesStateless Appli

3、cation,Stateful Application,Batch Jobs(Data Process,AI training/Inference,Science Computing)Why run batch jobs on Kubernetes Single infrastructure:manages both online and offline workloads.Scalability:scales your offline jobs horizontally.Cost Optimization:shares resource in on premises and run on s

4、erverless in cloud.Multi-Tenancy:isolates with namespace,resource quota,and kqueue.Portability:allows to migrate offline jobs in different cloud providers.Ecosystem:provides operation capabilities,e.g.monitoring,logging,and security.SparkApache Spark is a unified analytics engine for large-scale dat

5、a processing.It provides high-level APIs in Java,Scala,Python and R,and an optimized engine that supports general execution graphs.It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing,pandas API on Spark for pandas workloads,MLlib for machine l

6、earning,GraphX for graph processing,and Structured Streaming for incremental computation and stream processing.Spark CoreSpark SQLStandaloneHadoop YARNKubernetesMLlibStreamingGraphXSpark on Kubernetes./bin/spark-submit-master k8s:/https:/:-deploy-mode cluster-name spark-pi-class org.apache.spark.exa