a-story-of-managing-kubernetes-watch-events-end-to-end-flow-in-extremely-large-clusters-nanomao-zhong-shi-fa-lia-kubernetes-watchguo-guo-xiao-zha-bo-tang-ant-group.pdf

1、A Story of Managing K8s Watch Events End-to-end Flow in Extremely Large ClustersBo Tang,Ant GroupAug,2024OutlineThe Watch Mechanism and The Importance of Watch in KubernetesDefinition of Watch SLO and What It Brings to UsWhat We Did to Optimize Watch Flow and the Benefits We GotSummary and Future Pl

2、ansWatchMechanismWatchSLOOptimizationSummaryK8s OverviewKubernetes is getting boring?Let us investigate K8s moreYou will get what you wantStable SkeletonComprehensive extensibilityDesign PhilosophyAfter KubernetesBefore KubernetesK8s Birds-eyes ViewSchedulerKubeletControllersWhen Cluster Gets LargeK

3、CMSchedulerKubeletCustom ControllersKube-proxyWhen Cluster Gets LargeProblems arise when cluster gets large:Large number of nodes and pods.Large number of CRDsHeavy traffichigh churn rate of resources.The provisioning path is longDifferent usages of different controllersUsers requirements diverse/va

4、ry a lotAlmost every problem is connected to Watch mechanism.Weve done optimization for each component of Kubernetes system,but the link/connection part seems missing.Watch Procedure1.Apiserver obtains data from etcd,decodes it,formats it,and sends it to apiservers internal cache.2.Based on the watc

5、hed data,apiserver builds its cache internally,which is called the WatchCache object.It includes a full set of data and a circular array.*The full set of data is a set of data that includes indexing and is used for various client list requests,list-by-label requests,and list-by-field requests.*The c

6、ircular array contains the latest watchCache event and is used for various client watch requests.3.The apiserver internally iterates through all of the cacheWatchers for a resource and sends each cacheWatcher the change events it is interested in one by one.4.When the client receives data from apise