方勇-Flink OLAP作业调度和查询执行的改进.pdf

1、方勇/字节跳动基础架构工程师Improvements of Job Scheduler and Improvements of Job Scheduler and Query Execution on Flink OLAPQuery Execution on Flink OLAP背景背景BackgroundBackground问题和分析问题和分析Problems Problems&A&Analysisnalysis调度执行优化调度执行优化Scheduler&ExecutionScheduler&ExecutionOptimizationOptimization未来计划未来计划FutureFut

2、ure#1#1#2#2#3#3#4#4#1#1背景背景BackgroundBackgroundHTAP Architecture In ByteDanceHTAP Architecture In ByteDanceMySQLProxyAP EngineCatalogConnectorMetaServiceHTAP StoreOther HTAPComponetsSubmit QueryFetch ResultSubmit JobFetch ResultScanFetchFlink SQLGatewayFlink ClusterWhy Can Flink For OLAPWhy Can Flin

3、k For OLAPPrestoPresto ClusterClusterFlink OLAP Flink OLAP ClusterClusterFlink Session Flink Session ClusterClusterTaskManagerFlink sql gatewayFlink sql gatewayFlink sql gatewayFlink sql gatewayConnectorSchedulerPolicyVSVSResourceManagerJobMasterCatalogTaskManagerDeploymentFailoverFlinkFlinkSessionS

4、essionClusterClusterWorkerParserOptimizerParserOptimizerParserCoordinatorCoordinatorOptimizerJobManagerSchedulerPolicyMetadataManagerResourceManagerWorkerConnectorDeploymentFailoverMPPMPPJob PlanJob PlanRuntimeRuntimeManagerManagerExecutionExecutionManagerManagerFailoverFailoverManagerManagerJobMana

5、gerJobManagerWhy Choose Flink In HTAPWhy Choose Flink In HTAPFlink OLAP Flink OLAP ClusterClusterFlink Session Flink Session ClusterClusterTaskManagerFlinkFlink SQL gatewaySQL gatewayFlinkFlink SQL gatewaySQL gatewayConnectorSchedulerPolicyResourceManagerJobMasterCatalogTaskManagerDeploymentFailover

6、FlinkFlinkSessionSessionClusterClusterParserOptimizerParserOptimizerJobManagerJobManager引擎统一Streaming Batch OLAP生态支持跨数据源查询性能优势TPC-DS 基准测试#2 2问题和分析问题和分析Problems&AnalysisProblems&AnalysisRest 协议，SQL 查询特点问题要求模式架构Session On K8s，MPPBatch+Eager Scheduler秒级和毫秒级小作业作业频繁启停，资源碎片Latency+QPS 要求现状和问题现状和问题接入How?Ho