2019年Flink流批一体的技术架构以及在阿里的实践.pdf

1、Flink 流批一体的技术架构以及在阿里的实践高级技术专家目录目录 需求和挑战 Flink 架构简介 流批一体的入口 SQL 流批一体的大规模实践 在线机器学习平台 总结和展望主流架构对比主流架构对比LambdaKappa 执行引擎同时具备多种能力1.低延迟的流计算2.高吞吐、高稳定性的批处理 用户角度：编程接口统一。一份代码，一样的结果 开发人员角度：架构统一，代码复用流批一体系统的需求和挑战流批一体系统的需求和挑战Flink 架构简介架构简介StorageHDFS,HBase,KafkaLocalSingle JVMClusterStandalone,YARN,K8S,DeployDist

2、ributed Streaming DataflowRuntimeDataStream APIAPIDataSet APIWord Countval lines:DataStreamString=env.readFromQueue(address)val words:DataStreamWord=lines.flatmap(line)=split(line)val counts:DataStreamInt=words .keyBy(“word”).sum(“frequency”)counts.addSink(new RollingSink(path)计算模型的核心抽象表达作业逻辑的DAG主要由

3、点和边构成1.点：算子（operator），包含主要的计算逻辑2.边：数据流通管道，可以运行在多种介质上（网络、文件、内存）Streaming Dataflowsource1/2source2/2flatmap1/2flatmap2/21/2aggregate2/2aggregatesink1/1Word Count：批处理是流计算的特例批处理是流计算的特例opopUnbounded StreamUnbounded StreamopopopopopopUnbounded StreamUnbounded StreamPipelinePipelineStream JobStream Jobopop

4、opopopopopopPipeline/PersistPipeline/PersistBounded StreamBounded StreamBounded StreamBounded StreamBatch JobBatch Job流批统一的入口流批统一的入口-SQL-|USER_SCORES|-|User|Score|Time|-|Julie|7|12:01|Frank|3|12:03|Julie|1|12:03|Frank|2|12:06|Julie|4|12:07|-USER_SCORES is a source table/stream.Batch Mode:12:07 SELEC

5、T Name,SUM(Score),MAX(Time)FROM USER_SCORES GROUP BY Name;-|Name|Score|Time|-|Julie|12|12:07|Frank|5|12:06|-Stream Mode:12:01 SELECT Name,SUM(Score),MAX(Time)FROM USER_SCORES GROUP BY Name;-12:04,now)|-|Name|Score|Time|-|Julie|12|12:07|Frank|5|12:06|-|-12:01,12:04)|-|Name|Score|Time|-|Julie|8|12:03|

6、Frank|3|12:03|-|-|-inf,12:01)|-|Name|Score|Time|-|-|-1.流有 Early fire2.最终结果一致流批统一的架构流批统一的架构LocalSingle JVMCloudGCE,EC2ClusterStandalone,YARNRuntimeDistributed Streaming DataflowDataStream APIStream ProcessingDataSet APIBatch ProcessingTable API&SQLRelationalRuntimeDataStream APIStream ProcessingDataS