王峰：阿里新一代实时计算引擎-Blink技术解密（22页）.pdf

1、阿里新一代实时计算引擎Blink技术解密王峰资深技术专家姓名：王峰花名：莫问2006.4 硕士毕业于北航计算机系2006.4 目前 阿里巴巴集团资深技术专家负责实时计算北京团队3.Alibaba Blink1.流式计算架构2.Apache Flink4.Blink应用案例3.Alibaba Blink1.流式计算架构2.Apache Flink4.Blink应用案例流式计算模型UserCodeUserCodeUserCode对于不间断的数据流持续执行计算ProducerUsercodestate counters:sum,max,min,average,etc.windows:time/cou

2、nt window machine learning:model parameters有状态的流式计算什么是有状态的计算?需要处理跨多条记录信息的计算状态一致性exactly onceat least once状态管理checkpoint,recoveryupgrade,versionSliding window(no overlap)Tumbling window(with overlap)Session window(punctuated by a gap of inactivity)Time-driven(examples:every 30 seconds)流式计算中灵活的Window机

3、制Data-driven(examples:every 1000 records)Event-Time(Created Time)流式计算中乱序数据处理Core Feature is WatermarkOut-of-Order Records3.Alibaba Blink1.流式计算架构2.Apache Flink4.Blink应用案例LatencyVolume/ThroughputState&AccuracyExactly-once semanticsEvent time processing10s of millions events/secfor stateful application

4、sLatency down to the millisecondshttp:/flink.apache.orgFlink as a“true”Streaming Computing Engine完备的Streaming语义真正的流式执行引擎，高吞吐，低延迟提供Exactly-Once的一致性保证灵活丰富的window支持支持Out-of-Order乱序数据处理部署在多种调度系统StandaloneYARN,Mesos丰富的状态存储支持Java Heap+HDFSRocksdb+HDFSSQLTable APIDataStream/DataSet APIStateful Streaming Pr

5、ocessingLow-level Building Block(streams,state,event time)Core APIsDeclarative DSLHigh-level LanguageProgramming ModelFlink Checkpoints&Recovery(Chandy-Lamport)state(embedded key-value store)persistent storage(HDFS)sourcesnapshotoperatorSinkoperatorFlink Streaming Applicationoffset3.Alibaba Blink1.流

6、式计算架构2.Apache Flink4.Blink应用案例Blink在阿里的成长路线调研Flink架构最好的流式计算引擎尚未在大规模生产中验证启动Blink完善Flink架构和稳定性将Flink在阿里生产上线Blink双11上线搜索推荐全实时链路实时商品、报表、在线学习Blink统一阿里实时流计算统一服务阿里集团实时计算业务基于Blink的StreamCompute-2.02015.102015.82016.112017.5Blink拥抱Apache Flink开源社区Blink全面兼容Apache Flink