8-任庆盛-高谟-Flink Connector 的架构解析和最佳开发实践-已美化TY-定稿_compressed.pdf

1、Flink Connector Flink Connector 的架构解析和最佳开发实践的架构解析和最佳开发实践 Architecture and Development Practice of Flink Connectors 任庆盛 Qingsheng Ren 阿里巴巴 Alibaba Group 高赟 Yun Gao 阿里巴巴 Alibaba Group ConnectorConnector Testing FrameworkFramework # #2 2 SourceAPIAPI # #1 1 Sink Sink API # #3 3 SourceSourceAPI#1 开发者需要自

2、己处理复杂的线程模 型，开发和测试门槛高 Source 的任务发现、分工和读取 数据的逻辑杂糅在一起，导致如 Kafka、Kinesis 等 Source 的实现 非常复杂 与 Flink Runtime 的过度耦合，例 如 Checkpoint 锁、对算子无锁 Actor/Mailbox 线程模型的影响等 #2#2#1#1#3#3 动机 Motivation Developers have to deal with complex threading model, which adds a high bar to developing and testing Logic for work d

3、iscovery and reading the data is intermingled, leading to complex implementations like the Kafka and Kinesis source Over-coupling with Flink runtime, such as checkpoint lock, effect on Actor/Mailbox threading model FLIP-27 SourceSourceAPI Source Split Enumerator Source Reader Task Task Task Source Reader Source Reader Source Reader Split Enumerator JobMaster 分配 Splits、发送全局水位 Assign splits, send gl