2王金海-3-.pptx趣头条基于FlinkClickHouse构建实时数据平台__compressed.pdf

1、 ? ? ? ? About me 01/ 02/ ? ? ? ? ? ? Content ? Business scenario and current situation analysis ? Flink-to-Hive ? Flink-to-ClickHouse ? Future ? 01 Business scenario and current situation Analysis ? Business scenario and current situation Analysis ? Business scenario and current situation Analysis

2、? 02 Flink-to-Hive ? Hour level architecture DatabaseDatabaseKafkaKafkaFlinkFlinkHdfsHdfs alter table . add alter table . add partitionpartition . . locationlocation . HiveHive SchedulerScheduler ? StreamingFileSink API ? ? ? ? ? Exactly-Once ParticipantParticipantParticipantParticipant CoordinatorC

3、oordinator 2PC 2PC ackack commitcommit SourceSourceWindowWindowSinkSink Job Job ManagerManager StateState checkpoint barriercheckpoint barrier snapshotStatesnapshotState notifyCheckpointnotifyCheckpoint CompleteComplete StateStateStateState barrierbarrierbarrierbarrier prepareprepare StreamingFileSink implements CheckpointedFunction, CheckpointListener commitPendingFile restoreInProgressFile trunc