劲松-之信Apache Paimon 实时湖仓存储底座.pdf

1、李劲松/Paimon PMC Chair 关于我关于我阿里云开源大数据表存储团队负责人PMC Member of Apache FlinkCommitter of Apache IcebergPMC Chair of Apache PaimonContent实时湖仓PART ONE应用场景PART THREEApache PaimonPART TWO前沿技术PART FOURPARTPART ONEONE 实时湖仓实时湖仓数据架构的时效性演进数据架构的时效性演进ADSADSODSODSDWDDWDDWSDWSETLETLETLETLETLETLQueryQueryQueryQuery批式数仓：

2、批计算+Hive 表+OLAPETL 时效性:天或小时级查询时效性:分钟或秒级BatchBatchBatchBatchBatchBatchADSADSODSODSDWDDWDDWSDWSETLETLETLETLETLETLQueryQueryStreamStreamStreamStreamStreamStream实时数仓：流计算+Kafka+OLAPETL 时效性:秒或分钟级查询时效性:秒或毫秒级数据架构的时效性演进数据架构的时效性演进ADSADSODSODSDWDDWDDWSDWSETLETLETLETLETLETLQueryQueryQueryQuery实时湖仓：流批计算+湖格式+OLAP

3、ETL 时效性:分钟或天级查询时效性:分钟或秒级ADSADSODSODSDWDDWDDWSDWSETLETLETLETLETLETLQueryQueryStreamStreamStreamStreamStreamStream实时数仓：流计算+Kafka+OLAPETL 时效性:秒或分钟级查询时效性:秒或毫秒级BatchBatchStreamStreamBatchBatchStreamStreamBatchBatchStreamStream实时湖仓实时湖仓是批式数仓的原地升级是批式数仓的原地升级时效性 T+1写：覆盖全分区读：扫描全分区时效性分钟级流式更新批式更新流式读取文件过滤解锁完整的大数据

4、全生态流批一体的计算、OLAP数据架构的选择数据架构的选择批式数仓实时湖仓实时数仓PARTPART TWOTWO ApacheApache PaimonPaimonApacheApache IcebergIcebergShared Database Storage for Big DataApacheApache IcebergIceberg：HiveHive 存储的格式升级存储的格式升级对象存储友好ACID transactionsINSERT&UPDATE&DELETETime Travel and rollbackSchema EvolutionTag&BranchApacheApach

5、e PaimonPaimon：IcebergIceberg 的实时化升级的实时化升级流式主键更新变更日志消费ApacheApache PaimonPaimon：湖格式：湖格式 +LSMLSM 写入写入：攒内存 Buffer Append 写。压缩压缩：Minor Compact 减少写放大。清理清理：默认自动清理过期的快照。读取读取：单 bucket 内多路有序归并。The Past,Present and Future of PaimonThe Past,Present and Future of Paimon2022.012023.032023.122024.032024.10Initia

6、lized in Apache FlinkSub Project of Apache FlinkOriginal name-Flink Table StoreBecame Apache Incubating ProjectRename to Apache PaimonOpen to others rather than FlinkGA Version ReleaseProduction ReadyFor Both Streaming&BatchBecame Apache Top-Level-Project Graduated from Apache Software FoundationInt