Apache Paimon：面向数据和多模态人工智能的统一湖存储兼容 Apache Iceberg.pdf

1、Data LakeFile FormatsReal-Time Lake Format200820162017202220232025The upgrade table format of Hive,high-performance format for huge analytic tablesIncubated Flink-Table-StoreStreaming+Lake FormatApache Paimon(Flink-Table-Store)Real-Time Lake FormatApache Paimon-1.0-1.2 ReleaseTable Format for Increm

2、ental updates with Apache SparkHive Metastore+Hive TablesBuilt on HDFS for Data lake2013File Format:EfficientCompression and scanningApache Paimon Real-Time Lake Format-Timeline of Industry DevelopmentLake FormatsApache FlinkApache HudiApache ORCApache ParquetApache HiveApache Paimon Accelerating th

3、e Data Analytics on Lakehouse ArchitectureData LakeApache Paimon(Real-Time Lake Format compatible with Iceberg)BronzeSilverGoldenStreaming&BatchEnd-to-End Minutes LatencyStreaming&BatchStreaming LakehouseReal-Time IngestionReal-Time UpdateOLAPOnline QueryExternal Data SourcesDatabaseMessage QueueLSM

4、+ParquetApache Paimon Key Designs for Streaming Updates Data is appended as a file and written to Level 0 of LSMAsynchronous Minor Compact Balanced Write and ReadBenchmarkKey DesignsLog Structured Merge TreeReal-TimeUpdate&ChangelogCDC Ingestion Schema Evolution Real-Time Update Benchmark for Paimon

5、,Hudi and Iceberg Using Serverless Flink in Alibaba Cloud,Lower is Better020040060080010001200PaimonHudiIcebergServerless Flink Streaming Update:TPC-H Update Time Cost1.0 X2.5 X3.75 XL0L1L2Minor/MajorAsync CompactionLSM in BucketData FileUsing Paimon CDC to Sync CDC from KafkaUsing Flink CDC to Sync

6、 Database DirectlyApache Paimon Ingestion with Schema Evolution Unified snapshot reading and incremental readingNested Schema EvolutionDatabaseODSFlink CDCDatabaseODSPaimon CDCCDCSnapshotFilesInitalNear Real-Time QueryApache Paimons flourishing development in the industryTaobao&Tmall(Alibaba Group)1