利用 StarRocks 实现 Lakehouse 的实时分析：从数据摄取到洞察高效处理数据.pdf

1、Real-Time Lakehouse Analytics with StarRocks:Powering Efficient Data Processing from Ingestion to InsightZhen Fan|Staff Engineer|Alibaba Could Yan Zhang|Staff Engineer|CelerData01Near-real-timeStreaming Lakehouse02Real-time Streaming Lakehouse03A Real-WorldUse Case04StarRocksOverview05StarRocks onPa

2、imon TableNear-real-time Streaming Lakehouse01Lambda Architecturebatch pipelinestreamingpipeline Maintain two sets of code Not support transactions cannot easily do small updates insert overwrite:rewrite whole partitions Scaling up is tricky:all metadata in HMS bottleneck as your data grows Query Pe

3、rformance isnt great HMS lacks file-level metadata and indexes metadata is pretty simple and coarse-grained Bad query performance especially worse for point queriesHadoop EcosystemFlink Ecosystem Mayjor pain points:Evolution to Lakehouse Single source of truth avoid data silos:ingest data into cloud

4、 storage everyone actually agrees on this version of data unified BI and AI storage and metadata Great support for transactions:handle ACID transactions well and make fine-grained updates without overwriting the whole partitions Scales smoothly Lakehouse format is stored in cloud storage find files

5、from metadata without“list”operation Powerful query features MVCC,time travel,schema evolution etc.file-level stats:compute engines can leverage data skipping(predicates pushdown,columns pruning)Mayjor Advantages:Data Lakehouse Lakehouse Format ComparisonFeatureIcebergDelta LakeHudiOriginal PurposeE

6、nhanced Hive replacementSpark-native batch&stream processingIncremental updates/upsertsCore StrengthDe Facto Standard+Hive replacement+MergeInto capabilityInside Databricks ecosystemIncremental processingInteroperabilityHigh(Works with widely adopted compute engines)Limited(Spark/Databricks focused)