使用 Apache Flink + Paimon + Doris 构建实时 Lakehouse.pdf

1、Apache Flink+DorisReal-Time Lakehouse SolutionChristine XuTech VP VeloDB01IntroductionWhat is Apache Doris02Lakehouse SolutionApache Flink+Paimon+Doris03Community Introduction What is Apache Doris01What is Apache DorisReal-Time AnalyticsLakeHouse AnalyticsObservabilityData warehousingLightning FastL

2、eading performance on benchmark like ClickBench,TPC-H,TPC-DSEasy to UseMulti-ScenarioFriendly for first-time userwith low operational costOne Platform,All AnalyticsVectorized query enginePipeline execution modelCost based optimizerColumnar StorageSecondary indexesMaterialized viewSmart CachingBare m

3、etal,Cloud VMs,K8s supportReplica balancing and Self-healingMySQL protocol compatibilityStandard ANSI SQLBI reports and dashboardsAD-hoc queriesMachine learning Production analytics productsWhy Choose Apache DorisLakehouse Solution Apache Flink+Paimon+Doris02Data Warehouse&Data LakeData Warehouse St

4、ructured data Proprietary storage system Schema-on-Write SQL query interfaces Reporting&Analytics High performance Robust data management High data qualityData Lake Structured,semi structured data Open file format Shared storage：HDFS,Object Storage Schema-less API,NoSQL Data analysis,data mining,mac

5、hine learning,etc.Open ecosystem Flexibility Low storage cost High storage cost Limited flexibility Vendor lock-in Poor performance High data management cost Data quality issuesLambda&KappaLambda Separate Batch&Stream Ideal Use Cases：oRealtime-insight and accurate historical data for analysis or cor

6、rection Trade-offs to ConsideroTwo Codebases to MaintainoChallenging to Ensure Data ConsistencyoWasted Resources from Duplicated ComputationKappa Only Stream Ideal Use Cases:oBest for Real-Time Performance with Replayable Streams Trade-offs to ConsideroSacrifices historical data access capabilitieso