2-5 Presto+Alluxio 加速 Iceberg 数据湖访问.pdf

1、Presto+Alluxio加速加速Iceberg数据湖数据湖访问访问BeinanSep 11,2022AgendaPresto&Alluxio01Presto overview and Presto+Alluxio overviewAlluxio&Iceberg02Alluxio and Iceberg ArchitectureBest Practices03Data Consistency and PrivacyFuture Work04Future work of the open-source community2Presto&Alluxio013Presto OverviewDist

2、ributed SQL Query EngineANSI SQL on Hive data warehouse,Hudi,Iceberg,Kafka,Druid and etc.Designed to be interactiveAccess to petabytes of dataOpen- CasesAd-hocBI toolsDashboardA/B testingETL4Prestos HistorySource:varada5Presto Architecture6Presto+Alluxio OverviewWhy Presto+Alluxio?7Unified Namespace

3、Alluxio is the only data sourceNo code change to applicationsModernize Data PlatformHybrid-and multi-cloud data lakeData movementData Sharing Between ComputeBetween compute enginesAcross entire data pipeline*Performance*Accelerate queries(no guarantee,maybe yes in some queries,especially when co-loc

4、ated)Alluxio speeds up the I/OHow Presto Works with AlluxioPrestoHiveMetastorelocation=s3:/bucket/tableRead/WriteMetadataRead/WriteDataPrestoAlluxioMounted to AlluxioHiveMetastorelocation=alluxio:/tableRead/WriteMetadataRead/WriteData8Presto+Alluxio ArchitectureCo-located deployment9PrestoCoordinato

5、rAlluxioMasterPresto WorkerAlluxio WorkerPresto WorkerAlluxio WorkerPresto+Alluxio ArchitectureDisaggregated deployment10AlluxioServerAlluxioServerBig Data QueryBig Data ETLModel TrainingAlluxioServerAB/path1/file1/path2/file2CABCAAlluxio&Iceberg021112Apache IcebergAn open table format for huge anal

6、ytic datasetsSchema evolutionHidden partitioningPartition layout evolutionTime travelVersion rollback13Data LayersTransitioning the Lake from Files to Tables14Schema EvolutionAdd-add a new columnDrop-remove an existing columnRename-rename an existing columnUpdate-widen the type of a column,struct fi