基于对象存储构建实时云原生数仓-张雁飞.pdf

1、DatabendA modern warehouse with Rust for your massive-scale analyticshttps:/ and Data WarehouseOLTPHigh concurrencyLow latencyRow-orientedResource isolation?Duplicate data?Row+Column-orientedComplex queryQuery speedColumn-orientedMySQL etc.ClickHouse etc.HTAPOLAPTiDB etc.大数据分析遇到了什么”新”问题？传统数仓为什么无法解决这

2、些“新”问题？新一代实时弹性数仓如何设计？使用 Rust 从 0 到 1 研发一款数仓是种什么体验？大纲大纲Bohu TANG(张雁飞)Co-Creator of Databend:https:/ and MySQL(TokuDB)重度贡献者Database Kernel|Distributed Database|Data Warehousehttps:/bohutang.me/当今当今（2023）大数据新问题大数据新问题 01 全球数据指数级增长全球数据指数级增长1024PB=1EB,1024EB=1ZB 大数据量下的资源利用率问题,IO-bound-CPU-bound如何让系统更加智能，根

3、据查询模式自动创建索引？越跑越快.如何面向 Warehouse+Datalake 需求设计？挑战挑战 SQL 生成执行计划 执行计划包含多个算子(Operator)算子需要被执行:Pull-Based 或 Push-Based 为什么不适合对象存储？Cloud-Native Execution 设计设计 执行计划被编排成 Pipelines Pull-Based+Push-Based 双模式 负载感知(Workload-Aware)，运行时动态扩展Cloud-Native Execution 设计设计 Metadata 缓存 Index 缓存 Data 缓存 缓存介质：Memory/Local

4、 Disk/Redis 等Cloud-Native Caching 设计设计 ParserSQL Query Optimizer Executor Storage Catalog数仓单体功能模块数仓单体功能模块Meta Services(User/Schema)Metadata Security TransactionCompute Services(Query Engine+Table Format)Executor Executor ExecutorStorage Services(AWS S3)Data Data DataSQL Query数仓模块微服务化数仓模块微服务化 Databen

5、d 架构架构 Databend 架构架构 Databend 架构架构 Databend 架构架构 Cluster Key(age)Age=20Age=35Age=80min:5max:100min:15max:40min:20max:60file2file3f i l e 1Automatic Tuning Cluster Key(age)Age=20Depth:2Age=35Depth:3Age=80Depth:1min:5max:100min:15max:40min:20max:60file2file3f i l e 1select from table where age=20 Auto

6、matic Tuning Cluster Key(age)Age=20Depth:2Age=35Depth:3Age=80Depth:1min:5max:100min:15max:40min:20max:60file2file3f i l e 1select from table where age=80 Automatic Tuning Cluster Key(age)Age=20Depth:2Age=35Depth:3Age=80Depth:1min:5max:100min:15max:40min:20max: