康琪KangQi - 十荟团基于Flink SQL与Zeppelin构建实时数仓的实践 .pdf

1、康琪/十荟团大数据工程师,Apache Flink Contributor 十荟团基于Flink SQL与Zeppelin构建实时数仓的实践Flink SQL&Zeppelin-Based Practices of Real-Time Data Warehouse at NiceTuan实时数仓建设Construction of Real-Time Data Warehouse#1数仓平台化Data Warehouse Platformization#2Flink SQL 增强Enhancements of Flink SQL#3未来规划Future Planning#4#1实时数仓建设Con

2、struction of Real-Time Data Warehouse演进历程Progress of Evolution20182019202020212022Spark StreamingSimple,sporadic jobsInsular(Silo)Flink 1.8 1.9(DataStream API)Starting from dashboardsStill insularFlink 1.9&1.11(Moving to SQL)More real-time requirementsData integration WarehousingFlink 1.13(Flink SQL

3、&CEP)Platforming(Zeppelin)Standardization当前规模Current Scale60+Streaming Jobs800+Virtual Cores100+BusinessIndicators300 mil.Ingestion per DaySubject-OrientedIntegratedTime-VariantNon-Volatile指导思想IdeologyBusiness fields within enterpriseConsistency across systemsOver a long-term horizonDecision-Making

4、SupportStable storage&modelReal-time circumstances?分层设计Layered DesignBusiness log/Tracked events/DB binlog ODSBehavior/Order/Refund/Complaint/Tickets DWDP-UV/CTR/Sales/Promotion/Compensation/Bonus DWSDashboard/Heat map/Risk control/Real-Time OLAP APPUser/Partner/Groupon/Merchandise/Location DIM架构与数据

5、链路Architecture&Data LinkageDBLogsDIMODSDWDDWDDWSDWSAPPSQL大一统“One SQL to Rule Them All”Flink Table/SQLCatalogStream ProcessingBatch ProcessingOLTP QueriesOLAP Queries实时OLAP支持Real-Time OLAP SupportLoginBrowseSearchAdd to cartPlace order50k records/batch,1min timeoutPartitioned by day,TTL 2 yearsMateri

6、alized Views(Stable incremental aggr.)Funnel(windowFunnel/xFunnel)123RetentionPath analysisAd-hoc queries数据校验Data VerificationSemi-automatic,3-way checkingAvoid frequent replays in Kafka(page cache pollution)Redundant storage?SQLSQLIndicatorSQL#2数仓平台化Data Warehouse Platformization统一化与挑战Unification&C