基于 Flink + Hologres 为 Lazada 构建高效的产品选择平台.pdf

1、An Efficient Product Selection Platform for LazadaBased on Flink+HologresRichard Chen|Head of Supply Tech,Lazada01Background02Solution03Benefits04FutureBackground01AnalyticsPool SchedulingPool ManagementPoolMergingData Validation/ExportSelection Real-Time AnalysisReport ManagementJoint AnalysisProdu

2、ct AnalysisDataset MergingData QueryData TransformationDatasetManagementData ServicesWhat is Product Selection in E-commerce?Scattered Data SilosData from different applications is isolated,hindering efficiencySlow Full SyncSearch engine based indexing wasslow,hard to make T+1DBusinessSideTechSideCo

3、nclusion:A comprehensive solution is needed to excel and balance Data Volumes,Efficiency,and Cost.Poor Data FreshnessUpdates rely on T+1 offline methods,impacting decision qualityLow Data RichnessLimited storage and computing powerprevents more metrics and tagsInsufficient Write CapacityDifficult to

4、 write incremental data with high TPSHigh Cost in Machine Resource and MaintenanceLimited scalability in data rows and columns due to uneconomical costKey ChallengesBatch/Stream IntegrationLarge size while time-sensitiveHeterogeneous Data SourcesMultiple data sources,including MaxCompute,MySQL,MQ,et

5、cDiversifiedFormatsDifficult to consolidate and unify dynamic business data with various formats Technical OpportunitiesSolution02Runtime(Containers/Resource Scheduling/Task Management)Stream-ProcessingProduct Selection AbilityHologresMaxComputeMQ(Kafka/MetaQ/etc.)DB（Mysql/Hologres/etc.）Data InputBa

6、tch-ProcessingStream-Batch IntegrationProduct Analysis AbilityMaxComputeMQ(Kafka/MetaQ/etc.)DB（Mysql/Hologres/etc.）Data OutputHigh Level Architecture70%Offline data(T+1D)30%Incremental Update70%of the metrics come from the offline data warehouse50%Batch Incremental Update(T+1H)30%of the data support