从数据流到可执行洞察：Grab 利用 Flink 进行实时分析和数据质量提升的历程.pdf

1、From Data Streams to Actionable InsightsGrabs Journey with Flink in Real-Time Analytics and Data QualityCalvin TranSenior Software EngineerYuanzhe LiuSenior Software Engineer-Get to know us-Flink at Grab-Real-Time Analytics and Data Quality-Q&AAgendaGet to know usYuanzhe LiuCalvin TranStreaming Appl

2、ications&InfrastructureSenior Software Engineer,Data ServicesStreaming Applications&InfrastructureSenior Software Engineer600+Flink at Grab 200+Flink Pipelines Kafka StreamsFlink at Grab-FlinkSQL PlatformUI/UXSelf-servedSQLFlink at Grab-FlinkSQL PlatformFlink at Grab-FlinkSQL PlatformReal-Time Analy

3、tics and Data QualityMerchant ReportingA business use case in Grab to help merchants tracking their advertisements campaign Case01Merchant Reporting BackgroundMerchant Report on the App-Metrics for different advertisements campaigns-These metrics are generated by ETL jobs with a delay of 4 hoursMerc

4、hant Reporting BackgroundList of all the campaigns for a merchant.Metrics like AdSpend,Impressions,Clicks,CTR.Merchant Reporting BackgroundSingle Campaign Reporting-Helps advertisers make better decisions for their ads-ETL pipeline so delay 3 hrsMerchant Reporting ChallengesDelayed Metrics:-Campaign

5、 performance reports delayed by 3-4 hours,causing confusion for merchant users.Incident Handling:-ETL backfills take 5-6 hours for 1 day,and large-scale backfills can take up to a week.Realtime Merchant ReportingFast Reporting Layer-Real-time metrics via Flink pipelines for instant campaign performa

6、nce visibility.Reduce the the delay from 3-4 hours to secondsData Reconciliation Layer-Daily corrections using golden tables to ensure accuracy.Topic ATopic BTopic CReport AReport BReport CFiltered TopicFlinkSQL PipelineFlinkSQL PipelineFlinkSQLReal-Time Data Quality MonitoringAn internal data platf