使用增量处理构建指标存储.pdf

1、2024 Databricks Inc.All rights reserved1BUILDING BUILDING METRICS METRICS STORE WITH STORE WITH INCREMENTAL INCREMENTAL PROCESSINGPROCESSINGHang LiHang LiJune 2024June 20242024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reservedInstacart Ads Measurement TeamInstacart Ads Measur

2、ement Team2About UsAbout UsInstacart Ads2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved Challenges of Building Business Metrics Importance of Metrics Stores The Power of Incremental Processing Testing and Monitoring Case Study Q&A3AgendaAgenda2024 Databricks Inc.All ri

3、ghts reserved2024 Databricks Inc.All rights reservedInconsistency in the metrics definitionInconsistent metrics derived from different sourcesError introduced during clone and editInconsistent application and enforcement of policies,such as PII,financial controls and cost-effectiveness.Batch process

4、ing with static lookback windows doesnt scale well to increasing data volumesRedundant reprocessing leads to a waste of time and computational resourcesSlow processing delays the availability of insights for decision makingGaps in reviewInsufficient testingNo unit test during development phaseConsis

5、tencyScalabilityReliabilityConsistencyScalabilityReliability4ChallengesChallenges42024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reservedInconsistency in the metrics definitionInconsistent metrics derived from different sourcesError introduced during clone and editInconsistent

6、 application and enforcement of policies,such as PII,financial controls and cost-effectiveness.Batch query doesnt scale well to increasing data volumesRedundant reprocessing leads to a waste of time and computational resourcesSlow processing delays the availability of insights for decision makingGap