更好地协同工作：在流数据流中更改数据馈送.pdf

1、Better TogetherChange Data Feed in a Streaming Data FlowMattias Moser Scott GordonIntroductionsMattias MoserData Architect at 84.51Scott GordonData Engineer at 84.51About 84.51“We are a retail data science,insights and media company.We help The Kroger Co.,consumer packaged goods companies,agencies,p

2、ublishers and affiliates create more personalized and valuable experiences for shoppers across the path to purchase.Powered by cutting-edge science,we utilize first-party retail data from over 62 million U.S.households sourced by the Kroger Plus loyalty program to fuel a more customer-centric journe

3、y using 84.51 Insights,84.51Loyalty Marketing and our retail media advertising solution,Kroger Precision Marketing.”About 84.51 Change Data Capture:Streaming source exampleChange Data Capture:Delta Lake exampleAgendaChange Data Capturefrom a Streaming SourceMattias MoserData Architect at 84.51What i

4、s Change Data Capture?Definition:“The process of recognizing when data has changed in a source system so a downstream system can take an action based on that change.”Source System1.INSERT new records2.UPDATE existing records3.DELETE existing recordsExample:Kroger Online OrdersSycing Orders DB to Dat

5、abricksCDC Example Events_timestamp_operationorder_idcustomer_idstatusitems5/16/25 10:00:00insert2001311opensku:MILK1G,qty:1,sku:EGGS12PK,qty:15/16/25 10:00:35insert2002402opensku:BREADWHT,qty:2,sku:BANANA,qty:65/16/25 10:01:10update1784212shippedsku:CHKNBRST2LB,qty:1,sku:SPINACHBAG,qty:25/16/25 10:

6、01:20update2002402opensku:BREADWHT,qty:2,sku:BANANA,qty:6,sku:PB16OZ,qty:15/16/25 10:01:45delete1620CDC Streaming with DLTvalue:_timestamp:2025-05-16T10:00:00Z,_operation:insert,order_id:2001,customer_id:311,status:open,items:sku:MILK1G,qty:1,sku:EGGS12PK,qty:1,topic:orders_cdc,partition:0,offset:12