流式处理架构偏移发现和受控缓解.pdf

1、Streaming Schema Drift Discovery,and Controlled MitigationPresented by:Alexander VanadioDatabricks2023PrologueAbout meI have been doing some combination of Software/Data/ML Engineering for about 15 years Consulted for a Fortune 100 company for a few years to help with their PB scale big data problem

2、sBig fan of computer science,music,and memes “Alex,I have a time sensitive query that I need to run against a Delta table,but dont see some fields that should be there.Whats going on!?”1_DAIS_Title_SlideSchema DriftSchema DriftMoving the goalsThe data you ingested into your Delta Table today,might b

3、e different tomorrowSome keys might be so sparse that you didnt even know they were thereVendors and upstream data providers usually arent going coordinate with you before they change somethingData Providers Are Ready“Cant we just set.option(mergeSchema,True)on our stream?”Merge Schema=TrueWere putt

4、ing a lot of trust in our data providers and removing human gatesA table with 10 columns today could have 100 tomorrowThis is especially bad when incoming data contains dynamically generated keys“How about schema evolution within Auto Loader?”Now were onto somethingAuto Loader Schema Evolution4 Uniq

5、ue Strategies What if we want to utilize the badRecordsPathoption?What if we dont want to add all new columns to our table?What if we dont want to stop the stream(prioritize low-latency and availability)Auto Loader Schema EvolutionSelecting the rescue strategyIncoming data that doesnt match the inte

6、rnal schema file,ends up in a new String column called _rescued_dataThis is actually serialized JSON data(e.g.”key”:“value”)Can we use this for drift detection?.1_DAIS_Title_SlideDrift DetectionQuantifying DriftLeveraging the _rescued_data columnEvery single record that comes in has well-formed JSON