通过 CDC、Apache Spark™ 流和 Delta Lake 解锁近实时数据复制.pdf

1、Unlocking Near Real Time Data Replication with CDC,Apache Spark Streaming,and Delta LakeDatabricks2023Ivan Peng and Phani NalluriHow many orders did DoorDash do yesterday?How many orders did DoorDash do yesterday?Get me data from databasesselect*from table_nameGet me data from databases,fastselect*f

2、rom table_name where updated_at$LATEST_DATEmergeGet me data from databases,fast and as the schema changesselect*from information.schemas where name=table_name;mergeselect*from information.schemas where name=table_name;reconcilepageincompatibleselect*from information.schemas where name=table_name;mer

3、geselect*from information.schemas where name=table_name;reconcilepageincompatiblex1000Somewhere in there is a migration from Redshift to Snowflake,and building a whole orchestration system around the tasks HistoryAKA the State of Data at DoorDash,2020 90%of 1000 DB tables were dumped to Snowflake vi

4、a naive dump Incremental tables required:Table to have an updated_at fieldIndex on that fieldApplication to update that field on every write operation CDC was present,but in its infancy at DoorDashProject PeptoAlleviating indigestion of data processingRequirementsHave better data freshness than 24 h

5、oursOwn our data on a modern Lakehouse platformHandle schema evolution and backfillsEnable analytical workloads that otherwise would have been run on the production databasesDesign TenetsLean into CDC/Kafka across all database flavorsBuild a self-serve platform to democratize onboarding of tablesWri

6、te-once,read manyLeverage streaming checkpointing to bypass late-arriving dataOperational simplicityProject PeptoWhat we are not A coupled service with databases A real-time system that feeds into online servicesProject PeptoHighlighted Design Decisions Not-ka