单元测试：如何克服结构化流媒体中的挑战（重复）.pdf

1、2024 Databricks Inc.All rights reservedBartosz Konieczny|freelance data engineer|Bartosz Konieczny|freelance data engineer|1UNIT TESTS:HOW TO UNIT TESTS:HOW TO OVERCOME CHALLENGES OVERCOME CHALLENGES IN STRUCTURED IN STRUCTURED STREAMINGSTREAMING2024 Databricks Inc.All rights reserved2Freelance data

2、 engineer and Current focus:stream processingData Engineering Design Patterns Blogging:https:/ KoniecznyBartosz Konieczny22024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved3Challenge 01:Challenge 01:STRUCTURING THE STRUCTURING THE CODE AND THE CODE AND THE DATAFLOW MODELD

3、ATAFLOW MODEL2024 Databricks Inc.All rights reserved4DATAFLOW MODEL 101DATAFLOW MODEL 101WHATWHEREWHENHOWWHATWHEREWHENHOW.results your job generates.the job computes the results(event time).the job materializes the results(processing time).the job handles accumulated data.results your job generates.

4、the job computes the results(event time).the job materializes the results(processing time).the job handles accumulated data2024 Databricks Inc.All rights reserved5DATAFLOW MODEL 101DATAFLOW MODEL 101WHATWHEREWHENHOWWHATWHEREWHENHOW.results your job generates.the job computes the results(event time).

5、the job materializes the results(processing time).the job handles accumulated data.results your job generates.the job computes the results(event time).the job materializes the results(processing time).the job handles accumulated dataAND APACHE SPARK STRUCTURED STREAMINGAND APACHE SPARK STRUCTURED ST

6、REAMING.transformation logic.watermark.micro.transformation logic.watermark.micro-batch,triggerbatch,trigger.output mode.output mode2024 Databricks Inc.All rights reserved6DATAFLOW MODEL 101DATAFLOW MODEL 101WHATWHEREWHENHOWWHATWHEREWHENHOWProject organization Unit Tests Project organization Unit Te