2020年终大会-大数据架构：5-2.pdf

1、Data Quality Architecture at Tubi 沈达 Tubi: Stream Freely democratize content and make premium content accessible to everyone. AVOD Data Quality Issues Data Pipelines Overview Solutions and Workflow Related Work and Future Work Contents Data Quality Issues Impacts of Poor Data Quality Explicit Data Q

2、uality Issues: Weird data analysis report Crashing of data pipelines and ML pipelines Data skew and stuck query (the pride of a data engineer) Implicit Data Quality Issues: v Misleading business decisions v Upsetting Data Scientists or ML Engineers v Slowing down development speed v Explicit Issues:

3、 Structured Data For All DataType NULL/null/Null/undefined/unknown Duplication of distinct value For Specific DataType Empty/Truncated/Garbage String Negative of non-negative values Precision Malformatted JSON Possible Root Cause of Implicit Issues JVM: Thread Safety issues JDK: (new SimpleDateFormat(yyyy-MM-dd HH:m:SS) .parse(0000-00-00 00:00:00) Spark: Jira Issues labeled with correctness Hive U