4-4Flink-1.11 Hive集成与批流一体(3).pdf

1、0 1000 2000 3000 4000 5000 6000 7000 8000 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 Flink vs Hive TPC-DS 10 TB FlinkHive on MR SQL Client TableEnvironment CatalogManagerCatalog GenericInMemoryCatalogHiveCatalog Hive Metastore HiveShim referen

2、ceinheritance Streaming Data StreamingFileWriter Finished Files Pending Files In-progress Files Finished Files Pending Files In-progress Files Bucket Finished Files Pending Files In-progress Files Finished Files Pending Files In-progress Files Bucket Subtask Subtask StreamingFileCommitter Commit Mes

3、sage PartitionCommitTrigger PartitionCommitPolicy Partitions ready to commit Metastore Success File ContinuousFileMonitoringFunction ContinuousFileReaderOperator Splits FileSystem HiveContinuousMonitoringFunction ContinuousFileReaderOperator Splits FileSystem Metastore Streaming Data LookupJoinRunner FileSystemLookupFunction Hive Load & Cache Join Key Batch Analytics Batch Analytics Online Streams