2-2 StarRocks 中的实时更新.pdf

1、|StarRocks的实时更新StarRocks的实时更新常冰琳|Outline Real-time update use cases Common approaches Updates in StarRocks Ongoing&future works|实时更新需求01|Why?Traditional OLAPT+1 batch ETL,high latencyIncremental append only,no updateAppend update&merge-on-read,poor query performanceNew requirement in real-time analy

2、ticsrealtime data hot data volatile dataTP-AP sync pipelineIn database ELT|Use Case:full row upsert/deleteFull row upsert(or delete)most common formMySQLinsert into on duplicate key updateStarRocksunique key load(upsert)primary key load(upsert/delete)TP-AP CDC sync|Merge-on-ReadMergeFast write,slow

3、readExamples:Various LSM TreesHudi Merge-on-Read TableStarRocks Unique Key|Copy-on-WriteSlow write,Fast readExample:Delta LakeHudi Copy-on-WriteIcebergSnowflake|Copy-on-WriteSlow write,Fast readExample:Delta LakeHudi Copy-on-WriteIcebergSnowflakecheck overlapping filesidentify insert/updaterewrite o

4、verlapping files|Copy-on-WriteSlow write,Fast readExample:Delta LakeHudi Copy-on-WriteIcebergSnowflake|Delta StoreSlow(a bit)write,Fast readExample:KuduMany TP/HTAP DatabasesPrimaryIndexDeltaStore|Delete+InsertSlow(a bit)write,Fast readExample:SQL Server column storeAlibaba ADB,HologresStarRocks pri

5、mary key tablexxxxxDeleteBitmapPrimaryIndex|StarRocks的实时更新02|System OverviewFEBETablet 1Tablet 2Tablet 3BETablet 2Tablet 3Tablet 4BETablet 1Tablet 3Tablet 4BETablet 1Tablet 2Tablet 4|System OverviewFEBETablet 1Tablet 2Tablet 3BETablet 2Tablet 3Tablet 4BETablet 1Tablet 3Tablet 4BETablet 1Tablet 2Tabl

6、et 41.writeWrite TXN|System OverviewFEBETablet 1Tablet 2Tablet 3BETablet 2Tablet 3Tablet 4BETablet 1Tablet 3Tablet 4BETablet 1Tablet 2Tablet mit with versionWrite TXN|Inside a TabletRocksDBMetav1v2v9v10DelVectorPrimaryIndexRowset 0Rowset 1Rowset 2Rowset 3.Rowset N.|MetadataPB saved in RocksDBCached