李钰、唐云-Flink State：2021年的后端改进和发展.pdf

1、李钰/阿里巴巴资深技术专家，ASF Member,Apache Flink&HBase PMC唐云/阿里巴巴技术专家，Apache Flink committerFlink State-Backend Improvements and Evolution in 2021SnapshotSnapshotimprovementimprovementState Backend State Backend improvementimprovementFuture WorkFuture Work#1#2#3#1 1State Backend ImprovementState Backend Improv

2、ement支持状态访问延迟监控：state latency trackingState Storestart:System.nanoTimeend:System.nanoTimeLatency:end-start(record it every N operation access)Operation支持状态访问延迟监控：state latency trackingx配置项默认值描述state.backend.latency-track.keyed-state-enabledfalse是否启用状态访问时的性能监控state.backend.latency-track.sample-interv

3、al100每多少次操作时进行一次状态操作性能采样，越小越精确，但是会影响日常访问性能state.backend.latency-track.history-size128保留的采样数据个数。越大越精确相关配置https:/nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#state-backends-latency-tracking-optionsFRocksDB 版本升级(5.17.2-6.20.3)x支持ARM平台WIP 借助 deleteRange 加速扩并发速度更细粒度的WriteBuf

4、fer内存管控RocksDB Backend 内存管控优化 RecapxFlink申明一个state时，对应RocksDB的一个column family（独立内存）Flink在slot-sharing机制下，一个slot内可以存在多个包含keyed state的operatorFlink不限制用户在一个operator内申明的state数目RocksDB Backend 内存管控优化 RecapxBlock CachereserveDataBlockIndexBlockFilterBlockarena block Write buffer managerarena block arena b

5、lock cachecachecacheRocksDB Write buffer Manager+Block CacheRocksDB Backend 内存管控优化 RecapxRocksDB Write buffer Manager+Block Cache*RocksDB upgraded with 256KB dummy reserved in block cachearena block?8MB)Write buffer managerreserveRocksDB Block Cache256KB*Cache/sahrds shardsreserve.RocksDB Backend 内存

6、管控优化xRocksDB Write buffer Manager+Block CacheFLINK-19238 sanity check for RocksDB arena block sizearena block?8MB)Write buffer managermanager size32MBmutable limit28MBarena block?8MB)arena block?8MB)arena block?8MB)Write buffer managerarena block?8MB)arena block?8MB)new comearena block?8MB)Immutable