告别脆弱的管道：Kafka 和 Iceberg 的声明式方法.pdf

《告别脆弱的管道：Kafka 和 Iceberg 的声明式方法.pdf》由会员分享，可在线阅读，更多相关《告别脆弱的管道：Kafka 和 Iceberg 的声明式方法.pdf（44页珍藏版）》请在三个皮匠报告上搜索。

1、Adi PolakNo More Fragile Pipelines:Kafka and Iceberg the Declarative WayAdi PolakJune,2025123Adi PolakOpen questionsMaterialization:How to update this table when schema changes?Monitor all these jobs and find out when they breakWhat to do about bad dataHow often to create new parquet files(cost vs o

2、ptimality vs latency)MaintenanceWhats the best cost/optimality tradoff?How to actually schedule and run there?Adi PolakAuthor of 2 books(OReilly)People Manager,Software Engineer&Data/MLStreaming,big data field,including ML at scaleAdiPolakSchema evolutionConsistency&PartitionSmall files&CompactionCh

3、allengesAdi PolakAdi PolakConfluent Tableflow solution Multi trillion events events every hour Kafka=Iceberg Kafka=DeltaLake Trillion multi-versioned schemas 11Adi PolakKafkaKafka Producer“serialize”Kafka Consumer”deserialize”Broker,topic 1,topic 2Adi PolakAdi PolakSchema EvolutionAdi PolakHow to up

4、date this table when schema changes?Formsts edge-casesSemantic of tables append,upsert.Dealing with bad dataautomatically infer scehmaConvert and maintin the divive(Kafka-Iceberg)Adi PolakKafka vs.Iceberg formatsKey-value record of bytesCommon data formats:Avro,Json,ProtobufSQL data typesTables with

5、 columnsRows stored in Parquetk1:v1k2:v2k3:v3k3:v3NameStatecompanyAdi PolakCAConfluentDenny LeeWADatabricksAdi PolakKafka vs.Iceberg formatsNameStatecompanyAdi PolakCAConfluentDenny LeeWADatabricksk1:v1k2:v2k3:v3k3:v3Adi PolakDeserialize bytesApply schema evolutionUpdate Iceberg schema123Kafka topic

6、 can have multiple schema versionsv1:col1v2:col1,col2v3:col3k1:v1k2:v2k3:v3k3:v3Adi PolakKafka Producer“serialize”Kafka Consumer”deserialize”Broker,topic 1,topic 2topic 1How to deserialize?Compatibility modes:BackwardsForwardsFullProvide SerDe(Serializer/Deserializer)implementationAlert/DLQ when:Inc