Java与大数据架构：3. Large-scale near-real-time (NRT) data analytics platform empowered by Apache Flink - Ying Xu & Kailash Hassan Dayanand, Lyft.pdf

1、Apache Flink empowered large-scale near real-time (NRT) data analytics platform Ying Xu, Streaming Platform, Lyft Inc Kailash HD, Streaming Platform, Lyft Inc Streaming data scenarios at Lyft Architecture of near real-time data analytics platform Deep dive on platform design and fault tolerance Summ

2、arization and future directions Agenda Streaming data scenarios at Lyft About Lyft MISSION: Improve peoples life with the worlds best transportation Streaming data scenarios at Lyft Streaming Events Enrichment Real-time Adaptive Pricing Fraud and Anomaly Detection minute ML Feature Engineering secon

3、ds minuteminute Near Real-time Interactive Query 5 minutes Lyfts data analytics platform architecture Backend Services Mobile app PubSub Events KCL PERSISTENCEBatch ETL Presto, Hive Client, and BI Tools Issues of the legacy platform Persisted data cannot be ready for query in near real-time Streaming persistence using KCL exhibit limited performance Presence of too many small fi les limits perform