2017年喜马拉雅数据计算平台xql.pdf

1、喜马拉雅数据计算平台XQL2017.11Outline XQL总览 系统演进过程 周边产品 经验总结 未来展望XQL总览 研发背景 总体架构 使用人群与场景XQL总览 研发背景XQL总览 研发背景?XQL总览 研发背景?CoreSQLXQL总览 总体架构?XQL总览 总体架构 memory：4T spark Task：200w+XQL job：4000+dataSource：hdfs、hive、hbase、es、kafka、mysql、pg fileFormat：parquet、orc、csv、json、xmlXQL总览 使用人群与场景系统演进过程 v1版本 v2版本 v3版本系统演进过程 v

2、1版本架构?v1?系统演进过程 v1版本通信逻辑clientSingle Spark Application on YarnSpark Driver JVMMaster共享SparkContextSpark Executor JVMSpark Executor JVMSpark Executor JVMSpark Executor JVMHttp serverLocalProxySpark RpcDownloadHandlerWorker隔离SparkSessionHDFSWorker启动注册clientSingle Spark Application on YarnSpark Driver

3、JVMMaster共享SparkContextSpark Executor JVMSpark Executor JVMSpark Executor JVMSpark Executor JVMHttp serverLocalProxySpark RpcDownloadHandlerWorker隔离SparkSessionHDFSWorker启动注册用户提交clientSingle Spark Application on YarnSpark Driver JVMMaster共享SparkContextSpark Executor JVMSpark Executor JVMSpark Execut

4、or JVMSpark Executor JVMHttp serverLocalProxySpark RpcDownloadHandlerWorker隔离SparkSessionHDFSWorker启动注册用户提交请求可用地址clientSingle Spark Application on YarnSpark Driver JVMMaster共享SparkContextSpark Executor JVMSpark Executor JVMSpark Executor JVMSpark Executor JVMHttp serverLocalProxySpark RpcDownloadHan

5、dlerWorker隔离SparkSessionHDFSWorker启动注册用户提交请求可用地址返回异步IDclientSingle Spark Application on YarnSpark Driver JVMMaster共享SparkContextSpark Executor JVMSpark Executor JVMSpark Executor JVMSpark Executor JVMHttp serverLocalProxySpark RpcDownloadHandlerWorker隔离SparkSessionHDFSWorker启动注册用户提交请求可用地址返回异步ID分配执行c

6、lientSingle Spark Application on YarnSpark Driver JVMMaster共享SparkContextSpark Executor JVMSpark Executor JVMSpark Executor JVMSpark Executor JVMHttp serverLocalProxySpark RpcDownloadHandlerWorker隔离SparkSessionHDFSWorker启动注册用户提交请求可用地址返回异步ID分配执行写HDFSclientSingle Spark Application on YarnSpark Driver