何涛-Vineyard：开源分布式内存数据管理引擎（GOTC深圳会场）（23页）.pdf

1、综合技术专场何涛 2021年 8月 1日阿里巴巴：开源分布式数据管理引擎Why bother1.Sharing data efficiently(with“0-copy”)between libraries is easy within a single python process2.It is not as easy to do so across processes/runtimes on a single machinePossible with plasma from Apache Arrow,a local object store using shared-memory3.Wha

2、t about processing big data that cannot fit into a single machine,and involving different workloads?Use vineyard+K8s!PyData is the de-facto standard for data analysisThere are lots of libraries for different workloads(image credit:https:/coiled.io/blog/pydata-dask/)Big data analytical pipelinesAn an

3、ti-fraud pipelineLoad data,ETLLabel PropogationFraud detection with DNNPost ETL processingSQLML/TensorflowGraph Computation（LPA）/GRAPESQLBig data analytical pipelinesObversation：A typical big data application involves various kinds of workloads,and thus involves multiple dedicated systems for each w

4、orkloadThese dedicated systems typically shares intermediate data with external file systemsThe workflow is often organized as a chain/DAG,and each individual task only gets invoked after their prerequisite tasks are completedAn anti-fraud pipelineData Extraction and PreprocessingData Warehouse(Hive

5、)、Distributed FS(HDFS),Object Store(S3)Postprocessing and ETLETL（SQL）SQLGraph Analytical AlgorithmsGRAPETrain/Infer with a NN modelTensorflow/PytorchBig data analytical pipelinesProblem：Production-ready systems(Hive,Tensorflow,)are hard to develop.Sharing data with external file systems has huge I/O

6、 cost.Applying cross-task optimization(pipelining)on tasks is challenging.An anti-fraud pipelineData Extraction and PreprocessingData Warehouse(Hive)、Distributed FS(HDFS),Object Store(S3)Postprocessing and ETLETL（SQL）SQLGraph Analytical AlgorithmsGRAPETrain/Infer with a NN modelTensorflow/PytorchBig