2.利智超 - 阿里云峰会上海-开发者大会-AZ-V1（20页）.pdf

1、Analytics Zoo:A Unified Data Analytics+AI PlatformAnalyticsAnalytics-Zoo:Zoo:统一的大数据分析+AI+AI平台利智超Intel 软件架构师大数据分析和人工智能创新院Why Analytics-ZooReal-World ML/DL Applications Are Complex Data Analytics Pipelines“Hidden Technical Debt in Machine Learning Systems”,Sculley et al.,Google,NIPS 2015 PaperUnified

2、Big Data Analytics PlatformChasm b/w Deep Learning and Big Data CommunitiesReal-world users(big data users,data scientists,analysts,etc.)Deep learning expertsThe The ChasmChasmLarge-Scale Image Recognitionhttps:/ Spark jobsNo changes to the Spark or Hadoop clusters neededData parallelEach Spark task

3、 runs the same model on a subset of the data(batch)“Zero”code changeDirectly support TensorFlow,Keras and Caffe ModelSeamlessly deployed on production big data clustersOnly need to install on driver node.Whats Analytics-ZooAnalytics-Zoo:Unified Analytics+AI Platform for BigDataAnalytics-Zoo:Run as S

4、tandard Spark Programs13Partition 1Partition 2Partition nTraining SetSampleSampleSampleWorkerWorkerWorkerDriver2221113334Broadcast W(800MB)to each worker in each iterationEach task computes G(800MB)in each iterationEach task sends G(800MB)for aggregation in each iterationTraining samples cached in w

5、orker memoryDistributed Training in Analytics-ZooPeerPeer-2 2-Peer Peer AllAll-Reduce Reduce synchronizationsynchronizationDistributed TF&Keras on Spark#pyspark codetrain_rdd=spark.hadoopFile().map()dataset=TFDataset.from_rdd(train_rdd,)#tensorflow codeimport tensorflow as tfslim=tf.contrib.slimimag

6、es,labels=dataset.tensorswith slim.arg_scope(lenet.lenet_arg_scope():logits,end_points=lenet.lenet(images,)loss=tf.reduce_mean(tf.losses.sparse_softmax_cross_entropy(logits=logits,labels=labels)#distributed training on Sparkoptimizer=TFOptimizer.from_loss(loss