SPARK CONNECT 中的依赖关系管理：简单、隔离、强大.pdf

1、2024 Databricks Inc.All rights reservedAkhil Gudesa,Hyukjin Kwon Akhil Gudesa,Hyukjin Kwon-R&D DatabricksR&D DatabricksJune 2024June 20241Dependency Management Dependency Management in Spark Connect:in Spark Connect:Simple,Isolated,Simple,Isolated,PowerfulPowerful2024 Databricks Inc.All rights reser

2、ved2024 Databricks Inc.All rights reserved2How can Dependencies Help You?How can Dependencies Help You?Custom ETLCustom ETLBring your own,or 3rd party libraries in your data transformation pipelinesUse OSS AI librariesUse OSS AI librariesHave the freedom to experiment with the wide variety of existi

3、ng ML open source libraries2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved3How can Dependencies Help You?How can Dependencies Help You?Python PackagesJARsCustom LibrariesPython PackagesJARsCustom LibrariesDataDataTransformed DataTransformed DataUser Defined FunctionUse

4、r Defined Function2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved4Dependencies in“Classic”SparkDependencies in“Classic”SparkCluster ScopedCluster ScopedEnvironment/dependencies are shared between all usersInterInter-User ConflictsUser ConflictsUsers may require differe

5、nt versions of the same dependencyStaticStaticUpdating previously-set dependencies requires a Driver restartShared DependenciesShared Dependencies2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reservedWhat is Spark Connect?What is Spark Connect?52024 Databricks Inc.All rights

6、reserved2024 Databricks Inc.All rights reserved6DEPENDENCY DEPENDENCY ISOLATIONISOLATIONvia Spark Connectvia Spark Connect2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved7Dependencies in Spark ConnectDependencies in Spark ConnectSpark Connect Spark Connect ClientClientS