Edward Jones 的 Unity Catalog 实施与演变.pdf

1、Unity Catalog Implemntation&its Evolution at Edward JonesDatta Rao06-11-2025The great growling engine of change technology-Alvin TofflerCloud Analytics Journey3Databricks&Analytics Data Lake20251234202212342023123420241234(1)DatabricksActivation for Hadoop decommission(2)Databricks&ADLS for Analytic

2、s use cases(3)Databricks conference&reviews(4)Unity Catalog Design&deploy(5)Framework to register all incoming data assets to UC(6)Migrate hive metastore to UC(8)Prep for Cloud v2.0&scale out UC(9)DR preparedness(7)5K+securable objects in UCCloud V1.xCloud V2.0+Analytics Hub in Cloud V1.x4Current se

3、tupAnalytics Hub in Cloud V1.xUC design aligning to the Medallion architectureoBronze layer catalog for landing and maintaining the raw datasets in its original format.oSilver layer has two catalogs for Enterprise and for Point Solution aligning datasets which are guard railed by the enterprise and

4、business taxonomies.oGold layer has application catalog,with portfolio&product hierarchical alignment of applications with each of it having autonomous UC Schemas.oUtility layer to accommodate the centralized configuration items of all layers.Cataloging all data assets of tables,views or volumes to

5、Enterprise Data Catalog.Process&infrastructure defined to develop&train the Data Science models with production quality data.Limited or early stage of integration with other enterprise tools.5Current setupAnalytics Hub in Cloud V1.xBuilt on single storage account.User provisioning of access rights a

6、re at the UC Schema level where some schemas are heavily crowded with tons of entities.All analytics use cases fit into the Medallion architecture which has different expectations of SLAs,RTO/RPO.Limited number of Catalogs.Both managed&external table locations are in same storage account posing issu