使用 Node.js 和增量共享编写数据共享应用.pdf

1、Building Data Sharing Applications using Node.jsWill GirtenLead Specialist Solutions Architect at DatabricksDatabricks2023Who am I?Will Girten,Lead SSA at DatabricksBS in Computer Engineering from the University of Delaware Author of the Node.js connector for Delta SharingJoined Databricks in 2019 S

2、pecializes in data warehousing and performance tuning BI workloads for Financial Services.Prior to Databricks,Will worked as a Data Architect helping federal customers build intelligent data lakes in HealthCare and Government verticals1_DAIS_Title_SlideWhy Node.js?Top Programming LanguagesAcross rep

3、os created on GitHub in 2022RubyC+C#TypescriptJavaPythonJavascriptSource:https:/ is evolving into a language built for communicating data insights Built on top of the V8 JavaScript engine#2.Cross-platformCompatible with all major operating systems#3.Real-time appsAn event-driven,non-blocking I/O mod

4、el#1.FastThe Node.js Event LoopRequestRequestRequestWorkerWorkerWorkerEvent QueueThread PoolEvent LoopRegister CallbackOperation CompleteExecute CallbackWhat makes Node.js great?Sharing large datasets with Delta SharingOverview of Delta SharingThe industrys first open data sharing protocol1.Share li

5、ve data without copying out of the data lake2.Support a wide range of clients,like Node.js3.Strong security,auditing,and governance4.Efficiently scale to massive datasetsHow it works“under the hood”The industrys first open data sharing protocolDelta Sharing ServerDelta TableQuery“sales”tableGet late

6、st snapshothttps:/ are short-lived file URLshttps:/ worth solvingThe hidden cost to powering your APIs&frontendsStreaming&Batch Sources Step 1:Raw(Bronze)Step 2:Transformed(Silver)Step 3:Feature/Aggs(Gold)Step 4:COPY Key-Value StoreThe hidden data maintenance cost!Who maintains t