Delta Tensor：Delta Lake 中的高效向量和张量存储.pdf

1、2024 Databricks Inc.All rights reservedDELTA TENSORDELTA TENSOREfficient Tensor Efficient Tensor Storage in Delta LakeStorage in Delta LakeLiaoliao Liu,Liam Bao,Zhiyu WuLiaoliao Liu,Liam Bao,Zhiyu Wu12024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reservedLiaoliao LiuLiam BaoZh

2、iyu WuLiaoliao LiuLiam BaoZhiyu Wu2Our teamOur teamIncoming SDE AWS AthenaCS Graduate Northeastern University2Incoming SWE SnowflakeCS Graduate Northeastern UniversityCS Graduate Northeastern Universityl liu.liaonortheastern.eduiu.liaonortheastern.edubaobao.zhiwnortheastern.eduzhiwnortheastern.eduwu

3、wu.zhiyunortheastern.eduzhiyunortheastern.edu2024 Databricks Inc.All rights reserved3ContentsContents01.01.02.02.03.03.04.04.05.05.IntroductionIntroductionProblem statementProblem statementMethodsMethodsExperimentsExperimentsResultsResults06.06.FTSFFTSF CSFCSF BSGSBSGSConclusionConclusion2024 Databr

4、icks Inc.All rights reserved4INTRODUCTIONINTRODUCTION01.01.2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reserved Exponential growth of AI/ML applications Inefficient methods for tensor storageRedundant storage spaceSlow processing speed5BackgroundBackgroundRelated worksRelat

5、ed works Focus on vectors(1d arrays)rather than tensors(nd arrays)Miss cloud-native environment:cloud storage offers more than just disksObjectiveObjective Efficient tensor storage in cloud object storage2024 Databricks Inc.All rights reserved6PROBLEM PROBLEM STATEMENTSTATEMENT02.02.2024 Databricks

6、Inc.All rights reserved2024 Databricks Inc.All rights reservedClientsClients7Problem StatementProblem StatementHow to store tensors efficiently?How to store tensors efficiently?7Our Methods!Our Methods!ptptptptptptptptptptptptptptptptptpt2024 Databricks Inc.All rights reserved2024 Databricks Inc.All