6544 - Extreme Savings through Modern Storage Architecture.pdf

1、Kyle LambExtreme Savings through Modern Storage ArchitectureExtreme Savings through Modern Storage ArchitectureKyle LambARTIFICIAL INTELLIGENCE(AI)NASEast-west cluster traffic slows transactions as clusters scale.Not designed for NVMe.S h a r e d-n o t h i n g A r c h i t e c t u r eIssues with Lega

2、cy ArchitectureParallel Architecture EnhancementsDisaggregated,SharedDisaggregated,Shared-Everything(DASE)ArchitectureEverything(DASE)ArchitectureDisaggregated Shared Everything ArchitectureData PlatformUNIVERSAL DATA SPACEElements in Hyperscale FlashTransactions Data in SCMTablesFilesObjectsK8S Vol

3、umesIndexesMetadataWrite BufferDISAGGREGATION FABRIC(ETHERNET,INFINIBAND)STATELESS CONTAINERSK8S CSICinder,Manilla S3,S3A(Hadoop)NVMe/TCP(Future)Trino&Spark PluginsSQLTablesVMware HCLSMB v2.1 v 3.0NFS v3 v4.1RDMA&GPUDirect Clients/File ServersHPC/AIData ProtectionDB/Big DataConnection StateScalabili

4、ty and ResilienceVAST Data 2024 Overview6Generation 1Asymmetric Architecture*(1H25)Enterprise NAS&Object StorageNFS,SMB,S3,GPUDirect,VAAI*Block StorageAccessScalable Data AnalysisPython,Trino,SpaarkSQL,DremioScalable Computing*Events,LambdasDataStoreStoreDataBaseBaseData PlatformCombined Storage all

5、ows for Cost savings across many PlatformsDataEngineEngineVDP:Data Services,Protocols&Deployment PlatformsSC24 Update7High-PerformanceData IngestionKafka,Python SDK,Parquet,Spark&TrinoNVMe/TCPInferenceMicroservicesK8S Operator,Vector SearchLocallyLocally-Decodable Codes:Decodable Codes:EC provides l

6、ow overhead data protectionWrite Shaping with Large scale Write Write Shaping with Large scale Write BuffersBuffersLow Cost QLCNearest Neighbour Algorithm Nearest Neighbour Algorithm Data ReductionData ReductionGlobal De-dupe with fine grained resolutionSignificant Data Reduction Significant Data Re