一个常见问题和一个古老的理念：我们如何构建资产库.pdf

1、A Common Problem and an Ancient IdeaHow We Built Our Asset LibraryAgendaWhat is AlexandriaWhat is Asset LibraryHow we solved itWhat problems we hadThe resultWhat is Asset LibraryAlexandria is the asset library management service.What is AlexandriaAlexandria is the asset library management service.Wh

2、at is Alexandrialibrary_header id;owner;protection_level;folders;Data Modelsasset_record id;asset_type;file_name;file_size;color_infos;Alexandria is the asset library management service.What is AlexandriaData StorageWhy not a database?Because we need:Near infinite scalabilityCost effectivenessStrong

3、 global consistencyWhat we talk about when we talk about consistencyA Distributed System has:Multiple nodes/machines/processes that might lead to concurrency issuesMultiple copies of data that might be out of syncAt the highest level of consistency guarantee,a distributed system works as if:Single p

4、rocessSingle copy of dataButGCS is object storage.Its not designed to be fast.Read PerformanceIn memory cacheRead PerformanceIn memory cacheConsistent hashingConsistent HashingRead PerformanceIn memory cacheConsistent hashingRead PerformanceIn memory cacheConsistent hashingPostgres SidecarWrite Perf

5、ormanceProblem#1-We have some extremely large librariesSolution-Multi segments organized as a prefix treeasset_record id:“asset1”,Write PerformanceProblem#1-We have some extremely large librariesSolution-Multi segments organized as a prefix treeasset_record id:“asset1”,binary(“asset1”)-01001101-Segm

6、ent 01Write PerformanceProblem#2-Write rate limitSolution-Requests coalescingWrite PerformanceProblem#2-Write rate limitSolution-Requests coalescingWrite PerformanceProblem#3-long tail of write latencyOur ProblemsUser ExperienceLatency long tailComplicated request coalescing logi