cubefs-boosts-efficiency-of-ai-production-cubefspan-aichang-zha-yi-chi-he-oppo.pdf

1、CubeFS Boosts Efficiency of AI ProductionOPPO Chi.HeContentContent01Introduction to CubeFS02The practice of CubeFS on OPPOs machine learning platform03Future of CubeFSProject Project overviewoverviewCubeFS is a next-generation cloud-native open-source storage product hosted under the CNCF.It feature

2、s complete file and object storage capabilities and is currently in the incubation stage.Official website:https:/cubefs.io/ArchitectureArchitectureMaster:Managing resource metadata in the cluster.Ensuring the consistency and high availability of cluster metadata through the Raft protocol.Volume:A fi

3、le system that allows clients to access data from containers.Data Partition:The minimum management unit of file data sharding.Replica Subsystem:Managing data partition.Erasure Code Subsystem:Managing erasure code stripe.Meta Partition:The minimum management unit of file metadata.Metadata Subsystem:M

4、anaging meta partition.Object Subsystem:Object gateway compatible with S3 semantics.Client:Providing access interfaces for mounting file systems.Metadata subsystemMetadata subsystemMeta Partition Splitting:Achieving dynamic scaling by splitting the management scope of meta partitions,without trigger

5、ing data migration tasks.Full in-memory caching strategy：Improving the access speed of metadata.Multi-raft:Ensuring strong data consistency and high availability.Periodic snapshots:Metadata is periodically persisted to disk on a per-partition basis for backup and recovery purposes.Replica subsystemR

6、eplica subsystemLarge/Small file storage optimization:Stored through fragmentation/aggregation.Pre-allocated TinyExtents reduce network overhead.Context-aware replication:Different replication strategies are employed between replicas based on different write patterns to improve replication efficienc