Methods and apparatus to organize an object store namespace
Inventors
Pogde, Prashant • Wagle, Siddharth Jivan • Gangumalla, Uma Maheswara Rao • Agarwal, Arpit Ashok
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12373462-B2
Publication Date
2025-07-29
Expiration Date
Abstract
Disclosed examples create at least first and second database shards in a leader node, the leader node located in a consensus ring; and cause replication of first namespace metadata in the at least the first and second database shards of the leader node and in at least first and second database shards in a follower node, the follower node located in the consensus ring.
Core Innovation
The described invention organizes object-store namespace management by distributing namespace metadata across database shards in a namespace management cluster that includes a consensus ring with a leader node and follower nodes. The system creates at least first and second database shards in the leader node, where the leader node is located in the consensus ring, and replicates first namespace metadata to the at least the first and second database shards in the leader node. The replication also includes causing replication of the namespace metadata to at least first and second database shards in corresponding ones of namespace databases in the follower node, where the follower node is located in the consensus ring.
Namespace metadata is sharded into multiple database shards per node, and updates are propagated across the nodes using transaction commands written to per-node command logs and drained to achieve an atomic, strongly consistent replication within the consensus ring. The system further introduces cross-shard metadata, including a bucket or volume/bucket-to-shard mapping, stored in master shards for locating sharded namespace metadata across the database shards. Cross-shard metadata is generated and replicated so that shard description information is available for both leader and follower roles.
The invention supports scalability by creating additional database shards and local data stores when shard capacity or throughput reaches threshold capacity. Shard scaling is described as adding and de-provisioning shards, and creating additional shards can be triggered by capacity constraints to increase input/output operations (IOPS) for executing transaction commands in the leader node’s command log. The document also describes reduced time and parallelism for committing multiple namespace update transactions using sharded databases.
Claims Coverage
The provided independent claims are clm-00001, clm-00009, and clm-00016. Across these independent claims, the inventive coverage centers on consensus-ring database shards (leader and follower nodes), replication of namespace metadata across corresponding database shards, and generation/replication of cross-shard metadata used to describe shard layouts, with refinements including transaction-command-based replication, shard scaling triggers, and performance-oriented shard creation.
Consensus-ring sharded replication of namespace metadata from leader to follower
Create at least first and second database shards in a leader node located in a consensus ring, and cause replication of first namespace metadata across the at least the first and second database shards of the leader node and across at least first and second database shards in corresponding ones of first and second namespace databases in a follower node located in the consensus ring.
Cross-shard metadata describing database shards with transactional command-log replication
Establish database shards for a leader node and a follower node of a consensus ring; generate cross-shard metadata to describe the database shards; and initiate transactions to command logs in the leader node and the follower node, where the transactions write namespace metadata and the cross-shard metadata in first ones of the database shards in the leader node and replicate the namespace metadata and the cross-shard metadata to second ones of the database shards in corresponding ones of namespace databases in the follower node.
Creating leader shards and replicating namespace metadata across corresponding follower shards in a consensus ring
Create at least first and second database shards in a leader node located in a consensus ring, and cause replication of first namespace metadata across the at least the first and second database shards of the leader node and across at least first and second database shards in corresponding ones of first and second namespace databases in a follower node located in the consensus ring.
The independent claims collectively cover sharding namespace metadata into database shards within a leader and follower deployed on a consensus ring, replicating namespace metadata across corresponding leader and follower shards, and generating cross-shard metadata that describes the database shards and is written to and replicated via transactions to command logs. Dependent claims further refine replication mechanisms, scaling triggers, and inclusion of volume information in cross-shard metadata, and may include shard creation for IOPS.
Stated Advantages
Provides atomic, strongly consistent replication across nodes within the consensus ring.
Supports scalability by adding and de-provisioning shards when shard capacity or throughput reaches threshold capacity.
Can reduce time by enabling parallelism for committing multiple namespace update transactions via sharded databases.
Can increase input/output operations (IOPS) for executing transaction commands by creating additional database shards.
Documented Applications
Namespace management for an object store namespace, including namespace metadata operations such as rename/move/delete/link, executed by client devices with distributed sharded metadata replication across a consensus ring cluster.
Interested in licensing this patent?