如何手动定位Ceph对象位置？CRUSH算法手动推演咨询

Great question! Understanding how CRUSH locates objects manually is a fantastic way to deepen your grasp of Ceph's distributed mechanics. Let's break this down into two clear parts: checking object locations directly, and manually walking through the CRUSH logic to find replicas.

一、手动查看对象位置的快速方法 #

Ceph provides built-in tools to directly query where an object is stored—no need to reverse-engineer everything from scratch.

使用ceph osd map命令 #

This is the most straightforward way to get an object's placement. Run:

ceph osd map <pool-name> <object-name>

示例输出解析：

osdmap e123 pool 'mypool' (1) object 'myobject' -> pg 1.42 (1.2a) -> up ([0,2,5], p0) acting ([0,2,5], p0)

• pg 1.42: The PG (Placement Group) that holds the object.
• up ([0,2,5]): The OSDs currently marked as "up" and responsible for the object.
• acting ([0,2,5]): The OSDs actively serving the object (usually matches up unless there's rebalancing).

补充：使用rados命令验证 #

You can also get basic object stats including placement hints with:

rados -p <pool-name> stat <object-name>

This will show you the PG ID associated with the object, which you can then map to OSDs using ceph osd map <pool-name> <pg-id>.

二、手动推演CRUSH副本定位的步骤 #

If you want to replicate CRUSH's logic manually, you'll need to work through its core steps: calculating the PG ID, then applying the CRUSH rule set to map that PG to OSDs.

步骤1：获取Pool的关键参数 #

First, gather details about your target pool:

• Pool ID & PG count:

  ceph osd pool ls detail | grep <pool-name>
  # 输出示例：pool 1 'mypool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 ...
  

  Note down the pool ID (e.g., 1), pg_num (e.g., 128), and crush_rule (e.g., 0).
• CRUSH rule details:

  ceph osd crush rule dump <rule-name-or-id>
  

  Example rule output:

  rule replicated_rule {
      id 0
      type replicated
      min_size 1
      max_size 10
      step take default
      step chooseleaf firstn 0 type host
      step emit
  }
  

  This rule tells CRUSH to:
  1. Start with the default root bucket.
  2. Select n leaf buckets (hosts), where n is the pool's replication size (firstn 0 uses the pool's size).
  3. Pick one OSD from each selected host and output them as the replica set.

步骤2：计算对象对应的PG ID #

CRUSH uses the rjenkins hash of the object name to map it to a PG. To calculate this manually:

1. Implement the rjenkins hash algorithm (here's a quick Python snippet):

   def rjenkins_hash(key):
       h = 0
       for c in key:
           h += ord(c)
           h += (h << 10)
           h ^= (h >> 6)
       h += (h << 3)
       h ^= (h >> 11)
       h += (h << 15)
       return h & 0xFFFFFFFF  # 保持32位无符号整数
   
   # 替换为你的对象名和pool的pg_num
   object_name = "myobject"
   pg_num = 128
   
   hash_val = rjenkins_hash(object_name)
   pg_id = hash_val % pg_num
   print(f"Calculated PG ID: {pg_id}")
   

2. The resulting pg_id is the placement group that holds the object. Combine it with the pool ID to get the full PG identifier: <pool-id>.<pg-id> (e.g., 1.42).

步骤3：模拟CRUSH的OSD选择逻辑 #

Now, use the PG identifier and CRUSH rule to find the replica OSDs:

1. Hash the PG identifier: Compute the rjenkins hash of the full PG string (e.g., rjenkins_hash("1.42")). This hash acts as the "random seed" CRUSH uses to select buckets/OSDs.
2. Traverse the CRUSH map per the rule:
   • Start with the root bucket (default in our example). List all child buckets (e.g., racks, hosts) under it, each with their weight.
   • Use the hash value to perform a weighted random selection of buckets, following the rule's steps. For chooseleaf firstn 0 type host, you'll select pool-size distinct hosts (ensuring fault domain isolation), then pick one active OSD from each host.
   • Exclude any OSDs that are marked down or out in the cluster (check with ceph osd tree).

关键注意事项 #

• CRUSH Map State: Your manual calculation must match the current CRUSH map (use ceph osd crush dump to get the latest map). OSD weights, bucket structure, and OSD states all affect the result.
• Replication vs Erasure Code: EC pools use different CRUSH rules (e.g., step chooseleaf indep instead of firstn), so adjust your logic accordingly.
• PGP Num: If pgp_num differs from pg_num, CRUSH uses pgp_num for placement to allow gradual rebalancing. Use pgp_num in your PG ID calculation if that's the case.

内容的提问来源于stack exchange，提问作者yasin lachini