diff --git a/docs/offload-mechanism.md b/docs/offload-mechanism.md
new file mode 100644
index 0000000000..5fe72e2db3
--- /dev/null
+++ b/docs/offload-mechanism.md
@@ -0,0 +1,331 @@
+# Mooncake SSD Offload 机制
+
+## 1. 核心概念
+
+Offload 是 Mooncake 将数据从 **DRAM（MEMORY副本）** 迁移到 **本地 SSD（LOCAL_DISK副本）** 的过程。与 Eviction（直接丢弃）不同，Offload 将数据持久化到磁盘，后续可通过 Load 路径读回。
+
+```
+MEMORY副本 ──Offload──→ LOCAL_DISK副本 ──Promotion──→ MEMORY副本
+ │ │
+ └──Eviction（丢弃） └──Disk Eviction（丢弃）
+```
+
+## 2. 核心数据流
+
+### 2.1 Offload（内存 → SSD）
+
+```mermaid
+sequenceDiagram
+ participant FS as FileStorage (Client)
+ participant M as MasterService
+
+ loop 每隔 heartbeat_interval (默认10s)
+ FS->>M: OffloadObjectHeartbeat(client_id, enable_offloading)
+ M-->>FS: 返回 offloading_objects {key→size}
+ end
+
+ Note over FS: 执行 OffloadObjects()
+ FS->>FS: BatchQuerySegmentSlices() 从内存读数据
+ FS->>FS: StorageBackend::BatchOffload() 写入SSD
+ FS->>M: NotifyOffloadSuccess(keys, metadatas)
+ Note over M: 释放MEMORY副本refcnt
添加LOCAL_DISK副本(COMPLETE)
+```
+
+### 2.2 Load（SSD → 请求方）
+
+```mermaid
+sequenceDiagram
+ participant RC as 请求方Client
+ participant M as MasterService
+ participant TC as 目标Client (FileStorage)
+
+ RC->>M: Get/BatchGet(keys)
+ M-->>RC: 返回 LOCAL_DISK 副本位置
+ RC->>TC: batch_get_offload_object(keys)
+ TC->>TC: 从SSD读取到ClientBuffer
+ TC-->>RC: 返回 batch_id + RDMA地址
+ RC->>RC: TransferEngine RDMA零拷贝拉取
+ RC->>TC: release_offload_buffer(batch_id)
+```
+
+## 3. 触发时机与 Key 选取
+
+系统有两种 offload 触发模式，由 `offload_on_evict` 开关控制：
+
+### 模式 A：PutEnd 即入队（默认，`offload_on_evict=false`）
+
+```mermaid
+flowchart TD
+ A[Client 调用 PutEnd] --> B{enable_offload?
!offload_on_evict?}
+ B -->|Yes| C[将该对象的所有已完成
MEMORY副本加入 offloading_queue]
+ B -->|No| D[不做任何offload操作]
+ C --> E[副本 refcnt++ 防止被evict]
+ E --> F[等心跳线程取出执行]
+```
+
+- **选取标准**：所有 PutEnd 完成的对象**无差别入队**
+- **无筛选逻辑**：不区分冷热，全部 offload
+
+### 模式 B：Eviction 时入队（`offload_on_evict=true`）
+
+```mermaid
+flowchart TD
+ A[内存使用率 > eviction_high_watermark] --> B[BatchEvict 开始淘汰]
+ B --> C{遍历候选对象}
+ C --> D{已有 LOCAL_DISK 副本?}
+ D -->|Yes| E[安全,直接evict MEMORY副本]
+ D -->|No| F{offload队列达到上限?}
+ F -->|No| G[PushOffloadingQueue
refcnt++ 保护]
+ F -->|Yes| H{offload_force_evict?}
+ H -->|Yes| I[强制evict,数据丢失]
+ H -->|No| J[跳过,保留数据]
+ G --> K[等心跳线程取出执行]
+```
+
+- **选取标准**：由 `BatchEvict` 决定候选对象，基于 **lease_timeout 时间排序**（近似 LRU）
+- **两轮扫描**：第一轮淘汰无 soft pin 的对象，第二轮淘汰有 soft pin 的对象（需 `allow_evict_soft_pinned_objects=true`）
+- **保护机制**：入队时 `refcnt++` 防止 offload 期间被 evict
+
+### Master 端 Eviction 流程
+
+```mermaid
+flowchart TD
+ A[EvictionThreadFunc 后台线程] --> B{内存使用率 >
eviction_high_watermark?}
+ B -->|No| C[休眠,继续监测]
+ B -->|Yes| D[计算本次evict目标量]
+ D --> E[BatchEvict
按lease_timeout排序选候选]
+ E --> F{offload_on_evict模式?}
+ F -->|No| G[直接evict MEMORY副本]
+ F -->|Yes| H[尝试先offload再evict]
+```
+
+## 4. Offload 与 Eviction 的关系
+
+| 维度 | Offload | Eviction |
+|------|---------|----------|
+| 目的 | 将数据持久化到 SSD | 释放内存空间 |
+| 数据去向 | 本地 SSD 文件 | 丢弃 |
+| 数据可恢复 | 是（通过 Load/Promotion） | 否 |
+| 触发者 | 心跳线程（定时） | Eviction 后台线程（水位触发） |
+| 副本变化 | MEMORY → LOCAL_DISK | MEMORY → 删除 |
+
+**协同关系**：
+- Offload 是 Eviction 的**前置安全网**——先持久化再释放，避免数据丢失
+- `offload_on_evict=true` 时二者紧密耦合：eviction 候选先尝试 offload，成功后才释放内存
+- `offload_on_evict=false` 时二者独立：PutEnd 时入 offload 队列，eviction 按自己逻辑运行
+
+## 5. 四种配置组合
+
+| 组合 | enable_offload | offload_on_evict | offload_force_evict | 行为 |
+|------|:-:|:-:|:-:|------|
+| A（默认） | true | false | false | PutEnd 立即入 offload 队列，eviction 独立运行 |
+| B | true | true | false | eviction 时才尝试 offload，失败则跳过（保留数据） |
+| C | true | true | true | eviction 时先 offload，队列满则强制 evict（数据丢失） |
+| D | true | false | true | 等同 A（force_evict 无效） |
+
+## 6. Promotion（SSD → 内存热提升）
+
+当 `promotion_on_hit=true` 时，频繁访问的 LOCAL_DISK 数据自动提升回内存：
+
+```mermaid
+flowchart TD
+ A[Get 命中 LOCAL_DISK 副本] --> B[TryPushPromotionQueue]
+ B --> C{准入检查}
+ C -->|频率 >= threshold| D{内存水位 < 高水位?}
+ C -->|频率不足| Z[跳过]
+ D -->|Yes| E{去重:无MEMORY副本且无进行中任务?}
+ D -->|No| Z
+ E -->|Yes| F{队列 < promotion_queue_limit?}
+ E -->|No| Z
+ F -->|Yes| G[加入promotion队列]
+ F -->|No| Z
+ G --> H[心跳线程取出
分配MEMORY副本→SSD读取→RDMA写入]
+```
+
+- **频率统计**：Count-Min Sketch，阈值 `promotion_admission_threshold`（默认 2）
+- **每次心跳限 1 个** promotion 任务（`kMaxPerHeartbeat=1`）
+
+## 7. 关键代码索引
+
+### 7.1 Master 端（`mooncake-store/src/master_service.cpp`）
+
+| 函数 | 行号 | 职责 |
+|------|------|------|
+| `EvictionThreadFunc()` | :3160 | 后台线程，监测内存水位，触发 `BatchEvict` |
+| `BatchEvict()` | :4466 | 核心淘汰逻辑，按 lease_timeout 选候选对象，内部定义 `try_evict_or_offload` lambda(:4515) 处理 offload/evict 分支 |
+| `OffloadObjectHeartbeat()` | :2618 | 客户端心跳入口，返回 `offloading_objects` 队列给客户端 |
+| `NotifyOffloadSuccess()` | :2705 | 处理客户端 offload 完成通知：释放 MEMORY 副本 refcnt，添加 LOCAL_DISK 副本 |
+| `PushOffloadingQueue()` | :2744 | 将 key 入 offload 队列，根据副本的 segment 名称定位目标客户端 |
+| `TryPushPromotionQueue()` | :2823 | Get 命中 LOCAL_DISK 时调用，经四重准入检查后将 key 加入 promotion 队列 |
+| `PromotionObjectHeartbeat()` | :2919 | 返回待 promotion 任务（每次心跳限 1 个） |
+| `PromotionAllocStart()` | :2948 | 为 promotion 分配 MEMORY 副本（PROCESSING 状态） |
+| `NotifyPromotionSuccess()` | :3041 | 确认 promotion 完成：标记 MEMORY 副本 COMPLETE，释放 LOCAL_DISK refcnt |
+
+**PutEnd 中的 offload 触发**（:1390）：
+
+```cpp
+if (enable_offload_ && !offload_on_evict_) {
+ metadata.VisitReplicas(/* MEMORY + COMPLETED */, [&](Replica& replica) {
+ auto result = PushOffloadingQueue(key, replica);
+ if (result) {
+ replica.inc_refcnt(); // 防止 offload 期间被 evict
+ }
+ });
+}
+```
+
+**BatchEvict 中的 try_evict_or_offload**（:4515）：
+
+```cpp
+auto try_evict_or_offload = [&](const std::string& key, ObjectMetadata& metadata, ...) {
+ if (!offload_on_evict_) return metadata.size * evict_replicas(metadata); // 直接淘汰
+
+ if (has_local_disk_replica(metadata))
+ return metadata.size * evict_replicas(metadata); // 已有SSD副本，安全淘汰
+
+ if (offload_force_evict_ && offload_queued >= offload_cap)
+ return metadata.size * evict_replicas(metadata); // 队列满，强制淘汰
+
+ // 尝试入 offload 队列
+ auto result = PushOffloadingQueue(key, replica);
+ if (result) { replica.inc_refcnt(); /* 保护 */ return ...; }
+
+ if (offload_force_evict_) return metadata.size * evict_replicas(metadata); // 入队失败，强制淘汰
+ return 0; // 跳过，保留数据
+};
+```
+
+### 7.2 Client 端（`mooncake-store/src/file_storage.cpp`）
+
+| 函数 | 行号 | 职责 |
+|------|------|------|
+| `Heartbeat()` | :495 | 心跳主循环：拉取 offload 任务 → `OffloadObjects()` → `ProcessPromotionTasks()` |
+| `OffloadObjects()` | :341 | 执行 offload：从内存读数据 → 写 SSD → 通知 Master |
+| `BatchGet()` | :300 | Load 路径：从 SSD 读到 ClientBuffer，返回 RDMA 可访问地址 |
+| `ProcessPromotionTasks()` | :534 | 驱动 promotion：拉取任务 → 分配 MEMORY → SSD 读取 → RDMA 写入 |
+
+### 7.3 RPC 通信层（`mooncake-store/src/master_client.cpp`）
+
+| 函数 | 行号 | 职责 |
+|------|------|------|
+| `OffloadObjectHeartbeat()` | :941 | RPC 封装：客户端 → Master 拉取 offload 任务 |
+| `NotifyOffloadSuccess()` | :963 | RPC 封装：客户端 → Master 确认 offload 完成 |
+| `PromotionObjectHeartbeat()` | :977 | RPC 封装：客户端 → Master 拉取 promotion 任务 |
+| `PromotionAllocStart()` | :985 | RPC 封装：客户端 → Master 请求分配 promotion 的 MEMORY 副本 |
+| `NotifyPromotionSuccess()` | :998 | RPC 封装：客户端 → Master 确认 promotion 完成 |
+
+## 8. 控制开关与环境变量详解
+
+### 8.1 Master 端开关
+
+配置文件：`mooncake-store/include/master_config.h`，可通过 `master.yaml` 或命令行参数设置。
+
+#### `enable_offload`（默认 false）
+
+- **false**：SSD offload 完全禁用。客户端调用 `OffloadObjectHeartbeat(enable_offloading=false)` 时，Master 清空该客户端的 offload 队列并释放所有 refcnt。对象只有 MEMORY 副本，内存不足时直接 eviction 丢弃。
+- **true**：启用 offload。客户端 `FileStorage` 初始化时注册 LOCAL_DISK segment（`MountLocalDiskSegment`），心跳线程开始工作。PutEnd 或 eviction 时对象可被加入 offload 队列。
+
+#### `offload_on_evict`（默认 false）
+
+- **false（模式 A）**：PutEnd 完成后**立即**将该对象的所有 MEMORY 副本加入 offload 队列。意味着所有写入的数据都会尽快下沉到 SSD，内存中的副本仅作为 RDMA 访问源存在，直到 offload 完成后由 Master 释放。
+- **true（模式 B/C）**：PutEnd 时不做任何 offload 操作。只有当内存使用率超过 `eviction_high_watermark_ratio` 触发 `BatchEvict` 时，才将候选淘汰对象入 offload 队列。**区别**：模式下 B 对象在内存充裕时不会被 offload，仅在被选中淘汰时才持久化到 SSD。
+
+#### `offload_force_evict`（默认 false）
+
+- **false**：在 `offload_on_evict=true` 模式下，如果 offload 队列已满（达到 `offloading_queue_limit_ * kOffloadCapRatio` = 25000），超出上限的候选对象**跳过淘汰**，数据保留在内存中。这会导致本轮 eviction 无法释放足够内存，Master 会打印 WARNING。
+- **true**：在 `offload_on_evict=true` 模式下，offload 队列满时不再跳过，而是**强制 eviction 丢弃数据**。适用于宁可丢失数据也要保证内存可用的场景。
+- **注意**：此开关仅在 `offload_on_evict=true` 时生效。单独设置（模式 D）无任何效果。
+
+#### `promotion_on_hit`（默认 false）
+
+- **false**：LOCAL_DISK 副本被 Get 命中后，后续访问始终走 SSD Load 路径（读磁盘 → staging buffer → RDMA 传输）。
+- **true**：Get 命中 LOCAL_DISK 副本时，`TryPushPromotionQueue()` 被调用，通过 Count-Min Sketch 统计访问频率。频率达到 `promotion_admission_threshold` 的 key 被加入 promotion 队列，心跳线程将其从 SSD 提升回 MEMORY 副本。**效果**：热点数据自动回到内存，后续访问走 RDMA 零拷贝路径，避免 SSD I/O 延迟。
+
+#### `promotion_admission_threshold`（默认 2）
+
+- Count-Min Sketch 的频率阈值。key 被访问的次数（近似）达到此值才有资格 promotion。
+- **设为 1**：任何被访问一次的 LOCAL_DISK key 立即被加入 promotion 队列。适合缓存空间充裕的场景。
+- **设为更大值（如 5）**：需要多次访问才 promotion。避免一次性访问冷数据占用 promotion 资源。
+
+#### `promotion_queue_limit`（默认 50000）
+
+- 全局（所有 shard 共享）的待 promotion 任务上限。
+- **达到上限**：`TryPushPromotionQueue` 中的容量门控拒绝新任务，热 key 暂时留在 SSD。
+- 此上限同时控制 `promotion_in_flight_` 计数器，防止 promotion 占用过多内存。
+
+#### `eviction_high_watermark_ratio`（默认 0.85）
+
+- 内存使用率触发 eviction 的阈值。
+- **设高（如 0.95）**：容忍更高的内存使用率，eviction 触发更晚，留给 offload 的时间窗口更短。
+- **设低（如 0.70）**：更早触发 eviction，内存更充裕，但在 `offload_on_evict=true` 模式下会提前开始 offload。
+
+#### `eviction_ratio`（默认 0.05）
+
+- 每轮 `BatchEvict` 的目标回收比例（相对于总内存）。
+- **设大（如 0.10）**：每轮淘汰更多对象，eviction 频率更低但每轮耗时更长。
+- **设小（如 0.02）**：每轮淘汰少量对象，更平滑但 eviction 线程更频繁工作。
+
+### 8.2 Client 端环境变量
+
+#### `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH`（默认 `/data/file_storage`）
+
+- SSD 上的存储目录路径。BucketStorageBackend 在此目录下创建 `.bucket` 和 `.meta` 文件。
+- **不设置**：使用默认路径，需确保该目录存在且有写入权限。
+- **建议**：设置为 NVMe SSD 挂载点，如 `/nvme/mooncake_offload`。
+
+#### `MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR`（默认 `bucket_storage_backend`）
+
+- `bucket_storage_backend`：默认推荐。多对象合入桶文件（256MB/桶，500 key/桶），支持 FIFO/LRU 淘汰，支持重启恢复。
+- `file_per_key_storage_backend`：每个对象一个文件。适合调试，大规模场景下文件数爆炸。
+- `offset_allocator_storage_backend`：单文件 + 偏移分配器，1024 分片元数据。高并发性能好，但**不支持重启恢复**（启动时 truncates）。
+
+#### `MOONCAKE_OFFLOAD_BUCKET_SIZE_LIMIT_BYTES`（默认 256MB）
+
+- BucketStorageBackend 单个桶文件的大小上限。配置定义在 `mooncake-store/include/storage_backend.h` 的 `BucketBackendConfig::bucket_size_limit`（:181-182）。
+- **分组逻辑**（`GroupOffloadingKeysByBucket()`，`storage_backend.cpp:1880`）：心跳返回的 offload 对象按此大小打包分组。对象被依次加入当前桶，直到桶数据量达到 256MB 或 500 个 key 为止。凑不满一桶的剩余对象暂存在 `ungrouped_offloading_objects_` 中，等下次心跳凑满再写入。
+- **设小（如 64MB）**：桶更小更密集，淘汰粒度更细（LRU/FIFO 淘汰时整桶删除，浪费空间更少），但文件数量增多。
+- **设大（如 512MB）**：减少文件数，但淘汰时整桶删除可能浪费更多有效数据。
+- **注意**：单个对象大小超过此限制时会被跳过（:1911 打印 ERROR 日志）。
+
+#### `MOONCAKE_OFFLOAD_BUCKET_KEYS_LIMIT`（默认 500）
+
+- BucketStorageBackend 单个桶文件的 key 数量上限。配置定义在 `BucketBackendConfig::bucket_keys_limit`（:184）。
+- 与 `bucket_size_limit` 共同控制分组，任一条件先达到即封桶。
+
+#### `MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY`（默认 `none`）
+
+- BucketStorageBackend 的 SSD 空间淘汰策略。配置定义在 `BucketBackendConfig::eviction_policy`。
+- `none`：不淘汰。SSD 写满后 offload 失败。
+- `fifo`：淘汰最早创建的桶。
+- `lru`：淘汰最久未被读取的桶（通过 `last_access_ns_` 原子计数器追踪）。
+
+#### `MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES`（默认 1280MB）
+
+- Load 路径的 staging buffer 大小。从 SSD 读取数据时先写入此 buffer，再通过 RDMA 传输。
+- **设小**：并发 Load 能力受限，大对象可能需要排队等待 buffer 槽位。
+- **设大**：支持更多并发 Load，但占用更多 Host 内存。
+- 此 buffer 会被注册到 Transfer Engine 用于 RDMA 访问。
+
+#### `MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES`（默认 2TB）
+
+- SSD 磁盘使用上限。达到上限后 BucketStorageBackend 触发淘汰（如有 eviction policy）。
+- **设为 0**：BucketBackend 默认使用磁盘物理容量的 90%。
+
+#### `MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS`（默认 10）
+
+- 客户端心跳线程的间隔。每次心跳执行：(1) 拉取 offload 任务 (2) 执行 OffloadObjects (3) 执行 ProcessPromotionTasks。
+- **设小（如 3）**：offload/promotion 响应更快，但 Master RPC 压力增大。
+- **设大（如 30）**：减少 RPC 开销，但数据在内存中停留更久，promotion 延迟更高。
+
+#### `MOONCAKE_OFFLOAD_USE_URING`（默认 false）
+
+- **false**：使用标准 POSIX I/O（pread/pwrite）。
+- **true**：使用 Linux io_uring 异步 I/O。每个线程拥有独立的 io_uring ring（无锁），ClientBuffer 注册为 fixed buffer 避免 mmap 开销，配合 O_DIRECT 绕过页缓存。**仅 Linux 可用**。
+
+### 8.3 内部硬编码常量
+
+| 常量 | 值 | 说明 |
+|------|----|------|
+| `offloading_queue_limit_` | 50000 | 单客户端 offload 队列最大长度（`master_service.h`） |
+| `kOffloadCapRatio` | 0.5 | `offload_force_evict` 的触发阈值 = `offloading_queue_limit_ * 0.5` = 25000 |
+| `kMaxPerHeartbeat` | 1 | 每次心跳最多返回 1 个 promotion 任务，防止阻塞 |
diff --git a/docs/source/design/ssd-balance-allocation.md b/docs/source/design/ssd-balance-allocation.md
new file mode 100644
index 0000000000..5b314c8563
--- /dev/null
+++ b/docs/source/design/ssd-balance-allocation.md
@@ -0,0 +1,264 @@
+# SSD负载均衡分配策略设计文档
+
+## 1. 概述
+
+### 1.1 问题背景
+
+现有 `FreeRatioFirstAllocationStrategy` 在选择segment时只考虑DDR空闲比例，忽略了SSD水位。这导致以下问题：
+
+- 一个segment的DDR空闲但SSD已满时，数据仍被分配到该segment
+- 后续eviction时无法offload到SSD（因为SSD已满），DDR产生backpressure
+- 最终DDR被填满，整个节点无法接受新写入
+
+### 1.2 解决方案
+
+新增 `SsdBalanceAllocationStrategy`，按SSD空闲比例做负载均衡：
+
+- 默认只看SSD水位（alpha=0），优先选择SSD空闲的节点
+- SSD达到高水位时禁止向该节点写入，但不驱逐SSD数据
+- DDR达到驱逐水位时临时禁止写入，水位下降后自动恢复
+
+### 1.3 适用场景
+
+多节点集群中每个节点有DDR+本地SSD的分层存储环境。
+
+## 2. 设计目标
+
+| 目标 | 说明 |
+|------|------|
+| SSD比例均衡 | 按SSD空闲比例选择segment，优先写入SSD空闲的节点 |
+| SSD驱逐保护 | SSD达到高水位时禁止写入，绝不驱逐SSD数据（避免数据丢失） |
+| DDR准入控制 | 每个segment的DDR达到准入水位时禁止向该segment分配，自动fallback到其他segment |
+| 全满暂停 | 所有节点DDR都满时暂停所有put，返回 DDR_ADMISSION_REJECTED（-201），不触发eviction |
+
+## 3. 核心算法
+
+### 3.1 SSD比例计算
+
+```
+ssd_free_ratio = (ssd_total_capacity - ssd_used_bytes) / ssd_total_capacity
+```
+
+- 无SSD信息的segment：`ssd_free_ratio = 1.0`（不约束）
+- `ssd_used_bytes` 通过 `std::atomic` 跟踪，在offload成功时递增，磁盘驱逐时递减
+
+### 3.2 候选采样与排序
+
+```
+1. 采样 min(6 * replica_num, total_segments) 个候选segment
+2. 排除SSD使用率 >= ssd_high_watermark_ratio 的segment
+3. 按ssd_free_ratio降序排序
+4. 从top-N候选中尝试分配
+5. 如果replica_num未满足，fallback到随机分配
+```
+
+### 3.3 SSD高水位保护
+
+当segment的SSD使用率 >= `ssd_high_watermark_ratio`（默认0.90）时：
+
+- **禁止**向该segment分配新数据
+- **绝不驱逐**SSD上的已有数据（驱逐意味着数据不可恢复丢失）
+- SSD数据只能通过以下方式释放：
+ - 正常promotion（访问命中后提升回DDR）
+ - TTL过期（软pin到期后自动清理）
+- SSD水位下降后，节点自动恢复可写状态
+
+### 3.4 DDR写入准入控制（per-segment）
+
+通过 `--ddr_admission_watermark_ratio`（默认 0.0，即禁用）设定每个 segment 的 DDR 准入水位。
+当 segment 的 DDR 使用率 >= 该水位时：
+
+- 分配策略**跳过**该 segment，尝试分配到其他 segment
+- 所有 segment 都被跳过时，返回 `DDR_ADMISSION_REJECTED`（-201）
+- **不设置** `need_mem_eviction_`（避免触发 eviction 驱逐已有数据，DDR 数据零丢失）
+- 其他 segment DDR 下降（eviction 释放空间或 offload 完成）后自动恢复
+
+与 eviction 的关系：
+- `ddr_admission_watermark_ratio`（如 0.90）应设得**低于** `eviction_high_watermark_ratio`（0.95）
+- 准入阻写先于 eviction 驱逐发生，保护 DDR 数据不被驱逐
+- 如果所有 segment 都超过准入水位也无 eviction 触发，put 暂停直到有 segment 释放空间
+
+使用方式：
+
+```bash
+./mooncake_master --allocation_strategy=ssd_balance \
+ --ddr_admission_watermark_ratio=0.90
+```
+
+## 4. 决策流程
+
+### 4.1 AllocateAndInsertMetadata流程
+
+```
+AllocateAndInsertMetadata()
+│
+├── 获取AllocatorManager和SsdMetricsProvider
+│
+├── 调用 SsdBalanceAllocationStrategy::Allocate()
+│ │
+│ ├── 处理preferred segments
+│ │ ├── 检查SSD水位，跳过高水位segment
+│ │ └── 检查DDR准入水位，跳过超标segment
+│ │
+│ ├── 候选采样 + SSD比例排序
+│ │ ├── 排除excluded/used segments
+│ │ ├── 排除SSD高水位segments
+│ │ ├── 排除DDR准入水位超标的segments
+│ │ └── 按ssd_free_ratio降序排序，取top-N
+│ │
+│ ├── Fallback随机分配
+│ │ └── 同样排除SSD高水位和DDR准入超标segments
+│ │
+│ └── 返回结果
+│ ├── 有可用segment → replicas
+│ ├── 被DDR准入拒绝 → DDR_ADMISSION_REJECTED（-201）
+│ │ └── 不设need_mem_eviction_，保护DDR数据
+│ └── 其他原因失败 → NO_AVAILABLE_HANDLE（-200）
+│ └── 设need_mem_eviction_，触发eviction释放空间
+│
+└── 返回结果给客户端
+```
+
+### 4.2 SSD水位检查
+
+```
+isSsdHighWatermark(segment_name)
+│
+├── 查询SsdMetricsProvider
+│ ├── total = getSsdTotalCapacity(segment_name)
+│ └── used = getSsdUsedBytes(segment_name)
+│
+├── total <= 0?
+│ └── 返回false（无SSD信息，不阻塞）
+│
+└── used/total >= ssd_high_watermark_ratio?
+ ├── YES → 排除该segment
+ └── NO → 允许分配
+```
+
+### 4.3 DDR准入水位检查
+
+```
+isDdrHighWatermark(segment_name)
+│
+├── ddr_admission_watermark_ <= 0.0?
+│ └── 返回false（未启用DDR准入）
+│
+├── ddr_admission_watermark_ >= 1.0?
+│ └── 返回false（显式禁用）
+│
+├── 查询SsdMetricsProvider
+│ └── ratio = getDdrUsedRatio(segment_name)
+│ └── MasterMetricManager.get_segment_mem_used_ratio()
+│
+└── ratio >= ddr_admission_watermark_?
+ ├── YES → 排除该segment
+ └── NO → 允许分配
+```
+
+## 5. SSD使用量追踪
+
+### 5.1 数据结构
+
+`LocalDiskSegment` 新增字段：
+
+```cpp
+std::atomic ssd_used_bytes{0};
+```
+
+### 5.2 更新时机
+
+| 事件 | 操作 | 触发位置 |
+|------|------|----------|
+| offload成功 | `ssd_used_bytes += data_size` | `NotifyOffloadSuccess` |
+| 磁盘replica被驱逐 | `ssd_used_bytes -= object_size` | `EvictDiskReplica` |
+
+### 5.3 暴露接口
+
+通过 `SsdMetricsProvider` 接口：
+
+```cpp
+class SsdMetricsProvider {
+ virtual int64_t getSsdTotalCapacity(const std::string& segment_name) const = 0;
+ virtual int64_t getSsdUsedBytes(const std::string& segment_name) const = 0;
+ virtual double getDdrUsedRatio(const std::string& segment_name) const {
+ return 0.0; // 默认不检查DDR
+ }
+};
+```
+
+`ScopedLocalDiskSegmentAccess` 实现该接口，通过 segment_name → client_id → LocalDiskSegment 查找。
+
+## 6. 配置参数
+
+### 6.1 Master 启动参数
+
+| 参数 | 默认值 | 说明 |
+|------|--------|------|
+| `--allocation_strategy` | `random` | 设为 `ssd_balance` 启用本策略 |
+| `--ssd_high_watermark_ratio` | `0.90` | SSD使用率上限，超过则禁止向该节点写入 |
+| `--ddr_admission_watermark_ratio` | `0.0` | DDR准入水位（0.0 = 禁用），低于此值则禁止向该segment分配 |
+
+### 6.2 环境变量（存储后端驱逐保护）
+
+| 变量 | 默认值 | 说明 |
+|------|--------|------|
+| `MOONCAKE_OFFLOAD_DISABLE_SSD_EVICTION` | `false` | 强制禁止SSD驱逐，即使 eviction_policy 非 NONE 也不驱逐 |
+
+### 6.3 错误码
+
+| 错误码 | 值 | 触发条件 |
+|--------|-----|----------|
+| `NO_AVAILABLE_HANDLE` | -200 | 分配失败（段满或其他原因），触发 eviction |
+| `DDR_ADMISSION_REJECTED` | -201 | DDR准入水位拒绝分配，**不触发** eviction |
+
+启用方式：
+
+```bash
+./mooncake_master --allocation_strategy=ssd_balance \
+ --ssd_high_watermark_ratio=0.90 \
+ --ddr_admission_watermark_ratio=0.90
+```
+
+## 7. 代码结构
+
+### 7.1 新增/修改文件
+
+| 文件 | 变更类型 | 说明 |
+|------|----------|------|
+| `include/allocation_strategy.h` | 修改 | 新增 `SsdMetricsProvider` 接口（含 `getDdrUsedRatio`）、`SsdBalanceAllocationStrategy` 类（含 `isDdrHighWatermark`）、更新工厂函数 |
+| `include/types.h` | 修改 | `AllocationStrategyType` 枚举新增 `SSD_BALANCE`；新增 `DDR_ADMISSION_REJECTED` 错误码 |
+| `include/segment.h` | 修改 | `LocalDiskSegment` 新增 `ssd_used_bytes`；`ScopedLocalDiskSegmentAccess` 实现 `SsdMetricsProvider`（含 `getDdrUsedRatio`） |
+| `src/segment.cpp` | 修改 | 实现 `getSsdTotalCapacity`、`getSsdUsedBytes`、`getDdrUsedRatio` |
+| `include/master_config.h` | 修改 | 新增 `ssd_high_watermark_ratio`、`ddr_admission_watermark_ratio` 配置字段 |
+| `src/master.cpp` | 修改 | 新增 `--ssd_high_watermark_ratio`、`--ddr_admission_watermark_ratio` gflag |
+| `include/master_service.h` | 修改 | 新增 `ssd_high_watermark_ratio_` 成员 |
+| `src/master_service.cpp` | 修改 | 分配策略传 SSD/DDR provider、SSD使用量追踪、分发 DDR_ADMISSION_REJECTED（不触发 eviction） |
+| `src/client_service.cpp` | 修改 | 处理 `DDR_ADMISSION_REJECTED` 错误码（日志 + 重试） |
+| `include/storage_backend.h` | 修改 | `BucketBackendConfig` 新增 `disable_ssd_eviction` 字段 |
+| `src/storage_backend.cpp` | 修改 | `PrepareEviction` 检查 `disable_ssd_eviction`；`IsEnableOffloading` 跳过eviction分支 |
+
+### 7.2 类继承关系
+
+```
+AllocationStrategy (抽象基类)
+├── RandomAllocationStrategy
+│ └── FreeRatioFirstAllocationStrategy
+│ └── SsdBalanceAllocationStrategy ← 新增
+└── CxlAllocationStrategy
+
+SsdMetricsProvider (抽象接口)
+└── ScopedLocalDiskSegmentAccess ← 新增实现
+```
+
+## 8. 验证方案
+
+详见 `mooncake-wheel/tests/verify_ssd_balance.py` 和 `tests/ssd_balance_test_guide.md`。
+
+| 测试 | 验证内容 |
+|------|----------|
+| `load_balancing` | 2个Client不对称SSD，验证数据按SSD空闲比例分布 |
+| `ssd_high_watermark_blocking` | SSD达到90%高水位后offload完成，验证新分配被拒绝 + 初始数据可读 |
+| `ssd_eviction_protection` | 启用FIFO驱逐+`MOONCAKE_OFFLOAD_DISABLE_SSD_EVICTION=true`，验证已有SSD数据不被驱逐 |
+| `ddr_admission` | 设置 `--ddr_admission_watermark_ratio=0.90`，DDR满时拒绝写入，不触发eviction |
+| `all_ssd_full` | 所有节点SSD满后全局拒绝，释放后恢复 |
diff --git a/docs/yh/log-reference.md b/docs/yh/log-reference.md
new file mode 100644
index 0000000000..089fd1b855
--- /dev/null
+++ b/docs/yh/log-reference.md
@@ -0,0 +1,336 @@
+# Mooncake Store 日志参考手册
+
+本文档描述 `get` / `get_batch` / `put` / `put_batch` 四个操作的全链路日志输出。
+
+日志来源三个层次：
+- **Python 绑定层** — `mooncake-integration/store/store_py.cpp`
+- **核心逻辑层** — `mooncake-store/src/real_client.cpp`
+- **传输服务层** — `mooncake-store/src/client_service.cpp`
+
+---
+
+## 1. `get` 日志链路
+
+正常路径日志按调用顺序：
+
+```
+store_py::get
+ ├ get start
+ ├ real_client::get_buffer_internal
+ │ ├ query_success
+ │ ├ replica_selected
+ │ ├ [SSD 路径] ssd_read_detail
+ │ └ get_breakdown
+ ├ client_service::Get
+ │ └ transfer_read_completed
+ ├ client_service::TransferData
+ │ └ transfer_data op[READ]
+ └ get complete
+```
+
+### 1.1 Python 绑定层 — `store_py.cpp::get`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `get start` | INFO | `get start key[{key}]` | 操作开始 |
+| `get complete` | INFO | `get complete key[{key}] rc[0] size[{size}] elapsed_us[{us}]` | 操作成功完成 |
+| `get complete` | INFO | `get complete key[{key}] rc[-1] elapsed_us[{us}]` | 操作失败 |
+| `get_slow` | WARNING | `get_slow key[{key}] size[{size}] elapsed_us[{us}]` | 耗时超过 3ms 触发慢操作告警 |
+
+### 1.2 核心逻辑层 — `real_client.cpp::get_buffer_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `query_success` | INFO | `query_success key[{key}] replicas[{n}]` | Master 查询成功，返回 n 个副本 |
+| `replica_selected` | INFO | `replica_selected key[{key}] type[{type}] endpoint[{ip:port}] size[{bytes}]` | Memory/LocalDisk 副本选中，含 endpoint |
+| `replica_selected` | INFO | `replica_selected key[{key}] type[disk] file_path[{path}] size[{bytes}]` | Disk 副本选中，含文件路径 |
+| `get_breakdown` | INFO | `get_breakdown key[{key}] query_us[{t1}] select_us[{t2}] alloc_us[{t3}] read_us[{t4}] total_us[{total}] type[{type}] status[{status}]` | 分阶段耗时汇总 |
+
+**`get_breakdown` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `query_us` | Master 查询耗时（微秒） |
+| `select_us` | 副本选择耗时 |
+| `alloc_us` | 缓冲区分配耗时 |
+| `read_us` | 数据读取耗时（RDMA/文件IO/SSD RPC） |
+| `total_us` | 总耗时 |
+| `type` | 副本类型：`memory` / `local_disk` / `disk` |
+| `status` | 结果：`read_ok` / `read_fail` / `ssd_ok` / `ssd_fail` |
+
+### 1.3 传输服务层 — `client_service.cpp::Get`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `transfer_read_completed` | INFO | `transfer_read_completed key[{key}] elapsed_us[{us}] data_size[{bytes}] cache_hit[{0/1}]` | RDMA/文件传输完成 |
+| `transfer_read_failed` | ERROR | `transfer_read_failed key={key}` | 传输失败 |
+| `lease_expired_before_data_transfer_completed` | WARNING | `lease_expired_before_data_transfer_completed key={key}` | 租约过期 |
+
+### 1.4 传输引擎层 — `client_service.cpp::TransferData`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `transfer_data` | INFO | `transfer_data op[READ] submit_us[{t1}] wait_us[{t2}] result[{code}]` | 传输耗时拆分 |
+
+**字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `submit_us` | 提交传输请求耗时 |
+| `wait_us` | 等待传输完成耗时 |
+| `result` | 传输结果，`OK` 表示成功 |
+
+### 1.5 SSD Offload 路径 — `real_client.cpp::batch_get_into_offload_object_internal`
+
+仅当副本类型为 `local_disk`（远端 SSD）时触发。
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `ssd_read_detail` | INFO | `ssd_read_detail endpoint[{ip:port}] num_keys[{n}] total_size[{bytes}] elapsed_ms[{ms}] batch_id[{id}]` | SSD RPC 读取详情 |
+
+---
+
+## 2. `get_batch` 日志链路
+
+```
+store_py::get_batch
+ ├ get_batch start
+ ├ real_client::batch_get_buffer_internal
+ │ ├ batch_query_result
+ │ ├ [逐 key] replica_selected (无此日志，batch 不逐 key 输出)
+ │ ├ [SSD 路径] ssd_read_detail
+ │ └ batch_get_breakdown
+ ├ client_service::BatchGet
+ │ └ batch_get_transfer_complete
+ ├ client_service::TransferData (多次)
+ │ └ transfer_data op[READ]
+ └ get_batch complete
+```
+
+### 2.1 Python 绑定层 — `store_py.cpp::get_batch`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `get_batch start` | INFO | `get_batch start num_keys[{n}]` | 操作开始 |
+| `get_batch complete` | INFO | `get_batch complete num_keys[{n}] success[{s}] rc[0] elapsed_us[{us}]` | 操作成功完成 |
+| `get_batch complete` | INFO | `get_batch complete num_keys[{n}] rc[-1] elapsed_us[{us}]` | 操作失败 |
+| `get_batch_slow` | WARNING | `get_batch_slow num_keys[{n}] elapsed_us[{us}]` | 耗时超过 10ms 触发慢操作告警 |
+
+### 2.2 核心逻辑层 — `real_client.cpp::batch_get_buffer_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `batch_query_result` | INFO | `batch_query_result num_keys[{n}] num_found[{f}]` | 批量查询结果，f 为找到的 key 数 |
+| `batch_get_breakdown` | INFO | `batch_get_breakdown num_keys[{n}] query_us[{t1}] prep_us[{t2}] read_us[{t3}] total_us[{total}] batch_get_ops[{m}] ssd_offload_ops[{s}] success[{ok}]` | 分阶段耗时汇总 |
+
+**`batch_get_breakdown` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `query_us` | 批量 Master 查询耗时 |
+| `prep_us` | 准备阶段耗时（副本选择 + 缓冲区分配，逐 key 循环） |
+| `read_us` | 数据读取耗时（BatchGet + SSD RPC） |
+| `total_us` | 总耗时 |
+| `batch_get_ops` | 走 BatchGet 的 key 数（MEMORY + DISK 副本） |
+| `ssd_offload_ops` | 走 SSD RPC 的 key 数（LOCAL_DISK 副本） |
+| `success` | 成功读取的 key 数 |
+
+### 2.3 传输服务层 — `client_service.cpp::BatchGet`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `batch_get_transfer_complete` | INFO | `batch_get_transfer_complete num_keys[{n}] success[{s}] elapsed_us[{us}] pending_count[{c}]` | 批量传输完成 |
+
+**字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `pending_count` | 总传输任务数（提交的 TransferFuture 数量） |
+
+### 2.4 传输引擎层 — 同 `get` 的 `transfer_data`
+
+### 2.5 SSD Offload 路径 — 同 `get` 的 `ssd_read_detail`
+
+---
+
+## 3. `put` 日志链路
+
+```
+store_py::put
+ ├ put start
+ ├ real_client::put_internal
+ │ └ put_result
+ ├ client_service::Put
+ │ ├ put_start_success (或 OBJECT_ALREADY_EXISTS)
+ │ └ put_end_success
+ ├ client_service::TransferData
+ │ └ transfer_data op[WRITE]
+ └ put complete
+```
+
+### 3.1 Python 绑定层 — `store_py.cpp::put`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `put start` | INFO | `put start key[{key}] size[{bytes}]` | 操作开始 |
+| `put complete` | INFO | `put complete key[{key}] rc[{ret}] elapsed_us[{us}]` | 操作完成，rc=0 成功 |
+| `put_slow` | WARNING | `put_slow key[{key}] size[{bytes}] elapsed_us[{us}]` | 耗时超过 3ms 触发慢操作告警 |
+
+### 3.2 核心逻辑层 — `real_client.cpp::put_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `put_result` | INFO | `put_result key[{key}] rc[0] size[{bytes}]` | Put 成功 |
+| `put_result` | INFO | `put_result key[{key}] rc[{code}] size[{bytes}]` | Put 失败，code 为错误码 |
+
+### 3.3 传输服务层 — `client_service.cpp::Put`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `put_start` | INFO | `put_start key[{key}] rc[OBJECT_ALREADY_EXISTS]` | 对象已存在，直接返回成功 |
+| `put_start_success` | INFO | `put_start_success key[{key}] replicas[{n}]` | Master 分配 replica 成功 |
+| `put_end_success` | INFO | `put_end_success key[{key}] transfer_us[{us}] data_size[{bytes}]` | Put 完成，数据写入成功 |
+
+**`put_end_success` 字段说明：**
+
+| 字段 | 含义 |
+|------|------|
+| `transfer_us` | 传输阶段总耗时（含磁盘写入 + RDMA 传输） |
+| `data_size` | 写入数据大小 |
+
+### 3.4 传输引擎层 — `client_service.cpp::TransferData`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `transfer_data` | INFO | `transfer_data op[WRITE] submit_us[{t1}] wait_us[{t2}] result[{code}]` | 传输耗时拆分 |
+
+字段含义同 GET 路径的 `transfer_data`。
+
+---
+
+## 4. `put_batch` 日志链路
+
+```
+store_py::put_batch
+ ├ put_batch start
+ ├ real_client::put_batch_internal
+ │ └ batch_put_result
+ ├ client_service::BatchPut
+ │ ├ batch_put start
+ │ └ batch_put complete
+ ├ client_service::TransferData (多次)
+ │ └ transfer_data op[WRITE]
+ └ put_batch complete
+```
+
+### 4.1 Python 绑定层 — `store_py.cpp::put_batch`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `put_batch start` | INFO | `put_batch start num_keys[{n}] total_size[{bytes}]` | 操作开始 |
+| `put_batch complete` | INFO | `put_batch complete num_keys[{n}] rc[{ret}] elapsed_us[{us}]` | 操作完成，rc=0 成功 |
+| `put_batch_slow` | WARNING | `put_batch_slow num_keys[{n}] elapsed_us[{us}]` | 耗时超过 10ms 触发慢操作告警 |
+
+### 4.2 核心逻辑层 — `real_client.cpp::put_batch_internal`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `batch_put_result` | INFO | `batch_put_result num_keys[{n}] num_failed[{f}]` | 批量 Put 结果 |
+
+### 4.3 传输服务层 — `client_service.cpp::BatchPut`
+
+| 关键字 | 级别 | 格式 | 说明 |
+|--------|------|------|------|
+| `batch_put start` | INFO | `batch_put start num_keys[{n}]` | 批量 Put 传输开始 |
+| `batch_put complete` | INFO | `batch_put complete num_keys[{n}] num_failed[{f}] transfer_us[{us}] total_size[{bytes}]` | 批量 Put 完成（正常路径） |
+| `batch_put complete` | INFO | `batch_put complete num_keys[{n}] num_failed[{f}] total_size[{bytes}]` | 批量 Put 完成（prefer_same_node 路径，无 transfer_us） |
+
+### 4.4 传输引擎层 — 同 `put` 的 `transfer_data`
+
+---
+
+## 5. 附录：PerfPoint 打点与日志对照表
+
+PerfPoint 定义在 `mooncake-integration/store/mooncake_perf_points.def`。
+使用 `ubdiag show` 可查看实时性能数据，配合日志进行交叉分析。
+
+### GET 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `GET_STORE_PY_GET` | store_py.cpp::get | Get | `get start` / `get complete` |
+| `GET_BUFFER_INTERNAL` | store_py.cpp::get | GetBuffer | `get_breakdown` |
+| `GET_INTERNAL_QUERY` | real_client.cpp::get_buffer_internal | Query | `query_success` |
+| `GET_INTERNAL_SELECT_REPLICA` | real_client.cpp::get_buffer_internal | SelectReplica | `replica_selected` |
+| `GET_INTERNAL_ALLOC_BUFFER` | real_client.cpp::get_buffer_internal | AllocBuffer | `get_breakdown` alloc_us |
+| `GET_INTERNAL_SSD_READ` | real_client.cpp::get_buffer_internal | SSDRead | `ssd_read_detail` |
+| `GET_INTERNAL_MEM_READ` | real_client.cpp::get_buffer_internal | MemRead | `transfer_read_completed` |
+| `GET_INTERNAL_DISK_READ` | real_client.cpp::get_buffer_internal | DiskRead | `transfer_read_completed` |
+| `GET_SSD_OFFLOAD_RPC` | real_client.cpp::batch_get_into_offload_object_internal | OffloadRpc | `ssd_read_detail` |
+| `GET_SSD_TRANSFER_DATA` | real_client.cpp::batch_get_into_offload_object_internal | TransferData | `ssd_read_detail` |
+| `GET_SSD_RELEASE_BUFFER` | real_client.cpp::batch_get_into_offload_object_internal | ReleaseBuffer | — |
+| `GET_SINGLE_FIND_REPLICA` | client_service.cpp::Get | FindReplica | `transfer_read_completed` |
+| `GET_SINGLE_HOT_CACHE` | client_service.cpp::Get | HotCache | `transfer_read_completed` cache_hit |
+| `GET_SINGLE_TRANSFER_READ` | client_service.cpp::Get | TransferRead | `transfer_read_completed` |
+| `GET_SINGLE_RELEASE_CACHE` | client_service.cpp::Get | ReleaseCache | — |
+| `GET_SINGLE_ASYNC_CACHE` | client_service.cpp::Get | AsyncCache | — |
+| `GET_SINGLE_TRANSFER_FULL` | client_service.cpp::TransferData | TransferData | `transfer_data op[READ]` |
+| `GET_SINGLE_TRANSFER_SUBMIT` | client_service.cpp::TransferData | Submit | `transfer_data` submit_us |
+| `GET_SINGLE_TRANSFER_WAIT` | client_service.cpp::TransferData | Wait | `transfer_data` wait_us |
+
+### GET BATCH 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `GET_STORE_PY_GET_BATCH` | store_py.cpp::get_batch | GetBatch | `get_batch start` / `get_batch complete` |
+| `GET_BATCH_BUFFER_INTERNAL` | store_py.cpp::get_batch | BatchGetBuffer | `batch_get_breakdown` |
+| `GET_BATCH_INTERNAL_QUERY` | real_client.cpp::batch_get_buffer_internal | BatchQuery | `batch_query_result` |
+| `GET_BATCH_INTERNAL_PREPARATION` | real_client.cpp::batch_get_buffer_internal | Preparation | `batch_get_breakdown` prep_us |
+| `GET_BATCH_INTERNAL_SELECT_REPLICA` | real_client.cpp::batch_get_buffer_internal | SelectReplica | — |
+| `GET_BATCH_INTERNAL_ALLOC_BUFFER` | real_client.cpp::batch_get_buffer_internal | AllocBuffer | — |
+| `GET_BATCH_INTERNAL_SSD_READ` | real_client.cpp::batch_get_buffer_internal | SSDRead | `ssd_read_detail` |
+| `GET_BATCH_INTERNAL_MEMDISH_READ` | real_client.cpp::batch_get_buffer_internal | MemDiskRead | `batch_get_transfer_complete` |
+| `GET_BATCH_FIND_REPLICA` | client_service.cpp::BatchGet | FindReplica | — |
+| `GET_BATCH_HOT_CACHE` | client_service.cpp::BatchGet | HotCache | — |
+| `GET_BATCH_SUBMIT` | client_service.cpp::BatchGet | Submit | — |
+| `GET_BATCH_WAIT` | client_service.cpp::BatchGet | Wait | — |
+| `GET_BATCH_RELEASE_CACHE` | client_service.cpp::BatchGet | ReleaseCache | — |
+| `GET_BATCH_ASYNC_CACHE` | client_service.cpp::BatchGet | AsyncCache | — |
+
+### PUT 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `PUT_STORE_PY_PUT` | store_py.cpp::put | Put | `put start` / `put complete` |
+| `PUT_INTERNAL_FULL` | store_py.cpp::put | PutBuffer | `put_result` |
+| `PUT_INTERNAL_ALLOC_BUFFER` | real_client.cpp::put_internal | AllocBuffer | — |
+| `PUT_INTERNAL_MEM_COPY` | real_client.cpp::put_internal | MemCopy | — |
+| `PUT_INTERNAL_SPLIT_SLICES` | real_client.cpp::put_internal | SplitSlices | — |
+| `PUT_SINGLE_FULL` | client_service.cpp::Put | TransferPut | `put_end_success` |
+| `PUT_SINGLE_PUT_START` | client_service.cpp::Put | PutStart | `put_start_success` |
+| `PUT_SINGLE_DISK_WRITE` | client_service.cpp::Put | DiskWrite | `put_end_success` |
+| `PUT_SINGLE_TRANSFER_WRITE` | client_service.cpp::Put | TransferWrite | `put_end_success` |
+| `PUT_SINGLE_PUT_END` | client_service.cpp::Put | PutEnd | `put_end_success` |
+| `PUT_SINGLE_PUT_REVOKE` | client_service.cpp::Put | PutRevoke | — |
+| `PUT_SINGLE_TRANSFER_FULL` | client_service.cpp::TransferData | TransferData | `transfer_data op[WRITE]` |
+| `PUT_SINGLE_TRANSFER_SUBMIT` | client_service.cpp::TransferData | Submit | `transfer_data` submit_us |
+| `PUT_SINGLE_TRANSFER_WAIT` | client_service.cpp::TransferData | Wait | `transfer_data` wait_us |
+
+### PUT BATCH 侧
+
+| PerfPoint 名称 | 定义位置 | 标签 | 对应日志关键字 |
+|----------------|---------|------|---------------|
+| `PUT_STORE_PY_PUT_BATCH` | store_py.cpp::put_batch | PutBatch | `put_batch start` / `put_batch complete` |
+| `PUT_BATCH_INTERNAL_FULL` | store_py.cpp::put_batch | BatchPutBuffer | `batch_put_result` |
+| `PUT_BATCH_INTERNAL_ALLOC_BUFFER` | real_client.cpp::put_batch_internal | AllocBuffer | — |
+| `PUT_BATCH_INTERNAL_MEM_COPY` | real_client.cpp::put_batch_internal | MemCopy | — |
+| `PUT_BATCH_INTERNAL_SPLIT_SLICES` | real_client.cpp::put_batch_internal | SplitSlices | — |
+| `PUT_BATCH_FULL` | client_service.cpp::BatchPut | TransferBatchPut | `batch_put complete` |
+| `PUT_BATCH_CREATE_OPS` | client_service.cpp::BatchPut | CreateOps | — |
+| `PUT_BATCH_PUT_START` | client_service.cpp::StartBatchPut | PutStart | — |
+| `PUT_BATCH_SUBMIT` | client_service.cpp::SubmitTransfers | Submit | — |
+| `PUT_BATCH_DISK_WRITE` | client_service.cpp::SubmitTransfers | DiskWrite | — |
+| `PUT_BATCH_WAIT` | client_service.cpp::WaitForTransfers | Wait | — |
+| `PUT_BATCH_PUT_END` | client_service.cpp::FinalizeBatchPut | PutEnd | — |
+| `PUT_BATCH_PUT_REVOKE` | client_service.cpp::FinalizeBatchPut | PutRevoke | — |
+| `PUT_BATCH_COLLECT_RESULTS` | client_service.cpp::BatchPut | CollectResults | — |
diff --git a/docs/yh/pipline.md b/docs/yh/pipline.md
new file mode 100644
index 0000000000..d5f7dd05aa
--- /dev/null
+++ b/docs/yh/pipline.md
@@ -0,0 +1,192 @@
+## 打点流程图
+
+### `get` 流程
+
+```mermaid
+flowchart TB
+ Start["store_py::get(key)
Python入口，释放GIL，调用get_buffer，返回结果
🔑 store_py.cpp::get/Get"] --> GetBufferCall["store_->get_buffer(key)
🔑 store_py.cpp::get/GetBuffer"]
+
+ GetBufferCall --> Internal["get_buffer_internal(key, allocator)
核心逻辑：查询→选副本→分配→读取
"]
+
+ Internal --> QueryPart["部分1: client_->Query(key)
向Master查询对象副本元数据
🔑 real_client.cpp::get_buffer_internal/Query"]
+ QueryPart --> SelectPart["部分2: SelectBestReplica
从副本列表中选择最优副本
🔑 real_client.cpp::get_buffer_internal/SelectReplica"]
+ SelectPart --> AllocPart["部分3: allocator->allocate
分配本地缓冲区
🔑 real_client.cpp::get_buffer_internal/AllocBuffer"]
+
+ AllocPart --> CheckDisk{is_local_disk_replica?}
+
+ CheckDisk -->|Yes| SSDPart["部分4a: batch_get_into_offload_object_internal
通过RPC从远端SSD读取数据
🔑 real_client.cpp::get_buffer_internal/SSDRead"]
+ SSDPart --> SSDRpc["步骤1: batch_get_offload_object()
RPC到远端节点，远端从SSD读数据到buffer
🔑 real_client.cpp::batch_get_into_offload_object_internal/OffloadRpc"]
+ SSDRpc --> SSDTransfer["步骤2: BatchGetOffloadObject()
Transfer Engine零拷贝搬数据到本地
🔑 real_client.cpp::batch_get_into_offload_object_internal/TransferData"]
+ SSDTransfer --> SSDRelease["步骤3: release_offload_buffer()
通知远端释放buffer(fire-and-forget)
🔑 real_client.cpp::batch_get_into_offload_object_internal/ReleaseBuffer"]
+ SSDRelease --> Done["返回"]
+
+ CheckDisk -->|No| ReadType{is_memory_replica?}
+
+ ReadType -->|Yes| MemRead["部分4b-Memory: client_->Get(key, filtered_qr, slices)
内存副本RDMA读取
🔑 real_client.cpp::get_buffer_internal/MemRead"]
+ ReadType -->|No| DiskRead["部分4b-Disk: client_->Get(key, filtered_qr, slices)
磁盘副本文件I/O读取
🔑 real_client.cpp::get_buffer_internal/DiskRead"]
+
+ MemRead --> ClientGetSub["Client::Get内部子步骤
"]
+ DiskRead --> ClientGetSub
+
+ ClientGetSub --> FindReplica["子步骤1: FindFirstCompleteReplica
🔑 client_service.cpp::Get/FindReplica"]
+ FindReplica --> HotCache["子步骤2: RedirectToHotCache
🔑 client_service.cpp::Get/HotCache"]
+ HotCache --> TransferRead["子步骤3: TransferRead → TransferData
🔑 client_service.cpp::Get/TransferRead"]
+ TransferRead --> TransferDetail["TransferData内部
🔑 client_service.cpp::TransferData/TransferData
├ submit → client_service.cpp::TransferData/Submit
└ future.get() → client_service.cpp::TransferData/Wait"]
+ TransferDetail --> ReleaseCache["子步骤4: ReleaseHotKey
🔑 client_service.cpp::Get/ReleaseCache"]
+ ReleaseCache --> AsyncUpdate["子步骤5: ProcessSlicesAsync
🔑 client_service.cpp::Get/AsyncCache"]
+ AsyncUpdate --> Done
+
+ style Start fill:#e8f5e9
+ style GetBufferCall fill:#c8e6c9
+ style Internal fill:#e3f2fd
+ style QueryPart fill:#fff3e0
+ style SelectPart fill:#fff3e0
+ style AllocPart fill:#fff3e0
+ style SSDPart fill:#fce4ec
+ style SSDRpc fill:#fce4ec
+ style SSDTransfer fill:#fce4ec
+ style SSDRelease fill:#fce4ec
+ style MemRead fill:#bbdefb
+ style DiskRead fill:#ffccbc
+ style ClientGetSub fill:#e3f2fd
+ style FindReplica fill:#f3e5f5
+ style HotCache fill:#f3e5f5
+ style TransferRead fill:#f3e5f5
+ style TransferDetail fill:#e0f2f1
+ style ReleaseCache fill:#f3e5f5
+ style AsyncUpdate fill:#f3e5f5
+```
+
+### `get_batch` 流程
+
+```mermaid
+flowchart TB
+ Start["store_py::get_batch(keys)
Python入口，释放GIL，调用batch_get_buffer，返回结果
🔑 store_py.cpp::get_batch/GetBatch"] --> BatchGetBufferCall["store_->batch_get_buffer(keys)
🔑 store_py.cpp::get_batch/BatchGetBuffer"]
+
+ BatchGetBufferCall --> Internal["batch_get_buffer_internal(keys, allocator)
核心逻辑：批量查询→选副本→分配→读取
"]
+
+ Internal --> QueryPart["部分1: client_->BatchQuery(keys)
批量向Master查询副本元数据
🔑 real_client.cpp::batch_get_buffer_internal/BatchQuery"]
+ QueryPart --> LoopPart["部分2: 循环逐key处理
├ SelectBestReplica → real_client.cpp::batch_get_buffer_internal/SelectReplica
└ allocator->allocate → real_client.cpp::batch_get_buffer_internal/AllocBuffer"]
+
+ LoopPart --> CheckDisk{有 LOCAL_DISK 副本?}
+
+ CheckDisk -->|Yes| SSDPart["部分3a: batch_get_into_offload_object_internal
🔑 real_client.cpp::batch_get_buffer_internal/SSDRead"]
+ SSDPart --> SSDRpc["步骤1: batch_get_offload_object()
🔑 real_client.cpp::batch_get_into_offload_object_internal/OffloadRpc"]
+ SSDRpc --> SSDTransfer["步骤2: BatchGetOffloadObject()
🔑 real_client.cpp::batch_get_into_offload_object_internal/TransferData"]
+ SSDTransfer --> SSDRelease["步骤3: release_offload_buffer()
🔑 real_client.cpp::batch_get_into_offload_object_internal/ReleaseBuffer"]
+
+ CheckDisk -->|No| MemDiskRead["部分3b: client_->BatchGet(keys, query_results, slices)
批量读取内存/磁盘副本
🔑 real_client.cpp::batch_get_buffer_internal/MemDiskRead"]
+
+ MemDiskRead --> BatchGetSub["Client::BatchGet内部子步骤
"]
+
+ BatchGetSub --> SubmitLoop["提交阶段 [循环]
├ FindFirstCompleteReplica → client_service.cpp::BatchGet/FindReplica
├ RedirectToHotCache → client_service.cpp::BatchGet/HotCache
└ submit → client_service.cpp::BatchGet/Submit"]
+ SubmitLoop --> WaitLoop["等待阶段 [循环]
├ future.get() → client_service.cpp::BatchGet/Wait
├ ReleaseHotKey → client_service.cpp::BatchGet/ReleaseCache
└ ProcessSlicesAsync → client_service.cpp::BatchGet/AsyncCache"]
+
+ SSDRelease --> Done["返回"]
+ WaitLoop --> Done
+
+ style Start fill:#e8f5e9
+ style BatchGetBufferCall fill:#c8e6c9
+ style Internal fill:#e3f2fd
+ style QueryPart fill:#fff3e0
+ style LoopPart fill:#fff3e0
+ style SSDPart fill:#fce4ec
+ style SSDRpc fill:#fce4ec
+ style SSDTransfer fill:#fce4ec
+ style SSDRelease fill:#fce4ec
+ style MemDiskRead fill:#bbdefb
+ style BatchGetSub fill:#e3f2fd
+ style SubmitLoop fill:#f3e5f5
+ style WaitLoop fill:#f3e5f5
+```
+
+### `put` 流程
+
+```mermaid
+flowchart TB
+ Start["store_py::put(key, value)
Python入口，释放GIL，调用store_->put()
🔑 store_py.cpp::put/Put"] --> PutCall["store_->put(key, value, config)
🔑 store_py.cpp::put/PutBuffer"]
+
+ PutCall --> Internal["put_internal(key, value, config, allocator)
核心逻辑：分配→拷贝→切分→写入
"]
+
+ Internal --> AllocPart["部分1: allocator->allocate
分配本地缓冲区(RDMA注册内存)
🔑 real_client.cpp::put_internal/AllocBuffer"]
+ AllocPart --> CopyPart["部分2: memcpy
将用户数据拷贝到分配的缓冲区
🔑 real_client.cpp::put_internal/MemCopy"]
+ CopyPart --> SplitPart["部分3: split_into_slices
按kMaxSliceSize切分为多个Slice
🔑 real_client.cpp::put_internal/SplitSlices"]
+ SplitPart --> ClientPutPart["部分4: client_->Put(key, slices, config)
🔑 client_service.cpp::Put/TransferPut"]
+
+ ClientPutPart --> PutStart["子步骤1: master_client_.PutStart(key)
向Master申请分配replica handle
若返回OBJECT_ALREADY_EXISTS则直接返回成功
🔑 client_service.cpp::Put/PutStart"]
+
+ PutStart --> CheckDisk{storage_backend_存在
且有磁盘副本?}
+
+ CheckDisk -->|Yes| DiskWrite["子步骤2a: PutToLocalFile(key, slices, disk_descriptor)
将数据写入本地磁盘(仅处理一个磁盘副本)
🔑 client_service.cpp::Put/DiskWrite"]
+
+ CheckDisk -->|No| MemReplicaLoop["子步骤2b: 遍历所有内存副本
对每个内存副本调用TransferWrite"]
+
+ DiskWrite --> MemReplicaLoop
+
+ MemReplicaLoop --> TransferWrite["TransferWrite → TransferData(replica, slices, WRITE)
🔑 client_service.cpp::Put/TransferWrite"]
+
+ TransferWrite --> TransferDetail["TransferData内部
🔑 client_service.cpp::TransferData/TransferData
├ transfer_submitter_->submit() → client_service.cpp::TransferData/Submit
└ future->get() 阻塞等待传输完成 → client_service.cpp::TransferData/Wait"]
+
+ TransferDetail --> CheckTransfer{传输是否成功?}
+
+ CheckTransfer -->|失败| PutRevoke["子步骤3a: master_client_.PutRevoke(key, MEMORY)
撤销本次Put操作，释放已分配的replica
🔑 client_service.cpp::Put/PutRevoke"]
+
+ CheckTransfer -->|成功| PutEnd["子步骤3b: master_client_.PutEnd(key, MEMORY)
确认Put完成，replica正式生效
🔑 client_service.cpp::Put/PutEnd"]
+
+ PutRevoke --> Done["返回"]
+ PutEnd --> Done
+
+ style Start fill:#e8f5e9
+ style PutCall fill:#c8e6c9
+ style Internal fill:#e3f2fd
+ style AllocPart fill:#fff3e0
+ style CopyPart fill:#fff3e0
+ style SplitPart fill:#fff3e0
+ style ClientPutPart fill:#bbdefb
+ style PutStart fill:#f3e5f5
+ style DiskWrite fill:#fce4ec
+ style MemReplicaLoop fill:#f3e5f5
+ style TransferWrite fill:#f3e5f5
+ style TransferDetail fill:#e0f2f1
+ style PutRevoke fill:#ffcdd2
+ style PutEnd fill:#c8e6c9
+```
+
+### `put_batch` 流程
+
+```mermaid
+flowchart TB
+ Start["store_py::put_batch(keys, values)
Python入口，释放GIL，调用store_->put_batch()
🔑 store_py.cpp::put_batch/PutBatch"] --> PutBatchCall["store_->put_batch(keys, values, config)
🔑 store_py.cpp::put_batch/BatchPutBuffer"]
+
+ PutBatchCall --> Internal["put_batch_internal(keys, values, config, allocator)
核心逻辑：逐key分配→拷贝→切分→批量写入
"]
+
+ Internal --> LoopPart["部分1: 循环逐key处理
对每个key执行以下3步:
├ allocator->allocate → 🔑 real_client.cpp::put_batch_internal/AllocBuffer
│ (分配本地缓冲区，RDMA注册内存)
├ memcpy → 🔑 real_client.cpp::put_batch_internal/MemCopy
│ (将用户数据拷贝到分配的缓冲区)
└ split_into_slices → 🔑 real_client.cpp::put_batch_internal/SplitSlices
(按kMaxSliceSize切分为多个Slice)"]
+
+ LoopPart --> BatchPutPart["部分2: client_->BatchPut(keys, batched_slices, config)
🔑 client_service.cpp::BatchPut/TransferBatchPut"]
+
+ BatchPutPart --> CreateOps["子步骤1: CreatePutOperations(keys, batched_slices)
为每个key创建PutOperation对象，包含key和对应的slices
🔑 client_service.cpp::BatchPut/CreateOps"]
+
+ CreateOps --> StartBatch["子步骤2: StartBatchPut(ops, config)
调用master_client_.BatchPutStart(keys, slice_lengths, config)
Master为每个key分配replica handle，返回到op.replicas中
分配失败的op标记错误，后续步骤跳过
🔑 client_service.cpp::StartBatchPut/PutStart"]
+
+ StartBatch --> SubmitPhase["子步骤3: SubmitTransfers(ops)
对每个未失败的op，逐个提交传输任务:
├ 若storage_backend_存在且有磁盘副本:
│ 调用PutToLocalFile写入本地磁盘 → 🔑 client_service.cpp::SubmitTransfers/DiskWrite
├ 遍历op中所有内存副本:
│ 调用transfer_submitter_->submit(replica, slices, WRITE)
│ 返回TransferFuture存入op.pending_transfers → 🔑 client_service.cpp::SubmitTransfers/Submit
└ 若任一replica提交失败，标记op错误，清空pending_transfers"]
+
+ SubmitPhase --> WaitPhase["子步骤4: WaitForTransfers(ops)
对每个有pending_transfers的op:
├ 遍历所有TransferFuture，调用future.get()阻塞等待传输完成 → 🔑 client_service.cpp::WaitForTransfers/Wait
└ 若任一传输失败，记录首个错误，标记op失败"]
+
+ WaitPhase --> Finalize["子步骤5: FinalizeBatchPut(ops)
根据每个op的结果分类处理:
├ 传输成功的op: 调用master_client_.BatchPutEnd(keys) → 🔑 client_service.cpp::FinalizeBatchPut/PutEnd
│ 确认Put完成，replica正式生效，标记op成功
├ 传输失败但已分配replica的op: 调用master_client_.BatchPutRevoke(keys) → 🔑 client_service.cpp::FinalizeBatchPut/PutRevoke
│ 撤销Put操作，释放已分配的replica
└ 未分配replica的op(早期失败): 无需清理"]
+
+ Finalize --> CollectResults["子步骤6: CollectResults(ops)
从每个PutOperation中收集结果
OBJECT_ALREADY_EXISTS视为成功
🔑 client_service.cpp::BatchPut/CollectResults"]
+
+ CollectResults --> Done["返回"]
+
+ style Start fill:#e8f5e9
+ style PutBatchCall fill:#c8e6c9
+ style Internal fill:#e3f2fd
+ style LoopPart fill:#fff3e0
+ style BatchPutPart fill:#bbdefb
+ style CreateOps fill:#f3e5f5
+ style StartBatch fill:#f3e5f5
+ style SubmitPhase fill:#f3e5f5
+ style WaitPhase fill:#f3e5f5
+ style Finalize fill:#f3e5f5
+ style CollectResults fill:#f3e5f5
+```
\ No newline at end of file
diff --git a/docs/yh/put_get_logic.md b/docs/yh/put_get_logic.md
new file mode 100644
index 0000000000..12b836eca3
--- /dev/null
+++ b/docs/yh/put_get_logic.md
@@ -0,0 +1,649 @@
+# Put/Get 底层逻辑详解
+
+---
+
+## 第一部分：Put/PutBatch
+
+### Put 的完整底层逻辑（单 key）
+
+#### 第1层：Python 绑定（store_py.cpp L2531）
+
+```
+store.put(key, value, config)
+```
+1. 将 Python buffer 转为 C++ `std::span`
+2. **释放 GIL**（`py::gil_scoped_release`）
+3. 调用 `store_->put(key, span, config)`
+
+#### 第2层：RealClient（real_client.cpp L1584 → L1535）
+
+`put()` 调用 `put_internal()`，逻辑如下：
+
+1. **参数校验**：检查 client 是否初始化、allocator 是否存在
+2. **分配缓冲区**：`client_buffer_allocator_->allocate(value.size_bytes())`
+ - 为什么需要分配？因为用户的数据可能在任意内存位置，而 RDMA 传输要求内存必须是"注册过的"（MR，Memory Region）。`client_buffer_allocator_` 分配的就是已注册的 RDMA 内存
+3. **拷贝数据**：`memcpy(buffer_handle.ptr(), value.data(), value.size_bytes())`
+ - 把用户数据从普通内存拷贝到 RDMA 注册内存
+4. **切分 Slice**：`split_into_slices(buffer_handle)`
+ - 如果数据量大于 `kMaxSliceSize`（默认 256KB），需要切成多个 Slice。每个 Slice 是一段连续内存的描述符（指针+长度），对应一次 RDMA 写操作
+5. 调用 `client_->Put(key, slices, config)` 进入第3层
+
+#### 第3层：Client 服务层（client_service.cpp L1237）
+
+`Client::Put()` 逻辑如下：
+
+1. **PutStart**：`master_client_.PutStart(key, slice_lengths, config)`
+ - 向 Master 发送 RPC，申请为这个 key 分配 replica
+ - Master 选择目标 segment，返回 replica 描述符列表（每个描述符包含目标内存地址、大小、传输端点等）
+ - 如果 key 已存在，返回 `OBJECT_ALREADY_EXISTS`，直接返回成功
+ - 如果没有可用空间，返回 `NO_AVAILABLE_HANDLE`
+
+2. **处理磁盘副本**（如果有的话）：
+ - **逆序遍历** replica 列表（`rbegin/rend`），找磁盘类型的副本
+ - 调用 `PutToLocalFile(key, slices, disk_descriptor)` 写本地磁盘
+ - **只处理一个磁盘副本就 break**
+
+ > **为什么只处理一个磁盘副本？** 因为磁盘副本是写入本地存储后端的，一个 Client 只有一个 `storage_backend_`，即使 Master 分配了多个磁盘副本描述符，当前 Client 只能写入自己本地的磁盘，无需重复写入。注释也明确说明：`// Only one disk replica is needed`。
+ >
+ > **那其他磁盘副本怎么办？** Master 分配的多个磁盘副本描述符指向不同节点的磁盘。当前 Client 只负责写入自己本地的那个磁盘副本（通过 `storage_backend_`），其他节点的磁盘副本由 Master 协调其他节点来写入，或者由 Master 在后续的 rebalance 流程中补齐。Client 的职责就是：本地有 `storage_backend_` 就写一个本地磁盘副本，其余的不管。
+ >
+ > **为什么必须先处理磁盘副本？** 因为 `PutToLocalFile` 是异步的——数据拼接在调用线程同步完成，但实际磁盘 I/O 和 `PutEnd(DISK)`/`PutRevoke(DISK)` 在 `write_thread_pool_` 中异步执行。先启动磁盘写入，可以尽早触发异步的 `PutEnd(DISK)`，避免与后续内存副本的 `PutEnd(MEMORY)` 产生竞态。
+
+3. **处理内存副本**：
+ - 遍历 replica 列表，找内存类型的副本
+ - 对**每个**内存副本调用 `TransferWrite(replica, slices)`
+
+4. **TransferWrite → TransferData**：
+ - `transfer_submitter_->submit(replica, slices, WRITE)` — 提交异步 RDMA/TCP 写传输
+ - 返回 `TransferFuture`
+ - `future->get()` — 阻塞等待传输完成
+ - 传输层根据协议选择策略：RDMA 直接写远端内存，TCP 通过 socket 传输
+
+5. **传输结果处理**：
+ - 成功：`master_client_.PutEnd(key, MEMORY)` — 通知 Master 本次 Put 完成，replica 正式生效
+ - 失败：`master_client_.PutRevoke(key, MEMORY)` — 通知 Master 撤销本次 Put，释放已分配的 replica
+
+---
+
+### PutBatch 的完整底层逻辑（批量 key）
+
+#### 第1层：Python 绑定（store_py.cpp L2574）
+
+```
+store.put_batch(keys, values, config)
+```
+1. 将所有 Python buffer 转为 `std::vector>`
+2. **释放 GIL**
+3. 调用 `store_->put_batch(keys, spans, config)`
+
+#### 第2层：RealClient（real_client.cpp L1682 → L1599）
+
+`put_batch()` 调用 `put_batch_internal()`，逻辑如下：
+
+1. **参数校验**：检查 keys 和 values 大小是否匹配
+2. **循环逐 key 处理**（串行）：
+ - 对每个 key-value 对：
+ - `allocator->allocate(value.size_bytes())` — 分配 RDMA 注册内存
+ - `memcpy(buffer_handle.ptr(), value.data(), value.size_bytes())` — 拷贝数据
+ - `split_into_slices(buffer_handle)` — 切分 Slice
+ - 将所有 key 的 slices 收集到 `batched_slices` map 中
+3. 调用 `client_->BatchPut(keys, ordered_batched_slices, config)` 进入第3层
+
+#### 第3层：Client 服务层（client_service.cpp L2055）
+
+`Client::BatchPut()` 逻辑如下，分为 **6 个阶段**：
+
+1. **CreatePutOperations**：为每个 key 创建 `PutOperation` 对象，包含 key 和对应的 slices
+
+2. **StartBatchPut**：
+ - 调用 `master_client_.BatchPutStart(keys, slice_lengths, config)`
+ - 一次 RPC 批量为所有 key 分配 replica
+ - 每个返回结果可能是成功（带 replica 列表）或失败
+ - 失败的 op 标记错误，后续阶段跳过
+
+3. **SubmitTransfers**（提交阶段）：
+ - 对每个未失败的 op：
+ - 如果有磁盘副本：**逆序遍历找磁盘副本 → `PutToLocalFile()` 写磁盘 → 只处理一个就 break**（与 Put 相同逻辑）
+ - 对每个内存副本：`transfer_submitter_->submit(replica, slices, WRITE)` 提交异步传输
+ - 返回的 `TransferFuture` 存入 `op.pending_transfers`
+ - 如果任一 replica 提交失败，标记 op 错误，清空 pending_transfers
+
+4. **WaitForTransfers**（等待阶段）：
+ - 对每个有 pending_transfers 的 op：
+ - 遍历所有 `TransferFuture`，调用 `future.get()` 阻塞等待
+ - 如果任一传输失败，记录首个错误，标记 op 失败
+ - 注意：即使有失败，也会等待所有 future 完成，避免资源泄漏
+
+5. **FinalizeBatchPut**（收尾阶段）：
+ - 将 op 分为三类：
+ - **传输成功的 op**：调用 `master_client_.BatchPutEnd(successful_keys)` — 批量确认，replica 正式生效
+ - **传输失败但已分配 replica 的 op**：调用 `master_client_.BatchPutRevoke(failed_keys)` — 批量撤销，释放 replica
+ - **早期失败的 op**（未分配 replica）：无需清理
+
+6. **CollectResults**：
+ - 从每个 PutOperation 收集结果
+ - `OBJECT_ALREADY_EXISTS` 视为成功
+ - 返回 `vector>`
+
+---
+
+### Put 关键区别总结
+
+| 维度 | Put（单key） | PutBatch（批量） |
+|------|-------------|----------------|
+| Master RPC | 1次 PutStart + 1次 PutEnd/Revoke | 1次 BatchPutStart + 1次 BatchPutEnd + 1次 BatchPutRevoke |
+| 传输方式 | 同步：submit → 立即 get() 等待 | 异步批量：先 submit 所有 → 再统一 wait 所有 |
+| 失败处理 | 传输失败立即 PutRevoke | 传输失败后统一在 FinalizeBatchPut 中 BatchPutRevoke |
+| 磁盘副本 | 只处理1个本地磁盘副本 | 每个op只处理1个本地磁盘副本 |
+| 数据拷贝 | 1次 memcpy + split | N次 memcpy + split（逐key串行） |
+
+---
+
+### 拷贝路径 vs 零拷贝路径
+
+| API | 是否 memcpy | buffer 来源 | 底层调用 |
+|-----|-----------|------------|---------|
+| `put` / `put_parts` / `put_batch` | 是 | `client_buffer_allocator_->allocate()` 分配 | `client_->Put()` / `client_->BatchPut()` |
+| `put_from` / `batch_put_from` / `batch_put_from_multi_buffers` | 否（零拷贝） | 用户提供的外部 buffer 指针 | `client_->Put()` / `client_->BatchPut()` |
+
+拷贝路径中，`client_buffer_allocator_` 分配的 buffer 是注册过 RDMA 的内存区域，`memcpy` 将用户数据拷贝进去后才能进行零拷贝 RDMA 传输。而 `put_from`/`batch_put_from` 系列要求用户提前通过 `register_buffer()` 注册内存，从而跳过 memcpy 步骤。
+
+---
+
+### 零拷贝路径详解：put_from / batch_put_from / batch_put_from_multi_buffers
+
+#### 三者的差异
+
+| API | 每个 key 对应的 buffer | Slice 构造方式 | 底层调用 |
+|-----|----------------------|--------------|---------|
+| `put_from` | 1个连续 buffer | 按 `kMaxSliceSize` 手动切片 | `client_->Put()` |
+| `batch_put_from` | 1个连续 buffer | 按 `kMaxSliceSize` 手动切片 | `client_->BatchPut()` |
+| `batch_put_from_multi_buffers` | 多个不连续 buffer | 每个 buffer 直接作为一个 Slice | `client_->BatchPut()` |
+
+**除了 buffer 来源和 Slice 构造方式不同，三者进入 `client_->Put()` / `client_->BatchPut()` 之后的流程完全相同**——都要经历 PutStart → 磁盘副本处理 → 内存副本传输 → PutEnd/Revoke 的完整流程。
+
+`batch_put_from_multi_buffers` 的典型场景：一个对象的数据分散在多个不连续的 GPU 内存区域中（例如 vLLM 中 KV cache 的多个 layer tensor），每个区域单独注册，然后直接作为 Slice 传入，无需先拼接成连续内存。
+
+#### register_buffer 的实现与收益
+
+`register_buffer_internal` 的调用链：
+
+```
+RealClient::register_buffer_internal(buffer, size)
+ → Client::RegisterLocalMemory(buffer, size, location, ...)
+ → TransferEngine::registerLocalMemory(buffer, size, ...)
+ → RdmaTransport::registerLocalMemoryInternal(buffer, size, ...)
+ → RdmaContext::registerMemoryRegion(addr, length, access)
+ → ibv_reg_mr(pd_, addr, length, access) // CPU 内存
+ → ibv_reg_dmabuf_mr(pd_, ..., dmabuf_fd, access) // GPU 内存
+```
+
+**RDMA 内存注册做了什么？**
+
+1. **CPU 内存**：调用 `ibv_reg_mr()` 将内存页锁定（pin），并注册到 RDMA 保护域（Protection Domain），获取 `lkey`/`rkey`。锁定后，RDMA NIC 可以直接通过 DMA 访问这些内存，无需 CPU 介入。
+2. **GPU 内存**：通过 CUDA 的 DMA-BUF 机制获取文件描述符，再调用 `ibv_reg_dmabuf_mr()` 注册，RDMA NIC 可以直接从 GPU 显存读取数据并发送到远端，无需先拷贝到 CPU。
+
+**register_buffer 的核心收益：**
+
+1. **消除数据拷贝**：`put_internal` 需要将数据 memcpy 到共享内存池的内部缓冲区，而 `put_from_internal` 直接从已注册的用户缓冲区创建 Slice，省去了这次拷贝。对于大块数据（如 GPU 上的 KV cache tensor），可以显著降低延迟和 CPU 开销。
+
+2. **RDMA 零拷贝传输**：`ibv_reg_mr()` 将内存页锁定（pin），RDMA NIC 可以直接通过 DMA 访问这些内存，无需 CPU 介入。未注册的内存无法被 RDMA NIC 直接访问。
+
+3. **GPU 内存直传**：通过 `ibv_reg_dmabuf_mr()` 注册 GPU 内存，RDMA NIC 可以直接从 GPU 显存读取数据并发送到远端，无需先拷贝到 CPU 再发送。
+
+4. **一次注册，多次使用**：`register_buffer` 通常在初始化阶段调用一次（例如 vLLM 启动时注册整个 KV cache 内存池），之后所有 `put_from`/`get_into` 操作都可以零拷贝地复用该注册。`ibv_reg_mr` 本身是一个昂贵的操作（涉及页表锁定和 NIC 映射），避免每次操作都重新注册是关键优化。
+
+5. **子区域支持**：通过 `resolve_registered_buffer`，注册一个大区域后，可以对其中的任意子区域进行零拷贝操作，灵活支持 tensor 切片等场景。
+
+---
+
+### PutToLocalFile：磁盘副本写入机制
+
+`PutToLocalFile`（client_service.cpp L2593）的完整流程：
+
+**阶段1：同步数据暂存（调用线程）**
+
+1. 遍历所有 Slice，计算总大小
+2. 对每个 Slice：
+ - 如果是 GPU 指针（`IsDevicePointer` 检测）→ 通过 Pinned Buffer Pool 做 D2H（Device-to-Host）拷贝，暂存到 `std::string`
+ - 如果是普通主机内存指针 → 直接 append 到 `std::string`
+3. 这一步必须在调用线程同步完成，因为 BatchPut 还未返回给 Python，GPU buffer 不会被复用
+
+**阶段2：异步磁盘写入（write_thread_pool_）**
+
+1. `storage_backend_->StoreObject(path, value, key)` 写入磁盘文件
+2. 写入成功 → `master_client_.PutEnd(key, DISK)` + 处理驱逐通知
+3. 写入失败 → `master_client_.PutRevoke(key, DISK)`
+
+**磁盘写入没有使用 RDMA 或高速网络通道**。内存副本通过 `TransferWrite` → `transfer_submitter_` 走传输引擎（可能使用 RDMA/TCP），但磁盘副本走的是纯本地文件 I/O 路径。
+
+底层文件 I/O 有三种实现（根据编译选项和配置选择）：
+
+| 后端 | 条件 | 特点 |
+|------|------|------|
+| **PosixFile** | 默认 | POSIX `preadv`/`pwritev`，普通本地文件 I/O |
+| **UringFile** | 编译时 `USE_URING` + 运行时配置 | Linux `io_uring` 异步 I/O，读操作启用 `O_DIRECT` 绕过页缓存 |
+| **ThreeFSFile** | 编译时 `USE_3FS` | 3FS 分布式文件系统（高性能用户态文件系统），通过 `hf3fs_reg_fd` 注册 |
+
+此外，存储后端有三种架构模式：
+
+| 模式 | 类名 | 特点 |
+|------|------|------|
+| `kFilePerKey` | `StorageBackendAdaptor` | 每个 key 一个文件，序列化为 protobuf |
+| `kBucket` | `BucketStorageBackend` | 多个 key 聚合到一个 bucket 文件，支持 FIFO/LRU 驱逐 |
+| `kOffsetAllocator` | `OffsetAllocatorStorageBackend` | 单一预分配数据文件 + offset allocator |
+
+**总结**：磁盘副本的写入就是普通的本地文件 I/O，没有 RDMA。可选的 `io_uring` 和 `3FS` 是本地 I/O 路径上的优化，不是网络传输加速。GPU 数据需要先做 D2H 拷贝到主机内存，再写入磁盘。
+
+---
+
+## 第二部分：Get/GetBatch
+
+### Get 的完整底层逻辑（单 key）
+
+#### 第1层：Python 绑定（store_py.cpp L398）
+
+```
+store.get(key) → py::bytes
+```
+
+1. **性能打点**：`UbDiag::PerfPoint pt(PerfKey::GET_STORE_PY_GET)`
+2. **初始化检查**：`is_client_initialized()` — 若未初始化，返回 `py::bytes("\\0", 0)`
+3. **释放 GIL**：`py::gil_scoped_release release_gil` — 避免阻塞 Python 其他线程
+4. **调用底层**：`store_->get_buffer(key)` — 返回 `shared_ptr`
+5. **空指针检查**：若 `buffer_handle` 为空，返回 `kNullString`
+6. **重新获取 GIL**：`py::gil_scoped_acquire acquire_gil`
+7. **数据转换**：将 `buffer_handle->ptr()` 和 `buffer_handle->size()` 转为 `pybind11::bytes` 返回
+
+#### 第2层：RealClient（real_client.cpp L2497 → L2374）
+
+`get_buffer()` 调用 `get_buffer_internal()`，逻辑如下：
+
+1. **参数校验**：检查 client 是否初始化、allocator 是否存在
+
+2. **查询元数据**：`client_->Query(key)`
+ - 成功 → 获取 replica 列表
+ - 失败且为 `OBJECT_NOT_FOUND` 或 `REPLICA_IS_NOT_READY` → 静默返回 nullptr
+ - 其他错误 → LOG(ERROR) + 返回 nullptr
+
+3. **选择最优副本**：`SelectBestReplica(replica_list, local_endpoints)`
+ - 优先级：**本地 MEMORY > 远端 MEMORY > LOCAL_DISK > DISK**
+ - 只考虑 `status == COMPLETE` 的副本
+ - 无可用副本 → 返回 nullptr
+
+4. **计算总大小**：`calculate_total_size(replica)`
+ - MEMORY → `buffer_descriptor.size_`
+ - DISK → `disk_descriptor.object_size`
+ - LOCAL_DISK → `local_disk_descriptor.object_size`
+ - total_length == 0 → 返回 nullptr
+
+5. **分配缓冲区**：`client_buffer_allocator->allocate(total_length)`
+ - 失败 → 返回 nullptr
+
+6. **分支处理（根据副本类型）**：
+
+ **[分支A] LOCAL_DISK 副本**：
+ - 调用 `batch_get_into_offload_object_internal(endpoint, objects)`
+ - 通过 RPC 从远端节点的 SSD 读取数据
+ - 数据直接写入分配的 buffer
+
+ **[分支B] MEMORY / DISK 副本**：
+ - `allocateSlices(slices, replica, buffer_handle->ptr())`：构造 Slice 描述符
+ - MEMORY：单个 `Slice{buffer_ptr, handle.size_}`
+ - DISK：按 `kMaxSliceSize` 分片
+ - `FilterQueryResult(query_result, replica)`：构造仅包含选定副本的 QueryResult，防止 Client::Get 内部选错副本
+ - `client_->Get(key, filtered_qr, slices)` → 进入第3层
+
+#### 第3层：Client 服务层（client_service.cpp L787）
+
+`Client::Get(key, query_result, slices)` 逻辑如下：
+
+1. **查找完整副本**：`FindFirstCompleteReplica(query_result.replicas, replica)`
+ - 遍历副本列表，找第一个 `status == COMPLETE` 的副本
+ - 失败 → `INVALID_REPLICA`
+
+2. **Hot Cache 检查**（仅 MEMORY 副本）：
+ - `RedirectToHotCache(object_key, replica)`
+ - 如果本地 Hot Cache 有该 key 的数据：
+ - 获取本地缓存块引用
+ - 大小匹配检查
+ - 修改 replica 的 `buffer_address` 指向本地缓存地址
+ - `transport_endpoint` 设为 `local_hostname_`（变为本地传输）
+ - 缓存未命中 → 不修改 replica，走正常远程传输
+
+3. **TransferRead**：执行实际数据传输
+ - `TransferRead(replica, slices)` → `TransferData(replica, slices, READ)`
+ - `transfer_submitter_->submit(replica, slices, READ)` → 返回 `TransferFuture`
+ - `future->get()` → 阻塞等待传输完成
+ - 传输策略选择：
+ - **本地传输**（源和目标在同一节点）→ `MemcpyWorkerPool` 异步 memcpy
+ - **远程传输**（跨节点）→ `TransferEngine` 提交 RDMA/TCP/CXL 传输请求
+ - **文件读取**（DISK 副本）→ `FilereadWorkerPool` 异步文件读取
+
+4. **释放 Hot Cache**：`hot_cache_->ReleaseHotKey(object_key)`（如果使用了缓存）
+
+5. **Hot Cache 频率准入**：`ShouldAdmitToHotCache(key, cache_used)`
+ - 使用 CountMinSketch 统计访问频率
+ - 超过阈值 → `ProcessSlicesAsync(key, slices, replica)` 异步将数据写入本地 Hot Cache
+ - `cache_used=true` 时跳过（已从缓存服务，无需再提升）
+
+6. **Lease 过期检查**：`query_result.IsLeaseExpired()` → `LEASE_EXPIRED` 错误
+
+---
+
+### GetBatch 的完整底层逻辑（批量 key）
+
+#### 第1层：Python 绑定（store_py.cpp L425）
+
+```
+store.get_batch(keys) → list[py::bytes]
+```
+
+1. **性能打点**：`PerfKey::GET_STORE_PY_GET_BATCH`
+2. **初始化检查**：未初始化返回 `{kNullString}`
+3. **释放 GIL**：`py::gil_scoped_release release_gil`
+4. **调用底层**：`store_->batch_get_buffer(keys)` — 返回 `vector>`
+5. **空结果检查**：若 `batch_data.empty()`，返回 `{kNullString}`
+6. **重新获取 GIL**
+7. **逐项转换**：遍历 `batch_data`，对每个非空项转为 `pybind11::bytes`，空项转为 `kNullString`
+
+#### 第2层：RealClient（real_client.cpp L2908 → L2682）
+
+`batch_get_buffer()` 调用 `batch_get_buffer_internal()`，逻辑如下：
+
+1. **参数校验**：检查 client 是否初始化、keys 是否为空
+
+2. **批量查询元数据**：`client_->BatchQuery(keys)`
+
+3. **逐 key 准备操作**：
+ - 遍历每个 key 的 query_result
+ - 查询失败（`OBJECT_NOT_FOUND` / `REPLICA_IS_NOT_READY`）→ 静默跳过
+ - `SelectBestReplica` 选择最优副本
+ - `calculate_total_size` 计算大小
+ - `client_buffer_allocator->allocate` 分配缓冲区
+ - `allocateSlices` 构造 slices
+ - **分类为两类操作**：
+ - `valid_ops`（MEMORY / DISK 副本）→ 走 `client_->BatchGet`
+ - `disk_ops`（LOCAL_DISK 副本）→ 走 SSD RPC
+
+4. **执行 MEMORY/DISK 批量传输**：
+ - 收集所有 valid_ops 的 keys、query_results、slices
+ - `client_->BatchGet(batch_keys, batch_query_results, batch_slices)` → 进入第3层
+ - 成功的 key → 将 buffer_handle 放入 `final_results`
+ - 失败的 key → LOG(ERROR)，对应位置保持 nullptr
+
+5. **执行 LOCAL_DISK 批量传输**：
+ - 按 `transport_endpoint` 分组
+ - 对每个 endpoint 调用 `batch_get_into_offload_object_internal(endpoint, objects)`
+ - 成功 → 放入 `final_results`
+ - 失败 → LOG(ERROR)
+
+#### 第3层：Client 服务层（client_service.cpp L1034）
+
+`Client::BatchGet(keys, query_results, slices)` 逻辑如下：
+
+1. **前置检查**：`transfer_submitter_` 是否初始化、query_results 大小是否匹配
+
+2. **分支判断**：
+ - `prefer_alloc_in_same_node=true` → 走 `BatchGetWhenPreferSameNode`（按 endpoint 分组批量提交）
+ - 默认路径 → 走下面的并行提交+等待流程
+
+3. **阶段A：并行提交所有传输**：
+ - 对每个 key：
+ - `FindFirstCompleteReplica` → 找 COMPLETE 副本
+ - `RedirectToHotCache` → 检查/重定向到本地缓存
+ - `transfer_submitter_->submit(replica, slices, READ)` → 异步返回 `TransferFuture`
+ - 存入 `pending_transfers: (index, key, future, replica, cache_used)`
+ - 提交失败 → 释放 Hot Cache + 记录错误
+
+4. **阶段B：等待所有传输完成**：
+ - 对每个 pending_transfer：
+ - `future.get()` → 等待结果
+ - 释放 Hot Cache（如果使用了）
+ - 成功 → 检查是否应提升到 Hot Cache（`ShouldAdmitToHotCache` → `ProcessSlicesAsync`）
+ - 失败 → `results[index] = error`
+
+5. **阶段C：批量 Lease 过期检查**：
+ - 统一用当前时间检查所有 query_results 的 lease
+ - 过期 → `LEASE_EXPIRED`
+
+---
+
+### Get 关键区别总结
+
+| 维度 | Get（单key） | GetBatch（批量） |
+|------|-------------|----------------|
+| Master RPC | 1次 Query | 1次 BatchQuery |
+| 传输方式 | 同步：submit → 立即 get() 等待 | 异步批量：先 submit 所有 → 再统一 wait 所有 |
+| 副本选择 | SelectBestReplica 选1个 | 每个 key 各自 SelectBestReplica |
+| Hot Cache | 单 key 检查/准入/释放 | 批量检查/准入/释放，统计缓存命中率 |
+| LOCAL_DISK | 单 key RPC | 按 endpoint 分组批量 RPC |
+| 错误处理 | 单 key 失败直接返回错误 | 每个 key 独立返回结果，互不影响 |
+
+---
+
+### Get 的副本类型与传输路径
+
+| 副本类型 | 数据位置 | 传输方式 | 代码路径 |
+|---------|---------|---------|---------|
+| **MEMORY** | 远端节点内存 | RDMA/TCP/CXL（远程）或 memcpy（本地） | `TransferRead` → `transfer_submitter_->submit(READ)` |
+| **DISK** | 本地磁盘文件 | 本地文件 I/O | `TransferRead` → `FilereadWorkerPool` |
+| **LOCAL_DISK** | 远端节点 SSD | RPC 到远端节点读 SSD | `batch_get_into_offload_object_internal` |
+
+**注意**：DISK 和 LOCAL_DISK 的区别——DISK 是本地磁盘文件，LOCAL_DISK 是远端节点的 SSD（名称容易混淆）。LOCAL_DISK 通过 offload RPC 让远端节点从其 SSD 读取数据后通过网络返回。
+
+---
+
+### 深度解析1：MEMORY 本地和远端都走 submitMemoryReadOperation 吗？
+
+**是的，本地和远端 MEMORY 副本都走 `submitMemoryReadOperation()`**，内部通过 `selectStrategy()` 自动区分：
+
+```
+submit(replica, slices, READ)
+│
+├─ replica.is_memory_replica() == true
+│ └─ submitMemoryReadOperation(handle, slices, 0)
+│ └─ selectStrategy(handle, slices)
+│ │
+│ ├─ isLocalTransfer(handle) == true
+│ │ → LOCAL_MEMCPY → submitMemcpyOperation()
+│ │ 直接 std::memcpy 或 gpu_staging::CopyAuto
+│ │ 提交到 MemcpyWorkerPool（1个线程，受内存带宽限制）
+│ │
+│ └─ isLocalTransfer(handle) == false
+│ → TRANSFER_ENGINE → submitTransferEngineOperation()
+│ engine_.openSegment(endpoint) + engine_.submitTransfer()
+│ RDMA/TCP/CXL 远程传输
+│
+└─ replica.is_memory_replica() == false（DISK）
+ └─ submitFileReadOperation()
+ → FilereadWorkerPool（本地文件 I/O，10个线程）
+```
+
+**`selectStrategy` 的判断逻辑**（transfer_task.cpp L798）：
+
+1. `memcpy_enabled_ == false` → 强制 `TRANSFER_ENGINE`（环境变量 `MC_STORE_MEMCPY` 控制，有 RDMA 时禁用 memcpy）
+2. `isLocalTransfer(handle) == true` → `LOCAL_MEMCPY`（比较 `handle.transport_endpoint_` 与本机端点）
+3. 默认 → `TRANSFER_ENGINE`
+
+**`isLocalTransfer` 的判断**：将副本的 `transport_endpoint_` 与 `engine_.getLocalIpAndPort()` 比较，相同则说明数据在本机，走 memcpy；否则走 Transfer Engine 远程传输。
+
+**注意**：`submit()` 的 `is_memory_replica() == false` 分支（即 `submitFileReadOperation`）只处理 DISK 副本，**不处理 LOCAL_DISK**。LOCAL_DISK 在 RealClient 层就被提前拦截，走独立的 RPC 路径。
+
+---
+
+### 深度解析2：LOCAL_DISK 为什么不能走策略模式？
+
+LOCAL_DISK 不能走 `TransferSubmitter::submit()` 的策略模式，根本原因是**数据不在本机，也不在共享文件系统上，而是在远端 Worker 节点的本地 SSD 上**。
+
+三种副本的数据位置和访问方式完全不同：
+
+| 副本类型 | 数据位置 | 访问方式 |
+|---------|---------|---------|
+| **MEMORY** | 某个节点的内存中（有 `buffer_address` + `transport_endpoint`） | 直接通过传输引擎 RDMA/TCP 读取，或本地 memcpy |
+| **DISK** | 本机的共享文件系统路径（有 `file_path`） | 本地文件 I/O（`FilereadWorkerPool`） |
+| **LOCAL_DISK** | 远端 Worker 节点的本地 SSD（有 `client_id` + `transport_endpoint`） | **两阶段**：先 RPC 让远端从 SSD 读到内存，再通过传输引擎拉取 |
+
+LOCAL_DISK 无法走策略模式的原因：
+
+**1. 需要先触发远端 SSD 读取**
+
+MEMORY 副本的数据已经在内存中，`buffer_address` 直接可用；DISK 副本的 `file_path` 在本机，可以直接文件 I/O。但 LOCAL_DISK 的数据在远端 SSD 上，**远端必须先执行一次 SSD → 内存的数据搬运**，客户端才能通过传输引擎读取。这个"远端 SSD 读取"步骤无法在 `submit()` 内部完成，因为 `submit()` 只负责本机侧的传输调度。
+
+**2. 两阶段协议需要状态协调**
+
+LOCAL_DISK 的完整读取流程：
+
+```
+客户端 远端 Worker 节点
+ | |
+ |--- RPC: batch_get_offload_object ------->| 1. FileStorage::BatchGet()
+ | (keys, sizes) | AllocateBatch() → 分配对齐内存缓冲区
+ | | BatchLoad() → 从 SSD 读取到缓冲区
+ |<--- RPC Response -----------------------| 返回 {pointers, transfer_engine_addr, gc_ttl_ms}
+ | |
+ |=== RDMA/传输引擎 READ ===================| 2. 传输引擎将远端缓冲区数据拉到本地
+ | source=本地slice地址 | (远端内存 → 本地内存/GPU显存)
+ | target=远端segment+pointer偏移 |
+ | |
+ |--- RPC: release_offload_buffer --------->| 3. 释放远端缓冲区（fire-and-forget）
+ | |
+```
+
+这个两阶段协议涉及：RPC 调用 → 等待远端 SSD I/O → 获取远端内存地址 → 传输引擎拉取 → 释放远端缓冲区。中间有多个状态需要协调（远端缓冲区的分配、GC 租约、主动释放），无法用 `submit()` 的单次提交+等待模式表达。
+
+**3. 远端缓冲区有生命周期管理**
+
+远端 Worker 为 LOCAL_DISK 读取分配了临时内存缓冲区，有 `gc_ttl_ms`（默认 5000ms）租约。如果客户端超时未完成传输，远端 GC 线程会回收缓冲区，数据丢失。客户端必须在传输完成后主动调用 `release_offload_buffer` 释放，加速缓冲区回收。这种跨节点的缓冲区生命周期管理超出了 `TransferSubmitter` 的职责范围。
+
+---
+
+### 深度解析3：LOCAL_DISK 的读取速度比 DISK 快吗？
+
+**通常 LOCAL_DISK 更快**，原因如下：
+
+**1. 传输路径对比**
+
+| 阶段 | DISK | LOCAL_DISK |
+|------|------|------------|
+| 数据位置 | 本机共享文件系统 | 远端 Worker SSD |
+| 读取方式 | 本地文件 I/O（preadv/io_uring） | 远端 SSD → 远端内存 → RDMA → 本地 |
+| 目标内存 | 只能写 CPU 内存 | 可直接写 GPU 内存 |
+| 网络开销 | 无 | 有（RPC + RDMA） |
+
+**2. 为什么 LOCAL_DISK 通常更快？**
+
+- **SSD 性能优势**：LOCAL_DISK 存储在 Worker 节点的本地 NVMe SSD 上，随机读写性能远高于 DISK 使用的共享文件系统（可能是 NFS、CephFS 等网络文件系统，或普通 HDD）
+- **并行读取**：LOCAL_DISK 的远端 Worker 使用专用线程池并行从 SSD 读取（`coro_io::post`），然后通过 RDMA 高速传输回来；DISK 的本地文件 I/O 虽然也用 `FilereadWorkerPool`（10线程），但受限于共享文件系统的 I/O 能力
+- **GPU 直写**：LOCAL_DISK 通过 RDMA 传输可以直接写入 GPU 内存（用户通过 `register_buffer` 预注册），而 DISK 的本地文件 I/O 只能写 CPU 内存，如果目标是 GPU 内存还需要额外的 D2H 中转
+- **io_uring 优化**：LOCAL_DISK 的远端 Worker 读取 SSD 时可以使用 io_uring + O_DIRECT 零拷贝读取，避免内核态拷贝
+
+**3. 什么情况下 DISK 可能更快？**
+
+- 数据量很小（网络开销占比大）
+- 本地共享文件系统使用 NVMe SSD 且网络带宽有限
+- RDMA 网络不可用，回退到 TCP 传输
+
+**4. 速度排序总结**
+
+```
+本地 MEMORY（memcpy）> 远端 MEMORY（RDMA）> LOCAL_DISK（远端SSD+RDMA）> DISK（本地文件I/O）
+```
+
+这个排序与 `SelectBestReplica` 的优先级一致：系统自动选择最快的可用副本。
+
+---
+
+### 深度解析4：为什么 LOCAL_DISK 优先级高于 DISK？
+
+`SelectBestReplica` 的优先级：**本地 MEMORY > 远端 MEMORY > LOCAL_DISK > DISK**
+
+LOCAL_DISK 优先于 DISK 的原因：
+
+**1. 数据来源的可靠性不同**
+
+| 维度 | DISK | LOCAL_DISK |
+|------|------|------------|
+| 数据位置 | Master 管理的共享文件系统路径（`file_path`） | Worker 节点的本地 SSD |
+| 创建者 | Master 在 PutStart 时自动创建 | Worker 完成数据卸载后通知 Master 创建 |
+| 归属关系 | 无 client_id，全局共享 | 绑定到特定 client_id + transport_endpoint |
+| 生命周期 | 随对象存在 | 绑定到客户端，客户端失活时被清理 |
+
+LOCAL_DISK 的数据由活跃 Worker 写入并管理，数据新鲜且确定可用；而 DISK 是 Master 侧共享文件系统上的数据，可能存在路径解析、文件系统可用性等额外风险。
+
+**2. 传输路径的灵活性不同**
+
+- **LOCAL_DISK**：RPC + RDMA 传输路径，数据可以直接写入 GPU 内存（用户通过 `register_buffer` 预注册）
+- **DISK**：本地文件 I/O（`FilereadWorkerPool`），只能写 CPU 可寻址的内存，如果分配器返回了 GPU 内存则读取会失败，需要额外的临时 CPU 缓冲区中转
+
+**3. 架构定位不同**
+
+LOCAL_DISK 是较新的架构设计——当内存不足时，Worker 将 MEMORY 副本卸载（offload）到本地 SSD，形成 MEMORY → LOCAL_DISK 的降级路径。这是数据生命周期中的自然降级，数据仍然由活跃 Worker 管理。而 DISK 是更早期的、由 Master 直接管理的共享存储机制，属于完全不同的存储层。
+
+**4. transport_endpoint 的含义差异**
+
+| 副本类型 | transport_endpoint 含义 |
+|---------|----------------------|
+| MEMORY | 内存段所在节点的传输引擎端点（RDMA NIC 地址） |
+| DISK | 无 transport_endpoint（DiskDescriptor 中没有此字段） |
+| LOCAL_DISK | 拥有该 SSD 数据的 Worker 节点的 **RPC 服务地址** |
+
+LOCAL_DISK 有明确的 `transport_endpoint`（Worker 的 RPC 地址），客户端可以直接发起远程读取；而 DISK 没有远程端点，只能本地文件 I/O。
+
+---
+
+### 深度解析5：为什么使用了 Hot Cache 就要释放？
+
+在 `Client::Get` 和 `Client::BatchGet` 中，如果使用了 Hot Cache（`cache_used=true`），在数据传输完成后必须调用 `ReleaseHotKey(key)`。这不是"释放缓存数据"，而是**释放对缓存块的引用计数**。
+
+**Hot Cache 的引用计数机制：**
+
+```
+GetHotKey(key) → ref_count++ （获取引用，保护缓存块不被驱逐）
+ ↓
+数据传输/memcpy → 从 blk->addr 读取数据到用户 buffer
+ ↓
+ReleaseHotKey(key) → ref_count-- （释放引用，允许缓存块重新可被驱逐）
+```
+
+**`GetHotKey` 做了什么？**（local_hot_cache.cpp L115）
+
+1. 在 `key_to_lru_it_` 中查找 key
+2. 找到后 `ref_count++`（原子操作）——这相当于对缓存块加了一把"读锁"
+3. `accessed = true`（延迟 LRU touch 标志）
+4. 返回 `HotMemBlock*` 指针
+
+**`RedirectToHotCache` 做了什么？**（client_service.cpp L1212）
+
+1. 调用 `GetHotKey(key)` 获取缓存块（`ref_count++`）
+2. 将 replica 的 `buffer_address_` 改为缓存块的内存地址 `blk->addr`
+3. 将 `transport_endpoint_` 改为本地地址（变为本地 memcpy 传输）
+
+**`ReleaseHotKey` 做了什么？**（local_hot_cache.cpp L136）
+
+1. 在 `key_to_lru_it_` 中查找 key
+2. `ref_count--`（原子操作）
+
+**如果不释放会怎样？**
+
+`ref_count` 永远不为 0，该缓存块在 `GetFreeBlock()` 中会被跳过，永远无法被驱逐和重用。随着时间推移，越来越多的块被"锁死"，可用缓存容量持续减少，最终导致 Hot Cache 完全失效——`GetFreeBlock()` 返回 `nullptr`，新数据无法进入缓存。
+
+**为什么不在传输前就释放？**
+
+因为 `RedirectToHotCache` 将传输目标重定向到了缓存块的内存地址。如果提前释放引用，缓存块可能被其他线程驱逐（`GetFreeBlock` 会驱逐 `ref_count == 0` 的块），其内存可能被新数据覆盖，导致当前传输读到脏数据。必须在 `future.get()` 确认传输完成后才能释放——此时 memcpy 已经完成，数据已经安全拷贝到用户 buffer 中。
+
+---
+
+## 第三部分：Put vs Get 对比
+
+| 维度 | Put | Get |
+|------|-----|-----|
+| Master 交互 | PutStart → PutEnd/PutRevoke | Query → 无需再通知 Master |
+| 副本处理 | 写入所有内存副本 + 1个磁盘副本 | 只读1个最优副本 |
+| 传输方向 | WRITE（本地→远端） | READ（远端→本地） |
+| 数据拷贝 | 需要 memcpy 到 RDMA 注册内存 | 需要 allocate 缓冲区接收数据 |
+| 零拷贝路径 | `put_from`（用户 buffer 已注册） | `get_into`（用户 buffer 已注册） |
+| Hot Cache | 不涉及 | 有频率准入机制（CountMinSketch） |
+| 失败恢复 | PutRevoke 撤销 replica | 无需撤销，换副本重试或返回错误 |
+| 磁盘写入 | `PutToLocalFile`（异步线程池写磁盘） | 不涉及（Get 只读磁盘） |
+| Lease | 不涉及 | 有 Lease 过期检查 |
diff --git a/docs/yh/transfer-engine-deep-dive.md b/docs/yh/transfer-engine-deep-dive.md
new file mode 100644
index 0000000000..47c2e3cf88
--- /dev/null
+++ b/docs/yh/transfer-engine-deep-dive.md
@@ -0,0 +1,1203 @@
+# Mooncake Transfer Engine 深度解析
+
+本文基于源码详细讲解 Mooncake Transfer Engine（TE）的初始化流程、读写机制以及 URMA 通信原理。
+
+---
+
+## 目录
+
+1. [整体架构概览](#1-整体架构概览)
+2. [核心数据结构](#2-核心数据结构)
+3. [初始化流程](#3-初始化流程)
+4. [内存注册](#4-内存注册)
+5. [读写流程](#5-读写流程)
+6. [URMA 通信详解](#6-urma-通信详解)
+7. [连接建立与握手](#7-连接建立与握手)
+8. [完成与状态查询](#8-完成与状态查询)
+
+---
+
+## 1. 整体架构概览
+
+Transfer Engine 采用**数据面与控制面分离**的分层设计。自顶向下可以理解为：用户 API 层、实现/调度层、传输后端层，以及负责发现、握手和段信息发布的元数据控制面。
+
+```
+┌─────────────────────────────────────────────────────────┐
+│ 用户 API 层 │
+│ TransferEngine (门面类) │
+├─────────────────────────────────────────────────────────┤
+│ 实现层 │
+│ TransferEngineImpl + MultiTransport │
+├──────────┬──────────┬──────────┬──────────┬─────────────┤
+│ RDMA │ TCP │ UB/URMA │ CXL │ NVLink/... │
+│Transport │Transport │Transport │Transport │ Transport │
+├──────────┴──────────┴──────────┴──────────┴─────────────┤
+│ TransferMetadata / Handshake (控制面) │
+│ etcd / HTTP / Redis / P2P Handshake │
+└─────────────────────────────────────────────────────────┘
+```
+
+**核心类关系：**
+
+```mermaid
+classDiagram
+ class TransferEngine {
+ -shared_ptr~TransferEngineImpl~ impl_
+ +init(metadata_conn, server_name, ip, port)
+ +registerLocalMemory(addr, length)
+ +submitTransfer(batch_id, entries)
+ +getTransferStatus(batch_id, task_id)
+ +allocateBatchID(batch_size)
+ }
+
+ class TransferEngineImpl {
+ -shared_ptr~TransferMetadata~ metadata_
+ -shared_ptr~MultiTransport~ multi_transports_
+ -shared_ptr~Topology~ local_topology_
+ -MemoryRegionMap local_memory_regions_
+ +init()
+ +submitTransfer()
+ +registerLocalMemory()
+ }
+
+ class MultiTransport {
+ -map~string, shared_ptr~Transport~~ transport_map_
+ +installTransport(proto, topo)
+ +submitTransfer(batch_id, entries)
+ +selectTransport(request, transport)
+ +allocateBatchID()
+ }
+
+ class Transport {
+ <>
+ +submitTransferTask(task_list)*
+ +getTransferStatus(batch_id, task_id)*
+ +registerLocalMemory(addr, len)*
+ }
+
+ class RdmaTransport {
+ -vector~shared_ptr~RdmaContext~~ context_list_
+ +submitTransferTask()
+ }
+
+ class UbTransport {
+ -vector~shared_ptr~UbContext~~ context_list_
+ +submitTransferTask()
+ }
+
+ class TcpTransport {
+ +submitTransferTask()
+ }
+
+ class TransferMetadata {
+ -shared_ptr~MetadataStoragePlugin~ storage_plugin_
+ -shared_ptr~HandShakePlugin~ handshake_plugin_
+ +getSegmentDescByID(id)
+ +updateLocalSegmentDesc()
+ +startHandshakeDaemon()
+ }
+
+ class Topology {
+ +discover(filter)
+ +selectDevice(location, retry)
+ +getHcaList()
+ }
+
+ TransferEngine --> TransferEngineImpl : impl_
+ TransferEngineImpl --> MultiTransport : multi_transports_
+ TransferEngineImpl --> TransferMetadata : metadata_
+ TransferEngineImpl --> Topology : local_topology_
+ MultiTransport --> Transport : transport_map_
+ Transport <|-- RdmaTransport
+ Transport <|-- UbTransport
+ Transport <|-- TcpTransport
+ RdmaTransport --> RdmaContext : context_list_
+ UbTransport --> UbContext : context_list_
+```
+
+### 源码位置索引
+
+| 组件 | 头文件 | 实现文件 |
+|------|--------|----------|
+| TransferEngine | `include/transfer_engine.h:42` | `src/transfer_engine.cpp:22` |
+| TransferEngineImpl | `include/transfer_engine_impl.h:54` | `src/transfer_engine_impl.cpp:77` |
+| MultiTransport | `include/multi_transport.h:23` | `src/multi_transport.cpp:69` |
+| Transport (基类) | `include/transport/transport.h:42` | - |
+| RdmaTransport | `include/transport/rdma_transport/rdma_transport.h:41` | `src/transport/rdma_transport/rdma_transport.cpp:60` |
+| UbTransport | `include/transport/kunpeng_transport/ub_transport.h` | `src/transport/kunpeng_transport/ub_transport.cpp:25` |
+| TransferMetadata | `include/transfer_metadata.h:43` | - |
+
+---
+
+## 2. 核心数据结构
+
+> 本节代码片段用于说明字段职责，保留了主路径相关字段；实际源码还包含不同编译选项下的扩展字段。
+
+### 2.1 TransferRequest — 传输请求
+
+```cpp
+// transport.h:58-67
+struct TransferRequest {
+ enum OpCode { READ, WRITE };
+ OpCode opcode; // 读或写
+ void *source; // 本地内存地址
+ SegmentID target_id; // 目标段 ID
+ uint64_t target_offset; // 目标偏移
+ size_t length; // 传输长度
+ int advise_retry_cnt = 0;
+};
+```
+
+### 2.2 Slice — 传输切片
+
+大块传输被拆分为多个 Slice，每个 Slice 是一次 RDMA/URMA 操作的基本单位。
+
+```cpp
+// transport.h:104-238
+struct Slice {
+ void *source_addr;
+ size_t length;
+ TransferRequest::OpCode opcode;
+ SegmentID target_id;
+ SliceStatus status; // PENDING -> POSTED -> SUCCESS/FAILED
+
+ union {
+ struct { /* rdma */ uint64_t dest_addr; uint32_t source_lkey;
+ uint32_t dest_rkey; volatile int *qp_depth; ... } rdma;
+ struct { /* ub/urma */ uint64_t dest_addr; volatile int *jetty_depth;
+ void *r_seg; void *l_seg; ... } ub;
+ struct { /* tcp */ uint64_t dest_addr; } tcp;
+ // ... 其他传输类型
+ };
+};
+```
+
+### 2.3 TransferTask — 传输任务
+
+一个 TransferRequest 对应一个 TransferTask，包含多个 Slice。
+
+```cpp
+// transport.h:281-312
+struct TransferTask {
+ volatile uint64_t slice_count = 0;
+ volatile uint64_t success_slice_count = 0;
+ volatile uint64_t failed_slice_count = 0;
+ volatile uint64_t transferred_bytes = 0;
+ volatile bool is_finished = false;
+ uint64_t total_bytes = 0;
+ BatchID batch_id = 0;
+ const TransferRequest *request = nullptr;
+ std::vector slice_list;
+
+#ifdef USE_EVENT_DRIVEN_COMPLETION
+ volatile uint64_t completed_slice_count = 0;
+#endif
+};
+```
+
+### 2.4 BatchDesc — 批次描述符
+
+```cpp
+// transport.h:314-335
+struct BatchDesc {
+ BatchID id; // 即 BatchDesc 指针本身
+ size_t batch_size;
+ std::vector task_list;
+ void *context; // 供具体 transport 扩展
+ int64_t start_timestamp;
+ std::atomic has_failure{false};
+ std::atomic is_finished{false};
+ std::atomic finished_transfer_bytes{0};
+
+#ifdef USE_EVENT_DRIVEN_COMPLETION
+ std::atomic finished_task_count{0};
+ std::mutex completion_mutex;
+ std::condition_variable completion_cv;
+#endif
+};
+```
+
+> **BatchID 本质上是 BatchDesc 指针的整型表示**（`transport.h:98-100`）：
+> ```cpp
+> static inline BatchDesc &toBatchDesc(BatchID id) {
+> return *reinterpret_cast(id);
+> }
+> ```
+
+### 2.5 SegmentDesc — 段描述符
+
+```cpp
+// transfer_metadata.h:88-108
+struct SegmentDesc {
+ std::string name; // 段名（通常是 ip:port）
+ std::string protocol; // "rdma" / "ub" / "tcp" 等
+ std::vector devices; // 网卡设备列表
+ Topology topology; // 拓扑选择矩阵
+ std::vector buffers; // 已注册的内存区
+ std::vector nvmeof_buffers;
+ std::string cxl_name;
+ uint64_t cxl_base_addr;
+ RankInfoDesc rank_info; // Ascend 场景
+ int tcp_data_port;
+};
+```
+
+数据结构之间的关系：
+
+```mermaid
+graph TD
+ A[BatchDesc] -->|task_list| B[TransferTask 1]
+ A -->|task_list| C[TransferTask 2]
+ B -->|slice_list| D[Slice 1]
+ B -->|slice_list| E[Slice 2]
+ B -->|slice_list| F[Slice 3]
+ C -->|slice_list| G[Slice 4]
+ B -->|request| H[TransferRequest]
+ C -->|request| I[TransferRequest]
+
+ style A fill:#f9f,stroke:#333
+ style B fill:#bbf,stroke:#333
+ style C fill:#bbf,stroke:#333
+ style D fill:#bfb,stroke:#333
+ style E fill:#bfb,stroke:#333
+ style F fill:#bfb,stroke:#333
+ style G fill:#bfb,stroke:#333
+```
+
+---
+
+## 3. 初始化流程
+
+### 3.1 整体初始化时序
+
+```mermaid
+sequenceDiagram
+ participant User as 用户代码
+ participant TE as TransferEngine
+ participant Impl as TransferEngineImpl
+ participant MT as MultiTransport
+ participant MD as TransferMetadata
+ participant Topo as Topology
+ participant RT as RdmaTransport/UbTransport
+
+ User->>TE: init(metadata_conn, server_name, ip, port)
+ TE->>Impl: init(...)
+ Impl->>Impl: setFilesLimit() 提升文件描述符限制
+ Impl->>Impl: parseHostNameWithPort() 解析地址端口
+ Impl->>MD: new TransferMetadata(conn_string)
+ Impl->>MT: new MultiTransport(metadata, server_name)
+ Impl->>MD: addRpcMetaEntry(server_name, desc)
+
+ alt auto_discover == true
+ Impl->>Topo: discover(filter)
+ Note over Topo: 扫描 RDMA/UB 设备
+ Impl->>MT: installTransport("rdma"/"ub", topology)
+ MT->>RT: new RdmaTransport() / UbTransport()
+ MT->>RT: install(server_name, metadata, topology)
+ RT->>RT: initializeRdmaResources()
+ Note over RT: 为每个 HCA 创建 Context
+ RT->>RT: allocateLocalSegmentID()
+ RT->>RT: startHandshakeDaemon()
+ RT->>MD: updateLocalSegmentDesc()
+ end
+```
+
+### 3.2 逐步详解
+
+#### 步骤 1：构造 TransferEngine
+
+```cpp
+// transfer_engine.cpp:22-24
+TransferEngine::TransferEngine(bool auto_discover)
+ : impl_(std::make_shared(auto_discover)) {}
+```
+
+`TransferEngine` 是纯门面类，所有调用直接转发给 `TransferEngineImpl`。
+
+#### 步骤 2：调用 init()
+
+源码位于 `transfer_engine_impl.cpp:77-368`，核心步骤：
+
+1. **提升系统资源限制**：调用 `setFilesLimit()` 将 RLIMIT_NOFILE 提升到最大值。
+
+2. **解析地址端口**：
+ ```cpp
+ auto [host_name, port] = parseHostNameWithPort(local_server_name);
+ local_server_name_ = local_server_name;
+ ```
+
+3. **配置 RPC 描述符**：
+ - Legacy/P2P 模式：使用 `local_server_name` 中指定的端口
+ - 新模式：自动发现本机 IP + 随机端口
+
+4. **创建元数据管理器**：
+ ```cpp
+ metadata_ = std::make_shared(metadata_conn_string);
+ ```
+ 支持的后端：`etcd://`、`http://`、`redis://`、`P2PHANDSHAKE`。
+
+5. **创建多传输管理器**：
+ ```cpp
+ multi_transports_ = std::make_shared(metadata_, local_server_name_);
+ ```
+
+6. **注册 RPC 元数据条目**：
+ ```cpp
+ metadata_->addRpcMetaEntry(local_server_name_, desc);
+ ```
+
+7. **自动拓扑发现与传输安装**（`auto_discover_ == true` 时）：
+
+ ```mermaid
+ flowchart TD
+ A[auto_discover?] -->|Yes| B[Topo.discover]
+ B --> C{有 HCA 设备?}
+ C -->|Yes, USE_UB| D[installTransport 'ub']
+ C -->|Yes, RDMA| E[installTransport 'rdma']
+ C -->|No, MC_FORCE_TCP| F[installTransport 'tcp']
+ C -->|Yes, NVLink| G[installTransport 'nvlink']
+ D --> H[initializeUbResources]
+ E --> I[initializeRdmaResources]
+ H --> J[allocateLocalSegmentID]
+ I --> J
+ J --> K[startHandshakeDaemon]
+ K --> L[updateLocalSegmentDesc]
+ ```
+
+ 设备选择逻辑（`transfer_engine_impl.cpp:236-364`）可以按优先级理解：
+ - `USE_ASCEND` / `USE_ASCEND_DIRECT` → 直接安装 Ascend 传输，跳过普通自动发现路径
+ - `USE_UBSHMEM` → 安装 `ubshmem`，并关闭普通 `auto_discover_`
+ - `USE_CXL` 且设置 `MC_CXL_DEV_PATH` → 额外安装 `CxlTransport`
+ - `auto_discover_ == true` → 自动发现或解析 `MC_CUSTOM_TOPO_JSON`
+ - `USE_UB` → 安装 `UbTransport`（URMA）
+ - `USE_ASCEND_HETEROGENEOUS` → 安装异构 Ascend 传输
+ - `USE_MACA` → 安装 MACA 传输
+ - `USE_MNNVL` / `USE_INTRA_NVLINK` → 根据 `MC_FORCE_MNNVL`、`MC_INTRANODE_NVLINK` 和 HCA 是否存在选择 `nvlink`、`nvlink_intra` 或 RDMA
+ - 默认路径：检测到 HCA 且未设置 `MC_FORCE_TCP`，或设置了 `MC_FORCE_HCA` → 安装 `RdmaTransport`；否则安装 `TcpTransport`
+ - `USE_HIP` → 额外安装 HIP GPU P2P 传输，可与跨节点 RDMA/TCP 路径共存
+
+### 3.3 Transport 安装流程
+
+以 `RdmaTransport` 为例（`rdma_transport.cpp:92-130`）：
+
+```cpp
+int RdmaTransport::install(string &local_server_name,
+ shared_ptr meta,
+ shared_ptr topo) {
+ metadata_ = meta;
+ local_server_name_ = local_server_name;
+ local_topology_ = topo;
+
+ // 1. 初始化 RDMA 资源（为每个 HCA 创建 Context）
+ ret = initializeRdmaResources();
+
+ // 2. 分配本地段 ID
+ ret = allocateLocalSegmentID();
+
+ // 3. 启动握手守护线程
+ ret = startHandshakeDaemon(local_server_name);
+
+ // 4. 发布段描述符到元数据服务
+ ret = metadata_->updateLocalSegmentDesc();
+}
+```
+
+`initializeRdmaResources()` 的实现（`rdma_transport.cpp:651-672`）：
+
+```cpp
+int RdmaTransport::initializeRdmaResources() {
+ auto hca_list = local_topology_->getHcaList();
+ for (auto &device_name : hca_list) {
+ auto context = make_shared(*this, device_name);
+ int ret = context->construct(
+ config.num_cq_per_ctx, // CQ 数量
+ config.num_comp_channels_per_ctx,
+ config.port, // IB 端口 (默认1)
+ config.gid_index, // GID 索引
+ config.max_cqe, // 最大 CQE 数
+ config.max_ep_per_ctx // 最大端点数
+ );
+ if (ret)
+ local_topology_->disableDevice(device_name);
+ else
+ context_list_.push_back(context);
+ }
+}
+```
+
+**RdmaContext 的构造**（`rdma_context.h:65-228`）为每个 RDMA NIC 分配：
+- `ibv_context` — 设备上下文
+- `ibv_pd` — Protection Domain
+- CQ 列表（`RdmaCq`） — 完成队列
+- Completion Channel — 事件通知通道
+- Endpoint Store — 连接端点管理
+
+---
+
+## 4. 内存注册
+
+### 4.1 注册流程
+
+```mermaid
+sequenceDiagram
+ participant User as 用户代码
+ participant TE as TransferEngine
+ participant Impl as TransferEngineImpl
+ participant MT as MultiTransport
+ participant RT as RdmaTransport
+ participant Ctx as RdmaContext
+ participant MD as TransferMetadata
+
+ User->>TE: registerLocalMemory(addr, len, location)
+ TE->>Impl: registerLocalMemory(...)
+ Impl->>Impl: checkOverlap(addr, len) 检查重叠
+ Impl->>MT: listTransports()
+ MT-->>Impl: [RdmaTransport, ...]
+
+ loop 每个 Transport
+ Impl->>RT: registerLocalMemory(addr, len, ...)
+ RT->>RT: preTouchMemory() 可选：大内存预触
+ loop 每个 Context (即每个 NIC)
+ RT->>Ctx: registerMemoryRegion(addr, len, access)
+ Ctx->>Ctx: ibv_reg_mr(pd, addr, len, access)
+ Note over Ctx: 获取 lkey/rkey
+ end
+ RT->>MD: addLocalMemoryBuffer(buffer_desc)
+ end
+ Impl->>Impl: insertMemoryRegionLocked(...)
+```
+
+### 4.2 RDMA 内存注册细节
+
+```cpp
+// rdma_transport.cpp:182-303
+int RdmaTransport::registerLocalMemoryInternal(void *addr, size_t length, ...) {
+ const int kBaseAccessRights = IBV_ACCESS_LOCAL_WRITE |
+ IBV_ACCESS_REMOTE_WRITE |
+ IBV_ACCESS_REMOTE_READ;
+
+ // 可选：大内存（>4GB）并行预触
+ if (context_list_.size() > 0 && length >= 4GB) {
+ preTouchMemory(addr, length); // 多线程 mmap 触页
+ }
+
+ // 并行或串行注册
+ for (auto &context : context_list_) {
+ context->registerMemoryRegion(addr, length, access_rights);
+ // 底层调用 ibv_reg_mr()
+ }
+
+ // 收集所有 context 的 lkey/rkey
+ for (auto &context : context_list_) {
+ buffer_desc.lkey.push_back(context->lkey(addr));
+ buffer_desc.rkey.push_back(context->rkey(addr));
+ }
+
+ // 添加到元数据
+ metadata_->addLocalMemoryBuffer(buffer_desc, update_metadata);
+}
+```
+
+### 4.3 URMA 内存注册
+
+```cpp
+// ub_transport.cpp:83-123
+int UbTransport::registerLocalMemory(void *addr, size_t length, ...) {
+ for (auto &context : context_list_) {
+ // 注册内存段，获取 target segment (tseg)
+ context->registerMemoryRegion((uint64_t)addr, length);
+ // 构建包含 tseg 信息的 buffer 描述符
+ context->buildLocalBufferDesc((uint64_t)addr, buffer_desc);
+ }
+ metadata_->addLocalMemoryBuffer(buffer_desc, update_metadata);
+}
+```
+
+URMA 底层调用 `urma_register_seg()` 注册内存，生成的 `tseg`（target segment）用于后续的远程内存访问。
+
+---
+
+## 5. 读写流程
+
+### 5.1 写操作完整时序
+
+```mermaid
+sequenceDiagram
+ participant User as 用户代码
+ participant TE as TransferEngine
+ participant Impl as TransferEngineImpl
+ participant MT as MultiTransport
+ participant RT as RdmaTransport
+ participant Ctx as RdmaContext
+ participant EP as RdmaEndPoint
+
+ User->>TE: batch_id = allocateBatchID(N)
+ TE->>MT: allocateBatchID(N)
+ MT-->>User: BatchID (= BatchDesc*)
+
+ User->>TE: submitTransfer(batch_id, [req1, req2])
+ TE->>Impl: submitTransfer(batch_id, entries)
+ Impl->>MT: submitTransfer(batch_id, entries)
+
+ Note over MT: 为每个 request 选择 transport
+ MT->>MT: selectTransport(req, transport)
+ Note over MT: 查找 target segment 的 protocol
从 transport_map_ 找到对应 Transport
+
+ MT->>MT: 创建 TransferTask，按 Transport 分组
+ MT->>RT: submitTransferTask(task_list)
+
+ Note over RT: 拆分 Slice + 选择设备
+ RT->>RT: 遍历每个 task
+ loop 每个 request，按 slice_size 分片
+ RT->>RT: Slice = getSliceCache().allocate()
+ RT->>RT: selectDevice() 选择 NIC
+ RT->>RT: 填充 rdma.source_lkey, dest_addr
+ RT-->>RT: slices_to_post[context].push(slice)
+ end
+
+ RT->>Ctx: submitPostSend(slices)
+ Ctx->>EP: endpoint.submitPostSend(slices, failed)
+ EP->>EP: 按 QP/CQ 可用深度分配 slice，构建 ibv_send_wr
+ EP->>EP: ibv_post_send(qp, wr)
+ Note over EP: RDMA WRITE 发送到远端
+
+ User->>TE: getTransferStatus(batch_id, task_id)
+ TE->>MT: getTransferStatus(...)
+ MT->>MT: 检查 task.slice_count == success + failed ?
+ MT-->>User: {COMPLETED, transferred_bytes}
+```
+
+### 5.2 submitTransferTask 详解
+
+以 RDMA 传输为例（`rdma_transport.cpp:456-574`）：
+
+```cpp
+Status RdmaTransport::submitTransferTask(
+ const vector &task_list) {
+
+ unordered_map, vector> slices_to_post;
+ const size_t kBlockSize = globalConfig().slice_size; // 默认 ~1MB
+
+ for (auto &task : task_list) {
+ auto &request = *task->request;
+
+ // 1. 尝试整体选择 device（优化：整个 request 用同一个 NIC）
+ selectDevice(local_segment_desc, (uint64_t)request.source,
+ request.length, request_buffer_id, request_device_id);
+
+ // 2. 按 slice_size 分片
+ for (uint64_t offset = 0; offset < request.length; offset += kBlockSize) {
+ Slice *slice = getSliceCache().allocate();
+ slice->source_addr = (char*)request.source + offset;
+ slice->length = merge_final ? request.length - offset : kBlockSize;
+ slice->opcode = request.opcode;
+ slice->rdma.dest_addr = request.target_offset + offset;
+
+ // 3. 选择目标设备（确定用哪个 NIC）
+ selectDevice(local_segment_desc, (uint64_t)slice->source_addr,
+ slice->length, buffer_id, device_id);
+
+ // 4. 填充 RDMA 密钥
+ slice->rdma.source_lkey =
+ local_segment_desc->buffers[buffer_id].lkey[device_id];
+
+ // 5. 按设备分组
+ slices_to_post[context_list_[device_id]].push_back(slice);
+ task.slice_count++;
+
+ // 6. 达到水位线时提前提交
+ if (nr_slices >= kSubmitWatermark) {
+ context->submitPostSend(entry.second);
+ slices_to_post.clear();
+ }
+ }
+ }
+ // 7. 提交剩余 slices
+ for (auto &entry : slices_to_post)
+ entry.first->submitPostSend(entry.second);
+}
+```
+
+### 5.3 Slice 分片策略
+
+```
+TransferRequest: length = 3.5 MB, slice_size = 1 MB
+ ┌──────┐──────┐──────┐─────┐
+ │ 1 MB │ 1 MB │ 1 MB │0.5MB│
+ └──────┘──────┘──────┘─────┘
+ Slice0 Slice1 Slice2 Slice3
+
+fragment_limit 控制最后一个分片的合并策略：
+若剩余长度 <= slice_size + fragment_limit，则合并为一个 Slice
+```
+
+### 5.4 RDMA submitPostSend
+
+```cpp
+// rdma_endpoint.h:144
+int RdmaEndPoint::submitPostSend(vector &slice_list,
+ vector &failed_slice_list) {
+ // 1. 根据 QP 深度和 CQ 剩余容量，把 slice 分配到可用 QP
+ int cq_remaining = globalConfig().max_cqe - *cq_outstanding_;
+ int qp_avail = max_wr_depth_ - wr_depth_list_[qp_index];
+
+ // 2. 构建工作请求
+ ibv_send_wr wr;
+ wr.opcode = (opcode == WRITE) ? IBV_WR_RDMA_WRITE : IBV_WR_RDMA_READ;
+ wr.wr.rdma.remote_addr = slice->rdma.dest_addr;
+ wr.wr.rdma.rkey = dest_rkey;
+ wr.sg_list = &sge; // {addr=source_addr, lkey, length}
+
+ // 3. 批量提交到硬件，并更新 QP/CQ outstanding 计数
+ ibv_post_send(qp_list_[qp_index], &wr, &bad_wr);
+}
+```
+
+当前实现不是简单 round-robin。`RdmaEndPoint::submitPostSend()` 会综合 `max_wr_depth_`、每个 QP 的 `wr_depth_list_` 和 CQ 的 outstanding 数，把待发送 slice 分 chunk 分发到多个 QP；如果 `ibv_post_send()` 部分失败，则把失败 slice 放入 `failed_slice_list`，后续由 worker 侧重试或标记失败。
+
+### 5.5 URMA submitPostSend
+
+```cpp
+// urma_endpoint.h 中
+int UrmaEndpoint::submitPostSend() {
+ // 1. 随机选择一个 Jetty
+ int jetty_index = SimpleRandom::Get().next(jetty_list_.size());
+
+ // 2. 构建工作请求
+ urma_jfs_wr_t wr;
+ wr.opcode = (opcode == WRITE) ? URMA_OPC_WRITE : URMA_OPC_READ;
+ wr.rw.src.sge[0].addr = slice->source_addr;
+ wr.rw.src.sge[0].tseg = slice->ub.l_seg; // 本地 segment
+ wr.rw.dst.sge[0].addr = slice->ub.dest_addr;
+ wr.rw.dst.sge[0].tseg = slice->ub.r_seg; // 远端 segment
+ wr.tjetty = remote_jetty; // 远端 Jetty 引用
+
+ // 3. 提交到硬件
+ urma_post_jetty_send_wr(jetty, &wr);
+}
+```
+
+URMA 路径同样有 outstanding 深度控制和失败重试。当前源码中 Jetty 选择是随机选择，不是 round-robin；完成事件由 `UbWorkerPool` 轮询 JFC 后调用 `markSuccess()` 或进入失败重试路径。
+
+### 5.6 设备选择策略
+
+`selectDevice()` 的核心逻辑（`rdma_transport.cpp:684-698`）：
+
+```mermaid
+flowchart TD
+ A[selectDevice: offset, length] --> B{遍历所有 BufferDesc}
+ B --> C{offset 在 buffer 范围内?}
+ C -->|No| B
+ C -->|Yes| D{topology.selectDevice}
+ D --> E{用 buffer.name 作 location}
+ E --> F{找到匹配设备?}
+ F -->|Yes| G[返回 buffer_id, device_id]
+ F -->|No| H[用 wildcard 重试]
+ H --> F2{找到?}
+ F2 -->|Yes| G
+ F2 -->|No| I[返回 ERR_ADDRESS_NOT_REGISTERED]
+```
+
+Topology 的选择矩阵会根据内存位置（NUMA node / GPU）优先选择距离最近的 NIC，减少跨 NUMA 访问。
+
+---
+
+## 6. URMA 通信详解
+
+### 6.1 URMA 是什么
+
+**URMA (Unified Remote Memory Access)** 是华为鲲鹏 (Kunpeng) 处理器专有的高速互联通信框架，基于 **UB (Unified Bus)** 总线。它提供了类似 RDMA 的编程模型，但针对鲲鹏芯片架构进行了深度优化。
+
+核心概念映射：
+
+| RDMA 概念 | URMA 对应 | 说明 |
+|-----------|-----------|------|
+| QP (Queue Pair) | Jetty | 通信通道 |
+| CQ (Completion Queue) | JFC (Jetty Factory Completion) | 完成队列 |
+| MR (Memory Region) | Segment (tseg) | 注册内存区 |
+| ibv_post_send | urma_post_jetty_send_wr | 提交工作请求 |
+| ibv_poll_cq | urma_poll_jfc | 轮询完成 |
+| LID/GID | EID (Endpoint ID) | 设备地址标识 |
+
+### 6.2 URMA 架构层次
+
+```mermaid
+graph TB
+ subgraph "Transfer Engine 层"
+ UB[UbTransport]
+ end
+
+ subgraph "URMA 抽象层"
+ UC[UrmaContext]
+ UE[UrmaEndpoint]
+ end
+
+ subgraph "URMA API"
+ U1[urma_init]
+ U2[urma_create_context]
+ U3[urma_register_seg / urma_import_seg]
+ U4[urma_create_jetty]
+ U5[urma_bind_jetty]
+ U6[urma_post_jetty_send_wr]
+ U7[urma_poll_jfc]
+ end
+
+ subgraph "硬件层"
+ HW[Kunpeng UB Bus
鲲鹏统一总线]
+ end
+
+ UB --> UC
+ UB --> UE
+ UC --> U1
+ UC --> U2
+ UC --> U3
+ UC --> U7
+ UE --> U4
+ UE --> U5
+ UE --> U6
+ U1 --> HW
+ U2 --> HW
+ U3 --> HW
+ U4 --> HW
+ U5 --> HW
+ U6 --> HW
+ U7 --> HW
+```
+
+### 6.3 UrmaContext 初始化
+
+```cpp
+// urma_endpoint.h:50-147
+class UrmaContext : public UbContext {
+ urma_context_t *urma_ctx_; // URMA 上下文
+ vector jfc_list_; // 完成队列列表
+ vector jfr_list_; // 接收队列列表
+ vector jfce_list_; // 完成事件队列
+ map local_tsegs_; // 本地注册段
+ map imported_segs_; // 导入的远端段
+};
+
+UrmaContext::construct(GlobalConfig &config) {
+ // 1. 创建完成事件队列 (JFCE)
+ urma_create_jfce(ctx, &jfce);
+
+ // 2. 创建完成队列 (JFC)
+ urma_create_jfc(ctx, jfce, &jfc);
+
+ // 3. 注册 EID
+ urma_register_eid_by_index(ctx, eid_index, &eid);
+
+ // 4. 启动后台工作线程池
+ worker_pool_->start();
+}
+```
+
+### 6.4 UrmaEndpoint 连接管理
+
+```cpp
+// urma_endpoint.h:150-196
+class UrmaEndpoint : public UbEndPoint {
+ vector jetties_; // Jetty 列表（类似 QP）
+ map imported_jetties_; // 远端 Jetty 引用
+};
+```
+
+Jetty 配置参数：
+```cpp
+urma_jetty_attr_t attr;
+attr.depth = 2048; // 最大工作请求数
+attr.trans_mode = URMA_TM_RC; // 可靠连接模式
+attr.priority = 15;
+attr.max_sge = 5; // 最大 SGE 数
+attr.rnr_retry = 7;
+attr.err_timeout = 17;
+```
+
+### 6.5 URMA 数据传输流程
+
+```mermaid
+sequenceDiagram
+ participant App as 应用层
+ participant UB as UbTransport
+ participant UC as UrmaContext
+ participant UE as UrmaEndpoint
+ participant HW as UB 硬件
+
+ Note over App: 写操作为例
+ App->>UB: submitTransferTask(task_list)
+
+ loop 每个 Slice
+ UB->>UB: 查找本地 l_seg
+ Note over UB: slice->ub.l_seg = context->localSegWithIndex(idx)
+ end
+
+ UB->>UC: submitPostSend(slices)
+ UC->>UE: endpoint->submitPostSend(slices, failed)
+
+ loop 每个 Slice
+ UE->>UE: 随机选择 Jetty
+ UE->>UE: 构建 urma_jfs_wr_t
+
+ Note over UE: wr.opcode = URMA_OPC_WRITE
wr.rw.src.sge[0] = {source_addr, l_seg}
wr.rw.dst.sge[0] = {dest_addr, r_seg}
wr.tjetty = remote_jetty
+
+ UE->>HW: urma_post_jetty_send_wr(jetty, &wr)
+ end
+
+ Note over HW: UB 总线直接写入远端内存
+
+ par 后台轮询线程
+ UE->>HW: urma_poll_jfc(jfc, wc, num)
+ HW-->>UE: 完成事件
+ UE->>UE: slice->markSuccess() / markFailed()
+ end
+```
+
+### 6.6 URMA 内存操作
+
+**本地注册：**
+```cpp
+int UrmaContext::registerMemoryRegion(uint64_t va, size_t length) {
+ urma_seg_attr_t attr;
+ attr.va = va;
+ attr.len = length;
+ attr.cacheable = URMA_NON_CACHEABLE;
+ attr.access = URMA_ACCESS_READ | URMA_ACCESS_WRITE | URMA_ACCESS_ATOMIC;
+ attr.token_policy = URMA_TOKEN_NONE;
+
+ urma_tseg_t *tseg;
+ urma_register_seg(urma_ctx_, &attr, tseg);
+ local_tsegs_[va] = tseg; // 缓存用于后续查找
+}
+```
+
+**远端导入：**
+```cpp
+void* UrmaContext::retrieveRemoteSeg(const string &remoteSegmentStr) {
+ // 反序列化远端 segment 信息
+ urma_import_attr_t attr;
+ attr.cacheable = URMA_NON_CACHEABLE;
+ attr.access = URMA_ACCESS_READ | URMA_ACCESS_WRITE;
+ attr.mapping = URMA_SEG_NOMAP;
+
+ urma_tseg_t *tseg;
+ urma_import_seg(urma_ctx_, &attr, tseg);
+ imported_segs_[remoteSegmentStr] = tseg;
+}
+```
+
+远端 segment 信息通过元数据服务交换，在 Slice 提交时通过 `slice->ub.r_seg` 引用。
+
+### 6.7 完成轮询
+
+URMA 的完成轮询运行在后台线程中：
+
+```cpp
+// 工作线程循环
+while (running) {
+ urma_wc_t wc[kBatchSize];
+ int count = urma_poll_jfc(jfc, wc, kBatchSize);
+
+ for (int i = 0; i < count; i++) {
+ Slice *slice = (Slice*)wc[i].user_ctx;
+ if (wc[i].status == URMA_WC_SUCCESS)
+ slice->markSuccess();
+ else
+ slice->markFailed();
+ }
+}
+```
+
+---
+
+## 7. 连接建立与握手
+
+### 7.1 握手协议
+
+Transfer Engine 使用基于 RPC 的握手协议建立连接，而非 RDMA CM。
+
+```mermaid
+sequenceDiagram
+ participant A as Node A (Active)
+ participant Meta as 元数据服务
+ participant B as Node B (Passive)
+
+ Note over A,B: 阶段 1: 发现
+ A->>Meta: getSegmentDescByName("node_b")
+ Meta-->>A: SegmentDesc{devices, protocol, ...}
+ A->>A: 获取 Node B 的 RPC 地址和设备信息
+
+ Note over A,B: 阶段 2: 主动握手
+ A->>A: 查找目标 NIC 对应的本地 Context
+ A->>A: 获取或创建 Endpoint
+ A->>B: RPC: sendHandshake(peer_server_name, local_desc)
+ Note over A,B: local_desc 包含:
local_nic_path, qp_num[], gid, lid
+
+ Note over B: 阶段 3: 被动连接
+ B->>B: 收到握手请求
+ B->>B: 查找本地 Context 和 Endpoint
+ B->>B: setupConnectionsByPassive(peer_desc, local_desc)
+ B->>B: 填充 local_desc (QP 号、GID、LID)
+ B-->>A: 返回 local_desc
+
+ Note over A,B: 阶段 4: 建立 QP 连接
+ A->>A: setupConnectionsByActive()
+ A->>A: doSetupConnection(peer_gid, peer_lid, peer_qp_num)
+
+ Note over A: RTU: 修改 QP 状态
INIT -> RTR -> RTS
+
+ B->>B: doSetupConnection(peer_gid, peer_lid, peer_qp_num)
+
+ Note over A,B: 连接建立完成，可以传输数据
+```
+
+### 7.2 RDMA QP 状态转换
+
+```mermaid
+stateDiagram-v2
+ [*] --> INIT: ibv_create_qp
+ INIT --> RTR: ibv_modify_qp
(dest: peer_gid/lid/qp_num)
+ RTR --> RTS: ibv_modify_qp
(timeout, retry, rnr_retry)
+ RTS --> ERR: disconnect / error
+ ERR --> [*]: ibv_destroy_qp
+```
+
+`doSetupConnection` 的实现（`rdma_endpoint.cpp`）：
+
+```cpp
+int RdmaEndPoint::doSetupConnection(int qp_index,
+ const ibv_gid &peer_gid, uint16_t peer_lid, uint32_t peer_qp_num) {
+
+ // QP: INIT -> RTR (Ready to Receive)
+ ibv_qp_attr attr;
+ attr.qp_state = IBV_QPS_RTR;
+ attr.path_mtu = IBV_MTU_4096;
+ attr.dest_qp_num = peer_qp_num;
+ attr.rq_psn = 0;
+ attr.ah_attr.dlid = peer_lid;
+ attr.ah_attr.dgid = peer_gid;
+ ibv_modify_qp(qp, &attr, IBV_QP_STATE | IBV_QP_PATH_MTU | ...);
+
+ // QP: RTR -> RTS (Ready to Send)
+ attr.qp_state = IBV_QPS_RTS;
+ attr.sq_psn = 0;
+ attr.timeout = 14;
+ attr.retry_cnt = 7;
+ attr.rnr_retry = 7;
+ ibv_modify_qp(qp, &attr, IBV_QP_STATE | IBV_QP_TIMEOUT | ...);
+}
+```
+
+### 7.3 URMA Jetty 连接建立
+
+```mermaid
+sequenceDiagram
+ participant A as Node A (Active)
+ participant B as Node B (Passive)
+
+ A->>B: RPC sendHandshake(local_eid, jetty_nums)
+ B->>B: 创建远端 Jetty 引用: urma_import_jetty(eid, jetty_id)
+ B->>B: 绑定本地 Jetty: urma_bind_jetty(local_jetty, remote_jetty)
+ B-->>A: 返回 peer_eid, peer_jetty_nums
+ A->>A: urma_import_jetty(peer_eid, peer_jetty_id)
+ A->>A: urma_bind_jetty(local_jetty, remote_jetty)
+ Note over A,B: Jetty 绑定完成，可以传输
+```
+
+---
+
+## 8. 完成与状态查询
+
+### 8.1 默认完成机制：轮询聚合
+
+默认情况下，worker 线程负责轮询底层完成队列：
+- RDMA 路径调用 `ibv_poll_cq()`，根据 `ibv_wc.status` 判断成功或失败。
+- URMA 路径调用 `urma_poll_jfc()`，根据 completion record 判断成功或失败。
+- 成功时 `Slice::markSuccess()` 增加 `transferred_bytes` 和 `success_slice_count`。
+- 失败时 `Slice::markFailed()` 增加 `failed_slice_count`。
+
+用户侧通过 `getTransferStatus()` 查询单个 task，或通过 `getBatchTransferStatus()` 聚合整个 batch 的状态。未开启事件驱动完成时，batch 完成状态主要在这些查询函数中被聚合出来。
+
+### 8.2 可选事件驱动完成机制
+
+如果编译时启用了 `USE_EVENT_DRIVEN_COMPLETION`，`Slice::check_batch_completion()` 会在最后一个 slice 完成时推进 task/batch 级计数，并通过条件变量唤醒等待者。该路径适合 `submitTransferWithNotify()` 等需要完成后触发通知的场景。
+
+```mermaid
+flowchart TD
+ A[CQ 轮询线程] --> B[ibv_poll_cq / urma_poll_jfc]
+ B --> C{wc.status == SUCCESS?}
+ C -->|Yes| D[slice->markSuccess]
+ C -->|No| E[slice->markFailed]
+
+ D --> F[__atomic_fetch_add
task.transferred_bytes]
+ D --> G[__atomic_fetch_add
task.success_slice_count]
+ D --> H[check_batch_completion]
+
+ E --> I[__atomic_fetch_add
task.failed_slice_count]
+ E --> H
+
+ H --> J{最后一个 slice 完成?}
+ J -->|Yes| K[__atomic_store
task.is_finished = true]
+ K --> L[batch.finished_task_count++]
+ L --> M{最后一个 task 完成?}
+ M -->|Yes| N[batch.is_finished = true]
+ N --> O[completion_cv.notify_all]
+ M -->|No| P[返回]
+ J -->|No| P
+```
+
+### 8.3 用户侧状态查询
+
+```cpp
+// multi_transport.cpp:190-227
+Status MultiTransport::getTransferStatus(BatchID batch_id, size_t task_id,
+ TransferStatus &status) {
+ auto &task = batch_desc.task_list[task_id];
+ status.transferred_bytes = task.transferred_bytes;
+
+ uint64_t success = task.success_slice_count;
+ uint64_t failed = task.failed_slice_count;
+
+ if (success + failed == task.slice_count) {
+ status.s = failed ? FAILED : COMPLETED;
+ task.is_finished = true;
+ } else {
+ // 超时检测
+ if (globalConfig().slice_timeout > 0) {
+ for (auto &slice : task.slice_list) {
+ if (current_ts - slice->ts > kPacketDeliveryTimeout)
+ return TIMEOUT;
+ }
+ }
+ status.s = WAITING;
+ }
+}
+```
+
+Batch 级查询会遍历所有 task，聚合 `transferred_bytes`，并在所有 task 完成时设置 `batch_desc.is_finished` 和 `finished_transfer_bytes`：
+
+```cpp
+Status MultiTransport::getBatchTransferStatus(BatchID batch_id,
+ TransferStatus &status) {
+ if (batch_desc.is_finished.load(...) || task_count == 0) {
+ status.s = COMPLETED;
+ status.transferred_bytes = batch_desc.finished_transfer_bytes.load(...);
+ return Status::OK();
+ }
+
+ for (size_t task_id = 0; task_id < task_count; task_id++) {
+ getTransferStatus(batch_id, task_id, task_status);
+ // 任一 task FAILED，则 batch FAILED；
+ // 全部 COMPLETED，则 batch COMPLETED；否则 WAITING。
+ }
+}
+```
+
+### 8.4 典型使用模式
+
+```cpp
+// 完整的读写使用示例
+TransferEngine engine;
+engine.init("etcd://127.0.0.1:2379", "192.168.1.1:12345");
+
+// 注册本地内存
+engine.registerLocalMemory(buffer, buffer_size, "cpu:0");
+
+// 打开远程段
+SegmentHandle remote = engine.openSegment("192.168.1.2:12345");
+
+// 分配批次
+BatchID batch = engine.allocateBatchID(16);
+
+// 构造写请求
+std::vector requests;
+requests.push_back({
+ .opcode = TransferRequest::WRITE,
+ .source = local_buffer,
+ .target_id = remote,
+ .target_offset = 0,
+ .length = data_size
+});
+
+// 提交传输
+engine.submitTransfer(batch, requests);
+
+// 轮询单个 task 完成状态
+TransferStatus status;
+while (true) {
+ engine.getTransferStatus(batch, 0, status);
+ if (status.s == COMPLETED || status.s == FAILED) break;
+ // 可以做其他事情...
+}
+
+// 释放资源
+engine.freeBatchID(batch);
+```
+
+如果一个 batch 内提交了多个 request，更推荐查询 batch 级状态：
+
+```cpp
+TransferStatus batch_status;
+while (true) {
+ engine.getBatchTransferStatus(batch, batch_status);
+ if (batch_status.s == COMPLETED || batch_status.s == FAILED) break;
+}
+```
+
+---
+
+## 附录：调用链速查
+
+### 初始化调用链
+```
+TransferEngine::init()
+ └→ TransferEngineImpl::init()
+ ├→ TransferMetadata(conn_string) // 创建元数据客户端
+ ├→ MultiTransport(metadata, server_name) // 创建多传输管理器
+ ├→ Topology::discover() // 发现硬件拓扑
+ └→ MultiTransport::installTransport(proto)
+ └→ RdmaTransport::install()
+ ├→ initializeRdmaResources()
+ │ └→ RdmaContext::construct() // 每个NIC创建context
+ ├→ allocateLocalSegmentID()
+ ├→ startHandshakeDaemon()
+ └→ metadata_->updateLocalSegmentDesc()
+```
+
+### 写操作调用链
+```
+TransferEngine::submitTransfer(batch_id, entries)
+ └→ TransferEngineImpl::submitTransfer()
+ └→ MultiTransport::submitTransfer()
+ ├→ selectTransport(req) // 选transport
+ └→ RdmaTransport::submitTransferTask()
+ ├→ selectDevice() // 选NIC/buffer
+ ├→ Slice 分片
+ └→ RdmaContext::submitPostSend()
+ └→ RdmaEndPoint::submitPostSend()
+ └→ ibv_post_send() // 提交RDMA操作
+```
+
+### 默认完成查询调用链
+```
+CQ 轮询线程
+ └→ ibv_poll_cq()
+ └→ Slice::markSuccess() / markFailed()
+ ├→ task.success_slice_count / failed_slice_count
+ └→ task.transferred_bytes
+
+用户线程
+ └→ getTransferStatus()
+ └→ 检查 task.is_finished && slice 计数
+ └→ getBatchTransferStatus()
+ └→ 聚合所有 task 状态并更新 batch_desc.is_finished
+```
+
+### 事件驱动完成调用链（USE_EVENT_DRIVEN_COMPLETION）
+```
+CQ/JFC 轮询线程
+ └→ Slice::markSuccess() / markFailed()
+ └→ check_batch_completion()
+ ├→ task.completed_slice_count++
+ ├→ batch_desc.finished_task_count++
+ └→ batch_desc.completion_cv.notify_all()
+```
diff --git a/mooncake-common/FindUrma.cmake b/mooncake-common/FindUrma.cmake
index d2d93cc38b..0af8d1a7ca 100644
--- a/mooncake-common/FindUrma.cmake
+++ b/mooncake-common/FindUrma.cmake
@@ -4,7 +4,7 @@ include(FetchContent)
FetchContent_Declare(
urma
GIT_REPOSITORY https://atomgit.com/openeuler/umdk.git
- GIT_TAG v25.12.0
+ GIT_TAG v25.12.0.B081
)
FetchContent_MakeAvailable(urma)
diff --git a/mooncake-integration/CMakeLists.txt b/mooncake-integration/CMakeLists.txt
index 379a5e0387..6ee283eda7 100644
--- a/mooncake-integration/CMakeLists.txt
+++ b/mooncake-integration/CMakeLists.txt
@@ -100,6 +100,11 @@ if(WITH_STORE)
pybind11_add_module(store ${SOURCES} ${CACHE_ALLOCATOR_SOURCES}
store/store_py.cpp store/engram_store_py.cpp integration_utils.h)
set_target_properties(store PROPERTIES INSTALL_RPATH "$ORIGIN")
+
+ find_package(UbDiag REQUIRED)
+ target_include_directories(store PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/store)
+ target_link_libraries(store PRIVATE UbDiag::ubdiag_lib)
+
if(USE_ASCEND_DIRECT)
target_link_libraries(
store PUBLIC ascendcl transfer_engine glog::glog gflags::gflags
diff --git a/mooncake-integration/store/mooncake_perf_points.def b/mooncake-integration/store/mooncake_perf_points.def
new file mode 100644
index 0000000000..1f0d3191ee
--- /dev/null
+++ b/mooncake-integration/store/mooncake_perf_points.def
@@ -0,0 +1,89 @@
+// === Python绑定层 ===
+PERF_KEY_DEF(GET_STORE_PY_GET, "store_py.cpp::get", "Get")
+PERF_KEY_DEF(GET_STORE_PY_GET_BATCH, "store_py.cpp::get_batch", "GetBatch")
+
+// === RealClient核心逻辑层 ===
+PERF_KEY_DEF(GET_BUFFER_INTERNAL, "store_py.cpp::get", "GetBuffer")
+PERF_KEY_DEF(GET_BATCH_BUFFER_INTERNAL, "store_py.cpp::get_batch", "BatchGetBuffer")
+
+// === get_buffer_internal 子步骤 ===
+PERF_KEY_DEF(GET_INTERNAL_QUERY, "real_client.cpp::get_buffer_internal", "Query")
+PERF_KEY_DEF(GET_INTERNAL_SELECT_REPLICA, "real_client.cpp::get_buffer_internal", "SelectReplica")
+PERF_KEY_DEF(GET_INTERNAL_ALLOC_BUFFER, "real_client.cpp::get_buffer_internal", "AllocBuffer")
+PERF_KEY_DEF(GET_INTERNAL_SSD_READ, "real_client.cpp::get_buffer_internal", "SSDRead")
+PERF_KEY_DEF(GET_INTERNAL_MEM_READ, "real_client.cpp::get_buffer_internal", "MemRead")
+PERF_KEY_DEF(GET_INTERNAL_DISK_READ, "real_client.cpp::get_buffer_internal", "DiskRead")
+
+// === batch_get_buffer_internal 子步骤 ===
+PERF_KEY_DEF(GET_BATCH_INTERNAL_QUERY, "real_client.cpp::batch_get_buffer_internal", "BatchQuery")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_PREPARATION, "real_client.cpp::batch_get_buffer_internal", "Preparation")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_SELECT_REPLICA, "real_client.cpp::batch_get_buffer_internal", "SelectReplica")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_ALLOC_BUFFER, "real_client.cpp::batch_get_buffer_internal", "AllocBuffer")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_SSD_READ, "real_client.cpp::batch_get_buffer_internal", "SSDRead")
+PERF_KEY_DEF(GET_BATCH_INTERNAL_MEMDISH_READ,"real_client.cpp::batch_get_buffer_internal", "MemDiskRead")
+
+// === Client::Get (单key) ===
+PERF_KEY_DEF(GET_SINGLE_FULL, "client_service.cpp::Get", "TransferGet")
+PERF_KEY_DEF(GET_SINGLE_FIND_REPLICA, "client_service.cpp::Get", "FindReplica")
+PERF_KEY_DEF(GET_SINGLE_HOT_CACHE, "client_service.cpp::Get", "HotCache")
+PERF_KEY_DEF(GET_SINGLE_TRANSFER_READ, "client_service.cpp::Get", "TransferRead")
+PERF_KEY_DEF(GET_SINGLE_RELEASE_CACHE, "client_service.cpp::Get", "ReleaseCache")
+PERF_KEY_DEF(GET_SINGLE_ASYNC_CACHE, "client_service.cpp::Get", "AsyncCache")
+
+// === Client::Get 内 TransferData ===
+PERF_KEY_DEF(GET_SINGLE_TRANSFER_FULL, "client_service.cpp::TransferData", "TransferData")
+PERF_KEY_DEF(GET_SINGLE_TRANSFER_SUBMIT, "client_service.cpp::TransferData", "Submit")
+PERF_KEY_DEF(GET_SINGLE_TRANSFER_WAIT, "client_service.cpp::TransferData", "Wait")
+
+// === Client::BatchGet (批量) ===
+PERF_KEY_DEF(GET_BATCH_FULL, "client_service.cpp::BatchGet", "TransferBatchGet")
+PERF_KEY_DEF(GET_BATCH_FIND_REPLICA, "client_service.cpp::BatchGet", "FindReplica")
+PERF_KEY_DEF(GET_BATCH_HOT_CACHE, "client_service.cpp::BatchGet", "HotCache")
+PERF_KEY_DEF(GET_BATCH_SUBMIT, "client_service.cpp::BatchGet", "Submit")
+PERF_KEY_DEF(GET_BATCH_WAIT, "client_service.cpp::BatchGet", "Wait")
+PERF_KEY_DEF(GET_BATCH_RELEASE_CACHE, "client_service.cpp::BatchGet", "ReleaseCache")
+PERF_KEY_DEF(GET_BATCH_ASYNC_CACHE, "client_service.cpp::BatchGet", "AsyncCache")
+
+// === Python绑定层 ===
+PERF_KEY_DEF(PUT_STORE_PY_PUT, "store_py.cpp::put", "Put")
+PERF_KEY_DEF(PUT_STORE_PY_PUT_BATCH, "store_py.cpp::put_batch", "PutBatch")
+
+// === RealClient核心逻辑层 ===
+PERF_KEY_DEF(PUT_INTERNAL_FULL, "store_py.cpp::put", "PutBuffer")
+PERF_KEY_DEF(PUT_INTERNAL_ALLOC_BUFFER, "real_client.cpp::put_internal", "AllocBuffer")
+PERF_KEY_DEF(PUT_INTERNAL_MEM_COPY, "real_client.cpp::put_internal", "MemCopy")
+PERF_KEY_DEF(PUT_INTERNAL_SPLIT_SLICES, "real_client.cpp::put_internal", "SplitSlices")
+
+PERF_KEY_DEF(PUT_BATCH_INTERNAL_FULL, "store_py.cpp::put_batch", "BatchPutBuffer")
+PERF_KEY_DEF(PUT_BATCH_INTERNAL_ALLOC_BUFFER,"real_client.cpp::put_batch_internal", "AllocBuffer")
+PERF_KEY_DEF(PUT_BATCH_INTERNAL_MEM_COPY, "real_client.cpp::put_batch_internal", "MemCopy")
+PERF_KEY_DEF(PUT_BATCH_INTERNAL_SPLIT_SLICES,"real_client.cpp::put_batch_internal", "SplitSlices")
+
+// === Client::Put (单key) ===
+PERF_KEY_DEF(PUT_SINGLE_FULL, "client_service.cpp::Put", "TransferPut")
+PERF_KEY_DEF(PUT_SINGLE_PUT_START, "client_service.cpp::Put", "PutStart")
+PERF_KEY_DEF(PUT_SINGLE_DISK_WRITE, "client_service.cpp::Put", "DiskWrite")
+PERF_KEY_DEF(PUT_SINGLE_TRANSFER_WRITE, "client_service.cpp::Put", "TransferWrite")
+PERF_KEY_DEF(PUT_SINGLE_PUT_END, "client_service.cpp::Put", "PutEnd")
+PERF_KEY_DEF(PUT_SINGLE_PUT_REVOKE, "client_service.cpp::Put", "PutRevoke")
+
+// === Client::Put 内 TransferData ===
+PERF_KEY_DEF(PUT_SINGLE_TRANSFER_FULL, "client_service.cpp::TransferData", "TransferData")
+PERF_KEY_DEF(PUT_SINGLE_TRANSFER_SUBMIT, "client_service.cpp::TransferData", "Submit")
+PERF_KEY_DEF(PUT_SINGLE_TRANSFER_WAIT, "client_service.cpp::TransferData", "Wait")
+
+// === Client::BatchPut (批量) ===
+PERF_KEY_DEF(PUT_BATCH_FULL, "client_service.cpp::BatchPut", "TransferBatchPut")
+PERF_KEY_DEF(PUT_BATCH_CREATE_OPS, "client_service.cpp::BatchPut", "CreateOps")
+PERF_KEY_DEF(PUT_BATCH_PUT_START, "client_service.cpp::StartBatchPut", "PutStart")
+PERF_KEY_DEF(PUT_BATCH_SUBMIT, "client_service.cpp::SubmitTransfers", "Submit")
+PERF_KEY_DEF(PUT_BATCH_DISK_WRITE, "client_service.cpp::SubmitTransfers", "DiskWrite")
+PERF_KEY_DEF(PUT_BATCH_WAIT, "client_service.cpp::WaitForTransfers", "Wait")
+PERF_KEY_DEF(PUT_BATCH_PUT_END, "client_service.cpp::FinalizeBatchPut", "PutEnd")
+PERF_KEY_DEF(PUT_BATCH_PUT_REVOKE, "client_service.cpp::FinalizeBatchPut", "PutRevoke")
+PERF_KEY_DEF(PUT_BATCH_COLLECT_RESULTS, "client_service.cpp::BatchPut", "CollectResults")
+
+// === batch_get_into_offload_object_internal 子步骤 (SSD read) ===
+PERF_KEY_DEF(GET_SSD_OFFLOAD_RPC, "real_client.cpp::batch_get_into_offload_object_internal", "OffloadRpc")
+PERF_KEY_DEF(GET_SSD_TRANSFER_DATA, "real_client.cpp::batch_get_into_offload_object_internal", "TransferData")
+PERF_KEY_DEF(GET_SSD_RELEASE_BUFFER, "real_client.cpp::batch_get_into_offload_object_internal", "ReleaseBuffer")
diff --git a/mooncake-integration/store/store_py.cpp b/mooncake-integration/store/store_py.cpp
index ccb25d9d18..ee97415fe1 100644
--- a/mooncake-integration/store/store_py.cpp
+++ b/mooncake-integration/store/store_py.cpp
@@ -2,6 +2,7 @@
#include
#include
+#include
#include
#include
#include
@@ -17,6 +18,10 @@
#include "integration_utils.h"
+#define UBDIAG_PERF_DEF_FILE "mooncake_perf_points.def"
+#define UBDIAG_PROGRAM_NAME "mooncake_store"
+#include "ubdiag/auto_perf.h"
+
// Forward declaration for EngramStore bindings
namespace mooncake {
namespace engram {
@@ -395,8 +400,14 @@ class MooncakeStorePyWrapper {
}
pybind11::bytes get(const std::string &key) {
+ auto start = std::chrono::steady_clock::now();
+ LOG(INFO) << "get start key[" << key << "]";
+
+ UbDiag::PerfPoint pt(PerfKey::GET_STORE_PY_GET, UbDiag::PerfLevel::SUB_SYSTEM);
+ pt.Start();
if (!is_client_initialized()) {
LOG(ERROR) << "Client is not initialized";
+ pt.End(-1);
return pybind11::bytes("\\0", 0);
}
@@ -404,13 +415,31 @@ class MooncakeStorePyWrapper {
{
py::gil_scoped_release release_gil;
+ UbDiag::PerfPoint pt_full(PerfKey::GET_BUFFER_INTERNAL, UbDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
auto buffer_handle = store_->get_buffer(key);
+ pt_full.End(buffer_handle ? 0 : -1);
if (!buffer_handle) {
+ pt.End(-1);
py::gil_scoped_acquire acquire_gil;
+ auto elapsed_us = std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start).count();
+ LOG(INFO) << "get complete key[" << key << "] rc[-1] elapsed_us[" << elapsed_us << "]";
+ if (elapsed_us > 3000) {
+ LOG(WARNING) << "get_slow key[" << key << "] elapsed_us[" << elapsed_us << "]";
+ }
return kNullString;
}
py::gil_scoped_acquire acquire_gil;
+ pt.End(0);
+ auto size = buffer_handle->size();
+ auto elapsed_us = std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start).count();
+ LOG(INFO) << "get complete key[" << key << "] rc[0] size[" << size << "] elapsed_us[" << elapsed_us << "]";
+ if (elapsed_us > 3000) {
+ LOG(WARNING) << "get_slow key[" << key << "] size[" << size << "] elapsed_us[" << elapsed_us << "]";
+ }
return pybind11::bytes((char *)buffer_handle->ptr(),
buffer_handle->size());
}
@@ -418,18 +447,37 @@ class MooncakeStorePyWrapper {
std::vector get_batch(
const std::vector &keys) {
+ auto start = std::chrono::steady_clock::now();
+ LOG(INFO) << "get_batch start num_keys[" << keys.size() << "]";
+
+ UbDiag::PerfPoint pt(PerfKey::GET_STORE_PY_GET_BATCH, UbDiag::PerfLevel::SUB_SYSTEM);
+ pt.Start();
const auto kNullString = pybind11::bytes("\\0", 0);
if (!is_client_initialized()) {
LOG(ERROR) << "Client is not initialized";
+ pt.End(-1);
py::gil_scoped_acquire acquire_gil;
+ auto elapsed_us = std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start).count();
+ LOG(INFO) << "get_batch complete num_keys[" << keys.size() << "] rc[-1] elapsed_us[" << elapsed_us << "]";
return {kNullString};
}
{
py::gil_scoped_release release_gil;
+ UbDiag::PerfPoint pt_full(PerfKey::GET_BATCH_BUFFER_INTERNAL, UbDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
auto batch_data = store_->batch_get_buffer(keys);
+ pt_full.End(0);
if (batch_data.empty()) {
+ pt.End(-1);
py::gil_scoped_acquire acquire_gil;
+ auto elapsed_us = std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start).count();
+ LOG(INFO) << "get_batch complete num_keys[" << keys.size() << "] rc[-1] elapsed_us[" << elapsed_us << "]";
+ if (elapsed_us > 10000) {
+ LOG(WARNING) << "get_batch_slow num_keys[" << keys.size() << "] elapsed_us[" << elapsed_us << "]";
+ }
return {kNullString};
}
@@ -437,11 +485,21 @@ class MooncakeStorePyWrapper {
std::vector results;
results.reserve(batch_data.size());
+ size_t success_count = 0;
for (const auto &data : batch_data) {
+ if (data) success_count++;
results.emplace_back(
data ? pybind11::bytes((char *)data->ptr(), data->size())
: kNullString);
}
+ pt.End(0);
+ auto elapsed_us = std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start).count();
+ LOG(INFO) << "get_batch complete num_keys[" << keys.size() << "] success[" << success_count
+ << "] rc[0] elapsed_us[" << elapsed_us << "]";
+ if (elapsed_us > 10000) {
+ LOG(WARNING) << "get_batch_slow num_keys[" << keys.size() << "] elapsed_us[" << elapsed_us << "]";
+ }
return results;
}
}
@@ -2546,13 +2604,31 @@ PYBIND11_MODULE(store, m) {
[](MooncakeStorePyWrapper &self, const std::string &key,
py::buffer buf,
const ReplicateConfig &config = ReplicateConfig{}) {
+ auto start = std::chrono::steady_clock::now();
py::buffer_info info = buf.request(/*writable=*/false);
+ size_t size = static_cast(info.size);
+ LOG(INFO) << "put start key[" << key << "] size[" << size << "]";
+
+ UbDiag::PerfPoint pt(PerfKey::PUT_STORE_PY_PUT, UbDiag::PerfLevel::SUB_SYSTEM);
+ pt.Start();
py::gil_scoped_release release;
- return self.store_->put(
+ UbDiag::PerfPoint pt_full(PerfKey::PUT_INTERNAL_FULL, UbDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
+ auto ret = self.store_->put(
key,
std::span(static_cast(info.ptr),
static_cast(info.size)),
config);
+ pt_full.End(ret == 0 ? 0 : -1);
+ pt.End(ret == 0 ? 0 : -1);
+
+ auto elapsed_us = std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start).count();
+ LOG(INFO) << "put complete key[" << key << "] rc[" << ret << "] elapsed_us[" << elapsed_us << "]";
+ if (elapsed_us > 3000) {
+ LOG(WARNING) << "put_slow key[" << key << "] size[" << size << "] elapsed_us[" << elapsed_us << "]";
+ }
+ return ret;
},
py::arg("key"), py::arg("value"),
py::arg("config") = ReplicateConfig{})
@@ -2590,21 +2666,41 @@ PYBIND11_MODULE(store, m) {
const std::vector &keys,
const std::vector &buffers,
const ReplicateConfig &config = ReplicateConfig{}) {
+ auto start = std::chrono::steady_clock::now();
+
+ UbDiag::PerfPoint pt(PerfKey::PUT_STORE_PY_PUT_BATCH, UbDiag::PerfLevel::SUB_SYSTEM);
+ pt.Start();
// Convert pybuffers to spans without copying
std::vector infos;
std::vector> spans;
infos.reserve(buffers.size());
spans.reserve(buffers.size());
+ size_t total_size = 0;
for (const auto &buf : buffers) {
infos.emplace_back(buf.request(/*writable=*/false));
const auto &info = infos.back();
+ total_size += static_cast(info.size);
spans.emplace_back(static_cast(info.ptr),
static_cast(info.size));
}
+ LOG(INFO) << "put_batch start num_keys[" << keys.size() << "] total_size[" << total_size << "]";
+
py::gil_scoped_release release;
- return self.store_->put_batch(keys, spans, config);
+ UbDiag::PerfPoint pt_full(PerfKey::PUT_BATCH_INTERNAL_FULL, UbDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
+ auto ret = self.store_->put_batch(keys, spans, config);
+ pt_full.End(ret == 0 ? 0 : -1);
+ pt.End(ret == 0 ? 0 : -1);
+
+ auto elapsed_us = std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - start).count();
+ LOG(INFO) << "put_batch complete num_keys[" << keys.size() << "] rc[" << ret << "] elapsed_us[" << elapsed_us << "]";
+ if (elapsed_us > 10000) {
+ LOG(WARNING) << "put_batch_slow num_keys[" << keys.size() << "] elapsed_us[" << elapsed_us << "]";
+ }
+ return ret;
},
py::arg("keys"), py::arg("values"),
py::arg("config") = ReplicateConfig{})
diff --git a/mooncake-p2p-store/build.sh b/mooncake-p2p-store/build.sh
index 66eff5f107..72aca4b3c6 100644
--- a/mooncake-p2p-store/build.sh
+++ b/mooncake-p2p-store/build.sh
@@ -53,6 +53,10 @@ if [ -d "/usr/local/musa/lib" ]; then
EXT_LDFLAGS+=" -L/usr/local/musa/lib -lmusart"
fi
+if [ -e "/usr/lib64/liburma.so" ]; then
+ EXT_LDFLAGS+=" -L/usr/lib64 -lurma"
+fi
+
if [ "$USE_ETCD" = "ON" ]; then
if [ "$USE_ETCD_LEGACY" = "ON" ]; then
EXT_LDFLAGS+=" -letcd-cpp-api -lprotobuf -lgrpc++ -lgrpc"
diff --git a/mooncake-store/benchmarks/CMakeLists.txt b/mooncake-store/benchmarks/CMakeLists.txt
index 8798af8940..69716312c3 100644
--- a/mooncake-store/benchmarks/CMakeLists.txt
+++ b/mooncake-store/benchmarks/CMakeLists.txt
@@ -27,3 +27,8 @@ add_executable(allocation_strategy_bench allocation_strategy_bench.cpp)
target_link_libraries(
allocation_strategy_bench PRIVATE mooncake_store cachelib_memory_allocator
gflags::gflags glog::glog pthread)
+
+add_executable(stress_cluster_bench stress_cluster_bench.cpp)
+target_link_libraries(
+ stress_cluster_bench PRIVATE mooncake_store transfer_engine asio_shared
+ gflags::gflags glog::glog pthread)
diff --git a/mooncake-store/benchmarks/stress_cluster_bench.cpp b/mooncake-store/benchmarks/stress_cluster_bench.cpp
new file mode 100644
index 0000000000..6cd830e32a
--- /dev/null
+++ b/mooncake-store/benchmarks/stress_cluster_bench.cpp
@@ -0,0 +1,812 @@
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "gflags/gflags.h"
+#include "glog/logging.h"
+#include "real_client.h"
+
+namespace {
+constexpr size_t KB = 1024;
+constexpr size_t MB = 1024 * KB;
+constexpr size_t GB = 1024 * MB;
+
+const static int NR_SOCKETS =
+ numa_available() == 0 ? numa_num_configured_nodes() : 1;
+
+static void bindToSocket(int socket_id) {
+ if (numa_available() < 0) return;
+ cpu_set_t cpu_set;
+ CPU_ZERO(&cpu_set);
+ if (socket_id < 0 || socket_id >= numa_num_configured_nodes())
+ socket_id = 0;
+ struct bitmask* cpu_list = numa_allocate_cpumask();
+ numa_node_to_cpus(socket_id, cpu_list);
+ int nr_possible_cpus = numa_num_possible_cpus();
+ int nr_cpus = 0;
+ for (int cpu = 0; cpu < nr_possible_cpus; ++cpu) {
+ if (numa_bitmask_isbitset(cpu_list, cpu) &&
+ numa_bitmask_isbitset(numa_all_cpus_ptr, cpu)) {
+ CPU_SET(cpu, &cpu_set);
+ ++nr_cpus;
+ }
+ }
+ numa_free_cpumask(cpu_list);
+ if (nr_cpus > 0) {
+ sched_setaffinity(0, sizeof(cpu_set), &cpu_set);
+ }
+}
+} // namespace
+
+DEFINE_string(local_hostname, "localhost",
+ "Local hostname (with optional port, e.g. node1:12345)");
+DEFINE_string(metadata_server, "http://127.0.0.1:8080/metadata",
+ "Metadata server URL");
+DEFINE_string(master_server, "127.0.0.1:50051", "Master server address");
+DEFINE_string(protocol, "tcp", "Transport protocol: tcp, rdma, ub");
+DEFINE_string(device_name, "", "RDMA/UB device name (comma-separated)");
+DEFINE_uint64(global_segment_size, 4 * GB, "Global segment size in bytes");
+DEFINE_uint64(local_buffer_size, 512 * MB, "Local buffer size in bytes");
+DEFINE_bool(enable_ssd_offload, false, "Enable SSD offload on this client");
+DEFINE_string(ssd_offload_path, "", "SSD offload directory path");
+
+DEFINE_string(scenario, "local_memory",
+ "Benchmark scenario: local_memory, remote_memory, local_disk, "
+ "remote_disk");
+DEFINE_string(role, "writer",
+ "Node role: writer (prefill data) or reader (benchmark reads)");
+DEFINE_uint64(value_size, 4 * MB, "Size of each value in bytes");
+DEFINE_uint64(num_keys, 100, "Number of keys to write/read");
+DEFINE_uint64(batch_size, 32, "Batch size for put/get operations");
+DEFINE_uint64(num_threads, 1, "Number of concurrent reader threads");
+DEFINE_uint64(warmup_keys, 5, "Number of warmup keys (not counted in stats)");
+DEFINE_uint64(wait_seconds, 5,
+ "Seconds to wait before reading (for remote scenarios)");
+DEFINE_bool(verify, true, "Verify data integrity after read");
+DEFINE_uint64(replica_num, 1, "Number of replicas for each object");
+DEFINE_bool(hard_pin, false,
+ "Pin objects to prevent eviction during benchmark");
+
+using Clock = std::chrono::steady_clock;
+using Nanos = std::chrono::nanoseconds;
+
+inline int64_t ElapsedNanos(Clock::time_point t0, Clock::time_point t1) {
+ return std::chrono::duration_cast(t1 - t0).count();
+}
+
+inline double NanosToUs(int64_t ns) { return static_cast(ns) / 1000.0; }
+inline double NanosToMs(int64_t ns) { return static_cast(ns) / 1000000.0; }
+inline double NanosToSec(int64_t ns) { return static_cast(ns) / 1e9; }
+
+struct ThreadResult {
+ std::vector latencies_ns;
+ size_t total_bytes = 0;
+ size_t total_ops = 0;
+ size_t failed_ops = 0;
+};
+
+class BenchmarkStats {
+ public:
+ void InitThreads(size_t n, size_t expected_per_thread) {
+ thread_results_.resize(n);
+ expected_per_thread_ = expected_per_thread;
+ }
+
+ ThreadResult& GetThreadResult(size_t tid) { return thread_results_[tid]; }
+
+ void StartTimer() { start_ = Clock::now(); }
+ void StopTimer() { end_ = Clock::now(); }
+
+ double WallSeconds() const {
+ return NanosToSec(ElapsedNanos(start_, end_));
+ }
+
+ void Finalize() {
+ merged_latencies_ns_.clear();
+ total_bytes_ = 0;
+ total_ops_ = 0;
+ total_failed_ = 0;
+
+ for (auto& tr : thread_results_) {
+ merged_latencies_ns_.insert(merged_latencies_ns_.end(),
+ tr.latencies_ns.begin(),
+ tr.latencies_ns.end());
+ total_bytes_ += tr.total_bytes;
+ total_ops_ += tr.total_ops;
+ total_failed_ += tr.failed_ops;
+ }
+ std::sort(merged_latencies_ns_.begin(), merged_latencies_ns_.end());
+ }
+
+ double PercentileUs(double p) const {
+ if (merged_latencies_ns_.empty()) return 0.0;
+ double rank = (p / 100.0) * (merged_latencies_ns_.size() - 1);
+ size_t lo = static_cast(rank);
+ size_t hi = std::min(lo + 1, merged_latencies_ns_.size() - 1);
+ double frac = rank - lo;
+ int64_t ns_val = static_cast(
+ merged_latencies_ns_[lo] * (1.0 - frac) +
+ merged_latencies_ns_[hi] * frac);
+ return NanosToUs(ns_val);
+ }
+
+ double MeanLatencyUs() const {
+ if (merged_latencies_ns_.empty()) return 0.0;
+ int64_t sum = std::accumulate(merged_latencies_ns_.begin(),
+ merged_latencies_ns_.end(), int64_t(0));
+ return NanosToUs(sum / static_cast(merged_latencies_ns_.size()));
+ }
+
+ double ThroughputMBps() const {
+ double wall = WallSeconds();
+ return (wall > 0) ? (static_cast(total_bytes_) / MB) / wall : 0;
+ }
+
+ double OpsPerSec() const {
+ double wall = WallSeconds();
+ return (wall > 0) ? static_cast(total_ops_) / wall : 0;
+ }
+
+ void Print(const std::string& title) const {
+ std::cout << "\n";
+ std::cout << "========================================"
+ << "========================================\n";
+ std::cout << " " << title << "\n";
+ std::cout << "========================================"
+ << "========================================\n";
+ std::cout << std::fixed << std::setprecision(2);
+
+ double wall = WallSeconds();
+ std::cout << " Wall time: " << wall << " s\n";
+ std::cout << " Total ops: " << total_ops_
+ << " (failed: " << total_failed_ << ")\n";
+ std::cout << " Total data: " << FormatBytes(total_bytes_)
+ << "\n";
+ std::cout << " Throughput: " << ThroughputMBps() << " MB/s";
+ if (ThroughputMBps() > 1024) {
+ std::cout << " (" << ThroughputMBps() / 1024 << " GB/s)";
+ }
+ std::cout << "\n";
+ std::cout << " Ops/sec: " << OpsPerSec() << "\n";
+
+ if (!merged_latencies_ns_.empty()) {
+ size_t n = merged_latencies_ns_.size();
+ std::cout << "\n Latency (us) [n=" << n << "]\n";
+ std::cout << " Min: " << std::setw(12)
+ << NanosToUs(merged_latencies_ns_.front()) << "\n";
+ std::cout << " Mean: " << std::setw(12) << MeanLatencyUs()
+ << "\n";
+ std::cout << " P50: " << std::setw(12) << PercentileUs(50)
+ << "\n";
+ std::cout << " P90: " << std::setw(12) << PercentileUs(90)
+ << "\n";
+ std::cout << " P99: " << std::setw(12) << PercentileUs(99);
+ if (n < 100) std::cout << " (n<100)";
+ std::cout << "\n";
+ std::cout << " P999: " << std::setw(12) << PercentileUs(99.9);
+ if (n < 1000) std::cout << " (n<1000)";
+ std::cout << "\n";
+ std::cout << " Max: " << std::setw(12)
+ << NanosToUs(merged_latencies_ns_.back()) << "\n";
+ }
+ std::cout << "========================================"
+ << "========================================\n\n";
+ }
+
+ size_t total_bytes() const { return total_bytes_; }
+ size_t total_ops() const { return total_ops_; }
+ size_t total_failed() const { return total_failed_; }
+
+ private:
+ static std::string FormatBytes(size_t bytes) {
+ if (bytes == 0) return "0 B";
+ const char* units[] = {"B", "KB", "MB", "GB", "TB"};
+ int i = static_cast(std::floor(std::log2(bytes) / 10));
+ if (i > 4) i = 4;
+ double val = static_cast(bytes) / std::pow(1024, i);
+ std::ostringstream oss;
+ oss << std::fixed << std::setprecision(2) << val << " " << units[i];
+ return oss.str();
+ }
+
+ std::vector thread_results_;
+ std::vector merged_latencies_ns_;
+ size_t total_bytes_ = 0;
+ size_t total_ops_ = 0;
+ size_t total_failed_ = 0;
+ size_t expected_per_thread_ = 0;
+ Clock::time_point start_;
+ Clock::time_point end_;
+};
+
+class StressBenchmark {
+ public:
+ StressBenchmark()
+ : client_(mooncake::RealClient::create()),
+ buffer_(nullptr),
+ buffer_size_(0) {}
+
+ ~StressBenchmark() {
+ if (client_) {
+ for (auto& tb : thread_buffers_) {
+ if (tb.ptr) {
+ client_->unregister_buffer(tb.ptr);
+ numa_free(tb.ptr, tb.size);
+ }
+ }
+ thread_buffers_.clear();
+ if (buffer_) {
+ client_->unregister_buffer(buffer_);
+ numa_free(buffer_, buffer_size_);
+ buffer_ = nullptr;
+ }
+ client_->tearDownAll();
+ }
+ }
+
+ int Setup() {
+ int ret = client_->setup_real(
+ FLAGS_local_hostname, FLAGS_metadata_server,
+ FLAGS_global_segment_size, FLAGS_local_buffer_size, FLAGS_protocol,
+ FLAGS_device_name, FLAGS_master_server, nullptr, "",
+ FLAGS_enable_ssd_offload, FLAGS_ssd_offload_path);
+ if (ret != 0) {
+ LOG(ERROR) << "RealClient setup_real failed, ret=" << ret;
+ return ret;
+ }
+ LOG(INFO) << "RealClient setup succeeded"
+ << (FLAGS_enable_ssd_offload ? " (SSD offload enabled)" : "");
+
+ buffer_size_ = FLAGS_batch_size * FLAGS_value_size;
+ buffer_ = reinterpret_cast(numa_alloc_local(buffer_size_));
+ if (!buffer_) {
+ LOG(ERROR) << "Failed to allocate buffer of " << buffer_size_
+ << " bytes";
+ return -1;
+ }
+ std::memset(buffer_, 0, buffer_size_);
+
+ ret = client_->register_buffer(buffer_, buffer_size_);
+ if (ret != 0) {
+ LOG(ERROR) << "register_buffer failed, ret=" << ret;
+ return ret;
+ }
+ LOG(INFO) << "Registered buffer of " << buffer_size_ / MB << " MB";
+ return 0;
+ }
+
+ int RunWriter() {
+ LOG(INFO) << "=== WRITER MODE ===";
+ LOG(INFO) << "Writing " << FLAGS_num_keys << " keys, each "
+ << FLAGS_value_size / MB << " MB";
+
+ mooncake::ReplicateConfig config;
+ config.replica_num = FLAGS_replica_num;
+ config.with_hard_pin = FLAGS_hard_pin;
+
+ size_t written = 0;
+ size_t failed = 0;
+
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ std::string key = MakeKey(i);
+ FillBuffer(i);
+
+ auto t0 = Clock::now();
+ int ret = client_->put_from(key, buffer_, FLAGS_value_size, config);
+ auto t1 = Clock::now();
+
+ if (ret != 0) {
+ LOG(ERROR) << "put_from failed for key=" << key
+ << " ret=" << ret;
+ ++failed;
+ continue;
+ }
+ ++written;
+
+ if ((i + 1) % 10 == 0 || i == FLAGS_num_keys - 1) {
+ double elapsed_us = NanosToUs(ElapsedNanos(t0, t1));
+ LOG(INFO) << " Written " << (i + 1) << "/" << FLAGS_num_keys
+ << " last_latency=" << elapsed_us << " us";
+ }
+ }
+
+ LOG(INFO) << "Write complete: " << written << " succeeded, " << failed
+ << " failed";
+ LOG(INFO) << "Waiting " << FLAGS_wait_seconds
+ << " seconds for reader to connect...";
+ std::this_thread::sleep_for(std::chrono::seconds(FLAGS_wait_seconds));
+
+ return (failed > 0) ? -1 : 0;
+ }
+
+ int RunReader() {
+ LOG(INFO) << "=== READER MODE ===";
+ LOG(INFO) << "Scenario: " << FLAGS_scenario;
+ LOG(INFO) << "Reading " << FLAGS_num_keys << " keys with "
+ << FLAGS_num_threads
+ << " threads, batch_size=" << FLAGS_batch_size;
+
+ int buf_ret = AllocateThreadBuffers(FLAGS_num_threads);
+ if (buf_ret != 0) return buf_ret;
+
+ if (FLAGS_scenario == "remote_memory" ||
+ FLAGS_scenario == "remote_disk") {
+ LOG(INFO) << "Waiting " << FLAGS_wait_seconds
+ << " seconds for writer to finish prefill...";
+ std::this_thread::sleep_for(
+ std::chrono::seconds(FLAGS_wait_seconds));
+ }
+
+ int warmup_ret = DoWarmup();
+ if (warmup_ret != 0) {
+ LOG(WARNING) << "Warmup had errors, continuing anyway";
+ }
+
+ BenchmarkStats stats;
+ stats.InitThreads(FLAGS_num_threads,
+ FLAGS_num_keys / FLAGS_num_threads);
+ stats.StartTimer();
+
+ std::latch start_latch(static_cast(FLAGS_num_threads));
+ std::latch done_latch(static_cast(FLAGS_num_threads));
+ std::vector threads;
+
+ size_t keys_per_thread = FLAGS_num_keys / FLAGS_num_threads;
+ size_t remainder = FLAGS_num_keys % FLAGS_num_threads;
+
+ for (size_t t = 0; t < FLAGS_num_threads; ++t) {
+ size_t my_keys = keys_per_thread + (t < remainder ? 1 : 0);
+ size_t key_offset = t * keys_per_thread + std::min(t, remainder);
+
+ threads.emplace_back([&, t, my_keys, key_offset]() {
+ ReadWorker(t, my_keys, key_offset, stats, start_latch,
+ done_latch);
+ });
+ }
+
+ done_latch.wait();
+ stats.StopTimer();
+
+ for (auto& th : threads) {
+ th.join();
+ }
+
+ stats.Finalize();
+
+ std::string title = "READ BENCHMARK [" + FLAGS_scenario + "]";
+ stats.Print(title);
+
+ if (FLAGS_verify) {
+ int v = VerifyData();
+ if (v != 0) {
+ LOG(ERROR) << "Data verification FAILED";
+ } else {
+ LOG(INFO) << "Data verification PASSED";
+ }
+ }
+
+ return 0;
+ }
+
+ int RunLocalMemory() {
+ LOG(INFO) << "=== LOCAL MEMORY BENCHMARK ===";
+
+ int buf_ret = AllocateThreadBuffers(FLAGS_num_threads);
+ if (buf_ret != 0) return buf_ret;
+
+ mooncake::ReplicateConfig config;
+ config.replica_num = FLAGS_replica_num;
+ config.with_hard_pin = FLAGS_hard_pin;
+
+ LOG(INFO) << "Phase 1: Writing " << FLAGS_num_keys << " keys...";
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ std::string key = MakeKey(i);
+ FillBuffer(i);
+ int ret = client_->put_from(key, buffer_, FLAGS_value_size, config);
+ if (ret != 0) {
+ LOG(ERROR) << "put_from failed for key=" << key;
+ return ret;
+ }
+ if ((i + 1) % 50 == 0) {
+ LOG(INFO) << " Written " << (i + 1) << "/" << FLAGS_num_keys;
+ }
+ }
+ LOG(INFO) << "Write phase complete";
+
+ int warmup_ret = DoWarmup();
+ if (warmup_ret != 0) {
+ LOG(WARNING) << "Warmup had errors, continuing anyway";
+ }
+
+ LOG(INFO) << "Phase 2: Concurrent reads with " << FLAGS_num_threads
+ << " threads";
+
+ BenchmarkStats stats;
+ stats.InitThreads(FLAGS_num_threads,
+ FLAGS_num_keys / FLAGS_num_threads);
+ stats.StartTimer();
+
+ std::latch start_latch(static_cast(FLAGS_num_threads));
+ std::latch done_latch(static_cast(FLAGS_num_threads));
+ std::vector threads;
+
+ size_t keys_per_thread = FLAGS_num_keys / FLAGS_num_threads;
+ size_t remainder = FLAGS_num_keys % FLAGS_num_threads;
+
+ for (size_t t = 0; t < FLAGS_num_threads; ++t) {
+ size_t my_keys = keys_per_thread + (t < remainder ? 1 : 0);
+ size_t key_offset = t * keys_per_thread + std::min(t, remainder);
+
+ threads.emplace_back([&, t, my_keys, key_offset]() {
+ ReadWorker(t, my_keys, key_offset, stats, start_latch,
+ done_latch);
+ });
+ }
+
+ done_latch.wait();
+ stats.StopTimer();
+
+ for (auto& th : threads) {
+ th.join();
+ }
+
+ stats.Finalize();
+ stats.Print("LOCAL MEMORY READ BENCHMARK");
+
+ if (FLAGS_verify) {
+ int v = VerifyData();
+ LOG_IF(INFO, v == 0) << "Data verification PASSED";
+ LOG_IF(ERROR, v != 0) << "Data verification FAILED";
+ }
+
+ return 0;
+ }
+
+ int RunLocalDisk() {
+ LOG(INFO) << "=== LOCAL DISK BENCHMARK ===";
+ LOG(INFO) << "NOTE: Disk reads require Master with enable_offload=true "
+ << "and client with enable_ssd_offload=true";
+
+ int buf_ret = AllocateThreadBuffers(FLAGS_num_threads);
+ if (buf_ret != 0) return buf_ret;
+
+ mooncake::ReplicateConfig config;
+ config.replica_num = FLAGS_replica_num;
+ config.with_hard_pin = FLAGS_hard_pin;
+
+ LOG(INFO) << "Phase 1: Writing " << FLAGS_num_keys
+ << " keys (data may be offloaded to SSD)...";
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ std::string key = MakeKey(i);
+ FillBuffer(i);
+ int ret = client_->put_from(key, buffer_, FLAGS_value_size, config);
+ if (ret != 0) {
+ LOG(ERROR) << "put_from failed for key=" << key;
+ return ret;
+ }
+ if ((i + 1) % 50 == 0) {
+ LOG(INFO) << " Written " << (i + 1) << "/" << FLAGS_num_keys;
+ }
+ }
+ LOG(INFO) << "Write phase complete";
+
+ LOG(INFO) << "Waiting " << FLAGS_wait_seconds
+ << " seconds for offload/eviction to complete...";
+ std::this_thread::sleep_for(std::chrono::seconds(FLAGS_wait_seconds));
+
+ int warmup_ret = DoWarmup();
+ if (warmup_ret != 0) {
+ LOG(WARNING) << "Warmup had errors, continuing anyway";
+ }
+
+ LOG(INFO) << "Phase 2: Concurrent disk reads with " << FLAGS_num_threads
+ << " threads";
+
+ BenchmarkStats stats;
+ stats.InitThreads(FLAGS_num_threads,
+ FLAGS_num_keys / FLAGS_num_threads);
+ stats.StartTimer();
+
+ std::latch start_latch(static_cast(FLAGS_num_threads));
+ std::latch done_latch(static_cast(FLAGS_num_threads));
+ std::vector threads;
+
+ size_t keys_per_thread = FLAGS_num_keys / FLAGS_num_threads;
+ size_t remainder = FLAGS_num_keys % FLAGS_num_threads;
+
+ for (size_t t = 0; t < FLAGS_num_threads; ++t) {
+ size_t my_keys = keys_per_thread + (t < remainder ? 1 : 0);
+ size_t key_offset = t * keys_per_thread + std::min(t, remainder);
+
+ threads.emplace_back([&, t, my_keys, key_offset]() {
+ ReadWorker(t, my_keys, key_offset, stats, start_latch,
+ done_latch);
+ });
+ }
+
+ done_latch.wait();
+ stats.StopTimer();
+
+ for (auto& th : threads) {
+ th.join();
+ }
+
+ stats.Finalize();
+ stats.Print("LOCAL DISK READ BENCHMARK");
+
+ if (FLAGS_verify) {
+ int v = VerifyData();
+ LOG_IF(INFO, v == 0) << "Data verification PASSED";
+ LOG_IF(ERROR, v != 0) << "Data verification FAILED";
+ }
+
+ return 0;
+ }
+
+ int Run() {
+ if (FLAGS_scenario == "local_memory") {
+ return RunLocalMemory();
+ } else if (FLAGS_scenario == "local_disk") {
+ return RunLocalDisk();
+ } else if (FLAGS_scenario == "remote_memory" ||
+ FLAGS_scenario == "remote_disk") {
+ if (FLAGS_role == "writer") {
+ return RunWriter();
+ } else {
+ return RunReader();
+ }
+ } else {
+ LOG(ERROR) << "Unknown scenario: " << FLAGS_scenario;
+ return -1;
+ }
+ }
+
+ private:
+ static std::string MakeKey(size_t idx) {
+ return "bench_key_" + std::to_string(idx);
+ }
+
+ void FillBuffer(size_t seed) {
+ uint64_t* ptr = reinterpret_cast(buffer_);
+ size_t num_words = FLAGS_value_size / sizeof(uint64_t);
+ uint64_t pattern = static_cast(seed) * 0x9E3779B97F4A7C15ULL;
+ for (size_t w = 0; w < num_words; ++w) {
+ pattern = (pattern ^ (pattern >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ pattern = (pattern ^ (pattern >> 27)) * 0x94D049BB133111EBULL;
+ ptr[w] = pattern ^ (pattern >> 31);
+ }
+ }
+
+ bool CheckBuffer(size_t seed, const void* data, size_t size) const {
+ const uint64_t* ptr = reinterpret_cast(data);
+ size_t num_words = size / sizeof(uint64_t);
+ uint64_t pattern = static_cast(seed) * 0x9E3779B97F4A7C15ULL;
+ for (size_t w = 0; w < num_words; ++w) {
+ pattern = (pattern ^ (pattern >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ pattern = (pattern ^ (pattern >> 27)) * 0x94D049BB133111EBULL;
+ uint64_t expected = pattern ^ (pattern >> 31);
+ if (ptr[w] != expected) {
+ LOG(ERROR) << "Checksum mismatch at word " << w
+ << " for seed=" << seed << " expected=" << std::hex
+ << expected << " got=" << ptr[w] << std::dec;
+ return false;
+ }
+ }
+ return true;
+ }
+
+ int DoWarmup() {
+ if (FLAGS_warmup_keys == 0) return 0;
+ LOG(INFO) << "Warmup: reading " << FLAGS_warmup_keys << " keys...";
+
+ size_t warmup_end = std::min(static_cast(FLAGS_warmup_keys),
+ static_cast(FLAGS_num_keys));
+ for (size_t i = 0; i < warmup_end; ++i) {
+ std::string key = MakeKey(i);
+ int64_t ret = client_->get_into(key, buffer_, FLAGS_value_size);
+ if (ret < 0) {
+ LOG(WARNING) << "Warmup get_into failed for key=" << key
+ << " ret=" << ret;
+ }
+ }
+ LOG(INFO) << "Warmup complete";
+ return 0;
+ }
+
+ void ReadWorker(size_t tid, size_t my_keys, size_t key_offset,
+ BenchmarkStats& stats, std::latch& start_latch,
+ std::latch& done_latch) {
+ bindToSocket(tid % NR_SOCKETS);
+
+ ThreadResult& result = stats.GetThreadResult(tid);
+ result.latencies_ns.reserve(my_keys);
+
+ char* my_buf = thread_buffers_[tid].ptr;
+
+ start_latch.arrive_and_wait();
+
+ size_t ops = 0;
+ size_t failed = 0;
+ size_t bytes = 0;
+
+ if (FLAGS_batch_size <= 1) {
+ for (size_t i = 0; i < my_keys; ++i) {
+ size_t key_idx = key_offset + i;
+ std::string key = MakeKey(key_idx);
+
+ auto t0 = Clock::now();
+ int64_t ret = client_->get_into(key, my_buf, FLAGS_value_size);
+ auto t1 = Clock::now();
+
+ int64_t lat_ns = ElapsedNanos(t0, t1);
+
+ if (ret < 0) {
+ ++failed;
+ LOG_EVERY_N(ERROR, 100)
+ << "get_into failed key=" << key << " ret=" << ret;
+ } else {
+ bytes += static_cast(ret);
+ }
+ result.latencies_ns.push_back(lat_ns);
+ ++ops;
+ }
+ } else {
+ size_t per_key_buf = FLAGS_value_size;
+ size_t i = 0;
+ while (i < my_keys) {
+ std::vector keys;
+ std::vector bufs;
+ std::vector sizes;
+ size_t batch_end = std::min(i + FLAGS_batch_size, my_keys);
+ keys.reserve(batch_end - i);
+ bufs.reserve(batch_end - i);
+ sizes.reserve(batch_end - i);
+
+ for (size_t j = i; j < batch_end; ++j) {
+ size_t key_idx = key_offset + j;
+ keys.push_back(MakeKey(key_idx));
+ bufs.push_back(my_buf + (j - i) * per_key_buf);
+ sizes.push_back(FLAGS_value_size);
+ }
+
+ auto t0 = Clock::now();
+ auto results = client_->batch_get_into(keys, bufs, sizes);
+ auto t1 = Clock::now();
+
+ int64_t lat_ns = ElapsedNanos(t0, t1);
+ result.latencies_ns.push_back(lat_ns);
+
+ for (size_t k = 0; k < results.size(); ++k) {
+ if (results[k] < 0) {
+ ++failed;
+ } else {
+ bytes += static_cast(results[k]);
+ }
+ ++ops;
+ }
+
+ i = batch_end;
+ }
+ }
+
+ result.total_bytes = bytes;
+ result.total_ops = ops;
+ result.failed_ops = failed;
+
+ done_latch.arrive_and_wait();
+ }
+
+ int VerifyData() {
+ LOG(INFO) << "Verifying data integrity for " << FLAGS_num_keys
+ << " keys...";
+ int errors = 0;
+
+ for (size_t i = 0; i < FLAGS_num_keys; ++i) {
+ std::string key = MakeKey(i);
+ int64_t ret =
+ client_->get_into(key, buffer_, FLAGS_value_size);
+ if (ret < 0) {
+ LOG(ERROR) << "Verify: get_into failed for key=" << key;
+ ++errors;
+ continue;
+ }
+ if (!CheckBuffer(i, buffer_, static_cast(ret))) {
+ LOG(ERROR) << "Verify: data mismatch for key=" << key;
+ ++errors;
+ }
+ }
+
+ LOG(INFO) << "Verification complete: " << errors << " errors out of "
+ << FLAGS_num_keys << " keys";
+ return errors > 0 ? -1 : 0;
+ }
+
+ std::shared_ptr client_;
+ char* buffer_;
+ size_t buffer_size_;
+
+ struct ThreadBuffer {
+ char* ptr = nullptr;
+ size_t size = 0;
+ int numa_node = -1;
+ };
+ std::vector thread_buffers_;
+
+ int AllocateThreadBuffers(size_t num_threads) {
+ thread_buffers_.resize(num_threads);
+ size_t per_buf_size = FLAGS_batch_size * FLAGS_value_size;
+ for (size_t t = 0; t < num_threads; ++t) {
+ int node = t % NR_SOCKETS;
+ thread_buffers_[t].size = per_buf_size;
+ thread_buffers_[t].numa_node = node;
+ thread_buffers_[t].ptr =
+ reinterpret_cast(numa_alloc_onnode(per_buf_size, node));
+ if (!thread_buffers_[t].ptr) {
+ LOG(ERROR) << "Failed to allocate buffer for thread " << t
+ << " on NUMA node " << node;
+ return -1;
+ }
+ std::memset(thread_buffers_[t].ptr, 0, per_buf_size);
+ int ret = client_->register_buffer(thread_buffers_[t].ptr,
+ per_buf_size);
+ if (ret != 0) {
+ LOG(ERROR) << "register_buffer failed for thread " << t
+ << " on NUMA node " << node;
+ return ret;
+ }
+ }
+ LOG(INFO) << "Allocated " << num_threads << " thread buffers, each "
+ << per_buf_size / MB << " MB (NUMA-aware, "
+ << NR_SOCKETS << " sockets)";
+ return 0;
+ }
+};
+
+int main(int argc, char* argv[]) {
+ google::InitGoogleLogging(argv[0]);
+ gflags::ParseCommandLineFlags(&argc, &argv, true);
+
+ FLAGS_logtostderr = true;
+
+ LOG(INFO) << "Mooncake Stress Cluster Benchmark";
+ LOG(INFO) << " Scenario: " << FLAGS_scenario;
+ LOG(INFO) << " Protocol: " << FLAGS_protocol;
+ LOG(INFO) << " Value size: " << FLAGS_value_size / MB << " MB";
+ LOG(INFO) << " Num keys: " << FLAGS_num_keys;
+ LOG(INFO) << " Batch size: " << FLAGS_batch_size;
+ LOG(INFO) << " Num threads: " << FLAGS_num_threads;
+ LOG(INFO) << " Hard pin: " << (FLAGS_hard_pin ? "yes" : "no");
+ LOG(INFO) << " SSD offload: "
+ << (FLAGS_enable_ssd_offload ? "yes" : "no");
+
+ size_t total_data = FLAGS_num_keys * FLAGS_value_size;
+ if (total_data > FLAGS_global_segment_size * 9.5 / 10) {
+ LOG(WARNING) << "Total data (" << total_data / MB << " MB) may exceed "
+ << "95% of segment (" << FLAGS_global_segment_size / MB
+ << " MB). Master eviction may delete objects. "
+ << "Consider increasing --global_segment_size or "
+ << "decreasing --num_keys, or use --hard_pin=true.";
+ }
+
+ StressBenchmark bench;
+ int ret = bench.Setup();
+ if (ret != 0) {
+ LOG(ERROR) << "Benchmark setup failed";
+ return ret;
+ }
+
+ ret = bench.Run();
+ return ret;
+}
diff --git a/mooncake-store/include/allocation_strategy.h b/mooncake-store/include/allocation_strategy.h
index e9f656126c..0ca86e38d9 100644
--- a/mooncake-store/include/allocation_strategy.h
+++ b/mooncake-store/include/allocation_strategy.h
@@ -119,6 +119,16 @@ class AllocatorManager {
friend class SegmentSerializer; // for fork serialize
};
+class SsdMetricsProvider {
+ public:
+ virtual ~SsdMetricsProvider() = default;
+ virtual int64_t getSsdTotalCapacity(const std::string& segment_name) const = 0;
+ virtual int64_t getSsdUsedBytes(const std::string& segment_name) const = 0;
+ virtual double getDdrUsedRatio(const std::string& segment_name) const {
+ return 0.0;
+ }
+};
+
/**
* @brief Abstract interface for allocation strategy, responsible for
* allocating a slice (with one or more replicas) using available
@@ -166,6 +176,18 @@ class AllocationStrategy {
std::set(),
const ReplicaType replica_type = ReplicaType::MEMORY) = 0;
+ virtual tl::expected, ErrorCode> Allocate(
+ const AllocatorManager& allocator_manager, const size_t slice_length,
+ const size_t replica_num,
+ const std::vector& preferred_segments,
+ const std::set& excluded_segments,
+ const ReplicaType replica_type,
+ const SsdMetricsProvider* ssd_provider) {
+ (void)ssd_provider;
+ return Allocate(allocator_manager, slice_length, replica_num,
+ preferred_segments, excluded_segments, replica_type);
+ }
+
/**
* @brief Allocate one replica from the specified segment.
*
@@ -538,6 +560,210 @@ class FreeRatioFirstAllocationStrategy : public RandomAllocationStrategy {
}
};
+class SsdBalanceAllocationStrategy : public RandomAllocationStrategy {
+ public:
+ explicit SsdBalanceAllocationStrategy(
+ double ssd_high_watermark = kDefaultSsdHighWatermark,
+ double ddr_admission_watermark = 1.0)
+ : ssd_high_watermark_(ssd_high_watermark),
+ ddr_admission_watermark_(ddr_admission_watermark) {}
+
+ tl::expected, ErrorCode> Allocate(
+ const AllocatorManager& allocator_manager, const size_t slice_length,
+ const size_t replica_num,
+ const std::vector& preferred_segments,
+ const std::set& excluded_segments,
+ const ReplicaType replica_type,
+ const SsdMetricsProvider* ssd_provider) override {
+ if (slice_length == 0 || replica_num == 0) {
+ return tl::make_unexpected(ErrorCode::INVALID_PARAMS);
+ }
+
+ const auto& names = allocator_manager.getNames();
+ if (names.empty()) {
+ return tl::make_unexpected(ErrorCode::NO_AVAILABLE_HANDLE);
+ }
+
+ static thread_local std::mt19937 generator(std::random_device{}());
+
+ std::vector replicas;
+ replicas.reserve(replica_num);
+ std::set used_segments;
+ size_t ddr_rejected_count = 0;
+
+ // Handle preferred segments first
+ for (const auto& preferred_segment : preferred_segments) {
+ if (excluded_segments.contains(preferred_segment) ||
+ used_segments.contains(preferred_segment)) {
+ continue;
+ }
+ if (ssd_provider &&
+ isSsdHighWatermark(preferred_segment, ssd_provider)) {
+ continue;
+ }
+ if (ssd_provider &&
+ isDdrHighWatermark(preferred_segment, ssd_provider)) {
+ ddr_rejected_count++;
+ continue;
+ }
+
+ auto buffer = allocateSingle(allocator_manager, preferred_segment,
+ slice_length, generator);
+ if (buffer) {
+ replicas.emplace_back(std::move(buffer),
+ ReplicaStatus::PROCESSING, replica_type);
+ used_segments.insert(preferred_segment);
+ if (replicas.size() == replica_num) {
+ return replicas;
+ }
+ }
+ }
+
+ const size_t remaining = replica_num - replicas.size();
+
+ // Sample candidates and sort by SSD free ratio
+ size_t sample_count =
+ std::min(kCandidateMultiplier * remaining, names.size());
+
+ std::uniform_int_distribution start_dist(0, names.size() - 1);
+ size_t start_idx = start_dist(generator);
+
+ struct Candidate {
+ size_t name_idx;
+ double ssd_free_ratio;
+ };
+ std::vector candidates;
+ candidates.reserve(sample_count);
+
+ for (size_t i = 0; i < sample_count; ++i) {
+ size_t idx = (start_idx + i) % names.size();
+ const auto& name = names[idx];
+
+ if (excluded_segments.contains(name) ||
+ used_segments.contains(name)) {
+ continue;
+ }
+ if (ssd_provider && isSsdHighWatermark(name, ssd_provider)) {
+ continue;
+ }
+ if (ssd_provider && isDdrHighWatermark(name, ssd_provider)) {
+ ddr_rejected_count++;
+ continue;
+ }
+
+ double ssd_free_ratio =
+ getSegmentSsdFreeRatio(name, ssd_provider);
+ candidates.push_back({idx, ssd_free_ratio});
+ }
+
+ std::sort(candidates.begin(), candidates.end(),
+ [](const Candidate& a, const Candidate& b) {
+ return a.ssd_free_ratio > b.ssd_free_ratio;
+ });
+
+ for (const auto& candidate : candidates) {
+ if (replicas.size() >= replica_num) {
+ break;
+ }
+
+ const auto& name = names[candidate.name_idx];
+ auto buffer = allocateSingle(allocator_manager, name, slice_length,
+ generator);
+ if (buffer) {
+ replicas.emplace_back(std::move(buffer),
+ ReplicaStatus::PROCESSING, replica_type);
+ used_segments.insert(name);
+ }
+ }
+
+ if (replicas.size() >= replica_num) {
+ return replicas;
+ }
+
+ // Fallback: Random allocation for remaining replicas
+ std::uniform_int_distribution distribution(0, names.size() - 1);
+ size_t fallback_idx = distribution(generator);
+ const size_t max_retry = std::min(kMaxRetryLimit, names.size());
+ size_t try_count = 0;
+
+ while (replicas.size() < replica_num && try_count < max_retry) {
+ auto index = fallback_idx % names.size();
+ fallback_idx++;
+ try_count++;
+
+ const auto& name = names[index];
+
+ if (excluded_segments.contains(name) ||
+ used_segments.contains(name)) {
+ continue;
+ }
+ if (ssd_provider && isSsdHighWatermark(name, ssd_provider)) {
+ continue;
+ }
+ if (ssd_provider && isDdrHighWatermark(name, ssd_provider)) {
+ ddr_rejected_count++;
+ continue;
+ }
+
+ auto buffer = allocateSingle(allocator_manager, name, slice_length,
+ generator);
+ if (buffer) {
+ replicas.emplace_back(std::move(buffer),
+ ReplicaStatus::PROCESSING, replica_type);
+ used_segments.insert(name);
+ }
+ }
+
+ if (replicas.empty()) {
+ if (ddr_rejected_count > 0) {
+ return tl::make_unexpected(ErrorCode::DDR_ADMISSION_REJECTED);
+ }
+ return tl::make_unexpected(ErrorCode::NO_AVAILABLE_HANDLE);
+ }
+ return replicas;
+ }
+
+ using AllocationStrategy::Allocate;
+
+ private:
+ static constexpr size_t kMaxRetryLimit = 100;
+ static constexpr size_t kCandidateMultiplier = 6;
+ static constexpr double kDefaultSsdHighWatermark = 0.90;
+
+ const double ssd_high_watermark_;
+ const double ddr_admission_watermark_;
+
+ bool isSsdHighWatermark(const std::string& name,
+ const SsdMetricsProvider* ssd_provider) const {
+ int64_t total = ssd_provider->getSsdTotalCapacity(name);
+ if (total <= 0) return false;
+ int64_t used = ssd_provider->getSsdUsedBytes(name);
+ double used_ratio =
+ static_cast(used) / static_cast(total);
+ return used_ratio >= ssd_high_watermark_;
+ }
+
+ bool isDdrHighWatermark(const std::string& name,
+ const SsdMetricsProvider* provider) const {
+ if (ddr_admission_watermark_ <= 0.0 ||
+ ddr_admission_watermark_ >= 1.0)
+ return false;
+ double ratio = provider->getDdrUsedRatio(name);
+ return ratio >= ddr_admission_watermark_;
+ }
+
+ double getSegmentSsdFreeRatio(
+ const std::string& name,
+ const SsdMetricsProvider* ssd_provider) const {
+ if (!ssd_provider) return 1.0;
+ int64_t total = ssd_provider->getSsdTotalCapacity(name);
+ if (total <= 0) return 1.0;
+ int64_t used = ssd_provider->getSsdUsedBytes(name);
+ int64_t free_bytes = total - used;
+ return static_cast(free_bytes) / static_cast(total);
+ }
+};
+
class CxlAllocationStrategy : public AllocationStrategy {
public:
CxlAllocationStrategy() = default;
@@ -603,7 +829,8 @@ class CxlAllocationStrategy : public AllocationStrategy {
* @brief Factory function to create allocation strategy based on type
*/
inline std::shared_ptr CreateAllocationStrategy(
- AllocationStrategyType type) {
+ AllocationStrategyType type, double ssd_high_watermark = 0.90,
+ double ddr_admission_watermark = 1.0) {
switch (type) {
case AllocationStrategyType::RANDOM:
return std::make_shared();
@@ -611,6 +838,9 @@ inline std::shared_ptr CreateAllocationStrategy(
return std::make_shared();
case AllocationStrategyType::CXL:
return std::make_shared();
+ case AllocationStrategyType::SSD_BALANCE:
+ return std::make_shared(
+ ssd_high_watermark, ddr_admission_watermark);
default:
return std::make_shared();
}
diff --git a/mooncake-store/include/master_config.h b/mooncake-store/include/master_config.h
index 1f1bc5a9b1..422042222e 100644
--- a/mooncake-store/include/master_config.h
+++ b/mooncake-store/include/master_config.h
@@ -39,6 +39,8 @@ struct MasterConfig {
bool allow_evict_soft_pinned_objects;
double eviction_ratio;
double eviction_high_watermark_ratio;
+ double ddr_admission_watermark_ratio;
+ double ssd_high_watermark_ratio;
double nof_eviction_ratio;
double nof_eviction_high_watermark_ratio;
int64_t client_live_ttl_sec;
@@ -123,6 +125,10 @@ class MasterServiceSupervisorConfig {
RequiredParam eviction_ratio{"eviction_ratio"};
RequiredParam eviction_high_watermark_ratio{
"eviction_high_watermark_ratio"};
+ RequiredParam ddr_admission_watermark_ratio{
+ "ddr_admission_watermark_ratio"};
+ RequiredParam ssd_high_watermark_ratio{
+ "ssd_high_watermark_ratio"};
RequiredParam nof_eviction_ratio{"nof_eviction_ratio"};
RequiredParam nof_eviction_high_watermark_ratio{
"nof_eviction_high_watermark_ratio"};
@@ -195,6 +201,9 @@ class MasterServiceSupervisorConfig {
config.allow_evict_soft_pinned_objects;
eviction_ratio = config.eviction_ratio;
eviction_high_watermark_ratio = config.eviction_high_watermark_ratio;
+ ddr_admission_watermark_ratio =
+ config.ddr_admission_watermark_ratio;
+ ssd_high_watermark_ratio = config.ssd_high_watermark_ratio;
nof_eviction_ratio = config.nof_eviction_ratio;
nof_eviction_high_watermark_ratio =
config.nof_eviction_high_watermark_ratio;
@@ -333,6 +342,9 @@ class WrappedMasterServiceConfig {
double eviction_ratio = DEFAULT_EVICTION_RATIO;
double eviction_high_watermark_ratio =
DEFAULT_EVICTION_HIGH_WATERMARK_RATIO;
+ double ddr_admission_watermark_ratio =
+ DEFAULT_DDR_ADMISSION_WATERMARK_RATIO;
+ double ssd_high_watermark_ratio = 0.90;
double nof_eviction_ratio = DEFAULT_NOF_EVICTION_RATIO;
double nof_eviction_high_watermark_ratio =
DEFAULT_NOF_EVICTION_HIGH_WATERMARK_RATIO;
@@ -401,6 +413,9 @@ class WrappedMasterServiceConfig {
http_port = static_cast(config.metrics_port);
eviction_ratio = config.eviction_ratio;
eviction_high_watermark_ratio = config.eviction_high_watermark_ratio;
+ ddr_admission_watermark_ratio =
+ config.ddr_admission_watermark_ratio;
+ ssd_high_watermark_ratio = config.ssd_high_watermark_ratio;
nof_eviction_ratio = config.nof_eviction_ratio;
nof_eviction_high_watermark_ratio =
config.nof_eviction_high_watermark_ratio;
@@ -439,14 +454,16 @@ class WrappedMasterServiceConfig {
allocation_strategy_type = AllocationStrategyType::FREE_RATIO_FIRST;
} else if (config.allocation_strategy == "cxl") {
allocation_strategy_type = AllocationStrategyType::CXL;
+ } else if (config.allocation_strategy == "ssd_balance") {
+ allocation_strategy_type = AllocationStrategyType::SSD_BALANCE;
} else if (config.allocation_strategy == "random") {
allocation_strategy_type = AllocationStrategyType::RANDOM;
} else {
LOG(WARNING) << "Unrecognized allocation_strategy value: '"
<< config.allocation_strategy
<< "'. Defaulting to 'random'. "
- << "Valid options are: random, free_ratio_first, cxl "
- "(case-sensitive)";
+ << "Valid options are: random, free_ratio_first, cxl, "
+ "ssd_balance (case-sensitive)";
allocation_strategy_type = AllocationStrategyType::RANDOM;
}
@@ -489,6 +506,9 @@ class WrappedMasterServiceConfig {
http_port = static_cast(config.metrics_port);
eviction_ratio = config.eviction_ratio;
eviction_high_watermark_ratio = config.eviction_high_watermark_ratio;
+ ddr_admission_watermark_ratio =
+ config.ddr_admission_watermark_ratio;
+ ssd_high_watermark_ratio = config.ssd_high_watermark_ratio;
nof_eviction_ratio = config.nof_eviction_ratio;
nof_eviction_high_watermark_ratio =
config.nof_eviction_high_watermark_ratio;
@@ -551,6 +571,9 @@ class MasterServiceConfigBuilder {
double eviction_ratio_ = DEFAULT_EVICTION_RATIO;
double eviction_high_watermark_ratio_ =
DEFAULT_EVICTION_HIGH_WATERMARK_RATIO;
+ double ddr_admission_watermark_ratio_ =
+ DEFAULT_DDR_ADMISSION_WATERMARK_RATIO;
+ double ssd_high_watermark_ratio_ = 0.90;
double nof_eviction_ratio_ = DEFAULT_NOF_EVICTION_RATIO;
double nof_eviction_high_watermark_ratio_ =
DEFAULT_NOF_EVICTION_HIGH_WATERMARK_RATIO;
@@ -626,6 +649,17 @@ class MasterServiceConfigBuilder {
return *this;
}
+ MasterServiceConfigBuilder& set_ddr_admission_watermark_ratio(
+ double ratio) {
+ ddr_admission_watermark_ratio_ = ratio;
+ return *this;
+ }
+
+ MasterServiceConfigBuilder& set_ssd_high_watermark_ratio(double ratio) {
+ ssd_high_watermark_ratio_ = ratio;
+ return *this;
+ }
+
MasterServiceConfigBuilder& set_nof_eviction_ratio(double ratio) {
nof_eviction_ratio_ = ratio;
return *this;
@@ -869,6 +903,9 @@ class MasterServiceConfig {
double eviction_ratio = DEFAULT_EVICTION_RATIO;
double eviction_high_watermark_ratio =
DEFAULT_EVICTION_HIGH_WATERMARK_RATIO;
+ double ddr_admission_watermark_ratio =
+ DEFAULT_DDR_ADMISSION_WATERMARK_RATIO;
+ double ssd_high_watermark_ratio = 0.90;
double nof_eviction_ratio = DEFAULT_NOF_EVICTION_RATIO;
double nof_eviction_high_watermark_ratio =
DEFAULT_NOF_EVICTION_HIGH_WATERMARK_RATIO;
@@ -933,6 +970,9 @@ class MasterServiceConfig {
config.allow_evict_soft_pinned_objects;
eviction_ratio = config.eviction_ratio;
eviction_high_watermark_ratio = config.eviction_high_watermark_ratio;
+ ddr_admission_watermark_ratio =
+ config.ddr_admission_watermark_ratio;
+ ssd_high_watermark_ratio = config.ssd_high_watermark_ratio;
nof_eviction_ratio = config.nof_eviction_ratio;
nof_eviction_high_watermark_ratio =
config.nof_eviction_high_watermark_ratio;
@@ -1001,6 +1041,8 @@ inline MasterServiceConfig MasterServiceConfigBuilder::build() const {
config.allow_evict_soft_pinned_objects = allow_evict_soft_pinned_objects_;
config.eviction_ratio = eviction_ratio_;
config.eviction_high_watermark_ratio = eviction_high_watermark_ratio_;
+ config.ddr_admission_watermark_ratio = ddr_admission_watermark_ratio_;
+ config.ssd_high_watermark_ratio = ssd_high_watermark_ratio_;
config.nof_eviction_ratio = nof_eviction_ratio_;
config.nof_eviction_high_watermark_ratio =
nof_eviction_high_watermark_ratio_;
@@ -1065,6 +1107,8 @@ struct InProcMasterConfig {
std::optional cxl_path;
std::optional cxl_size;
std::optional eviction_high_watermark_ratio;
+ std::optional ddr_admission_watermark_ratio;
+ std::optional ssd_high_watermark_ratio;
std::optional root_fs_dir;
std::optional enable_disk_eviction;
std::optional quota_bytes;
@@ -1082,6 +1126,8 @@ class InProcMasterConfigBuilder {
std::optional cxl_path_ = std::nullopt;
std::optional cxl_size_ = std::nullopt;
std::optional eviction_high_watermark_ratio_ = std::nullopt;
+ std::optional ddr_admission_watermark_ratio_ = std::nullopt;
+ std::optional ssd_high_watermark_ratio_ = std::nullopt;
std::optional root_fs_dir_ = std::nullopt;
std::optional enable_disk_eviction_ = std::nullopt;
std::optional quota_bytes_ = std::nullopt;
@@ -1138,6 +1184,24 @@ class InProcMasterConfigBuilder {
return *this;
}
+ InProcMasterConfigBuilder& set_ddr_admission_watermark_ratio(double ratio) {
+ if (ratio < 0.0 || ratio > 1.0) {
+ throw std::invalid_argument(
+ "ddr_admission_watermark_ratio must be between 0.0 and 1.0");
+ }
+ ddr_admission_watermark_ratio_ = ratio;
+ return *this;
+ }
+
+ InProcMasterConfigBuilder& set_ssd_high_watermark_ratio(double ratio) {
+ if (ratio < 0.0 || ratio > 1.0) {
+ throw std::invalid_argument(
+ "ssd_high_watermark_ratio must be between 0.0 and 1.0");
+ }
+ ssd_high_watermark_ratio_ = ratio;
+ return *this;
+ }
+
InProcMasterConfigBuilder& set_root_fs_dir(const std::string& dir) {
root_fs_dir_ = dir;
return *this;
@@ -1168,6 +1232,8 @@ inline InProcMasterConfig InProcMasterConfigBuilder::build() const {
config.cxl_path = cxl_path_;
config.cxl_size = cxl_size_;
config.eviction_high_watermark_ratio = eviction_high_watermark_ratio_;
+ config.ddr_admission_watermark_ratio = ddr_admission_watermark_ratio_;
+ config.ssd_high_watermark_ratio = ssd_high_watermark_ratio_;
config.root_fs_dir = root_fs_dir_;
config.enable_disk_eviction = enable_disk_eviction_;
config.quota_bytes = quota_bytes_;
diff --git a/mooncake-store/include/master_service.h b/mooncake-store/include/master_service.h
index 68b841f3aa..26c1fc9478 100644
--- a/mooncake-store/include/master_service.h
+++ b/mooncake-store/include/master_service.h
@@ -1218,6 +1218,7 @@ class MasterService {
false}; // Set to trigger NoF eviction when allocation fails
const double eviction_ratio_; // in range [0.0, 1.0]
const double eviction_high_watermark_ratio_; // in range [0.0, 1.0]
+ const double ssd_high_watermark_ratio_; // in range [0.0, 1.0]
const double nof_eviction_ratio_; // in range [0.0, 1.0]
const double nof_eviction_high_watermark_ratio_; // in range [0.0, 1.0]
diff --git a/mooncake-store/include/real_client.h b/mooncake-store/include/real_client.h
index bcf9b67465..0f73edddb5 100644
--- a/mooncake-store/include/real_client.h
+++ b/mooncake-store/include/real_client.h
@@ -766,11 +766,22 @@ class RealClient : public PyClient {
}
};
+ struct UbSegmentDeleter {
+ size_t size = 0;
+ std::string protocol = "ub";
+ void operator()(void *ptr) const {
+ if (ptr && size > 0) {
+ free_memory(protocol.c_str(), ptr);
+ }
+ }
+ };
+
std::vector>
hugepage_segment_ptrs_;
std::vector> segment_ptrs_;
std::vector>
ascend_segment_ptrs_;
+ std::vector> ub_segment_ptrs_;
std::string protocol;
std::string device_name;
std::string local_hostname;
diff --git a/mooncake-store/include/segment.h b/mooncake-store/include/segment.h
index 8fcd2f875f..81f9f9f52a 100644
--- a/mooncake-store/include/segment.h
+++ b/mooncake-store/include/segment.h
@@ -86,6 +86,7 @@ struct LocalDiskSegment {
mutable Mutex offloading_mutex_;
bool enable_offloading;
int64_t ssd_total_capacity_bytes = 0; // last reported by client heartbeat
+ std::atomic ssd_used_bytes{0};
std::unordered_map GUARDED_BY(offloading_mutex_)
offloading_objects;
// Promotion-on-hit pending work for this client. Populated by master's
@@ -327,7 +328,7 @@ class ScopedAllocatorAccess {
* @brief RAII-style access to LocalDiskOffloadingQueues for thread-safe
* LocalDiskOffloadingQueue usage
*/
-class ScopedLocalDiskSegmentAccess {
+class ScopedLocalDiskSegmentAccess : public SsdMetricsProvider {
public:
explicit ScopedLocalDiskSegmentAccess(
std::unordered_map& client_by_name,
@@ -348,6 +349,11 @@ class ScopedLocalDiskSegmentAccess {
return client_local_disk_segment_;
}
+ // SsdMetricsProvider implementation
+ int64_t getSsdTotalCapacity(const std::string& segment_name) const override;
+ int64_t getSsdUsedBytes(const std::string& segment_name) const override;
+ double getDdrUsedRatio(const std::string& segment_name) const override;
+
private:
const std::unordered_map&
client_by_name_; // segment name -> client_id
diff --git a/mooncake-store/include/storage_backend.h b/mooncake-store/include/storage_backend.h
index a686fd9724..d1eb89c38d 100644
--- a/mooncake-store/include/storage_backend.h
+++ b/mooncake-store/include/storage_backend.h
@@ -190,6 +190,10 @@ struct BucketBackendConfig {
int64_t max_total_size = 0; // 0 = unlimited; evict when total_size_
// exceeds this threshold (bytes)
+ bool disable_ssd_eviction = false; // Force disable eviction regardless of
+ // eviction_policy. Set via
+ // MOONCAKE_OFFLOAD_DISABLE_SSD_EVICTION.
+
bool Validate() const;
static BucketBackendConfig FromEnvironment();
diff --git a/mooncake-store/include/types.h b/mooncake-store/include/types.h
index b93e488203..a8c9ac1318 100644
--- a/mooncake-store/include/types.h
+++ b/mooncake-store/include/types.h
@@ -89,6 +89,8 @@ static constexpr uint64_t DEFAULT_KV_SOFT_PIN_TTL_MS =
static constexpr bool DEFAULT_ALLOW_EVICT_SOFT_PINNED_OBJECTS = true;
static constexpr double DEFAULT_EVICTION_RATIO = 0.05;
static constexpr double DEFAULT_EVICTION_HIGH_WATERMARK_RATIO = 0.95;
+static constexpr double DEFAULT_DDR_ADMISSION_WATERMARK_RATIO =
+ 0.0; // 0.0 = use eviction_high_watermark_ratio
static constexpr double DEFAULT_NOF_EVICTION_RATIO = 0.05;
static constexpr double DEFAULT_NOF_EVICTION_HIGH_WATERMARK_RATIO = 0.95;
static constexpr int64_t DEFAULT_MASTER_VIEW_LEASE_TTL_SEC = 5; // in seconds
@@ -267,6 +269,8 @@ enum class ErrorCode : int32_t {
// Handle selection errors (Range: -200 to -299)
NO_AVAILABLE_HANDLE =
-200, ///< Memory allocation failed due to insufficient space.
+ DDR_ADMISSION_REJECTED =
+ -201, ///< Allocation rejected by DDR admission watermark.
// Version errors (Range: -300 to -399)
INVALID_VERSION = -300, ///< Invalid version.
@@ -405,6 +409,7 @@ enum class AllocationStrategyType {
RANDOM = 0, // Pure random allocation
FREE_RATIO_FIRST, // Free-ratio-first allocation
CXL, // CXL-specific allocation
+ SSD_BALANCE, // SSD-ratio-based load balancing
};
/**
diff --git a/mooncake-store/src/CMakeLists.txt b/mooncake-store/src/CMakeLists.txt
index 53dbcf9d28..d71043b78b 100644
--- a/mooncake-store/src/CMakeLists.txt
+++ b/mooncake-store/src/CMakeLists.txt
@@ -164,6 +164,13 @@ target_link_libraries(
PUBLIC cachelib_memory_allocator ${ETCD_WRAPPER_LIB} glog::glog gflags::gflags
${EXTRA_LIBS} asio_shared
PRIVATE transfer_engine)
+
+# UbDiag instrumentation
+find_package(UbDiag REQUIRED)
+target_link_libraries(mooncake_store PRIVATE UbDiag::ubdiag_lib)
+target_include_directories(mooncake_store PRIVATE
+ ${CMAKE_CURRENT_SOURCE_DIR}/../../mooncake-integration/store)
+
if(STORE_USE_ETCD)
add_dependencies(mooncake_store build_etcd_wrapper)
endif()
diff --git a/mooncake-store/src/client_service.cpp b/mooncake-store/src/client_service.cpp
index ee2c13345e..430d6212d5 100644
--- a/mooncake-store/src/client_service.cpp
+++ b/mooncake-store/src/client_service.cpp
@@ -20,6 +20,9 @@
#include "transfer_engine.h"
#include "topology.h"
+#define UBDIAG_PERF_DEF_FILE "mooncake_perf_points.def"
+#define UBDIAG_PROGRAM_NAME "mooncake_store"
+#include "ubdiag/auto_perf.h"
#include "transfer_task.h"
#include "transport/transport.h"
#include "config.h"
@@ -563,6 +566,21 @@ ErrorCode Client::InitTransferEngine(
LOG(ERROR) << "Failed to install CXL transport";
return ErrorCode::INTERNAL_ERROR;
}
+ } else if (protocol == "ub") {
+ if (!device_names.has_value() || device_names->empty()) {
+ LOG(ERROR) << "ub protocol requires device names when auto "
+ "discovery is disabled";
+ return ErrorCode::INVALID_PARAMS;
+ }
+ auto deviceName = device_names.value_or("bonding_dev_0");
+ auto devices = splitString(deviceName, ',', true);
+ transfer_engine_->getLocalTopology()->discover(devices);
+ transport = transfer_engine_->installTransport("ub", nullptr);
+ if (!transport) {
+ LOG(ERROR) << "Failed to install ub transport with specified "
+ "devices";
+ return ErrorCode::INTERNAL_ERROR;
+ }
} else {
LOG(ERROR) << "unsupported_protocol protocol=" << protocol;
return ErrorCode::INVALID_PARAMS;
@@ -808,9 +826,13 @@ tl::expected, ErrorCode> Client::BatchReplicaClear(
tl::expected Client::Get(const std::string& object_key,
const QueryResult& query_result,
std::vector& slices) {
+
// Find the first complete replica
Replica::Descriptor replica;
+ UbDiag::PerfPoint pt_find(PerfKey::GET_SINGLE_FIND_REPLICA, UbDiag::PerfLevel::MODULE);
+ pt_find.Start();
ErrorCode err = FindFirstCompleteReplica(query_result.replicas, replica);
+ pt_find.End(err == ErrorCode::OK ? 0 : -1);
if (err != ErrorCode::OK) {
if (err == ErrorCode::INVALID_REPLICA) {
LOG(ERROR) << "no_complete_replicas_found key=" << object_key;
@@ -821,15 +843,24 @@ tl::expected Client::Get(const std::string& object_key,
// Check local hot cache and update replica descriptor if cache hit
bool cache_used = false;
if (hot_cache_ && replica.is_memory_replica()) {
+ UbDiag::PerfPoint pt_hc(PerfKey::GET_SINGLE_HOT_CACHE, UbDiag::PerfLevel::MODULE);
+ pt_hc.Start();
cache_used = RedirectToHotCache(object_key, replica);
+ pt_hc.End(0);
}
auto t0_get = std::chrono::steady_clock::now();
+ UbDiag::PerfPoint pt_tread(PerfKey::GET_SINGLE_TRANSFER_READ, UbDiag::PerfLevel::MODULE);
+ pt_tread.Start();
err = TransferRead(replica, slices);
+ pt_tread.End(err == ErrorCode::OK ? 0 : -1);
// Release the cache block after transfer completes (memcpy is done)
if (hot_cache_ && cache_used) {
+ UbDiag::PerfPoint pt_rel(PerfKey::GET_SINGLE_RELEASE_CACHE, UbDiag::PerfLevel::MODULE);
+ pt_rel.Start();
hot_cache_->ReleaseHotKey(object_key);
+ pt_rel.End(0);
}
auto us_get = std::chrono::duration_cast(
@@ -844,11 +875,20 @@ tl::expected Client::Get(const std::string& object_key,
return tl::unexpected(err);
}
+ size_t data_size = 0;
+ for (const auto &s : slices) data_size += s.size;
+ LOG(INFO) << "transfer_read_completed key[" << object_key << "] elapsed_us[" << us_get
+ << "] data_size[" << data_size
+ << "] cache_hit[" << (cache_used ? 1 : 0) << "]";
+
// Frequency admission: only promote frequently accessed keys to hot cache.
// Skip when cache_used — data was already served from local cache, no need
// to re-promote or increment the CMS counter.
if (ShouldAdmitToHotCache(object_key, cache_used)) {
+ UbDiag::PerfPoint pt_async(PerfKey::GET_SINGLE_ASYNC_CACHE, UbDiag::PerfLevel::MODULE);
+ pt_async.Start();
ProcessSlicesAsync(object_key, slices, replica);
+ pt_async.End(0);
}
if (query_result.IsLeaseExpired()) {
@@ -1087,8 +1127,11 @@ std::vector> Client::BatchGet(
// Find the first complete replica for this key
Replica::Descriptor replica;
+ UbDiag::PerfPoint pt_find(PerfKey::GET_BATCH_FIND_REPLICA, UbDiag::PerfLevel::MODULE);
+ pt_find.Start();
ErrorCode err =
FindFirstCompleteReplica(query_result.replicas, replica);
+ pt_find.End(err == ErrorCode::OK ? 0 : -1);
if (err != ErrorCode::OK) {
if (err == ErrorCode::INVALID_REPLICA) {
LOG(ERROR) << "no_complete_replicas_found key=" << key;
@@ -1099,15 +1142,21 @@ std::vector> Client::BatchGet(
bool cache_used = false;
if (hot_cache_ && replica.is_memory_replica()) {
+ UbDiag::PerfPoint pt_hc(PerfKey::GET_BATCH_HOT_CACHE, UbDiag::PerfLevel::MODULE);
+ pt_hc.Start();
cache_used = RedirectToHotCache(key, replica);
+ pt_hc.End(0);
if (cache_used) {
total_cache_hits++;
}
}
// Submit transfer operation asynchronously
+ UbDiag::PerfPoint pt_submit(PerfKey::GET_BATCH_SUBMIT, UbDiag::PerfLevel::DEBUG);
+ pt_submit.Start();
auto future = transfer_submitter_->submit(replica, slices_it->second,
TransferRequest::READ);
+ pt_submit.End(future ? 0 : -1);
if (!future) {
// Release cache block if submit failed
if (hot_cache_ && cache_used) {
@@ -1129,11 +1178,17 @@ std::vector> Client::BatchGet(
// Wait for all transfers to complete
for (auto& [index, key, future, stored_replica, cache_used] :
pending_transfers) {
+ UbDiag::PerfPoint pt_wait(PerfKey::GET_BATCH_WAIT, UbDiag::PerfLevel::DEBUG);
+ pt_wait.Start();
ErrorCode result = future.get();
+ pt_wait.End(result == ErrorCode::OK ? 0 : -1);
// Release the cache block after transfer completes (memcpy is done)
if (hot_cache_ && cache_used) {
+ UbDiag::PerfPoint pt_rel(PerfKey::GET_BATCH_RELEASE_CACHE, UbDiag::PerfLevel::MODULE);
+ pt_rel.Start();
hot_cache_->ReleaseHotKey(key);
+ pt_rel.End(0);
}
if (result != ErrorCode::OK) {
LOG(ERROR) << "Transfer failed for key: " << key
@@ -1149,7 +1204,10 @@ std::vector> Client::BatchGet(
auto slices_it = slices.find(key);
if (slices_it != slices.end() &&
ShouldAdmitToHotCache(key, cache_used)) {
+ UbDiag::PerfPoint pt_async(PerfKey::GET_BATCH_ASYNC_CACHE, UbDiag::PerfLevel::MODULE);
+ pt_async.Start();
ProcessSlicesAsync(key, slices_it->second, stored_replica);
+ pt_async.End(0);
}
}
}
@@ -1181,6 +1239,15 @@ std::vector> Client::BatchGet(
} else {
VLOG(1) << "BatchGet completed for " << object_keys.size() << " keys";
}
+
+ size_t num_success = 0;
+ for (const auto& r : results) {
+ if (r.has_value()) num_success++;
+ }
+ LOG(INFO) << "batch_get_transfer_complete num_keys[" << object_keys.size()
+ << "] success[" << num_success << "] elapsed_us[" << us_batch_get
+ << "] pending_count[" << pending_transfers.size() << "]";
+
return results;
}
@@ -1212,6 +1279,8 @@ bool Client::RedirectToHotCache(const std::string& key,
tl::expected Client::Put(const ObjectKey& key,
std::vector& slices,
const ReplicateConfig& config) {
+ UbDiag::PerfPoint pt_full(PerfKey::PUT_SINGLE_FULL, UbDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
// Prepare slice lengths
std::vector slice_lengths;
for (size_t i = 0; i < slices.size(); ++i) {
@@ -1224,23 +1293,33 @@ tl::expected Client::Put(const ObjectKey& key,
}
// Start put operation
+ UbDiag::PerfPoint pt_start(PerfKey::PUT_SINGLE_PUT_START, UbDiag::PerfLevel::MODULE);
+ pt_start.Start();
auto start_result = master_client_.PutStart(key, slice_lengths, client_cfg);
+ pt_start.End(start_result ? 0 : -1);
if (!start_result) {
ErrorCode err = start_result.error();
if (err == ErrorCode::OBJECT_ALREADY_EXISTS) {
VLOG(1) << "object_already_exists key=" << key;
+ LOG(INFO) << "put_start key[" << key << "] rc[OBJECT_ALREADY_EXISTS]";
+ pt_full.End(0);
return {};
}
- if (err == ErrorCode::NO_AVAILABLE_HANDLE) {
+ if (err == ErrorCode::NO_AVAILABLE_HANDLE ||
+ err == ErrorCode::DDR_ADMISSION_REJECTED) {
LOG(WARNING) << "Failed to start put operation for key=" << key
- << PUT_NO_SPACE_HELPER_STR;
+ << PUT_NO_SPACE_HELPER_STR
+ << " (error=" << toString(err) << ")";
} else {
LOG(ERROR) << "Failed to start put operation for key=" << key
<< ": " << toString(err);
}
+ pt_full.End(-1);
return tl::unexpected(err);
}
+ LOG(INFO) << "put_start_success key[" << key << "] replicas[" << start_result.value().size() << "]";
+
// Record Put transfer latency (all replicas)
auto t0_put = std::chrono::steady_clock::now();
@@ -1253,7 +1332,10 @@ tl::expected Client::Put(const ObjectKey& key,
if (replica.is_disk_replica()) {
// Store to local file if storage backend is available
auto disk_descriptor = replica.get_disk_descriptor();
+ UbDiag::PerfPoint pt_disk(PerfKey::PUT_SINGLE_DISK_WRITE, UbDiag::PerfLevel::MODULE);
+ pt_disk.Start();
PutToLocalFile(key, slices, disk_descriptor);
+ pt_disk.End(0);
break; // Only one disk replica is needed
}
}
@@ -1262,15 +1344,23 @@ tl::expected Client::Put(const ObjectKey& key,
for (const auto& replica : start_result.value()) {
if (replica.is_memory_replica()) {
// Transfer data using allocated handles from all replicas
+ UbDiag::PerfPoint pt_tw(PerfKey::PUT_SINGLE_TRANSFER_WRITE, UbDiag::PerfLevel::MODULE);
+ pt_tw.Start();
ErrorCode transfer_err = TransferWrite(replica, slices);
+ pt_tw.End(transfer_err == ErrorCode::OK ? 0 : -1);
if (transfer_err != ErrorCode::OK) {
// Revoke put operation
+ UbDiag::PerfPoint pt_revoke(PerfKey::PUT_SINGLE_PUT_REVOKE, UbDiag::PerfLevel::MODULE);
+ pt_revoke.Start();
auto revoke_result =
master_client_.PutRevoke(key, ReplicaType::MEMORY);
+ pt_revoke.End(revoke_result ? 0 : -1);
if (!revoke_result) {
LOG(ERROR) << "Failed to revoke put operation";
+ pt_full.End(-1);
return tl::unexpected(revoke_result.error());
}
+ pt_full.End(-1);
return tl::unexpected(transfer_err);
}
}
@@ -1284,13 +1374,23 @@ tl::expected Client::Put(const ObjectKey& key,
}
// End put operation
+ UbDiag::PerfPoint pt_end(PerfKey::PUT_SINGLE_PUT_END, UbDiag::PerfLevel::MODULE);
+ pt_end.Start();
auto end_result = master_client_.PutEnd(key, ReplicaType::MEMORY);
+ pt_end.End(end_result ? 0 : -1);
if (!end_result) {
ErrorCode err = end_result.error();
LOG(ERROR) << "Failed to end put operation: " << err;
+ pt_full.End(-1);
return tl::unexpected(err);
}
+ size_t data_size = 0;
+ for (const auto &s : slices) data_size += s.size;
+ LOG(INFO) << "put_end_success key[" << key << "] transfer_us[" << us_put
+ << "] data_size[" << data_size << "]";
+
+ pt_full.End(0);
return {};
}
@@ -1313,9 +1413,11 @@ tl::expected Client::Upsert(const ObjectKey& key,
master_client_.UpsertStart(key, slice_lengths, client_cfg);
if (!start_result) {
ErrorCode err = start_result.error();
- if (err == ErrorCode::NO_AVAILABLE_HANDLE) {
+ if (err == ErrorCode::NO_AVAILABLE_HANDLE ||
+ err == ErrorCode::DDR_ADMISSION_REJECTED) {
LOG(WARNING) << "Failed to start upsert operation for key=" << key
- << PUT_NO_SPACE_HELPER_STR;
+ << PUT_NO_SPACE_HELPER_STR
+ << " (error=" << toString(err) << ")";
} else {
LOG(ERROR) << "Failed to start upsert operation for key=" << key
<< ": " << toString(err);
@@ -1470,11 +1572,14 @@ class PutOperation {
std::vector Client::CreatePutOperations(
const std::vector& keys,
const std::vector>& batched_slices) {
+ UbDiag::PerfPoint pt(PerfKey::PUT_BATCH_CREATE_OPS, UbDiag::PerfLevel::MODULE);
+ pt.Start();
std::vector ops;
ops.reserve(keys.size());
for (size_t i = 0; i < keys.size(); ++i) {
ops.emplace_back(keys[i], batched_slices[i]);
}
+ pt.End(0);
return ops;
}
@@ -1497,8 +1602,11 @@ void Client::StartBatchPut(std::vector& ops,
slice_lengths.emplace_back(std::move(slice_sizes));
}
+ UbDiag::PerfPoint pt_batch_start(PerfKey::PUT_BATCH_PUT_START, UbDiag::PerfLevel::MODULE);
+ pt_batch_start.Start();
auto start_responses =
master_client_.BatchPutStart(keys, slice_lengths, config);
+ pt_batch_start.End(start_responses.size() == ops.size() ? 0 : -1);
// Ensure response size matches request size
if (start_responses.size() != ops.size()) {
@@ -1606,7 +1714,10 @@ void Client::SubmitTransfers(std::vector& ops) {
const auto& replica = *it;
if (replica.is_disk_replica()) {
auto disk_descriptor = replica.get_disk_descriptor();
+ UbDiag::PerfPoint pt_disk(PerfKey::PUT_BATCH_DISK_WRITE, UbDiag::PerfLevel::MODULE);
+ pt_disk.Start();
PutToLocalFile(op.key, op.slices, disk_descriptor);
+ pt_disk.End(0);
break; // Only one disk replica is needed
}
}
@@ -1616,8 +1727,11 @@ void Client::SubmitTransfers(std::vector& ops) {
++replica_idx) {
const auto& replica = op.replicas[replica_idx];
if (replica.is_memory_replica()) {
+ UbDiag::PerfPoint pt_submit(PerfKey::PUT_BATCH_SUBMIT, UbDiag::PerfLevel::DEBUG);
+ pt_submit.Start();
auto submit_result = transfer_submitter_->submit(
replica, op.slices, TransferRequest::WRITE);
+ pt_submit.End(submit_result ? 0 : -1);
if (!submit_result) {
failure_context = "Failed to submit transfer for replica " +
@@ -1661,18 +1775,19 @@ void Client::WaitForTransfers(std::vector& ops) {
ErrorCode first_error = ErrorCode::OK;
size_t failed_transfer_idx = 0;
+ UbDiag::PerfPoint pt_wait(PerfKey::PUT_BATCH_WAIT, UbDiag::PerfLevel::MODULE);
+ pt_wait.Start();
for (size_t i = 0; i < op.pending_transfers.size(); ++i) {
ErrorCode transfer_result = op.pending_transfers[i].get();
if (transfer_result != ErrorCode::OK) {
if (all_transfers_succeeded) {
- // Record the first error for reporting
first_error = transfer_result;
failed_transfer_idx = i;
all_transfers_succeeded = false;
}
- // Continue waiting for all transfers to avoid resource leaks
}
}
+ pt_wait.End(all_transfers_succeeded ? 0 : -1);
if (all_transfers_succeeded) {
VLOG(1) << "All transfers completed successfully for key "
@@ -1727,7 +1842,10 @@ void Client::FinalizeBatchPut(std::vector& ops) {
// Process successful operations
if (!successful_keys.empty()) {
+ UbDiag::PerfPoint pt_end(PerfKey::PUT_BATCH_PUT_END, UbDiag::PerfLevel::MODULE);
+ pt_end.Start();
auto end_responses = master_client_.BatchPutEnd(successful_keys);
+ pt_end.End(end_responses.size() == successful_keys.size() ? 0 : -1);
if (end_responses.size() != successful_keys.size()) {
LOG(ERROR) << "BatchPutEnd response size mismatch: expected "
<< successful_keys.size() << ", got "
@@ -1758,7 +1876,10 @@ void Client::FinalizeBatchPut(std::vector& ops) {
// Process failed operations that need cleanup
if (!failed_keys.empty()) {
+ UbDiag::PerfPoint pt_revoke(PerfKey::PUT_BATCH_PUT_REVOKE, UbDiag::PerfLevel::MODULE);
+ pt_revoke.Start();
auto revoke_responses = master_client_.BatchPutRevoke(failed_keys);
+ pt_revoke.End(revoke_responses.size() == failed_keys.size() ? 0 : -1);
if (revoke_responses.size() != failed_keys.size()) {
LOG(ERROR) << "BatchPutRevoke response size mismatch: expected "
<< failed_keys.size() << ", got "
@@ -1898,6 +2019,8 @@ void Client::FinalizeBatchUpsert(std::vector& ops) {
std::vector> Client::CollectResults(
const std::vector& ops) {
+ UbDiag::PerfPoint pt(PerfKey::PUT_BATCH_COLLECT_RESULTS, UbDiag::PerfLevel::MODULE);
+ pt.Start();
std::vector> results;
results.reserve(ops.size());
@@ -1934,6 +2057,7 @@ std::vector> Client::CollectResults(
<< " keys" << PUT_NO_SPACE_HELPER_STR;
}
+ pt.End(0);
return results;
}
@@ -2031,20 +2155,33 @@ std::vector> Client::BatchPut(
const std::vector& keys,
std::vector>& batched_slices,
const ReplicateConfig& config) {
+ UbDiag::PerfPoint pt_full(PerfKey::PUT_BATCH_FULL, UbDiag::PerfLevel::KEY_MODULE);
+ pt_full.Start();
ReplicateConfig client_cfg = config;
if (protocol_ == "cxl") {
client_cfg.preferred_segment = local_hostname_;
}
+ LOG(INFO) << "batch_put start num_keys[" << keys.size() << "]";
std::vector ops = CreatePutOperations(keys, batched_slices);
if (client_cfg.prefer_alloc_in_same_node) {
if (client_cfg.replica_num != 1) {
LOG(ERROR) << "prefer_alloc_in_same_node is not supported with "
"replica_num != 1";
+ pt_full.End(-1);
return std::vector>(
keys.size(), tl::unexpected(ErrorCode::INVALID_PARAMS));
}
StartBatchPut(ops, client_cfg);
- return BatchPutWhenPreferSameNode(ops);
+ auto results = BatchPutWhenPreferSameNode(ops);
+ int num_failed = 0;
+ for (auto& r : results) if (!r) num_failed++;
+ size_t total_size = 0;
+ for (const auto& key_slices : batched_slices)
+ for (const auto& s : key_slices) total_size += s.size;
+ LOG(INFO) << "batch_put complete num_keys[" << keys.size()
+ << "] num_failed[" << num_failed << "] total_size[" << total_size << "]";
+ pt_full.End(0);
+ return results;
}
StartBatchPut(ops, client_cfg);
@@ -2059,7 +2196,17 @@ std::vector> Client::BatchPut(
}
FinalizeBatchPut(ops);
- return CollectResults(ops);
+ auto results = CollectResults(ops);
+ int num_failed = 0;
+ for (auto& r : results) if (!r) num_failed++;
+ size_t total_size = 0;
+ for (const auto& key_slices : batched_slices)
+ for (const auto& s : key_slices) total_size += s.size;
+ LOG(INFO) << "batch_put complete num_keys[" << keys.size()
+ << "] num_failed[" << num_failed << "] transfer_us[" << us
+ << "] total_size[" << total_size << "]";
+ pt_full.End(0);
+ return results;
}
tl::expected Client::Remove(const ObjectKey& key, bool force) {
@@ -2788,21 +2935,44 @@ void Client::PutToLocalFile(const std::string& key,
ErrorCode Client::TransferData(const Replica::Descriptor& replica_descriptor,
std::vector& slices,
TransferRequest::OpCode op_code) {
+ bool is_write = (op_code == TransferRequest::WRITE);
+ UbDiag::PerfPoint pt_full(is_write ? PerfKey::PUT_SINGLE_TRANSFER_FULL : PerfKey::GET_SINGLE_TRANSFER_FULL, UbDiag::PerfLevel::MODULE);
+ pt_full.Start();
if (!transfer_submitter_) {
LOG(ERROR) << "TransferSubmitter not initialized";
+ pt_full.End(-1);
return ErrorCode::INVALID_PARAMS;
}
+ auto t0_transfer = std::chrono::steady_clock::now();
+ UbDiag::PerfPoint pt_submit(is_write ? PerfKey::PUT_SINGLE_TRANSFER_SUBMIT : PerfKey::GET_SINGLE_TRANSFER_SUBMIT, UbDiag::PerfLevel::DEBUG);
+ pt_submit.Start();
auto future =
transfer_submitter_->submit(replica_descriptor, slices, op_code);
+ pt_submit.End(future ? 0 : -1);
if (!future) {
LOG(ERROR) << "Failed to submit transfer operation";
+ pt_full.End(-1);
return ErrorCode::TRANSFER_FAIL;
}
+ auto submit_us = std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - t0_transfer).count();
+
VLOG(1) << "Using transfer strategy: " << future->strategy();
- return future->get();
+ UbDiag::PerfPoint pt_wait(is_write ? PerfKey::PUT_SINGLE_TRANSFER_WAIT : PerfKey::GET_SINGLE_TRANSFER_WAIT, UbDiag::PerfLevel::DEBUG);
+ pt_wait.Start();
+ auto result = future->get();
+ pt_wait.End(result == ErrorCode::OK ? 0 : -1);
+ pt_full.End(result == ErrorCode::OK ? 0 : -1);
+
+ auto wait_us = std::chrono::duration_cast(
+ std::chrono::steady_clock::now() - t0_transfer).count() - submit_us;
+ LOG(INFO) << "transfer_data op[" << (is_write ? "WRITE" : "READ")
+ << "] submit_us[" << submit_us << "] wait_us[" << wait_us
+ << "] result[" << toString(result) << "]";
+ return result;
}
ErrorCode Client::TransferReadInternal(
@@ -2969,7 +3139,8 @@ void Client::ExecuteTask(const ClientTask& client_task) {
// Other errors (e.g., OBJECT_NOT_FOUND, REPLICA_NOT_FOUND) should
// not be retried
bool should_retry =
- (result == ErrorCode::NO_AVAILABLE_HANDLE) &&
+ (result == ErrorCode::NO_AVAILABLE_HANDLE ||
+ result == ErrorCode::DDR_ADMISSION_REJECTED) &&
(current_retry_count < assignment.max_retry_attempts);
if (should_retry) {
diff --git a/mooncake-store/src/master.cpp b/mooncake-store/src/master.cpp
index 8c559c0706..e44fe753d5 100644
--- a/mooncake-store/src/master.cpp
+++ b/mooncake-store/src/master.cpp
@@ -83,6 +83,10 @@ DEFINE_double(eviction_ratio, mooncake::DEFAULT_EVICTION_RATIO,
DEFINE_double(eviction_high_watermark_ratio,
mooncake::DEFAULT_EVICTION_HIGH_WATERMARK_RATIO,
"Ratio of high watermark trigger eviction in Memory");
+DEFINE_double(ddr_admission_watermark_ratio,
+ mooncake::DEFAULT_DDR_ADMISSION_WATERMARK_RATIO,
+ "Ratio above which DDR allocation is rejected (0.0 = use "
+ "eviction_high_watermark_ratio)");
DEFINE_double(nof_eviction_ratio, mooncake::DEFAULT_NOF_EVICTION_RATIO,
"Ratio of objects to evict when NoF SSD space is full");
DEFINE_double(nof_eviction_high_watermark_ratio,
@@ -172,7 +176,11 @@ DEFINE_string(memory_allocator, "offset",
"Memory allocator for global segments, cachelib | offset");
DEFINE_string(
allocation_strategy, "random",
- "Allocation strategy for segments, random | free_ratio_first | cxl");
+ "Allocation strategy for segments, random | free_ratio_first | cxl | "
+ "ssd_balance");
+DEFINE_double(ssd_high_watermark_ratio, 0.90,
+ "SSD usage ratio above which a segment is excluded from "
+ "allocation (0.0-1.0)");
DEFINE_bool(enable_http_metadata_server, false,
"Enable HTTP metadata server instead of etcd");
DEFINE_int32(http_metadata_server_port, 8080,
@@ -322,6 +330,9 @@ void InitMasterConf(const mooncake::DefaultConfig& default_config,
default_config.GetDouble("eviction_high_watermark_ratio",
&master_config.eviction_high_watermark_ratio,
FLAGS_eviction_high_watermark_ratio);
+ default_config.GetDouble("ddr_admission_watermark_ratio",
+ &master_config.ddr_admission_watermark_ratio,
+ FLAGS_ddr_admission_watermark_ratio);
default_config.GetDouble("nof_eviction_ratio",
&master_config.nof_eviction_ratio,
FLAGS_nof_eviction_ratio);
@@ -378,6 +389,9 @@ void InitMasterConf(const mooncake::DefaultConfig& default_config,
default_config.GetString("allocation_strategy",
&master_config.allocation_strategy,
FLAGS_allocation_strategy);
+ default_config.GetDouble("ssd_high_watermark_ratio",
+ &master_config.ssd_high_watermark_ratio,
+ FLAGS_ssd_high_watermark_ratio);
default_config.GetBool("enable_http_metadata_server",
&master_config.enable_http_metadata_server,
FLAGS_enable_http_metadata_server);
@@ -716,6 +730,19 @@ void LoadConfigFromCmdline(mooncake::MasterConfig& master_config,
!conf_set) {
master_config.allocation_strategy = FLAGS_allocation_strategy;
}
+ if ((google::GetCommandLineFlagInfo("ssd_high_watermark_ratio", &info) &&
+ !info.is_default) ||
+ !conf_set) {
+ master_config.ssd_high_watermark_ratio =
+ FLAGS_ssd_high_watermark_ratio;
+ }
+ if ((google::GetCommandLineFlagInfo("ddr_admission_watermark_ratio",
+ &info) &&
+ !info.is_default) ||
+ !conf_set) {
+ master_config.ddr_admission_watermark_ratio =
+ FLAGS_ddr_admission_watermark_ratio;
+ }
if ((google::GetCommandLineFlagInfo("enable_http_metadata_server", &info) &&
!info.is_default) ||
!conf_set) {
diff --git a/mooncake-store/src/master_service.cpp b/mooncake-store/src/master_service.cpp
index 00604a8c7d..1674a75e4c 100644
--- a/mooncake-store/src/master_service.cpp
+++ b/mooncake-store/src/master_service.cpp
@@ -115,6 +115,7 @@ MasterService::MasterService(const MasterServiceConfig& config)
allow_evict_soft_pinned_objects_(config.allow_evict_soft_pinned_objects),
eviction_ratio_(config.eviction_ratio),
eviction_high_watermark_ratio_(config.eviction_high_watermark_ratio),
+ ssd_high_watermark_ratio_(config.ssd_high_watermark_ratio),
nof_eviction_ratio_(config.nof_eviction_ratio),
nof_eviction_high_watermark_ratio_(
config.nof_eviction_high_watermark_ratio),
@@ -139,7 +140,9 @@ MasterService::MasterService(const MasterServiceConfig& config)
nof_segment_manager_(config.memory_allocator),
memory_allocator_type_(config.memory_allocator),
allocation_strategy_(
- CreateAllocationStrategy(config.allocation_strategy_type)),
+ CreateAllocationStrategy(config.allocation_strategy_type,
+ config.ssd_high_watermark_ratio,
+ config.ddr_admission_watermark_ratio)),
enable_snapshot_restore_(config.enable_snapshot_restore),
enable_snapshot_(config.enable_snapshot),
snapshot_backup_dir_(config.snapshot_backup_dir),
@@ -1148,6 +1151,8 @@ auto MasterService::AllocateAndInsertMetadata(
ScopedAllocatorAccess allocator_access =
segment_manager_.getAllocatorAccess();
const auto& allocator_manager = allocator_access.getAllocatorManager();
+ ScopedLocalDiskSegmentAccess ssd_access =
+ segment_manager_.getLocalDiskSegmentAccess();
std::vector preferred_segments;
if (!config.preferred_segment.empty()) {
@@ -1158,7 +1163,8 @@ auto MasterService::AllocateAndInsertMetadata(
auto allocation_result = allocation_strategy_->Allocate(
allocator_manager, value_length, config.replica_num,
- preferred_segments);
+ preferred_segments, std::set(),
+ ReplicaType::MEMORY, &ssd_access);
if (!allocation_result.has_value()) {
VLOG(1) << "Failed to allocate replicas for key=" << key
@@ -1168,8 +1174,11 @@ auto MasterService::AllocateAndInsertMetadata(
}
if (write_mode != ReplicaWriteMode::FLEXIBLE_DUAL_REPLICA) {
MasterMetricManager::instance().inc_put_start_alloc_failures();
- need_mem_eviction_ = true;
- return tl::make_unexpected(ErrorCode::NO_AVAILABLE_HANDLE);
+ if (allocation_result.error() !=
+ ErrorCode::DDR_ADMISSION_REJECTED) {
+ need_mem_eviction_ = true;
+ }
+ return tl::make_unexpected(allocation_result.error());
}
} else {
allocated_memory_replicas = allocation_result->size();
@@ -1817,12 +1826,38 @@ auto MasterService::EvictDiskReplica(const UUID& client_id,
[](const Replica& replica) { return replica.is_disk_replica(); });
MasterMetricManager::instance().dec_file_cache_nums();
} else if (replica_type == ReplicaType::LOCAL_DISK) {
+ // Sum sizes of LOCAL_DISK replicas being evicted for SSD tracking
+ int64_t evicted_size = 0;
+ metadata.VisitReplicas(
+ [&client_id](const Replica& replica) {
+ return replica.is_local_disk_replica() &&
+ replica.get_descriptor()
+ .get_local_disk_descriptor()
+ .client_id == client_id;
+ },
+ [&evicted_size](Replica& replica) {
+ evicted_size += static_cast(
+ replica.get_descriptor()
+ .get_local_disk_descriptor()
+ .object_size);
+ });
metadata.EraseReplicas([&client_id](const Replica& replica) {
return replica.is_local_disk_replica() &&
replica.get_descriptor()
.get_local_disk_descriptor()
.client_id == client_id;
});
+ // Decrement SSD usage tracking
+ if (evicted_size > 0) {
+ ScopedLocalDiskSegmentAccess ssd_access =
+ segment_manager_.getLocalDiskSegmentAccess();
+ auto& client_segments = ssd_access.getClientLocalDiskSegment();
+ auto disk_it = client_segments.find(client_id);
+ if (disk_it != client_segments.end()) {
+ disk_it->second->ssd_used_bytes.fetch_sub(
+ evicted_size, std::memory_order_relaxed);
+ }
+ }
} else {
LOG(ERROR) << "key=" << key
<< ", error=invalid_replica_type_for_eviction";
@@ -2707,9 +2742,13 @@ auto MasterService::NotifyOffloadSuccess(
const UUID& client_id, const std::vector& keys,
const std::vector& metadatas)
-> tl::expected {
+ // Track total SSD usage increment for this batch
+ int64_t total_ssd_increment = 0;
+
for (size_t i = 0; i < keys.size(); ++i) {
const auto& key = keys[i];
const auto& metadata = metadatas[i];
+ total_ssd_increment += metadata.data_size;
// Release refcnt and clear offloading task.
{
@@ -2739,6 +2778,19 @@ auto MasterService::NotifyOffloadSuccess(
return tl::make_unexpected(res.error());
}
}
+
+ // Update SSD usage tracking for this client
+ {
+ ScopedLocalDiskSegmentAccess ssd_access =
+ segment_manager_.getLocalDiskSegmentAccess();
+ auto& client_segments = ssd_access.getClientLocalDiskSegment();
+ auto disk_it = client_segments.find(client_id);
+ if (disk_it != client_segments.end()) {
+ disk_it->second->ssd_used_bytes.fetch_add(
+ total_ssd_increment, std::memory_order_relaxed);
+ }
+ }
+
return {};
}
diff --git a/mooncake-store/src/real_client.cpp b/mooncake-store/src/real_client.cpp
index 74cdd0a742..17f037ecdd 100644
--- a/mooncake-store/src/real_client.cpp
+++ b/mooncake-store/src/real_client.cpp
@@ -32,6 +32,9 @@
#include "default_config.h"
#include "shm_helper.h"
#include "memory_location.h"
+#define UBDIAG_PERF_DEF_FILE "mooncake_perf_points.def"
+#define UBDIAG_PROGRAM_NAME "mooncake_store"
+#include "ubdiag/auto_perf.h"
#ifdef USE_ASCEND_DIRECT
#include "acl/acl_rt.h"
#include "transport/ascend_transport/ascend_direct_transport/context_manager.h"
@@ -826,6 +829,9 @@ tl::expected RealClient::setup_internal(
if (this->protocol == "ascend" || this->protocol == "ubshmem") {
ascend_segment_ptrs_.emplace_back(
ptr, AscendSegmentDeleter{this->protocol});
+ } else if (this->protocol == "ub") {
+ ub_segment_ptrs_.emplace_back(ptr,
+ UbSegmentDeleter{mapped_size});
} else if (!seg_numa_nodes.empty() || should_use_hugepage) {
// NUMA-segmented or hugepage: track as mmap allocation for
// munmap cleanup
@@ -1082,6 +1088,7 @@ tl::expected RealClient::tearDownAll_internal() {
client_buffer_allocator_.reset();
port_binder_.reset();
hugepage_segment_ptrs_.clear();
+ ub_segment_ptrs_.clear();
segment_ptrs_.clear();
local_hostname = "";
device_name = "";
@@ -1649,22 +1656,33 @@ tl::expected RealClient::put_internal(
LOG(ERROR) << "Client buffer allocator is not provided";
return tl::unexpected(ErrorCode::INVALID_PARAMS);
}
+ UbDiag::PerfPoint pt_alloc(PerfKey::PUT_INTERNAL_ALLOC_BUFFER, UbDiag::PerfLevel::MODULE);
+ pt_alloc.Start();
auto alloc_result = client_buffer_allocator->allocate(value.size_bytes());
+ pt_alloc.End(alloc_result ? 0 : -1);
if (!alloc_result) {
LOG(ERROR) << "Failed to allocate buffer for put operation, key: "
<< key << ", value size: " << value.size();
return tl::unexpected(ErrorCode::INVALID_PARAMS);
}
auto &buffer_handle = *alloc_result;
+ UbDiag::PerfPoint pt_memcpy(PerfKey::PUT_INTERNAL_MEM_COPY, UbDiag::PerfLevel::MODULE);
+ pt_memcpy.Start();
memcpy(buffer_handle.ptr(), value.data(), value.size_bytes());
+ pt_memcpy.End(0);
+ UbDiag::PerfPoint pt_split(PerfKey::PUT_INTERNAL_SPLIT_SLICES, UbDiag::PerfLevel::MODULE);
+ pt_split.Start();
std::vector slices = split_into_slices(buffer_handle);
+ pt_split.End(0);
auto put_result = client_->Put(key, slices, config);
if (!put_result) {
+ LOG(INFO) << "put_result key[" << key << "] rc[" << static_cast(put_result.error()) << "] size[" << value.size_bytes() << "]";
return tl::unexpected(put_result.error());
}
+ LOG(INFO) << "put_result key[" << key << "] rc[0] size[" << value.size_bytes() << "]";
return {};
}
@@ -1725,8 +1743,11 @@ tl::expected RealClient::put_batch_internal(
for (size_t i = 0; i < keys.size(); ++i) {
auto &key = keys[i];
auto &value = values[i];
+ UbDiag::PerfPoint pt_alloc(PerfKey::PUT_BATCH_INTERNAL_ALLOC_BUFFER, UbDiag::PerfLevel::MODULE);
+ pt_alloc.Start();
auto alloc_result =
client_buffer_allocator->allocate(value.size_bytes());
+ pt_alloc.End(alloc_result ? 0 : -1);
if (!alloc_result) {
LOG(ERROR)
<< "Failed to allocate buffer for put_batch operation, key: "
@@ -1734,8 +1755,14 @@ tl::expected RealClient::put_batch_internal(
return tl::unexpected(ErrorCode::INVALID_PARAMS);
}
auto &buffer_handle = *alloc_result;
+ UbDiag::PerfPoint pt_memcpy(PerfKey::PUT_BATCH_INTERNAL_MEM_COPY, UbDiag::PerfLevel::MODULE);
+ pt_memcpy.Start();
memcpy(buffer_handle.ptr(), value.data(), value.size_bytes());
+ pt_memcpy.End(0);
+ UbDiag::PerfPoint pt_split(PerfKey::PUT_BATCH_INTERNAL_SPLIT_SLICES, UbDiag::PerfLevel::MODULE);
+ pt_split.Start();
auto slices = split_into_slices(buffer_handle);
+ pt_split.End(0);
buffer_handles.emplace_back(std::move(*alloc_result));
batched_slices.emplace(key, std::move(slices));
}
@@ -1756,6 +1783,14 @@ tl::expected RealClient::put_batch_internal(
auto results = client_->BatchPut(keys, ordered_batched_slices, config);
// Check if any operations failed
+ size_t num_failed = 0;
+ for (size_t i = 0; i < results.size(); ++i) {
+ if (!results[i]) {
+ num_failed++;
+ }
+ }
+ LOG(INFO) << "batch_put_result num_keys[" << keys.size() << "] num_failed[" << num_failed << "]";
+
for (size_t i = 0; i < results.size(); ++i) {
if (!results[i]) {
return tl::unexpected(results[i].error());
@@ -2500,8 +2535,16 @@ std::shared_ptr RealClient::get_buffer_internal(
return nullptr;
}
+ auto t0 = std::chrono::steady_clock::now();
+ auto t_query = t0, t_select = t0, t_alloc = t0;
+ std::string replica_type;
+
// Query the object info
+ UbDiag::PerfPoint pt_query(PerfKey::GET_INTERNAL_QUERY, UbDiag::PerfLevel::MODULE);
+ pt_query.Start();
auto query_result = client_->Query(key);
+ t_query = std::chrono::steady_clock::now();
+ pt_query.End(query_result ? 0 : -1);
if (!query_result) {
if (query_result.error() == ErrorCode::OBJECT_NOT_FOUND ||
query_result.error() == ErrorCode::REPLICA_IS_NOT_READY) {
@@ -2512,6 +2555,8 @@ std::shared_ptr RealClient::get_buffer_internal(
return nullptr;
}
+ LOG(INFO) << "query_success key[" << key << "] replicas[" << query_result.value().replicas.size() << "]";
+
const std::vector &replica_list =
query_result.value().replicas;
if (replica_list.empty()) {
@@ -2523,8 +2568,12 @@ std::shared_ptr RealClient::get_buffer_internal(
// then LOCAL_DISK, then DISK.
// LOCAL_DISK data is on a remote node's SSD — must use offload RPC.
// MEMORY / DISK are handled via client_->Get below.
+ UbDiag::PerfPoint pt_select(PerfKey::GET_INTERNAL_SELECT_REPLICA, UbDiag::PerfLevel::MODULE);
+ pt_select.Start();
auto local_endpoints = client_->GetLocalEndpoints();
const auto *best_replica = SelectBestReplica(replica_list, local_endpoints);
+ t_select = std::chrono::steady_clock::now();
+ pt_select.End(best_replica ? 0 : -1);
if (!best_replica) {
LOG(ERROR) << "No usable replica for key: " << key;
return nullptr;
@@ -2533,12 +2582,34 @@ std::shared_ptr RealClient::get_buffer_internal(
const auto &replica = *best_replica;
uint64_t total_length = calculate_total_size(replica);
+ // Set replica_type for breakdown log and build endpoint string
+ if (replica.is_memory_replica()) {
+ replica_type = "memory";
+ std::string endpoint = replica.get_memory_descriptor().buffer_descriptor.transport_endpoint_;
+ LOG(INFO) << "replica_selected key[" << key << "] type[" << replica_type
+ << "] endpoint[" << endpoint << "] size[" << total_length << "]";
+ } else if (replica.is_local_disk_replica()) {
+ replica_type = "local_disk";
+ std::string endpoint = replica.get_local_disk_descriptor().transport_endpoint;
+ LOG(INFO) << "replica_selected key[" << key << "] type[" << replica_type
+ << "] endpoint[" << endpoint << "] size[" << total_length << "]";
+ } else {
+ replica_type = "disk";
+ std::string file_path = replica.get_disk_descriptor().file_path;
+ LOG(INFO) << "replica_selected key[" << key << "] type[" << replica_type
+ << "] file_path[" << file_path << "] size[" << total_length << "]";
+ }
+
if (total_length == 0) {
return nullptr;
}
// Allocate buffer
+ UbDiag::PerfPoint pt_alloc(PerfKey::GET_INTERNAL_ALLOC_BUFFER, UbDiag::PerfLevel::MODULE);
+ pt_alloc.Start();
auto alloc_result = client_buffer_allocator->allocate(total_length);
+ t_alloc = std::chrono::steady_clock::now();
+ pt_alloc.End(alloc_result ? 0 : -1);
if (!alloc_result) {
LOG(ERROR) << "Failed to allocate buffer for get_buffer, key: " << key;
return nullptr;
@@ -2547,8 +2618,22 @@ std::shared_ptr RealClient::get_buffer_internal(
auto buffer_handle =
std::make_shared(std::move(*alloc_result));
+ auto log_breakdown = [&](const char *status) {
+ auto now = std::chrono::steady_clock::now();
+ auto query_us = std::chrono::duration_cast(t_query - t0).count();
+ auto select_us = std::chrono::duration_cast(t_select - t_query).count();
+ auto alloc_us = std::chrono::duration_cast(t_alloc - t_select).count();
+ auto read_us = std::chrono::duration_cast(now - t_alloc).count();
+ auto total_us = std::chrono::duration_cast(now - t0).count();
+ LOG(INFO) << "get_breakdown key[" << key << "] query_us[" << query_us << "] select_us[" << select_us
+ << "] alloc_us[" << alloc_us << "] read_us[" << read_us << "] total_us[" << total_us
+ << "] type[" << replica_type << "] status[" << status << "]";
+ };
+
if (best_replica->is_local_disk_replica()) {
// LOCAL_DISK: data is on remote node's SSD. Use offload RPC.
+ UbDiag::PerfPoint pt_ssd(PerfKey::GET_INTERNAL_SSD_READ, UbDiag::PerfLevel::MODULE);
+ pt_ssd.Start();
const auto &endpoint =
best_replica->get_local_disk_descriptor().transport_endpoint;
std::unordered_map> objects;
@@ -2556,11 +2641,14 @@ std::shared_ptr RealClient::get_buffer_internal(
key, std::vector{{buffer_handle->ptr(), total_length}});
auto read_result =
batch_get_into_offload_object_internal(endpoint, objects);
+ pt_ssd.End(read_result ? 0 : -1);
if (!read_result) {
LOG(ERROR) << "SSD read failed for key '" << key
<< "': " << toString(read_result.error());
+ log_breakdown("ssd_fail");
return nullptr;
}
+ log_breakdown("ssd_ok");
return buffer_handle;
}
@@ -2575,16 +2663,24 @@ std::shared_ptr RealClient::get_buffer_internal(
<< "Ensure client_buffer_allocator_ returns host memory.";
}
+ const PerfKey read_key = replica.is_memory_replica()
+ ? PerfKey::GET_INTERNAL_MEM_READ
+ : PerfKey::GET_INTERNAL_DISK_READ;
+ UbDiag::PerfPoint pt_read(read_key, UbDiag::PerfLevel::MODULE);
+ pt_read.Start();
std::vector slices;
allocateSlices(slices, replica, buffer_handle->ptr());
auto filtered_qr = FilterQueryResult(query_result.value(), replica);
auto get_result = client_->Get(key, filtered_qr, slices);
+ pt_read.End(get_result ? 0 : -1);
if (!get_result) {
LOG(ERROR) << "Get failed for key: " << key
<< " with error: " << toString(get_result.error());
+ log_breakdown("read_fail");
return nullptr;
}
+ log_breakdown("read_ok");
return buffer_handle;
}
@@ -2790,8 +2886,21 @@ RealClient::batch_get_buffer_internal(
return final_results;
}
+ auto t0 = std::chrono::steady_clock::now();
+ auto t_query = t0, t_prep = t0, t_read = t0;
+
// 1. Query metadata for all keys
+ UbDiag::PerfPoint pt_bquery(PerfKey::GET_BATCH_INTERNAL_QUERY, UbDiag::PerfLevel::MODULE);
+ pt_bquery.Start();
auto query_results = client_->BatchQuery(keys);
+ t_query = std::chrono::steady_clock::now();
+ pt_bquery.End(0);
+
+ size_t num_found = 0;
+ for (const auto &result : query_results) {
+ if (result) num_found++;
+ }
+ LOG(INFO) << "batch_query_result num_keys[" << keys.size() << "] num_found[" << num_found << "]";
// 2. Prepare for batch get: filter valid keys and prepare buffers
struct KeyOp {
@@ -2833,8 +2942,11 @@ RealClient::batch_get_buffer_internal(
// Select best replica: prefer local MEMORY, then any MEMORY,
// then LOCAL_DISK, then DISK.
+ UbDiag::PerfPoint pt_bsel(PerfKey::GET_BATCH_INTERNAL_SELECT_REPLICA, UbDiag::PerfLevel::MODULE);
+ pt_bsel.Start();
const auto *best_replica =
SelectBestReplica(query_result_values.replicas, local_endpoints);
+ pt_bsel.End(best_replica ? 0 : -1);
if (!best_replica) {
LOG(ERROR) << "No usable replica for key: " << key;
continue;
@@ -2847,7 +2959,10 @@ RealClient::batch_get_buffer_internal(
auto &allocator = client_buffer_allocator ? client_buffer_allocator
: client_buffer_allocator_;
+ UbDiag::PerfPoint pt_balloc(PerfKey::GET_BATCH_INTERNAL_ALLOC_BUFFER, UbDiag::PerfLevel::MODULE);
+ pt_balloc.Start();
auto alloc_result = allocator->allocate(total_size);
+ pt_balloc.End(alloc_result ? 0 : -1);
if (!alloc_result) {
LOG(ERROR) << "Failed to allocate buffer for key: " << key;
continue;
@@ -2889,12 +3004,16 @@ RealClient::batch_get_buffer_internal(
.slices = std::move(slices)});
}
+ t_prep = std::chrono::steady_clock::now();
+
if (valid_ops.empty() && disk_ops.empty()) {
return final_results;
}
// 3. Execute batch get for memory/disk replicas
if (!valid_ops.empty()) {
+ UbDiag::PerfPoint pt_bget(PerfKey::GET_BATCH_INTERNAL_MEMDISH_READ, UbDiag::PerfLevel::MODULE);
+ pt_bget.Start();
std::vector batch_keys;
std::vector batch_query_results;
std::unordered_map> batch_slices;
@@ -2922,10 +3041,13 @@ RealClient::batch_get_buffer_internal(
<< "': " << toString(batch_get_results[i].error());
}
}
+ pt_bget.End(0);
}
// 5. Execute batch get for LOCAL_DISK replicas via SSD RPC
if (!disk_ops.empty()) {
+ UbDiag::PerfPoint pt_bssd(PerfKey::GET_BATCH_INTERNAL_SSD_READ, UbDiag::PerfLevel::MODULE);
+ pt_bssd.Start();
// Group by transport endpoint
std::unordered_map>>
@@ -2973,6 +3095,26 @@ RealClient::batch_get_buffer_internal(
}
}
}
+ pt_bssd.End(0);
+ }
+
+ t_read = std::chrono::steady_clock::now();
+
+ size_t success_count = 0;
+ for (const auto &result : final_results) {
+ if (result) success_count++;
+ }
+ {
+ auto query_us = std::chrono::duration_cast(t_query - t0).count();
+ auto prep_us = std::chrono::duration_cast