Skip to content

[Conductor] conductor: tiered prefix indexer - #4041

Open
Misak2333 wants to merge 1 commit into
kvcache-ai:mainfrom
Misak2333:main
Open

[Conductor] conductor: tiered prefix indexer#4041
Misak2333 wants to merge 1 commit into
kvcache-ai:mainfrom
Misak2333:main

Conversation

@Misak2333

@Misak2333 Misak2333 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Description

Third slice of the Mooncake Conductor upstreaming series, on top of the
prefix index types and hash strategy. Adds PrefixCacheTable, the
in-memory index that answers cache-hit queries across GPU, CPU and disk
tiers:

  • prefixindex/prefix_indexer.{h,cpp}: engine registration and profile
    binding validation, the GPU and shared-tier store/remove/clear
    mutations, the Query prefix walk, and GetGlobalView. Blocks are keyed
    by ProjectedPrefix and carry per-tier owner sets, so one walk reports
    the longest match per instance plus per-dp-rank and per-tier hit
    counts, with optional cache_salt and instance filtering.
  • Two-level locking: a global context-map mutex guards the context
    lookup, and each ContextState has its own mutex. Contexts are held by
    shared_ptr so a query drops the map lock before touching state, and
    the lock order is fixed one way (map, then state) to keep it
    deadlock-free. Query does not reorder entries, so read-only traffic
    keeps a shared lock.
  • Capacity control: per-context block limit (200k by default, zero
    disables it) with insertion-order eviction that drops the oldest
    entries in batches down to 90% occupancy, and a cumulative eviction
    counter that surfaces stored/removed events going out of sync.
  • prefix_indexer_test_peer.h exposes the internals the tests need
    (presence snapshots, order-metadata sizes, state locks) without
    widening the public surface, through a single friend declaration.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Reshard (mooncake-reshard)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Test commands:

# Example: bash scripts/run_ci_test.sh

Test results:

  • Unit tests pass
  • Integration tests pass (if applicable)
  • Manual testing done (describe below)

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

Third slice of the Mooncake Conductor upstreaming series, on top of the
prefix index types and hash strategy. Adds PrefixCacheTable, the
in-memory index that answers cache-hit queries across GPU, CPU and disk
tiers

Co-authored-by: Misak2333 <167268798+Misak2333@users.noreply.github.com>
Co-authored-by: Chase-R <lolopop.rong@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant