Skip to content

[Store] Centralize client session lifecycle and resource ownership - #4180

Open
Aionw wants to merge 5 commits into
kvcache-ai:mainfrom
Aionw:feat/client-session
Open

Aionw wants to merge 5 commits into
kvcache-ai:mainfrom
Aionw:feat/client-session

Conversation

@Aionw

@Aionw Aionw commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Description

Centralize client registration, remount readiness, heartbeat/liveness monitoring, host hints, and offboarding lifecycle in ClientSessionManager instead of MasterService.

  • Add lock-scoped session admission, registration rollback, staged restore, and ordered state-change listeners.
  • Keep resource cleanup in the MasterService listener while the manager owns monitoring, offboarding workers, and the snapshot cleanup barrier.
  • Bind memory buffers and local-disk replicas to the exact client incarnation via read-only ClientSessionSharedPtr; replacing a session invalidates old segment generations rather than silently transferring their resources.
  • Check local-disk staleness using the bound session rather than a point-in-time registry complement.
  • Adapt snapshot restore, metrics, and tests. No RPC or snapshot wire-format change is intended.

Related work and scope

Searched existing issues/open PRs for client sessions and liveness before opening:

Draft blockers

This is a substantive change exceeding 500 LOC. A dedicated RFC has not yet been filed/linked. Human confirmation of reviewing every changed line is also pending; keep this PR in draft until both are addressed.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Mooncake Conductor (mooncake-conductor)
  • Reshard (mooncake-reshard)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Local selected-target build and tests passed. HA/snapshot tests were adapted but not run in this submission; full integration and hardware-specific configurations remain unvalidated.

Test commands:

CCACHE_DIR="$HOME/.cache/ccache/mooncake" \
CCACHE_BASEDIR="$(git rev-parse --show-toplevel)" \
cmake --build build --target client_session_manager_test client_liveness_test \
  segment_test master_service_test promotion_on_hit_test -j 8
ctest --test-dir build --output-on-failure -j 4 \
  -R '^(client_session_manager_test|client_liveness_test|segment_test|master_service_test|promotion_on_hit_test)$'
prek run --files $(git diff --name-only HEAD^ HEAD)
git diff --check HEAD^ HEAD

Test results:

  • Selected unit tests pass: 5/5 CTest targets (121.56 seconds).
  • Integration tests pass (if applicable)
  • Manual testing done (describe below)

All applicable lightweight hooks passed except mooncake-code-format: it requires unavailable clang-format 20 (the host provides 22.1.8). Changed-line clang-format 22 was used for the final inline-review edits, not claimed as a full PR formatting pass. The build emitted warnings from bundled fmt headers.

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

OpenAI Codex assisted with code changes, inline-review follow-up, validation, and PR preparation. Human end-to-end review and responsibility confirmation remain pending; no claim of completed human review is made.

Move registration and resource-operation serialization into replaceable client slots, keeping liveness observations independent of slow remount work. Preserve incarnation validation, restore coordination, and readiness accounting without long-held registry locks.

Cover concurrent admission, rollback, removal, restore, observer shutdown, and slot replacement. Add a disabled manual regression experiment for kvcache-ai#3936.

Validation: 249 tests passed across liveness, session manager, MasterService, HA, and segment suites; liveness/session suites repeated 50 times. The kvcache-ai#3936 fault-injection experiment also passed. File-scoped prek checks passed except the unavailable clang-format-20 hook; changed lines were checked with clang-format 22.

AI-assisted implementation and testing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant