Skip to content

[Store] Assert HA restore state from domain state, not metrics - #5

Open
Hubert-Zhu wants to merge 1 commit into
mainfrom
hzhu/store-tests-domain-state-oracle
Open

Hubert-Zhu wants to merge 1 commit into
mainfrom
hzhu/store-tests-domain-state-oracle

Conversation

@Hubert-Zhu

Copy link
Copy Markdown
Owner

Description

The standby restore and remount tests in master_service_ha_test.cpp read
master_allocated_mem_size to verify how many bytes each step accounted for.
That makes an observability gauge the oracle for business state, which
RFC #3158 explicitly
rules out, and it pins the tests to the legacy incremental metric writes
rather than to the authoritative state those writes describe.

This replaces all 11 such assertions with assertions on the authoritative
owners. The gauge is the sum of two distinct things, and the tests were
conflating them:

  • Bytes restored from a standby snapshot live in
    MasterService::standby_accounted_memory_bytes_ until ReMountSegment
    hands them to a real allocator. Before the remount these replicas hang off
    DummyBufferAllocator, which is never attached to the segment usage
    tracker, so they are invisible to the allocator-side view.
  • Bytes owned by a mounted segment live in that segment's allocator, already
    reachable via the existing SegmentAllocatedSizeForTesting helper.

Splitting the assertions along that boundary also lets each remount test
state that the handoff drained the standby ledger
(StandbyAccountedBytesForTesting(service) == 0), which the single aggregate
gauge could not express.

This is a prerequisite for removing the legacy DRAM gauge writers: those
tests would otherwise be the only thing keeping the incremental writes alive.

No production code and no production behavior changes.

Module

  • Mooncake Store (mooncake-store)

Type of Change

  • Refactor

How Has This Been Tested?

Test-only change, scoped to master_service_ha_test.cpp. The affected tests
are RestoreFailureKeepsExistingState,
RemountMakesRestoredMemoryReplicaReady and
RemountRestoresCachelibMemoryReplica.

Test commands:

bash scripts/run_ci_test.sh

Test results:

  • Unit tests pass
  • Integration tests pass (if applicable)
  • Manual testing done (describe below)

Not yet verified locally: the development machine has no cmake,
clang-format or pre-commit available, so this branch has not been compiled,
run or formatted locally. Every added line was checked by hand against the
80-column limit, but clang-format's exact wrapping is unverified.

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • AI tools were used (specify below)

Cursor was used to locate the gauge-based assertions, trace the standby
accounting lifecycle through RestoreFromStandbySnapshot and
ReMountSegment, and draft the replacement assertions.

Made with Cursor

The standby restore and remount tests read master_allocated_mem_size to
verify how many bytes each step accounted for. That makes an observability
gauge the oracle for business state, which RFC kvcache-ai#3158 explicitly rules out,
and it pins the tests to the legacy incremental metric writes rather than to
the authoritative state those writes describe.

Assert the two authoritative owners instead. Bytes restored from a standby
snapshot live in MasterService::standby_accounted_memory_bytes_ until
ReMountSegment hands them to a real allocator; before that they hang off
DummyBufferAllocator, which is never attached to the segment usage tracker
and so is invisible to the allocator-side view. Bytes owned by a mounted
segment live in that allocator, already reachable through the existing
SegmentAllocatedSizeForTesting helper.

Splitting the assertions along that boundary also lets each remount test
state that the handoff drained the standby ledger, which the single
aggregate gauge could not express. Guard the restore-failure comparison so
it cannot hold on two zeroes if the accounting is ever empty at that point,
which matters because the upcoming DRAM gauge work touches this bookkeeping.

No production behavior changes.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Hubert-Zhu
Hubert-Zhu force-pushed the hzhu/store-tests-domain-state-oracle branch from 2f56143 to ff122df Compare September 4, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant