Skip to content

Fix node→pivot write loss in set/delete/set sequences - #66

Merged
benitogf merged 3 commits into
masterfrom
fix/cluster-sync-vv-divergence
Jun 9, 2026
Merged

Fix node→pivot write loss in set/delete/set sequences#66
benitogf merged 3 commits into
masterfrom
fix/cluster-sync-vv-divergence

Conversation

@benitogf

@benitogf benitogf commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

Closes #65 — a node-originated write is no longer lost when a key is set, deleted from the pivot, then set again on the node.

  • The node's local write now reliably advances its version vector, so the pivot accepts it instead of mistaking it for an already-seen duplicate.
  • Closes two timing windows that caused the version vector to stay stale: a cross-operation bookkeeping mix-up between a concurrent pulled delete and a local set, and the bump landing after the change was broadcast (so the peer push read a stale vector).
  • Makes the end-to-end sync tests synchronise deterministically on observed sync events — exact-count wait groups — instead of time-based polling or deadlines. Now that the version-vector fix removes duplicate deliveries, each operation produces an exactly predictable number of events, so the tests can't flake or hang under load (the original CI failure was a timing-related hang).

Depends on the merged github.com/benitogf/ooo operation-aware post-write hook (benitogf/ooo#139); go.mod is pinned to it.

Test plan

  • Set → delete-from-pivot → set-on-node on the same key: the node's write reaches the pivot (no divergence).
  • End-to-end sync tests pass deterministically with no time-based waiting (no sleeps, no polling, no deadlines).
  • Full suite green under the race detector; validated ~6900 race iterations with zero divergence (was ~1/300).

🤖 Generated with Claude Code

…te loss

A node-originated write could be permanently dropped by the pivot in a
set/delete sequence on the same key: the local VV bump was skipped, so
the node pushed a stale VV the pivot rejected as VVEqual. Two causes,
both in how the pull bump-skip marks were consumed:

  - cross-op steal: the post-write bump consumed either the set OR the
    delete mark op-unaware, so a concurrent pulled delete's mark could
    suppress a local set's bump. Now consumes only the mark matching the
    write's own operation, via ooo's AfterWriteOp hook.

  - push-vs-bump race: the bump ran after the storage event was
    broadcast, so the event-driven peer push could read the pre-bump VV.
    ooo's AfterWriteOp fires before the broadcast, closing the window.

Rewrites testClusterSync to wait for observed-state convergence with a
bounded deadline instead of counting non-deterministic async events (the
old exact-count WaitGroups, which hung to the 60s package timeout on any
drift — the original CI failure). Validated ~6900 -race runs with zero
divergence (was ~1/300).

Requires github.com/benitogf/ooo with AfterWriteOp (>= main 8a306d1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@CBosch101 CBosch101 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes the node→pivot write loss by making the local VV bump operation-aware (consume only the mark matching this write's own op) and moving it onto ooo's AfterWriteOp so it lands before the broadcast; rewrites the cluster test to converge-or-fail-fast instead of counting non-deterministic async events. The two core race-fix claims hold against the ooo source, and both product consumers build clean against this PR. Approving.

Verified

  • ooo AfterWriteOp fires before the storage broadcast and before the legacy AfterWrite, and emits only "set"/"del" (ooo storage/layered.go:47-58,555,748) — substantiates both the "bump before broadcast" and "caller AfterWrite still sees post-bump VV" claims, and makes the op=="del"-else-set dispatch total.
  • Op dispatch lines up with the mark setters: trackSetbumpSkipSet, trackDeletebumpSkipDelete (sync.go:552-565); each pulled write later consumes its own matching mark, so the op-specific consume introduces no mark leak.
  • Breakage check — idnerdidx/bulk and idnerdidx/bundle both go mod tidy + go build ./... clean against this PR's pivot (which drags in the bumped ooo). Only pre-existing environmental failures remain, identical to baseline: libpcsclite/ebfe/scard (both repos) and -lstreamdock link in bundle/collector. pivot-adjacent packages test clean; bulk's lone failure (pivot/router TestRunUploadJob_PanicSurfacesAsError) fails identically at baseline.
  • The PR's failing check is a pre-existing flake, not this change: macos failed on TestClockDriftScenario (not in this diff) under -race -failfast -timeout 60s; the same commit's push workflow run passed all three platforms, and it passes locally under -race -count=3.
  • pivot suite green locally under -race (cluster sync ×3 + full suite). Did not reproduce the full ~6900-iteration figure — sampled only.

No blockers.

TestClockDriftScenario read pivot.VVManager directly right after
PivotWg.Wait(), but the pivot Set handler increments the leader counter
AFTER db.SetWithMeta returns — i.e. after the storage AfterWrite that
drives PivotWg.Done(). Waiting on PivotWg proves the pushed data landed,
not that the counter bumped, so the read raced the bump and intermittently
saw an empty/stale VV ("0 is not greater than 0", surfaced on the macOS CI
runner). The race is pre-existing on master and independent of this
branch's production change; the bump-after-write is correct in production
(it still precedes peer fanout) — only a test reading the VV directly can
observe the gap.

Fix it the way /testing-go-backend-async prescribes — a callback hook on
the component under test, not require.Eventually: VVManager.SetBumpObserver
fires once per counter increment, and the test pairs Add(1)/Wait() with
each push. The observer is disarmed before phase 4, whose pivot write bumps
via a path the test does not assert on. Production never sets the observer.

Verified: 1500 serial -race runs, 720 parallel runs under scheduling
pressure, and 8x full -race suite — zero failures (old rate ~1/300).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@benitogf
benitogf force-pushed the fix/cluster-sync-vv-divergence branch from e0e03f9 to 93a6531 Compare June 8, 2026 09:01
testClusterSync waited for cross-cluster sync via requireConverged — a
time.Sleep poll loop with a 20s deadline — plus a time.After wrapper around
the subscription-establishment barrier. Both violate the async testing rules
(no sleeps, no polling, no timing-based synchronisation).

That polling was a workaround for non-deterministic event counts that no
longer exist. The duplicate push-vs-pull deliveries that once made counts
unpredictable are gone now that the version-vector fix on this branch dedups
redundant syncs. Measured directly: every subscribed key delivers exactly one
websocket message per side per mutation (things/settings/items; set/push/
delete), so exact-count WaitGroups are sound again.

- deliv WaitGroup counts post-establishment ws deliveries; each operation
  arms Add(2) (pivot sub + node sub), triggers, Wait()s, then asserts.
- wsReady (count 8) gates on each subscription's initial snapshot; its
  time.After wrapper is replaced by a plain Wait().
- Policies have no ws subscription (custom HTTP routes), so they wait on a
  policyWrites WaitGroup driven by an authStorage AfterWrite, wired through
  FakeServer's new onPolicyWrite parameter.
- Removed requireConverged and the now-dead diagThings / tryGetThing /
  tryGetItem diagnostics.

Verified: 100 serial + 240 parallel -race runs of TestClusterSync, plus 6x
full -race suite — zero hangs, zero failures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@benitogf
benitogf merged commit 757b852 into master Jun 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cluster sync can permanently lose a node write after a cross-side delete

2 participants