Skip to content

Cluster sync can miss a real-time update under heavy load #73

Description

@benitogf

Summary

Under sustained heavy load a cluster node can miss a synced real-time update. It surfaced as an intermittent hang in the cluster sync test under repeated race runs (~1 in 300 iterations). Adopt the upstream reliability fix and make the sync test deterministic.

Motivation

The real-time event pipeline could drop a delivery when a subscriber stalled momentarily under load, leaving that subscriber's view silently behind storage. In the cluster sync integration test this showed up as a rare permanent hang under -race at high iteration counts; in production it would mean an occasional missed update on a busy node. The underlying store now offers an opt-in lossless delivery mode and stops leaking background watch workers — see ooo PR #149 — which this repo should pick up.

Acceptance

  • The cluster sync test runs cleanly under high-iteration -race (no intermittent hang) and fails loudly if any real-time event is ever dropped.
  • The store dependency is updated to the release with lossless delivery and the watch-worker leak fix.
  • The suite continues to build and pass after the update, including adapting to the dependency's removed request-gate API.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions