fix(mesh): wire EpochMaxWins into CRDT merge

[CatherineSue]

Description

Problem

Mesh v2 routes rate-limit counters through the generic CRDT OR-map (CrdtOrMap), with a per-prefix MergeStrategy. The rl: namespace is specified to use EpochMaxWins (spec §2.5): higher epoch wins, equal epoch takes the max count, and a tombstone partitions history so pre-tombstone inserts cannot resurrect a deleted shard. Before this PR:

• CrdtOrMap::merge only knew LastWriterWins semantics. Any rl: traffic gossiped over the eventual CRDT_BATCH wire path would merge incorrectly — a node that resets to {epoch: N+1, count: 0} could be overwritten by a stale gossip of {epoch: N, count: 100} because the older message has a higher Lamport timestamp.
• Operation-log compaction (append_with_strategy / compact / snapshot_and_truncate) was hardcoded to LWW. A single auto-compaction over 10K entries would collapse an rl: key to the wrong winner, and a snapshot pulled by a joining peer would replay the wrong value.
• There was no canonical on-the-wire shape for an rl: payload that survived a tombstone after compaction. Without tombstone_version embedded in the stored value, a peer receiving only the compacted Insert (the Remove having been folded away) had no way to know that earlier inserts at the same key are pre-tombstone and must be suppressed.

Solution

Wire EpochMaxWins through the CRDT merge end-to-end and normalize rl: payloads into a self-describing shard format that carries both the live-point frontier and the newest tombstone boundary. Every place that selects "the winner" — local apply, operation-log compaction, log snapshot, and remote merge — consults the per-key strategy through the same registry. The gateway adapter only sees EpochCount; the shard format is an internal CRDT detail.

Changes

Per-prefix merge strategy registry

• MergeStrategy enum surfaces as a crdt_kv export with two variants: LastWriterWins (default; worker:, policy:, config:) and EpochMaxWins (rl:). The earlier MaxValueWins placeholder was removed because exposing a configurable strategy with no implementation is a footgun.
• MeshKV::configure_crdt_prefix(prefix, strategy) now registers the strategy on the shared CrdtOrMap so apply and compaction both pick it up. Keeps the mesh crate application-agnostic — gateway adapters select the strategy at namespace setup time, not via mesh-side hardcoding.
• merge_strategy_for_key_from(strategies, key) does longest-prefix match against a snapshotted strategy table. The snapshot is taken once per merge/compact/append call to avoid reacquiring the read lock for every key inside the loop.

Apply-path: live store always reflects the configured strategy

• apply_insert_locked branches on the per-key strategy. LWW uses the existing record_insert_metadata path. EpochMaxWins routes through record_epoch_insert_metadata, which runs the merge against the current stored shard (with its embedded tombstone_version) plus any tombstone metadata, and only updates the store when the merged bytes actually change.
• apply_remove/remove likewise branch on the strategy. For EpochMaxWins, apply_epoch_remove_locked filters the stored shard per-point against the merged (existing ∪ incoming) tombstone — matching what compact_operations does in the log. If any live points survive, the filtered shard is written back with the new tombstone embedded; if all points are killed, the key is removed and metadata holds the tombstone for GC.
• Metadata for EpochMaxWins keys holds at most one entry: the newest live version when the shard is alive (the shard's embedded tombstone_version is the source of truth for the boundary), or the newest tombstone when the key is fully gone (so a delayed pre-tombstone insert cannot resurrect).

Stored shape: RateLimitShard as a self-describing CRDT state

• rl: values are no longer stored as the raw 16-byte (epoch, count) write payload. They are normalized into a RateLimitShard { live_points: Vec<LivePoint>, tombstone_version: Option<RateLimitVersion> } and bincode-serialized. The frontier holds the live points that could become the correct value after a future tombstone boundary, computed via suffix-best.
• The public decode_epoch_count accepts both the normalized shard form (what callers see on get() and via gossip) and the raw 16-byte payload (what local namespace subscribers can observe before CRDT normalization runs). The gateway adapter only ever sees EpochCount and never has to know about CRDT internals.
• state_from_insert_value accepts either shape on the insert boundary too, so a local subscriber that round-trips through put then subscribe doesn't blow up on its own raw payload.
• The compacted Insert emitted by RateLimitState::into_operation is the serialized shard (carrying any embedded tombstone_version), tagged with the newest live version's (timestamp, replica_id). That choice keeps the local op-log compact (one op per key after compaction) and preserves the original Lamport ordering for downstream merge.

Operation-log compaction: holistic per-key, allocation-free

• latest_epoch_max_wins_operation evaluates each key holistically: it folds all operations into a single RateLimitState and emits one canonical op. Earlier pairwise comparators mixed epoch ordering with timestamp ordering and produced order-dependent winners.
• Replaced a Vec<u8> allocation per pairwise compare with the holistic single-fold, dropping compaction allocations from O(N) to O(1) for an EpochMaxWins key.
• compact_operations recognizes tombstone cutoffs: pre-tombstone inserts are ignored, post-tombstone inserts can recreate the key and compete by EpochMaxWins. Malformed insert payloads are skipped (logged and continued) rather than aborting the whole compaction.
• append_with_strategy, compact_with_strategy, and snapshot_and_truncate all take a per-key strategy callback. None of these paths can accidentally rewrite an EpochMaxWins winner as an LWW winner.

Cross-replica merge: tombstone boundary must survive same-op-id collisions

• OperationLog::merge_with_strategy is strategy-aware. For LWW keys it dedups by (replica_id, timestamp) as before. For EpochMaxWins keys, on op-id collision it replaces the local op with epoch_max_wins::compact_operations([local, incoming]) so a compacted snapshot carrying an embedded tombstone_version overwrites the older raw payload at the same op-id.
• CrdtOrMap::merge filters unseen_operations per-key. LWW dedups by op-id; EpochMaxWins always replays, with merge_live_value.changed gating the store update so identical bytes remain a no-op. Without this, a receiver that already saw the post-tombstone raw insert would silently drop a compacted snapshot at the same op-id, never learn the tombstone boundary, and let a delayed pre-tombstone insert resurrect the deleted shard (spec §5.8 violation).
• The (timestamp, replica_id) short-circuit in record_epoch_insert_metadata was removed. The acceptance gate now belongs on the merged bytes (changed: bool), not on operation identity, because the same op-id can carry semantically different payloads under EpochMaxWins.

Equal-value insert no longer rewinds metadata

• When merge_live_value produces bytes equal to the current stored value (changed: false), record_epoch_insert_metadata returns without clearing the version vec. Previously, an older equal-value insert could erase newer metadata and let an intermediate tombstone delete the live value — silent data loss for the rl: reset path.

Strategy snapshotting at the right granularity

• compact_operation_log and append_operation snapshot the strategy table once and pass it into the operation-log helpers as a closure. The read lock on merge_strategies is acquired exactly once per gossip round, not once per key. Keeps per-key lookups lock-free under the hot path.

Test Plan

cargo test -p smg-mesh --lib — 141 tests pass, including the new EpochMaxWins-specific cases:

• test_epoch_max_wins_tombstone_filters_pre_tombstone_inserts_per_point — three operations (insert ε=5 @ ts=100, insert ε=6 @ ts=90, remove @ ts=95) arriving in separate merges. The post-tombstone live point survives (ε=5, count=100), the pre-tombstone high-epoch point is dropped. Matches spec §2.5.
• test_epoch_max_wins_live_store_matches_compacted_log_after_tombstone — same sequence, asserts that decode(store.get(key)) equals decode(value_of(compacted_log)). Catches any future drift between live apply and operation-log compaction.
• test_epoch_max_wins_tombstone_kills_all_points_removes_key — tombstone newer than every live point removes the key from the store, and a delayed pre-tombstone insert cannot resurrect it.
• test_epoch_max_wins_snapshot_only_propagation_preserves_tombstone_boundary — source replica compacts {I60 high-ε, T65, I70 low-ε} into one Insert at op-id (70, c) with embedded tombstone_version=65. Receiver applies snapshot, sees ε=6 count=1. A late peer's pre-tombstone insert at op-id (60, _) is filtered.
• test_epoch_max_wins_compacted_snapshot_applies_when_op_id_already_seen — the codex P1 case. Receiver already has the raw post-tombstone insert at op-id (70, c). Source sends compacted snapshot at the same op-id with the richer payload. Merge correctly applies it, embedding the tombstone, and a delayed pre-tombstone insert is then suppressed.
• compacted_live_state_remembers_tombstone_boundary, compacted_live_state_uses_newest_live_version, compact_operations_skips_malformed_inserts — direct tests for the holistic compaction logic.

Plus the existing 130+ tests covering LWW behavior, basic CRDT semantics, merge convergence, tombstone GC, and chunking — none regressed.

cargo clippy --workspace --all-targets --all-features -- -D warnings clean.

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets --all-features -- -D warnings passes
• Documentation updated: not needed; the spec at .claude/docs/mesh/mesh-v2-implementation-spec.md §2.5 and §5.8 was the source of truth and is unchanged. Module-level doc on epoch_max_wins.rs documents the stored-vs-raw shape distinction.
• (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

Summary by CodeRabbit

• New Features

  • Per-prefix configurable merge strategies (Last-Writer-Wins and Epoch-Max) with strategy-aware compaction, snapshots/merge, and operation persistence; configured prefixes are registered with the store.
  • Epoch-Max adds tombstone-aware epoch/count normalization to enforce deletes and prevent epoch “rewind.”

• Tests

  • Expanded tests covering compaction, tombstones, epoch tie-breaking, snapshots, and post-tombstone behaviors.

• Documentation

  • Adapter docs clarified decoding of epoch/count and normalized shard state.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2001a2e7-ef0f-4e56-ad48-c67a285bb7db

📥 Commits

Reviewing files that changed from the base of the PR and between cf38daf and 6274a61.

📒 Files selected for processing (2)

• crates/mesh/src/crdt_kv/crdt.rs
• crates/mesh/src/crdt_kv/kv_store.rs

 __________________________________________________________
< Your types are so loose they're in an open relationship. >
 ----------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

📝 Walkthrough

Walkthrough

Adds a per-prefix MergeStrategy registry (longest-prefix match), introduces MergeStrategy, refactors EpochMaxWins to normalized shard/tombstone state, and threads strategy-aware compaction/merge through OperationLog and CrdtOrMap; tests and MeshKV/adapter wiring updated.

Changes

Per-Prefix Merge Strategies for CRDT with Shard-Aware EpochMaxWins

Layer / File(s)	Summary
MergeStrategy Type & Module Exports crates/mesh/src/crdt_kv/merge_strategy.rs, crates/mesh/src/crdt_kv/mod.rs, crates/mesh/src/lib.rs	New MergeStrategy enum with LastWriterWins and EpochMaxWins; module/submodule and re-exports updated so MergeStrategy is sourced from crdt_kv.
EpochMaxWins Shard-Aware Deterministic Merging crates/mesh/src/crdt_kv/epoch_max_wins.rs	Refactor to normalized RateLimitShard encoding, two-path decode(), tombstone-aware merge_live_value/apply_tombstone, and compact_operations; tests updated for decoding, malformed inputs, idempotence/associativity, and compaction semantics.
CrdtOrMap Registry & Metadata Bridge crates/mesh/src/crdt_kv/crdt.rs	Add merge_strategies registry (longest-prefix ordering), register_merge_strategy(), strategy lookup, and ValueMetadata ↔ RateLimitVersion helpers.
OperationLog Strategy Parameterization crates/mesh/src/crdt_kv/operation.rs	Parameterize append/compact/snapshot/merge by per-key strategy callback; implement strategy-aware latest-operation selection and merge folding for EpochMaxWins collisions.
Strategy-Dispatched Insert & Remove Operations crates/mesh/src/crdt_kv/crdt.rs	Refactor insert/remove flows to build Operation and dispatch via apply_insert_locked() using per-key MergeStrategy; add record_epoch_insert_metadata() and apply_epoch_remove_locked() for EpochMaxWins and ensure tombstones recorded for never-seen keys.
Merge with Strategy-Aware Compaction & Unseen Operations crates/mesh/src/crdt_kv/crdt.rs	merge(&OperationLog) snapshots strategies, compacts local log per-key before apply, and treats unseen operations differently per strategy to preserve embedded tombstone metadata for EpochMaxWins.
KvStore Simplification crates/mesh/src/crdt_kv/kv_store.rs	Removed KvStore::upsert() and simplified DashMap usage; store mutations now use insert()/remove() only.
MeshKV Strategy Registration Wiring crates/mesh/src/kv.rs	MeshKV::new() registers config: with LastWriterWins; configure_crdt_prefix() registers configured prefixes on shared CrdtOrMap via register_merge_strategy().
Rate Limit Adapter Documentation & Test Updates model_gateway/src/mesh/adapters/rate_limit_sync.rs	Docs updated to document epoch/count payload and decoding; tests no longer assert fixed raw length and validate via decode_epoch_count + EpochCount.
Comprehensive EpochMaxWins & Strategy-Aware Test Suite crates/mesh/src/crdt_kv/tests.rs, crates/mesh/src/crdt_kv/crdt.rs	Extensive tests added for compaction/tombstone semantics, snapshot/truncate behavior, resurrection/non-rewind scenarios, and strategy-aware merge idempotence/order-independence.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant CrdtOrMap
  participant OperationLog
  participant EpochMaxModule
  Client->>CrdtOrMap: insert(key, value)
  CrdtOrMap->>CrdtOrMap: get_strategy_for_key(key)
  alt EpochMaxWins
    CrdtOrMap->>EpochMaxModule: record_epoch_insert_metadata(value, metadata)
    EpochMaxModule->>CrdtOrMap: maybe update stored value/metadata
  else LastWriterWins
    CrdtOrMap->>CrdtOrMap: record_lww_insert_metadata(value, metadata)
  end
  CrdtOrMap->>OperationLog: append_with_strategy(operation, strategy)
  OperationLog->>OperationLog: compact_with_strategy(strategy)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

• lightseekorg/smg#825: Related changes to OperationLog append/auto-compact behavior that this PR extends to be strategy-aware.
• lightseekorg/smg#1295: Prior work on epoch_max_wins that this change builds on and extends with normalized shard/state semantics.
• lightseekorg/smg#1299: Related edits around MeshKV prefix registration and wiring that intersect with this PR's strategy registration.

Suggested labels

tests

Suggested reviewers

• tonyluj
• llfl
• slin1237

Poem

A rabbit hops through prefix trees,
Registers strategies with nimble ease,
Epochs hum and tombstones gleam,
Shards agree and counters dream. 🐇✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix(mesh): wire EpochMaxWins into CRDT merge' directly describes the main objective of this PR: integrating the EpochMaxWins merge strategy into the CRDT merge system. The PR implements strategy-aware merge paths, operation-log compaction, and apply/remove operations across the mesh CRDT stack, with the primary change being this wiring of EpochMaxWins throughout the merge pipeline.
Docstring Coverage	✅ Passed	Docstring coverage is 90.09% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/mesh-epoch-max-wins-crdt

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6149e363e0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[gemini-code-assist]

Code Review

This pull request introduces a configurable MergeStrategy framework for the CRDT key-value store, enabling prefix-based conflict resolution policies such as LastWriterWins and EpochMaxWins. The implementation includes updates to the OperationLog for strategy-aware compaction and the addition of record_epoch_insert_metadata to handle epoch-based versioning. Review feedback highlights several performance optimization opportunities, including avoiding Vec allocations during value comparisons by decoding into structs and reducing lock contention by hoisting read locks out of tight loops during log compaction. Additionally, a correction was suggested for tombstone handling in metadata records to ensure consistency.

[claude]

Solid PR — the EpochMaxWins strategy is correctly wired through the CRDT merge, compaction, and operation-log paths. The candidate_wins logic correctly falls through to LWW for Remove operations and non-Insert pairs, and the record_epoch_insert_metadata correctly only suppresses on duplicate entries or newer tombstones (not newer non-tombstone inserts, which is the right call for value-based merge).

Summary: 0 🔴 Important · 1 🟡 Nit · 0 🟣 Pre-existing

The one nit: snapshot_and_truncate was not updated with a _with_strategy variant like compact and append were. No current callers, but it's a public method that would silently use LWW for EpochMaxWins keys.

Tests cover the core epoch-wins-over-stale-count and tombstone-wins-over-stale-insert scenarios well.

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

crates/mesh/src/crdt_kv/crdt.rs (1)

181-191: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Route local insert-like writes through the strategy-aware merge path.

These methods still call record_insert_metadata() and overwrite self.store directly before the EpochMaxWins logic in apply_insert() ever runs. For example, on an rl: key a later local write of encode(5, 100) will replace an existing encode(6, 0) even though merge/compaction would keep epoch 6. The local write path needs to reuse the same strategy-specific resolution as replicated inserts, and it needs a regression test for that case.

Also applies to: 220-230, 260-270, 299-310

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mesh/src/crdt_kv/crdt.rs` around lines 181 - 191, The local insert
path currently calls record_insert_metadata() and directly updates self.store
via self.store.upsert before the strategy-specific merge logic in
apply_insert()/EpochMaxWins runs; change the local insert-like handlers (the
blocks that call record_insert_metadata and self.store.upsert, and then
append_operation) to instead build the same Operation::insert and route it
through the same merge/application path used for replicated ops (e.g., call
apply_insert or the common apply_operation path after append_operation) so the
EpochMaxWins strategy and rl: key semantics are respected; update the code
blocks around the current self.store.upsert usage (and similar blocks at the
other noted ranges) to stop writing the store directly and reuse apply_insert,
and add a regression test that performs a local write encode(5,100) against an
existing encode(6,0) on an rl: key to assert epoch 6 is preserved.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh/src/crdt_kv/merge_strategy.rs`:
- Around line 8-9: The public enum variant MergeStrategy::MaxValueWins must not
be usable until its semantics are implemented; either remove the MaxValueWins
variant from the public MergeStrategy API or add a validation check wherever
MergeStrategy is instantiated/configured to reject MaxValueWins with an explicit
error. Locate the MergeStrategy definition and either make MaxValueWins
non-public/temporary (remove or comment out) or add a guard in the
configuration/constructor path (the code that parses or returns a MergeStrategy)
that returns an Err or validation failure if MergeStrategy::MaxValueWins is
selected; also add a unit test to assert the new validation so callers cannot
silently pick MaxValueWins while apply/compaction still treat it like
LastWriterWins.

In `@crates/mesh/src/crdt_kv/operation.rs`:
- Around line 252-255: snapshot_and_truncate() currently hardcodes
MergeStrategy::LastWriterWins when calling
latest_operations_by_key_with_strategy; change it to look up the per-key
registered merge strategy (the same lookup logic used by
compact_with_strategy()) so each key uses its configured MergeStrategy (e.g.,
EpochMaxWins vs LastWriterWins) during snapshotting and compaction; update the
closure passed to latest_operations_by_key_with_strategy in
snapshot_and_truncate() to retrieve the strategy for the given key instead of
always returning LastWriterWins.

---

Outside diff comments:
In `@crates/mesh/src/crdt_kv/crdt.rs`:
- Around line 181-191: The local insert path currently calls
record_insert_metadata() and directly updates self.store via self.store.upsert
before the strategy-specific merge logic in apply_insert()/EpochMaxWins runs;
change the local insert-like handlers (the blocks that call
record_insert_metadata and self.store.upsert, and then append_operation) to
instead build the same Operation::insert and route it through the same
merge/application path used for replicated ops (e.g., call apply_insert or the
common apply_operation path after append_operation) so the EpochMaxWins strategy
and rl: key semantics are respected; update the code blocks around the current
self.store.upsert usage (and similar blocks at the other noted ranges) to stop
writing the store directly and reuse apply_insert, and add a regression test
that performs a local write encode(5,100) against an existing encode(6,0) on an
rl: key to assert epoch 6 is preserved.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3f208cf6-27bf-43d1-854a-329467fc10b7

📥 Commits

Reviewing files that changed from the base of the PR and between cd332e0 and 6149e36.

📒 Files selected for processing (7)

• crates/mesh/src/crdt_kv/crdt.rs
• crates/mesh/src/crdt_kv/merge_strategy.rs
• crates/mesh/src/crdt_kv/mod.rs
• crates/mesh/src/crdt_kv/operation.rs
• crates/mesh/src/crdt_kv/tests.rs
• crates/mesh/src/kv.rs
• crates/mesh/src/lib.rs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bf16fcaf0d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh/src/crdt_kv/crdt.rs`:
- Around line 623-656: The merge logic currently treats any incoming bytes equal
to the incoming value as a "candidate wins" and rewrites stored epoch metadata
even when the merged bytes equal the existing local value, which can rewind the
stored (timestamp, replica_id); update the decision so we only replace
versions/new_metadata when the merged bytes actually differ from the current
stored bytes or when there is no current value. Concretely, change the
computation of candidate_wins_value and the subsequent check around
current/is_some() so that if current.as_deref() == Some(merged.as_slice()) you
do not clear/replace versions (i.e., preserve existing versions), referencing
the symbols candidate_wins_value, merged, current, versions, new_metadata,
epoch_max_wins::merge, and Self::compact_key_metadata to locate the code to
modify.

In `@crates/mesh/src/crdt_kv/operation.rs`:
- Around line 169-195: candidate_wins currently mixes epoch-based insert
comparisons with timestamp-based insert/remove comparisons causing
non-transitive ordering; change the MergeStrategy::EpochMaxWins branch (function
candidate_wins and the caller latest_operations_by_key_with_strategy) to stop
doing pairwise epoch-vs-timestamp reductions. Instead, for a given key compute
the epoch-max Insert winner across all Insert operations using
epoch_max_wins::winner semantics, separately compute the newest Tombstone/Remove
by (timestamp, replica_id), then compare those two deterministic winners
(epoch-max insert vs newest tombstone) using a single consistent rule to decide
survival; update candidate_wins to call those helpers or delegate to a new
function so the EpochMaxWins path makes a holistic decision rather than falling
back to the (timestamp, replica_id) pairwise comparison.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 4125fe15-96f3-4ed9-9df7-ff4d0d30ad4f

📥 Commits

Reviewing files that changed from the base of the PR and between 6149e36 and bf16fca.

📒 Files selected for processing (6)

• crates/mesh/src/crdt_kv/crdt.rs
• crates/mesh/src/crdt_kv/epoch_max_wins.rs
• crates/mesh/src/crdt_kv/kv_store.rs
• crates/mesh/src/crdt_kv/merge_strategy.rs
• crates/mesh/src/crdt_kv/operation.rs
• crates/mesh/src/crdt_kv/tests.rs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c2b791e87

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8287ddf4bb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh/src/crdt_kv/tests.rs`:
- Around line 552-586: The test function name
test_operation_log_epoch_max_wins_equal_insert_uses_newer_timestamp is
misleading because a tombstone removes the older insert rather than exercising
equal-epoch LWW tie-breaking; rename the test to
test_operation_log_epoch_max_wins_post_tombstone_insert_wins_over_pre_tombstone_equal_epoch
(update the function identifier and any references) to reflect the scenario
being tested (the code paths around OperationLog::append,
OperationLog::snapshot_and_truncate and MergeStrategy::EpochMaxWins remain
unchanged), or alternatively modify the test to remove the tombstone (keeping
the original name) if you want to directly test equal-epoch LWW tie-breaking.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: d46d3ccc-bc31-465f-908e-131fc429edff

📥 Commits

Reviewing files that changed from the base of the PR and between 5c2b791 and 8287ddf.

📒 Files selected for processing (2)

• crates/mesh/src/crdt_kv/operation.rs
• crates/mesh/src/crdt_kv/tests.rs

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh/src/crdt_kv/epoch_max_wins.rs`:
- Around line 342-369: compact_operations currently uses the try-operator on
state_from_insert_value(...) which returns None on a malformed `rl:` insert and
causes the whole compaction to drop the key; change the loop in
compact_operations so that decoding errors from state_from_insert_value are
handled locally (for Insert only) by skipping that operation (optionally
logging) and continuing to merge other operations instead of returning None,
then proceed to merge via current.merge(operation_state) and finally call
state.into_operation(key?) as before; reference the compact_operations function,
the match arm that calls state_from_insert_value, and the merge/into_operation
paths to locate where to replace the `?`-style early-return with a
continue-on-error behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: f94d3fef-6561-4add-8bc2-e32b5760d43a

📥 Commits

Reviewing files that changed from the base of the PR and between 8287ddf and 15e781f.

📒 Files selected for processing (6)

• crates/mesh/src/crdt_kv/crdt.rs
• crates/mesh/src/crdt_kv/epoch_max_wins.rs
• crates/mesh/src/crdt_kv/operation.rs
• crates/mesh/src/crdt_kv/replica.rs
• crates/mesh/src/crdt_kv/tests.rs
• model_gateway/src/mesh/adapters/rate_limit_sync.rs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 15e781f06d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 00dd0ba3cf

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c57f1ec86b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dfb1e6fcad

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

crates/mesh/src/crdt_kv/kv_store.rs (1)

20-21: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Stale doc comment references removed method.

The comment mentions upsert but this method was removed. Update to reflect current behavior.

📝 Suggested fix

-    /// Monotonically increasing counter, bumped on every insert/remove/upsert.
+    /// Monotonically increasing counter, bumped on every insert/remove.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mesh/src/crdt_kv/kv_store.rs` around lines 20 - 21, Update the stale
doc comment for the generation field (generation: Arc<AtomicU64>) to remove the
reference to the removed upsert method and reflect current mutation operations
that bump the counter (e.g., "bumped on every insert/remove/update" or list the
exact methods that now change generation), so the comment accurately describes
when generation is incremented.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@crates/mesh/src/crdt_kv/kv_store.rs`:
- Around line 20-21: Update the stale doc comment for the generation field
(generation: Arc<AtomicU64>) to remove the reference to the removed upsert
method and reflect current mutation operations that bump the counter (e.g.,
"bumped on every insert/remove/update" or list the exact methods that now change
generation), so the comment accurately describes when generation is incremented.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 57da17a6-ffbf-474f-a82d-d58250b790ed

📥 Commits

Reviewing files that changed from the base of the PR and between 2c32ef4 and cf38daf.

📒 Files selected for processing (10)

• crates/mesh/src/crdt_kv/crdt.rs
• crates/mesh/src/crdt_kv/epoch_max_wins.rs
• crates/mesh/src/crdt_kv/kv_store.rs
• crates/mesh/src/crdt_kv/merge_strategy.rs
• crates/mesh/src/crdt_kv/mod.rs
• crates/mesh/src/crdt_kv/operation.rs
• crates/mesh/src/crdt_kv/tests.rs
• crates/mesh/src/kv.rs
• crates/mesh/src/lib.rs
• model_gateway/src/mesh/adapters/rate_limit_sync.rs

[CatherineSue]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".