Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8ee1e75
fix(mesh): wire EpochMaxWins into CRDT merge
CatherineSue May 8, 2026
cd81842
fix(mesh): apply EpochMaxWins to local CRDT writes
CatherineSue May 9, 2026
4baebd9
fix(mesh): remove unimplemented MaxValueWins strategy
CatherineSue May 9, 2026
9b20166
fix(mesh): make operation-log snapshots strategy-aware
CatherineSue May 9, 2026
d82a302
fix(mesh): avoid EpochMaxWins compaction allocation
CatherineSue May 9, 2026
0b77522
fix(mesh): snapshot CRDT strategies for compaction
CatherineSue May 9, 2026
3435ec5
fix(mesh): snapshot CRDT strategies for append
CatherineSue May 9, 2026
beace98
fix(mesh): preserve EpochMaxWins metadata on equal values
CatherineSue May 12, 2026
c2092a7
fix(mesh): make EpochMaxWins log compaction deterministic
CatherineSue May 12, 2026
74bc1a6
fix(mesh): keep post-tombstone EpochMaxWins inserts
CatherineSue May 12, 2026
c17cf0f
fix(mesh): model rate-limit shards as CRDT state
CatherineSue May 12, 2026
936024b
refactor(mesh): simplify rate-limit shard encoding
CatherineSue May 12, 2026
a95df13
fix(mesh): skip malformed rate-limit compaction inserts
CatherineSue May 12, 2026
2af891a
fix(mesh): decode raw rate-limit write payloads
CatherineSue May 12, 2026
269c066
test(mesh): clarify rate-limit tombstone test name
CatherineSue May 12, 2026
9e26e0a
refactor(mesh): drop public merge_epoch_max_wins export
CatherineSue May 13, 2026
f6085e9
test(mesh): exercise snapshot-only tombstone propagation for EpochMax…
CatherineSue May 13, 2026
a5d017f
chore(mesh): drop unused crdt_kv re-exports after rebase
CatherineSue May 22, 2026
83cfe17
fix(mesh): apply EpochMaxWins removes per-point against shard frontier
CatherineSue May 23, 2026
5071254
fix(mesh): merge same-op-id EpochMaxWins payloads across snapshots
CatherineSue May 23, 2026
5ab5ba5
fix(mesh): record remote tombstones for never-seen CRDT keys
CatherineSue May 24, 2026
f9b2136
refactor(mesh): drop LWW-default OperationLog::merge/compact
CatherineSue May 24, 2026
d4310b0
chore(mesh): tighten MergeStrategy ergonomics and apply_tombstone
CatherineSue May 24, 2026
73c665b
fix(mesh): cap RateLimitShard bincode decode + narrow MergeStrategy doc
CatherineSue May 24, 2026
cf38daf
fix(mesh): preserve tombstone created_at on dominated EpochMaxWins re…
CatherineSue May 24, 2026
6274a61
docs(mesh): drop stale `/upsert` references in generation counter docs
CatherineSue May 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
403 changes: 360 additions & 43 deletions crates/mesh/src/crdt_kv/crdt.rs

Large diffs are not rendered by default.

608 changes: 526 additions & 82 deletions crates/mesh/src/crdt_kv/epoch_max_wins.rs

Large diffs are not rendered by default.

24 changes: 2 additions & 22 deletions crates/mesh/src/crdt_kv/kv_store.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ use std::sync::{
Arc,
};

use dashmap::{mapref::entry::Entry, DashMap};
use dashmap::DashMap;

// ============================================================================
// High-Performance In-Memory KV Storage - Concurrent-Safe Implementation Based on DashMap
Expand All @@ -17,7 +17,7 @@ use dashmap::{mapref::entry::Entry, DashMap};
#[derive(Debug, Clone)]
pub struct KvStore {
store: Arc<DashMap<String, Vec<u8>>>,
/// Monotonically increasing counter, bumped on every insert/remove/upsert.
/// Monotonically increasing counter, bumped on every insert/remove.
generation: Arc<AtomicU64>,
}

Expand All @@ -41,26 +41,6 @@ impl KvStore {
self.store.insert(key, value)
}

/// Atomically compute and update a key in a single DashMap entry operation.
pub fn upsert<F>(&self, key: String, updater: F) -> Vec<u8>
where
F: FnOnce(Option<&[u8]>) -> Vec<u8>,
{
self.generation.fetch_add(1, Ordering::Release);
match self.store.entry(key) {
Entry::Occupied(mut entry) => {
let new_value = updater(Some(entry.get().as_slice()));
entry.get_mut().clone_from(&new_value);
new_value
}
Entry::Vacant(entry) => {
let new_value = updater(None);
entry.insert(new_value.clone());
new_value
}
}
}

/// Get value by key
pub fn get(&self, key: &str) -> Option<Vec<u8>> {
self.store.get(key).map(|v| v.value().clone())
Expand Down
17 changes: 17 additions & 0 deletions crates/mesh/src/crdt_kv/merge_strategy.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
/// Merge strategy for CRDT namespaces. Determines how conflicts are resolved
/// when two nodes write the same key concurrently.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MergeStrategy {
/// Higher (version, replica_id) wins. Used for worker:*, policy:*, config:*.
LastWriterWins,
/// Compare epochs first, then max within same epoch.
///
/// The raw write payload at the put boundary MUST be exactly 16 bytes:
/// epoch (u64 big-endian) followed by count (i64 big-endian). The CRDT
/// normalizes stored and replicated values into a richer
/// `RateLimitShard` form internally (live-point frontier plus an
/// embedded tombstone boundary), so values observed by `get`, gossip
/// peers, and subscribers after normalization are larger than 16
/// bytes.
EpochMaxWins,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
4 changes: 3 additions & 1 deletion crates/mesh/src/crdt_kv/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,14 @@
mod crdt;
mod epoch_max_wins;
mod kv_store;
mod merge_strategy;
mod operation;
mod replica;

// Export core types
pub use crdt::CrdtOrMap;
pub use epoch_max_wins::{decode, encode, merge, EpochCount, EPOCH_MAX_WINS_ENCODED_LEN};
pub use epoch_max_wins::{decode, encode, EpochCount, EPOCH_MAX_WINS_ENCODED_LEN};
pub use merge_strategy::MergeStrategy;
pub use operation::OperationLog;

#[cfg(test)]
Expand Down
130 changes: 94 additions & 36 deletions crates/mesh/src/crdt_kv/operation.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
use std::collections::{HashMap, HashSet};
use std::collections::HashMap;

use serde::{Deserialize, Serialize};

use super::replica::ReplicaId;
use super::{epoch_max_wins, merge_strategy::MergeStrategy, replica::ReplicaId};

// ============================================================================
// Operation Type Definition - Atomic Unit of State Change
Expand Down Expand Up @@ -112,9 +112,16 @@ impl OperationLog {
/// new appends, not on every append. If compaction doesn't reduce below
/// threshold (very high key cardinality), the oldest entries are truncated.
pub fn append(&mut self, operation: Operation) {
self.append_with_strategy(operation, |_| MergeStrategy::LastWriterWins);
}

pub(super) fn append_with_strategy<F>(&mut self, operation: Operation, strategy_for_key: F)
where
F: Fn(&str) -> MergeStrategy,
{
self.operations.push(operation);
if self.operations.len() > Self::AUTO_COMPACT_THRESHOLD {
self.compact();
self.compact_with_strategy(strategy_for_key);
// If still over threshold after dedup (extremely high key cardinality
// >10K unique keys), truncate oldest entries. This drops state for the
// oldest keys, which will be re-synced from peers on the next merge.
Expand Down Expand Up @@ -159,34 +166,59 @@ impl OperationLog {
self.operations.is_empty()
}

fn latest_operations_by_key(&self) -> HashMap<String, Operation> {
let mut latest_by_key: HashMap<String, Operation> = HashMap::new();
fn latest_lww_operation<'a, I>(operations: I) -> Option<&'a Operation>
where
I: IntoIterator<Item = &'a Operation>,
{
operations
.into_iter()
.max_by_key(|operation| (operation.timestamp(), operation.replica_id()))
}

fn latest_epoch_max_wins_operation<'a>(
operations: impl IntoIterator<Item = &'a Operation>,
) -> Option<Operation> {
epoch_max_wins::compact_operations(operations)
}

fn latest_operations_by_key_with_strategy<F>(
&self,
strategy_for_key: F,
) -> HashMap<String, Operation>
where
F: Fn(&str) -> MergeStrategy,
{
let mut operations_by_key: HashMap<String, Vec<&Operation>> = HashMap::new();

for operation in &self.operations {
let key = operation.key().to_string();
match latest_by_key.get(&key) {
Some(current)
if (current.timestamp(), current.replica_id())
>= (operation.timestamp(), operation.replica_id()) => {}
_ => {
latest_by_key.insert(key, operation.clone());
}
}
operations_by_key
.entry(operation.key().to_string())
.or_default()
.push(operation);
}

latest_by_key
operations_by_key
.into_iter()
.filter_map(|(key, operations)| {
let latest = match strategy_for_key(&key) {
MergeStrategy::LastWriterWins => {
Self::latest_lww_operation(operations).cloned()
}
MergeStrategy::EpochMaxWins => {
Self::latest_epoch_max_wins_operation(operations)
}
}?;
Some((key, latest))
})
.collect()
}

/// Keep only latest operation per key to bound log growth.
///
/// This uses `latest_operations_by_key` LWW tie-breaking by `(timestamp, ReplicaId)`.
/// As a result, concurrent operations may be compacted away deterministically, so
/// `compact()` + `merge()` can be non-idempotent in raw log contents even though
/// `apply_operation` and `operation_id` guards keep state semantics safe.
/// Stronger concurrency retention would require vector-clock/version-vector metadata.
pub fn compact(&mut self) {
pub(super) fn compact_with_strategy<F>(&mut self, strategy_for_key: F)
where
F: Fn(&str) -> MergeStrategy,
{
self.operations = self
.latest_operations_by_key()
.latest_operations_by_key_with_strategy(strategy_for_key)
.into_values()
.collect::<Vec<_>>();
self.operations
Expand All @@ -199,9 +231,12 @@ impl OperationLog {
.retain(|operation| operation.timestamp() > watermark);
}

/// Build a latest-state snapshot and clear the operation log.
pub fn snapshot_and_truncate(&mut self) -> HashMap<String, Operation> {
let snapshot = self.latest_operations_by_key();
/// Build a latest-state snapshot with the configured merge strategy and clear the operation log.
pub fn snapshot_and_truncate<F>(&mut self, strategy_for_key: F) -> HashMap<String, Operation>
where
F: Fn(&str) -> MergeStrategy,
{
let snapshot = self.latest_operations_by_key_with_strategy(strategy_for_key);
self.operations.clear();
snapshot
}
Expand Down Expand Up @@ -233,20 +268,43 @@ impl OperationLog {
}
}

/// Merge another operation log.
///
/// INVARIANT: `Operation::operation_id()` (`ReplicaId`, `timestamp`) is unique per operation
/// because each replica's `LamportClock::tick()` is monotonic and never repeats a timestamp.
pub fn merge(&mut self, other: &OperationLog) {
let mut seen_ids: HashSet<(ReplicaId, u64)> = self
/// Per-key strategy-aware merge. For `EpochMaxWins` keys, an incoming
/// operation that collides on `(replica_id, timestamp)` with an existing
/// local op is folded via `epoch_max_wins::compact_operations` so a
/// compacted payload (carrying an embedded tombstone_version or richer
/// frontier) replaces the older raw payload at the same op id. LWW keys
/// dedup by op id.
pub(super) fn merge_with_strategy<F>(&mut self, other: &OperationLog, strategy_for_key: F)
where
F: Fn(&str) -> MergeStrategy,
{
let mut local_index: HashMap<(ReplicaId, u64), usize> = self
.operations
.iter()
.map(Operation::operation_id)
.enumerate()
.map(|(idx, op)| (op.operation_id(), idx))
.collect();

for operation in &other.operations {
if seen_ids.insert(operation.operation_id()) {
self.operations.push(operation.clone());
let op_id = operation.operation_id();
match local_index.get(&op_id).copied() {
None => {
local_index.insert(op_id, self.operations.len());
self.operations.push(operation.clone());
}
Some(local_idx) => {
if matches!(
strategy_for_key(operation.key()),
MergeStrategy::EpochMaxWins
) {
let local_op = self.operations[local_idx].clone();
if let Some(folded) =
epoch_max_wins::compact_operations([&local_op, operation])
{
self.operations[local_idx] = folded;
}
Comment on lines +281 to +305
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot May 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Include key in the merge collision index.

local_index is keyed only by (replica_id, timestamp). If one replica produces two operations for different keys at the same timestamp, this merge path aliases them as the same op: LWW drops the incoming update, and EpochMaxWins can fold bytes from one key into another.

🛠️ Localized fix
-        let mut local_index: HashMap<(ReplicaId, u64), usize> = self
+        let mut local_index: HashMap<(String, ReplicaId, u64), usize> = self
             .operations
             .iter()
             .enumerate()
-            .map(|(idx, op)| (op.operation_id(), idx))
+            .map(|(idx, op)| {
+                (
+                    (op.key().to_owned(), op.replica_id(), op.timestamp()),
+                    idx,
+                )
+            })
             .collect();

         for operation in &other.operations {
-            let op_id = operation.operation_id();
+            let op_id = (
+                operation.key().to_owned(),
+                operation.replica_id(),
+                operation.timestamp(),
+            );
             match local_index.get(&op_id).copied() {

Based on learnings, use INVARIANT: to document assumptions in safe code if global (replica_id, timestamp) uniqueness is intentional.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mesh/src/crdt_kv/operation.rs` around lines 281 - 305, local_index is
currently using only (ReplicaId, u64) from operation.operation_id(), which can
alias different keys produced at the same timestamp; fix by including the
operation key in the index key (e.g., use a tuple of (ReplicaId, timestamp,
key.clone()) when building and querying local_index in the merge routine that
iterates other.operations) so collisions are per-key; update all uses in this
function that call local_index.get/insert to use the new composite key, and if
the original (replica_id, timestamp) uniqueness was an intended invariant add an
INVARIANT comment near operation_id()/the merge to document that assumption
rather than relying on it implicitly.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining for this PR. The repro requires one replica producing two operations at the same (replica_id, timestamp) for different keys. That can't happen today: LamportClock::tick() is fetch_add(1, AcqRel), so each local op from a given replica gets a strictly unique timestamp. The invariant is one line of code, not a tribal-knowledge convention. If someone ever breaks Lamport monotonicity, the blast radius is far wider than this one HashMap — seen_operations in CrdtOrMap::merge, the dedup in apply_operation, every op_id-keyed structure breaks. A defensive layer here wouldn't actually protect against that class of regression; the right protection is to not change tick() without auditing all op-id consumers. Keying by (String, ReplicaId, u64) would also pull an extra String clone into the index (each entry already owns its key in the Operation itself) and slow down hashing. Open to revisiting if the invariant changes.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CatherineSue, understood — the LamportClock::tick() fetch_add(1, AcqRel) guarantee makes (replica_id, timestamp) a valid unique key, and you're right that any break there would affect every op-id–keyed structure in the mesh stack, not just this one. I'll drop the suggestion.

If you ever want to make the assumption explicit, a one-liner INVARIANT: comment near operation_id() or merge_with_strategy (e.g. "INVARIANT: each replica's LamportClock::tick() is strictly monotonic, so (replica_id, timestamp) is globally unique per operation") would fit the repo's annotation convention — but that's entirely up to you.

🐇 (a little deflated, but fair enough)


✏️ Learnings added
Learnt from: CatherineSue
Repo: lightseekorg/smg PR: 1469
File: crates/mesh/src/crdt_kv/operation.rs:281-305
Timestamp: 2026-05-24T16:58:08.546Z
Learning: In lightseekorg/smg's mesh crate (`crates/mesh/src/crdt_kv/operation.rs`), the `(ReplicaId, u64 timestamp)` pair from `Operation::operation_id()` is intentionally sufficient as a unique operation ID. `LamportClock::tick()` uses `fetch_add(1, AcqRel)`, guaranteeing strictly monotonic, non-repeating timestamps per replica. Do NOT suggest widening the merge/dedup index key to include the operation key (e.g., `(String, ReplicaId, u64)`), as the invariant holds and the extra String clone adds unnecessary allocation. If the invariant is ever broken, every op-id–keyed structure in the mesh stack (e.g., `seen_operations` in `CrdtOrMap::merge`, dedup in `apply_operation`) would be affected, not just this one HashMap.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


🧠 Learnings used
Learnt from: XinyueZhang369
Repo: lightseekorg/smg PR: 399
File: protocols/src/interactions.rs:505-509
Timestamp: 2026-02-19T03:08:50.192Z
Learning: In code reviews for Rust projects using the validator crate (v0.20.0), ensure that custom validation functions for numeric primitive types (e.g., f32, i32, u32, i16, etc.) accept the value by value, not by reference. Example: fn validate(value: f32) { ... }. The validator derive macro has a hardcoded list of numeric types that are passed by value, while all other types are passed by reference. Apply this guideline whenever validating numeric fields to align with the derive macro behavior.

Learnt from: slin1237
Repo: lightseekorg/smg PR: 489
File: model_gateway/src/core/token_bucket.rs:58-63
Timestamp: 2026-02-21T02:30:51.443Z
Learning: For lint-only/Clippy enforcement PRs in this repository, avoid introducing behavioral changes (e.g., new input validation or logic changes). Treat such PRs as non-functional changes and plan a separate follow-up issue/PR for hardening or behavior changes. This applies broadly to Rust files across the repo; during review, focus on lint/style corrections and clearly note any intentional exceptions. 

Learnt from: slin1237
Repo: lightseekorg/smg PR: 489
File: protocols/src/responses.rs:928-931
Timestamp: 2026-02-21T02:36:00.882Z
Learning: In Rust code across the repository, use the marker INVARIANT: to document assumptions in safe code. Reserve SAFETY: for explaining why unsafe blocks are sound. This improves clarity of invariants and safety reasoning. Example reference: protocols/src/responses.rs near validate_tool_choice_with_tools().

Learnt from: slin1237
Repo: lightseekorg/smg PR: 489
File: mesh/src/sync.rs:83-83
Timestamp: 2026-02-21T02:37:01.416Z
Learning: General Rust formatting rule: format! with implicit captures only supports simple identifiers, not full expressions like {state.model_id}. For cases where you want to interpolate a field or expression, bind the value first and interpolate the binding, e.g., let model_id = &state.model_id; and then use format!("policy:{}", model_id). In the specific file mesh/src/sync.rs, prefer format!("policy:{}", state.model_id) or bind to a local variable if you need named interpolation, to keep clarity and avoid unintended captures.

Learnt from: zhaowenzi
Repo: lightseekorg/smg PR: 807
File: model_gateway/src/middleware.rs:61-81
Timestamp: 2026-03-18T21:32:00.041Z
Learning: In Rust code using the http crate, HeaderMap::get() is effectively case-insensitive because HeaderName normalizes keys to lowercase on insertion and lookup. Do not require or perform explicit .to_lowercase() before HeaderMap::get() calls. Mark as not a concern for case-sensitivity in lookups; only consider normalization when inserting or comparing via HeaderName, not in lookups.

Learnt from: key4ng
Repo: lightseekorg/smg PR: 867
File: tui/src/app.rs:798-813
Timestamp: 2026-03-22T20:13:55.778Z
Learning: In this repo (lightseekorg/smg), treat the workspace `Cargo.toml`’s `package.rust-version` (MSRV) as the source of truth (e.g., `rust-version = "1.85"`). When reviewing Rust changes, do not flag usage of Rust language/library features that were stabilized on or before the MSRV (e.g., `Option::is_none_or`, stabilized in 1.82, is compatible with an MSRV of 1.85). Always verify the MSRV from the workspace `Cargo.toml` rather than relying on issue templates.

Learnt from: CatherineSue
Repo: lightseekorg/smg PR: 1254
File: crates/mesh/src/ping_server.rs:0-0
Timestamp: 2026-04-20T19:26:57.560Z
Learning: For lightseekorg/smg’s mesh crate, keep `next_generation()` as a `LazyLock<AtomicU64>` seeded once (not per-call) and incremented atomically to produce unique, monotonically increasing generation tags across all call sites. When reviewing changes to the client/server streaming batch emission code (e.g., `controller.rs` and `ping_server.rs`), ensure `next_generation()` is called inside the per-value loop so every `chunk_value(...)` gets a distinct generation. Do not replace this approach with inline `SystemTime::now()` calls; the atomic counter is preferred to avoid wall-clock resolution/consistency issues and to guarantee monotonicity.

}
}
}
}
}
Expand Down
Loading
Loading