Close lost-cancel window: match under the DashMap entry lock (#81)#100
Conversation
The match loop used pop_entry (index.pop_front + orders.remove) then a later reinsert, so a concurrent cancel(id) of the order being matched hit orders.remove -> None, silently no-op'd (no counter decrement), and the matcher re-rested the residual: the cancel was lost. New OrderQueue::match_front keeps the maker resident in the DashMap and runs the decision + commit under the per-entry shard lock (FrontAction = Remove/KeepInPlace/ReplaceAtTail/SetAside). A concurrent cancel serializes on the same shard lock, so it fully wins (removes + decrements) or fully loses (sees the residual) — never lost, never double-counted. ReplaceAtTail re-sequences in place rather than remove+push, so the guarantee holds for replenishment too. Stale index entries self-heal on the next sweep's Vacant branch. Verified by a loom protocol model (tests/loom/, non-vacuous) and std-thread stress tests; reviewed SOUND by concurrency-auditor. Preserves #39 price-time priority, #62 snapshot consistency, #64 in-place update, #65 taker-TIF + FOK dry-run + zero-visible guard. match∥match still requires caller serialization.
There was a problem hiding this comment.
Pull request overview
This PR closes the lost-cancel race between match_order and cancel on the same maker by introducing an in-place front-matching protocol: the matcher keeps the maker resident in the DashMap and commits the match outcome while holding the maker’s per-entry lock, so a concurrent cancel serializes on the same lock and cannot “slip through” a pop→reinsert gap.
Changes:
- Add
OrderQueue::match_frontplusFrontAction/FrontOutcometo atomically decide+commit front-maker mutations under the DashMap entry lock. - Refactor
PriceLevel::match_orderto usematch_front(supporting remove / keep-in-place / replace-at-tail / set-aside) while keeping counter updates outside the entry lock. - Add verification via a loom protocol model test (
cfg(loom)) and real-implementationstd::threadstress tests; wire loom as a cfg-gated dev dependency.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/loom/cancel_match.rs | Adds a loom model that exhaustively checks the cancel-vs-match protocol linearization under cfg(loom). |
| src/price_level/tests/level.rs | Adds real-implementation concurrent stress tests for match_order vs cancel (same id and different ids). |
| src/price_level/order_queue.rs | Introduces match_front and related enums to keep makers resident and commit under the per-entry lock. |
| src/price_level/level.rs | Refactors match_order to drive matching via match_front and updates concurrency rustdoc accordingly. |
| Cargo.toml | Registers cfg(loom) for linting and adds loom as a cfg-gated dev dependency + explicit loom test target. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is
❌ Your project status has failed because the head coverage (73.70%) is below the target coverage (75.00%). You can increase the head coverage or adjust the target coverage.
🚀 New features to boost your workflow:
|
…ark diagnostics - match_front's decide closure now borrows &OrderType instead of cloning the Arc (no extra refcount on the match path); the borrow ends before the get_mut/remove commit, protocol unchanged. - set_aside Vec<u64> -> HashSet<u64>: O(1) membership, removes the O(n^2) no-progress scan. - SetAside now carries maker_id + seq so the no-progress tracing::warn logs them as structured fields, outside the per-entry lock. No semantic change; linearization, counters, FOK dry-run, loom + stress tests unchanged. EOF )
|
Thanks, all three addressed (no semantic change): removed the hot-path Arc clone via a borrow, set_aside is now a HashSet (no O(n^2)), and the parked maker id/seq are logged as structured fields outside the lock. loom + stress tests still green. |
Summary
Closes the lost-cancel race:
match_orderconsumed a maker viapop_entry(
index.pop_front+orders.remove(id)) then re-inserted the partial residual.A concurrent
cancel(id)of the order being matched hitorders.remove(id)→None→ silently no-op'd (no counter decrement); the matcher then re-rested theresidual, so the user's cancel was lost. (Finding #3, the torn snapshot, was
already fixed by #62.)
Changes
OrderQueue::match_front: keeps the maker resident in theDashMapandruns the match decision + commit under the per-entry shard lock via
entry(id),committing a
FrontAction:Remove(full consume),KeepInPlace(partial, same seq — keeps price-timepriority),
ReplaceAtTail(replenishment: swap value+seq in place, re-keyindex — never leaves
orders),SetAside(no-progress park, preserves theDecide and document taker TIF / PostOnly / MarketToLimit responsibility at the level #65 zero-visible guard).
cancel/removeserializes on the same shard lock, so it eitherfully wins (removes original + decrements full counters) or fully loses (match
commits, cancel removes the residual) — never lost, never double-counted.
ReplaceAtTailre-sequences in place rather than remove+push, so the guaranteeholds for replenishment too. Stale
index→identries self-heal on the nextsweep's
Vacantbranch (next_seqis monotonic, so no live entry is evermistaken).
Verification
tests/loom/cancel_match.rs,#![cfg(loom)]): modelsthe per-entry-lock protocol (loom can't instrument dashmap/skiplist, documented)
and asserts conservation, counter↔queue lockstep, and no-lost-cancel. Verified
non-vacuous (the old remove-then-reinsert shape makes loom find the bug).
Run:
RUSTFLAGS="--cfg loom" cargo test --test loom_cancel_match.Barrier, deterministic, nosleep):match_order∥cancel(same id)(2000 iters; ~67% hit the exactwindow) and ∥
cancel(different ids).concurrency-auditor: SOUND — linearizable, counters lockstep across all 4actions, stale-index self-heal idempotent, no new race, FIFO + FOK dry-run
preserved.
Scope
match_order∥cancelis now safe (rustdoc updated). Two concurrent matcherson one level still require caller serialization (not expanded here). The
pre-existing
update_orderquantity-increase remove-then-push window isunchanged (out of scope; tracked separately).
Testing
cargo test: 408 (lib) + 1 (integration) + 9 (doctests), 0 failed; loom: 1passed under
--cfg loom.cargo clippy --all-targets --all-features -- -D warnings,cargo fmt --all --check,cargo build --release: clean.loomis adev-dependency under
cfg(loom)only (no runtime dep).Checklist
loomdev-only (cfg(loom)); no new runtime deps; no production.unwrap()Closes #81