Skip to content

Close lost-cancel window: match under the DashMap entry lock (#81)#100

Merged
joaquinbejar merged 2 commits into
mainfrom
issue-81-lost-cancel-fix
Jun 23, 2026
Merged

Close lost-cancel window: match under the DashMap entry lock (#81)#100
joaquinbejar merged 2 commits into
mainfrom
issue-81-lost-cancel-fix

Conversation

@joaquinbejar

Copy link
Copy Markdown
Owner

Summary

Closes the lost-cancel race: match_order consumed a maker via pop_entry
(index.pop_front + orders.remove(id)) then re-inserted the partial residual.
A concurrent cancel(id) of the order being matched hit orders.remove(id)
None → silently no-op'd (no counter decrement); the matcher then re-rested the
residual, so the user's cancel was lost. (Finding #3, the torn snapshot, was
already fixed by #62.)

Changes

  • New OrderQueue::match_front: keeps the maker resident in the DashMap and
    runs the match decision + commit under the per-entry shard lock via entry(id),
    committing a FrontAction:
  • A concurrent cancel/remove serializes on the same shard lock, so it either
    fully wins (removes original + decrements full counters) or fully loses (match
    commits, cancel removes the residual) — never lost, never double-counted.
    ReplaceAtTail re-sequences in place rather than remove+push, so the guarantee
    holds for replenishment too. Stale index→id entries self-heal on the next
    sweep's Vacant branch (next_seq is monotonic, so no live entry is ever
    mistaken).

Verification

  • loom protocol model (tests/loom/cancel_match.rs, #![cfg(loom)]): models
    the per-entry-lock protocol (loom can't instrument dashmap/skiplist, documented)
    and asserts conservation, counter↔queue lockstep, and no-lost-cancel. Verified
    non-vacuous (the old remove-then-reinsert shape makes loom find the bug).
    Run: RUSTFLAGS="--cfg loom" cargo test --test loom_cancel_match.
  • std-thread stress tests on the real impl (Barrier, deterministic, no
    sleep): match_ordercancel(same id) (2000 iters; ~67% hit the exact
    window) and ∥ cancel(different ids).
  • concurrency-auditor: SOUND — linearizable, counters lockstep across all 4
    actions, stale-index self-heal idempotent, no new race, FIFO + FOK dry-run
    preserved.

Scope

match_ordercancel is now safe (rustdoc updated). Two concurrent matchers
on one level still require caller serialization (not expanded here). The
pre-existing update_order quantity-increase remove-then-push window is
unchanged (out of scope; tracked separately).

Testing

cargo test: 408 (lib) + 1 (integration) + 9 (doctests), 0 failed; loom: 1
passed under --cfg loom. cargo clippy --all-targets --all-features -- -D warnings, cargo fmt --all --check, cargo build --release: clean. loom is a
dev-dependency under cfg(loom) only (no runtime dep).

Checklist

Closes #81

The match loop used pop_entry (index.pop_front + orders.remove) then a later
reinsert, so a concurrent cancel(id) of the order being matched hit
orders.remove -> None, silently no-op'd (no counter decrement), and the matcher
re-rested the residual: the cancel was lost.

New OrderQueue::match_front keeps the maker resident in the DashMap and runs the
decision + commit under the per-entry shard lock (FrontAction =
Remove/KeepInPlace/ReplaceAtTail/SetAside). A concurrent cancel serializes on the
same shard lock, so it fully wins (removes + decrements) or fully loses (sees the
residual) — never lost, never double-counted. ReplaceAtTail re-sequences in place
rather than remove+push, so the guarantee holds for replenishment too. Stale
index entries self-heal on the next sweep's Vacant branch.

Verified by a loom protocol model (tests/loom/, non-vacuous) and std-thread
stress tests; reviewed SOUND by concurrency-auditor. Preserves #39 price-time
priority, #62 snapshot consistency, #64 in-place update, #65 taker-TIF + FOK
dry-run + zero-visible guard. match∥match still requires caller serialization.
@joaquinbejar joaquinbejar added bug Something isn't working price-level Engine: level/order_queue/snapshot/statistics + execution results labels Jun 23, 2026
@joaquinbejar joaquinbejar self-assigned this Jun 23, 2026
@joaquinbejar joaquinbejar requested a review from Copilot June 23, 2026 23:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR closes the lost-cancel race between match_order and cancel on the same maker by introducing an in-place front-matching protocol: the matcher keeps the maker resident in the DashMap and commits the match outcome while holding the maker’s per-entry lock, so a concurrent cancel serializes on the same lock and cannot “slip through” a pop→reinsert gap.

Changes:

  • Add OrderQueue::match_front plus FrontAction/FrontOutcome to atomically decide+commit front-maker mutations under the DashMap entry lock.
  • Refactor PriceLevel::match_order to use match_front (supporting remove / keep-in-place / replace-at-tail / set-aside) while keeping counter updates outside the entry lock.
  • Add verification via a loom protocol model test (cfg(loom)) and real-implementation std::thread stress tests; wire loom as a cfg-gated dev dependency.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/loom/cancel_match.rs Adds a loom model that exhaustively checks the cancel-vs-match protocol linearization under cfg(loom).
src/price_level/tests/level.rs Adds real-implementation concurrent stress tests for match_order vs cancel (same id and different ids).
src/price_level/order_queue.rs Introduces match_front and related enums to keep makers resident and commit under the per-entry lock.
src/price_level/level.rs Refactors match_order to drive matching via match_front and updates concurrency rustdoc accordingly.
Cargo.toml Registers cfg(loom) for linting and adds loom as a cfg-gated dev dependency + explicit loom test target.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/price_level/order_queue.rs
Comment thread src/price_level/order_queue.rs Outdated
Comment thread src/price_level/level.rs
@codecov-commenter

codecov-commenter commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.64045% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/price_level/level.rs 88.70% 7 Missing ⚠️
src/price_level/order_queue.rs 85.18% 4 Missing ⚠️

❌ Your project status has failed because the head coverage (73.70%) is below the target coverage (75.00%). You can increase the head coverage or adjust the target coverage.

Files with missing lines Coverage Δ
src/price_level/order_queue.rs 92.68% <85.18%> (-2.11%) ⬇️
src/price_level/level.rs 92.51% <88.70%> (+0.02%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ark diagnostics

- match_front's decide closure now borrows &OrderType instead of cloning the Arc
  (no extra refcount on the match path); the borrow ends before the get_mut/remove
  commit, protocol unchanged.
- set_aside Vec<u64> -> HashSet<u64>: O(1) membership, removes the O(n^2)
  no-progress scan.
- SetAside now carries maker_id + seq so the no-progress tracing::warn logs them
  as structured fields, outside the per-entry lock.

No semantic change; linearization, counters, FOK dry-run, loom + stress tests
unchanged.
EOF
)
@joaquinbejar

Copy link
Copy Markdown
Owner Author

Thanks, all three addressed (no semantic change): removed the hot-path Arc clone via a borrow, set_aside is now a HashSet (no O(n^2)), and the parked maker id/seq are logged as structured fields outside the lock. loom + stress tests still green.

@joaquinbejar joaquinbejar merged commit 6d33fdc into main Jun 23, 2026
13 checks passed
@joaquinbejar joaquinbejar deleted the issue-81-lost-cancel-fix branch June 23, 2026 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working price-level Engine: level/order_queue/snapshot/statistics + execution results

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent match_order + cancel/match on the same level: lost cancel & torn snapshot (pre-existing lock-free gaps)

3 participants