From 0439d3440aaa2e5b4ebe4dd1213361fd7ca7b4af Mon Sep 17 00:00:00 2001 From: brianmmaina Date: Sun, 16 Aug 2026 12:22:44 -0700 Subject: [PATCH] readme: cut it down 368 lines to 315. The detail that got cut is in docs/ already, so the readme was repeating itself. Merged the benchmark commentary into two points instead of five, folded the crash-safety walkthrough into the paragraph that explains why the independent verifier is the part that matters, and collapsed the research section now that FARMER_2005.md carries the caveats. Test count was stale, 186 to 191. Gitignore __pycache__ and pdfs; the paper is not mine to redistribute. --- .gitignore | 7 ++ README.md | 281 ++++++++++++++++++++++------------------------------- 2 files changed, 121 insertions(+), 167 deletions(-) diff --git a/.gitignore b/.gitignore index cd242f8..d41b2f5 100644 --- a/.gitignore +++ b/.gitignore @@ -22,3 +22,10 @@ data/lobster/**/*.7z OME_*_PLAN.md *_PLAN.md notes/ + +# python bytecode +__pycache__/ +*.pyc + +# papers are not mine to redistribute +*.pdf diff --git a/README.md b/README.md index fbad8c6..3e5dd03 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ # Order Matching Machine A limit order book and matching engine in **C++17**, wrapped in a **networked -order gateway** with crash recovery. +order gateway** with crash recovery — then used to replicate a +[2005 PNAS paper](https://arxiv.org/abs/cond-mat/0309233) on market +microstructure. Clients connect over TCP, submit orders, and receive acks, fills and market data. A single matching thread owns the book; a write-ahead log makes the state @@ -12,7 +14,7 @@ survive being killed. No dependencies beyond GoogleTest. reimplementation of the book - Matching validated by differential replay against **269K NASDAQ LOBSTER messages** -- **186 tests**, green on clang and gcc, under ASan/UBSan and ThreadSanitizer +- **191 tests**, green on clang and gcc, under ASan/UBSan and ThreadSanitizer > **First C++ project.** I'm still learning the idioms and the tradeoffs below; > this is a learning sandbox, not battle-tested infrastructure. Feedback welcome. @@ -30,11 +32,10 @@ clients --TCP--> [network thread] --SPSC--> [matching thread] --> WAL ``` **The book is single-writer.** One thread owns every mutation, so there is no -lock anywhere near it. Concurrency lives entirely at the edges — network I/O and -market-data fanout — and the two SPSC queues are the seam. That is the whole -design in one sentence, and the rest follows from it: +lock anywhere near it. Concurrency lives entirely at the edges, and the two SPSC +queues are the seam. The rest follows from it: -- The network thread **never touches the book**. Not to check a price band, not +- The network thread **never touches the book** — not to check a price band, not to answer a cancel. Anything needing book state crosses the queue. - **Cancel-on-disconnect** travels the same queue as every other command, so a dropped connection needs no special path and no lock. @@ -49,7 +50,7 @@ design in one sentence, and the rest follows from it: ```bash cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build -ctest --test-dir build --output-on-failure # 186 tests +ctest --test-dir build --output-on-failure # 191 tests ./build/gateway --wal /tmp/ome.wal --port 9001 & python3 tools/smoke_client.py --port 9001 # send an order, print the ack @@ -69,23 +70,8 @@ Kill it and watch it come back: kill -9 $(pgrep -f 'build/gateway') ./build/gateway --wal /tmp/ome.wal --recover --port 9001 # recovered N commands, last seq N -# book digest after recovery: ... ``` -With periodic snapshots, recovery reads state rather than history: - -```bash -./build/gateway --wal /tmp/ome.wal --snapshot /tmp/ome.snap \ - --snapshot-every 100000 --port 9001 -# ... after a kill: -./build/gateway --wal /tmp/ome.wal --snapshot /tmp/ome.snap --recover --port 9001 -# restored 8020 orders from snapshot at seq 8020 -# recovered 1580 commands, last seq 9600 -``` - -The WAL is compacted behind each snapshot, so it holds the tail rather than the -whole history — 1,580 records instead of 9,600 above. - Needs network once on first configure (GoogleTest via FetchContent). Build type defaults to Release. Builds with `-Wall -Wextra -Werror`. @@ -93,9 +79,8 @@ defaults to Release. Builds with `-Wall -Wextra -Werror`. ## Measured results -Every number here comes from [`docs/BENCHMARK.md`](docs/BENCHMARK.md), which -states its own methodology and boundaries. Apple M4 Pro, Release, median of -repeated runs. +Apple M4 Pro, Release, median of repeated runs. Methodology and boundaries in +[`docs/BENCHMARK.md`](docs/BENCHMARK.md). | | | |---|---| @@ -103,14 +88,14 @@ repeated runs. | Order-to-ack p50 / p99 | 89 µs / 149 µs at 50,000 orders/sec, 10 clients | | Sustained throughput | **250,000 orders/sec** at 50 clients, nothing rejected | | WAL cost (`fsync`) | +11 µs p50, +52 µs p99 | -| WAL cost (`F_FULLFSYNC`) | 25× at the tail — see below | | Snapshot pause | 28 ms at 100,000 resting orders | -| Crash recovery | **50/50** WAL replay, **25/25** snapshot+tail, independent verifier | -| LOBSTER replay | ~270K messages parsed, applied and checked in ~0.5 s | +| Crash recovery | **50/50** WAL replay, **25/25** snapshot + tail | + +Two results worth stating outright: **Latency falls as load rises** — 185 µs at 100 orders/sec versus 45 µs at 5,000. Under load the matching thread's spin finds work and never parks; when -nearly idle every order pays a park-and-wake round trip. So "45 µs" is only +nearly idle every order pays a park-and-wake round trip. "45 µs" is only meaningful with its offered rate attached. **Media-level durability is expensive.** On macOS `fsync` returns once data @@ -125,143 +110,110 @@ quietly pick the weaker guarantee to look faster. ## Crash safety `tools/kill_test.sh` runs the gateway under load, `kill -9`s it at a random -moment, restarts with recovery, and checks the rebuilt book. **50/50 passing** -on WAL replay, and **25/25** on the snapshot+tail path -(`SNAPSHOT_EVERY=1500 tools/kill_test.sh 25`). - -The check that matters is not that recovery succeeds — it is what it is checked -*against*. `tools/wal_verify` rebuilds the book from the same log using a -separate implementation: `std::map` price levels, its own matching loop, no -shared code with `OrderBook`. Both must produce the same digest. Without that, -the test would be the engine agreeing with itself — a matching bug would produce -the same wrong book on both sides and pass. - -The harness was weaker than it looked at first. A deliberately broken verifier -passed 1/1, because the load generator quoted bids below asks and nothing ever -matched: both implementations agreed only that they could accumulate a book. It -now quotes crossing prices with varying sizes so orders partially fill and leave -real depth. With that fixed the broken verifier is caught immediately. +moment, restarts with recovery, and checks the rebuilt book. ```bash -tools/kill_test.sh 50 # WAL replay -SNAPSHOT_EVERY=1500 tools/kill_test.sh 25 # snapshot + tail +tools/kill_test.sh 50 # WAL replay 50/50 +SNAPSHOT_EVERY=1500 tools/kill_test.sh 25 # snapshot + tail 25/25 ``` +What matters is not that recovery succeeds but what it is checked *against*. +`tools/wal_verify` rebuilds the book from the same log using a separate +implementation — `std::map` price levels, its own matching loop, no shared code +with `OrderBook`. Both must produce the same digest. Otherwise the test is the +engine agreeing with itself, and a matching bug would produce the same wrong +book on both sides and pass. + +The harness was weaker than it looked. A deliberately broken verifier passed +1/1, because the load generator quoted bids below asks and nothing ever matched +— both implementations agreed only that they could accumulate a book. It now +quotes crossing prices with varying sizes, and the broken verifier is caught +immediately. + The snapshot path found a bug the unit tests could not: `truncate_before()` derived sequence numbers by counting from 1, which is right for a log that still -starts at 1 and wrong for one already truncated. The second truncation -renumbered every surviving record, producing a genuine sequence gap and a -gateway that correctly refused to start. It takes *two* truncations to see it, -so no single-truncation test would have. +starts at 1 and wrong for one already truncated. It takes *two* truncations to +see it. -**Scope, stated honestly.** `kill -9` destroys the process without letting it -flush, but it does not destroy the page cache, so everything already handed to -`write()` survives. This proves the recovery path, torn-tail handling, and the -append-before-apply ordering. It does **not** prove behavior under power loss, -which would additionally lose the group-commit window. +**Scope, stated honestly.** `kill -9` does not destroy the page cache, so +everything already handed to `write()` survives. This proves the recovery path, +torn-tail handling and append-before-apply ordering. It does **not** prove +behavior under power loss. --- -## How it works - -**Wire protocol** ([`docs/PROTOCOL.md`](docs/PROTOCOL.md)) — length-prefixed -binary, little-endian, `int64` tick prices, no floating point anywhere. -Serialization is field by field, never a struct `memcpy`: `sizeof(NewOrder)` is -24 on this build while its wire encoding is 22, and those two padding bytes are -uninitialized memory a struct copy would put on the wire. - -**Prices are integer ticks.** Not a style choice — durability asserts that a -recovered book's digest *equals* the original's, and a digest over doubles would -make that a float-equality claim dressed up as a guarantee. +## Research: replicating Farmer, Patelli & Zovko (2005) -**Write-ahead log** — the record is written before the command touches the book. -That ordering picks which failure mode a mid-command crash produces: -append-first leaves a record for a command that never reached the book, and -recovery replays it. Apply-first mutates the book for a command whose record -never landed, and nothing afterwards can detect it. +Full writeup and every caveat in [`docs/FARMER_2005.md`](docs/FARMER_2005.md). -**Recovery uses the live apply path** — the same function live traffic uses, not -a separate replay path. Otherwise "the recovered book is identical" would only -mean two implementations happen to agree. Verified by a 100-seed property test -over random command streams. - -**Risk checks live on the thread that owns the state they need.** Rate limiting -is on the network thread, so an over-limit client is refused before consuming -queue capacity. The price band is on the matching thread, because it is relative -to the last trade or the mid and only that thread may read the book. +[The paper](https://arxiv.org/abs/cond-mat/0309233) (PNAS 102(6):2254–2259) +predicts a stock's mean spread from its order flow alone, `ŝ = (μ/α)·f(ε)`, and +tests it **cross-sectionally** — across stocks, regressing `log s = A·log ŝ + B` +and asking whether A = 1. On 11 LSE stocks it gets A = 0.99 ± 0.10, R² = 0.96. ---- +[`analysis/farmer2005.py`](analysis/farmer2005.py) reruns that test on the five +LOBSTER symbols. **Both laws fail**, diffusion by four orders of magnitude — but +the failure is ordered by `dp/p_c`, the model's own nondimensional tick size, +which Equation 1 assumes away by taking `dp → 0`. A penny on a $27 stock is +seventeen times the characteristic price scale of its own order flow. -## Market data analysis - -[`analysis/stylized_facts.py`](analysis/stylized_facts.py) measures classic -microstructure facts directly from the LOBSTER message stream — no book -reconstruction, so none of it is affected by the replay accuracy gap. Results in -[`docs/STYLIZED_FACTS.md`](docs/STYLIZED_FACTS.md). - -Across five symbols: **91–96% of submitted orders are cancelled rather than -executed**, trade-sign autocorrelation runs **0.72–0.91 at lag 1** and decays -slowly, and trade-level returns show excess kurtosis of 12–24. - -Volatility clustering splits by tick constraint: clearly present in AAPL, AMZN -and GOOG at a 10-second horizon, and undetectable in INTC and MSFT — the two -low-priced names, pinned at a one-cent spread, where discrete bounce drowns the -signal. - -## Replicating Farmer, Patelli & Zovko (2005) - -[`analysis/farmer2005.py`](analysis/farmer2005.py) tests the two scaling laws -from [*The predictive power of zero intelligence in financial -markets*](https://arxiv.org/abs/cond-mat/0309233) (PNAS 102(6):2254–2259) -against the five LOBSTER symbols. Full writeup in -[`docs/FARMER_2005.md`](docs/FARMER_2005.md). - -The paper predicts a stock's mean spread and price diffusion rate from its -order flow alone — `ŝ = (μ/α)·f(ε)` — and tests it **cross-sectionally**, by -regressing `log s = A·log ŝ + B` and asking whether A = 1. - -Across all five symbols both laws fail, the diffusion rate by up to four orders -of magnitude. But the failure is not random: **the error is perfectly -rank-ordered by `dp/p_c`**, the model's own nondimensional tick size (ρ = 1.000, -exact p = 0.017 by enumerating all 120 permutations — a rank test rather than a -regression, because five points cannot support a regression). Equation 1 is -derived in the limit `dp → 0`, and INTC and MSFT sit at `dp/p_c` of 17 and 22 — -a penny on a $27 stock is seventeen times the characteristic price scale of its -own order flow. - -Inside the model's stated domain (`dp/p_c < 1`: GOOG, AAPL, AMZN) the spread -ratio is constant within a factor of 1.5, which is what the law predicts. So -the paper is not refuted here; it is confirmed in the weak sense five stocks -allow, and the tick constraint is what a naive replication would have -misreported as a refutation. - -That much is a correlation on five points, and `dp/p_c` is large for exactly -the two cheapest stocks — so plenty of other things could produce the same -ordering. [`tools/zi_paper.cpp`](tools/zi_paper.cpp) separates them by -simulating the paper's model on this project's matching engine at each stock's -measured parameters and its **real tick size**, where nothing about a cheap -stock is present except four flow parameters and `dp`: +That is a correlation on five points, and `dp/p_c` is large for exactly the two +cheapest stocks. [`tools/zi_paper.cpp`](tools/zi_paper.cpp) separates cause from +coincidence by simulating the paper's model on this project's matching engine at +each stock's measured parameters and its **real tick**, where nothing about a +cheap stock is present except four flow numbers and `dp`: | | dp/p_c | simulated ratio | real ratio | sim/real | |---|---:|---:|---:|---:| | GOOG | 0.21 | 0.66 | 4.36 | 0.15 | +| AAPL | 0.35 | 0.72 | 3.70 | 0.19 | | AMZN | 0.76 | 0.83 | 5.66 | 0.15 | | INTC | 17.30 | **32.23** | **39.75** | **0.81** | | MSFT | 22.03 | **40.87** | **50.35** | **0.81** | -**The tick alone reproduces 81% of the observed inflation**, the same fraction -for both constrained stocks, and the simulated inflation is perfectly -rank-ordered by `dp/p_c` (ρ = 1.000, p = 0.017). The remaining 19% is real and -is presumably what the model deletes — strategic quoting, hidden liquidity, -heterogeneous sizes. +**The tick alone reproduces 81% of the observed departure from the law** — the +same fraction for both constrained stocks — and the simulated inflation is +perfectly rank-ordered by `dp/p_c` (ρ = 1.000, exact p = 0.017). So the paper is +not refuted; ignoring its scope condition is what makes it look refuted. + +Two things this does not show, both kept in the writeup: the remaining 19% is +real, and the small-tick simulation runs ~25% *below* the mean-field prediction +rather than matching it. + +Earlier and narrower: [`docs/STYLIZED_FACTS.md`](docs/STYLIZED_FACTS.md) +measures cancel ratios, trade-sign autocorrelation and volatility clustering +straight from the message stream, and +[`docs/ZI_COMPARISON.md`](docs/ZI_COMPARISON.md) is a calibrated ZI baseline +that tests properties this paper never claims — and says so at the top. + +--- + +## Design decisions + +**Wire protocol** ([`docs/PROTOCOL.md`](docs/PROTOCOL.md)) — length-prefixed +binary, little-endian, `int64` tick prices, no floating point anywhere. +Serialization is field by field, never a struct `memcpy`: `sizeof(NewOrder)` is +24 on this build while its wire encoding is 22, and those two padding bytes are +uninitialized memory a struct copy would put on the wire. + +**Prices are integer ticks.** Durability asserts that a recovered book's digest +*equals* the original's, and a digest over doubles would make that a +float-equality claim dressed up as a guarantee. -It doubles as an independent exercise of the matching engine: a continuous -double auction driven for millions of events against an analytically known -answer, agreeing with it to within a constant wherever that answer applies. +**The WAL record is written before the command touches the book.** That ordering +picks which failure mode a mid-command crash produces: append-first leaves a +record for a command that never reached the book, and recovery replays it. +Apply-first mutates the book for a command whose record never landed, and +nothing afterwards can detect it. -The earlier stylized-facts comparison in -[`docs/ZI_COMPARISON.md`](docs/ZI_COMPARISON.md) tests properties this paper -never claims, and says so at the top. +**Recovery uses the live apply path**, not a separate replay path. Otherwise +"the recovered book is identical" would only mean two implementations happen to +agree. + +**Risk checks live on the thread that owns the state they need.** Rate limiting +is on the network thread, so an over-limit client is refused before consuming +queue capacity. The price band is on the matching thread, because it is relative +to the last trade or the mid and only that thread may read the book. --- @@ -278,14 +230,10 @@ loopback only. would get a second thread and a second book; a second thread on *one* book would need the locking this architecture exists to avoid. -**Benchmarks are loopback on one machine.** Not a network. The load generator -shares cores with the gateway, and at high client counts some of the tail is its -own scheduling. At 100 clients × 5,000 orders/sec the generator — not the -gateway — stops making progress, so that cell is blank rather than guessed. - -**The benchmark workload does not cross.** It measures the accept-and-acknowledge -path, not matching under contention. Published numbers were also taken with the -price band disabled and no subscriber attached. +**Benchmarks are loopback on one machine**, and the workload does not cross — it +measures the accept-and-acknowledge path, not matching under contention. At 100 +clients × 5,000 orders/sec the load generator, not the gateway, stops making +progress, so that cell is blank rather than guessed. **No self-trade prevention.** A session holding both sides will match itself. @@ -297,8 +245,7 @@ must not read an ack as "did not trade". **`BookUpdate` sequence numbers are not contiguous.** Conflation skips values by design; a client treating a gap as loss would flag healthy behavior as an error. - -**No delta encoding for market data** — every update is a full top-N snapshot. +No delta encoding either — every update is a full top-N snapshot. **LOBSTER replay accuracy is 70% at 1K events and 50% beyond.** Faithful reconstruction needs order-level state across the file window, and 21% of @@ -306,15 +253,16 @@ executions in this sample are against hidden liquidity that never appears in the book at all. **Snapshots pause the matching thread** — 28 ms at 100K resting orders. The fix -is measured and known (the copy is 2.2 ms while the write is 26 ms, so -serializing a copy on a background thread would remove most of it) and not yet +is measured and known (the copy is 2.2 ms while the write is 26 ms) and not yet implemented. Snapshots are off unless `--snapshot` is given. -**Cancel-on-disconnect is O(orders held).** A session disconnecting with a very -large book blocks the matching thread for the sweep. +**Cancel-on-disconnect is O(orders held)**, and **`poll()` is O(connections)** — +p50 goes 185 µs → 1,010 µs from 1 to 100 clients. `epoll`/`kqueue` are O(ready), +not yet worth the portability cost. -**`poll()` is O(connections).** p50 goes 185 µs → 1,010 µs from 1 to 100 -clients. `epoll`/`kqueue` are O(ready); not yet worth the portability cost. +**The replication is 5 stocks and 1 day**, against the paper's 11 stocks and 434 +days. At n = 5 the smallest attainable exact p-value *is* 0.017, so that result +is suggestive rather than established. --- @@ -327,9 +275,9 @@ clients. `epoll`/`kqueue` are O(ready); not yet worth the portability cost. | `include/ome/` | protocol, framing, sessions, WAL, snapshots, queues | | `src/net/tcp_server.cpp` | `poll()` event loop | | `bench/` | load generator and latency sweep | -| `tools/` | visualizer, live feed, WAL verifier, kill-test harness | -| `analysis/` | stylized facts from LOBSTER messages | -| `docs/` | protocol spec, benchmarks, measured market data facts | +| `tools/` | visualizer, live feed, WAL verifier, kill-test harness, ZI simulators | +| `analysis/` | LOBSTER measurement, calibration, the paper replication | +| `docs/` | protocol spec, benchmarks, market data facts, replication writeup | Sanitizer builds — `address`, `undefined`, `address,undefined`, or `thread` (thread and address are mutually exclusive): @@ -343,9 +291,10 @@ cmake --build build-asan && ctest --test-dir build-asan --output-on-failure ## LOBSTER data (optional) -Sample CSVs are **not** in git. Download a message + orderbook pair from +Sample CSVs are **not** in git. Download message + orderbook pairs from [LOBSTER samples](https://data.lobsterdata.com/info/DataSamples.php), place under `data/lobster/`, then see [`data/lobster/README.md`](data/lobster/README.md). +The replication needs all five symbols; everything else works with one. ```bash ./build/lobster_replay --messages --orderbook \ @@ -358,11 +307,9 @@ open tools/book_replay.html # load /tmp/replay.jsonl ## What I want to improve next - **LOBSTER parity** — order-level state across the file window, hidden - liquidity, cross and auction messages. Per-event golden checks rather than an - end snapshot. + liquidity, cross and auction messages. - **Copy-then-write snapshots**, to remove the matching-thread pause. - **A crossing benchmark**, to measure matching under contention rather than the accept path alone. -- **Synthetic order flow** calibrated to the LOBSTER data and run through the - same engine, to see which stylized facts survive when strategic behavior is - removed. +- **More trading days**, so the replication has error bars on the real side + rather than a single observation per stock.