From a3f80d57c402a2d0e792f121bbfd6f719182b509 Mon Sep 17 00:00:00 2001 From: brianmmaina Date: Sun, 16 Aug 2026 09:08:34 -0700 Subject: [PATCH 1/2] research: simulate the paper's model to test the tick explanation The dp/p_c result was a correlation on five points, and dp/p_c is large for exactly the two cheapest stocks in the sample. Plenty of things that separate a $30 stock from a $500 one would give the same ordering. Simulation holds everything else fixed. zi_paper implements the paper's model, the austere one, on our matching engine, and runs it at each stock's measured alpha, mu, delta, sigma and its real tick. Nothing about a cheap stock is present except four flow parameters and dp. The tick alone reproduces 81% of the observed inflation, the same fraction for both constrained stocks. Simulated inflation is perfectly rank-ordered by dp/p_c, rho = 1.000, p = 0.017, and inside a simulation that is much stronger than it was on real data because there is no confounder left. The remaining 19% is real and stays in the writeup. So does the small-tick ratio being 0.66 to 0.83 rather than 1: the simulation runs consistently below the mean field prediction and that is not rounded to agreement. Separate tool rather than a flag on zi_sim. zi_sim uses empirical sizes and an empirical placement histogram, so it is a better imitation of a market and a worse test of this paper. Mixing them would let a difference in the model look like a difference in the law. The width scan earned its place. Deposition intervals are semi-infinite and have to be truncated somewhere; if the answer moves with the truncation the boundary is setting the spread. The first version truncated to a fixed price box and scanned 32.23, 32.23, 32.23, 0.00, 32.23. Non-monotonic in the width is a bug, not a boundary effect: the book could pin its best bid against the top of the box, leaving the sell interval empty so no sell could ever arrive again. An absorbing one-sided state reporting a spread of zero. The box also made buy and sell rates depend on where the price sat inside it, breaking the model's equal-rates assumption. Anchoring each interval to the opposing quote fixes both. Without the scan the fixed-box version would have produced a plausible-looking table. Parameters come from farmer2005.measure_all rather than being measured again, so a difference in measurement cannot masquerade as a difference in the model. --- CMakeLists.txt | 3 + README.md | 24 +++ analysis/farmer2005.py | 39 ++-- analysis/validate_scaling.py | 166 +++++++++++++++++ docs/FARMER_2005.md | 91 +++++++++ tools/zi_paper.cpp | 349 +++++++++++++++++++++++++++++++++++ 6 files changed, 659 insertions(+), 13 deletions(-) create mode 100644 analysis/validate_scaling.py create mode 100644 tools/zi_paper.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 86fef3b..6532dfc 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -76,6 +76,9 @@ target_link_libraries(wal_verify PRIVATE order_book ome_warnings) add_executable(zi_sim tools/zi_sim.cpp) target_link_libraries(zi_sim PRIVATE order_book ome_warnings) +add_executable(zi_paper tools/zi_paper.cpp) +target_link_libraries(zi_paper PRIVATE order_book ome_warnings) + add_executable(loadgen bench/loadgen.cpp) target_link_libraries(loadgen PRIVATE order_book ome_warnings Threads::Threads) diff --git a/README.md b/README.md index 71c7664..fbad8c6 100644 --- a/README.md +++ b/README.md @@ -235,6 +235,30 @@ the paper is not refuted here; it is confirmed in the weak sense five stocks allow, and the tick constraint is what a naive replication would have misreported as a refutation. +That much is a correlation on five points, and `dp/p_c` is large for exactly +the two cheapest stocks — so plenty of other things could produce the same +ordering. [`tools/zi_paper.cpp`](tools/zi_paper.cpp) separates them by +simulating the paper's model on this project's matching engine at each stock's +measured parameters and its **real tick size**, where nothing about a cheap +stock is present except four flow parameters and `dp`: + +| | dp/p_c | simulated ratio | real ratio | sim/real | +|---|---:|---:|---:|---:| +| GOOG | 0.21 | 0.66 | 4.36 | 0.15 | +| AMZN | 0.76 | 0.83 | 5.66 | 0.15 | +| INTC | 17.30 | **32.23** | **39.75** | **0.81** | +| MSFT | 22.03 | **40.87** | **50.35** | **0.81** | + +**The tick alone reproduces 81% of the observed inflation**, the same fraction +for both constrained stocks, and the simulated inflation is perfectly +rank-ordered by `dp/p_c` (ρ = 1.000, p = 0.017). The remaining 19% is real and +is presumably what the model deletes — strategic quoting, hidden liquidity, +heterogeneous sizes. + +It doubles as an independent exercise of the matching engine: a continuous +double auction driven for millions of events against an analytically known +answer, agreeing with it to within a constant wherever that answer applies. + The earlier stylized-facts comparison in [`docs/ZI_COMPARISON.md`](docs/ZI_COMPARISON.md) tests properties this paper never claims, and says so at the top. diff --git a/analysis/farmer2005.py b/analysis/farmer2005.py index 4f233f1..1aa52c0 100644 --- a/analysis/farmer2005.py +++ b/analysis/farmer2005.py @@ -454,23 +454,23 @@ def report_regression(name, r, markdown=False): print(" confirmation.") -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("--symbols", default="AAPL,AMZN,GOOG,INTC,MSFT") - ap.add_argument("--skip-open", type=float, default=0.0, - help="seconds of the session to skip; the paper excludes the " - "opening auction, which LOBSTER samples already omit") - ap.add_argument("--markdown", action="store_true") - args = ap.parse_args() +def measure_all(symbols, skip_open=0.0, verbose=True): + """Measure every symbol. Shared with analysis/validate_scaling.py. + The simulator must run on exactly the parameters the empirical test used, + or a difference in the measurement would look like a difference in the + model. Same reason analysis/compare.py imports stylized_facts rather than + reimplementing it. + """ rows = [] - for sym in [s.strip().upper() for s in args.symbols.split(",") if s.strip()]: + for sym in symbols: m, b = find_pair(sym) if not m or not b: print(f"skipping {sym}: no message/orderbook pair", file=sys.stderr) continue - print(f"scanning {sym} ...", file=sys.stderr) - s = scan(sym, m, b, args.skip_open) + if verbose: + print(f"scanning {sym} ...", file=sys.stderr) + s = scan(sym, m, b, skip_open) p = parameters(s) eps, p_c, s_hat, d_hat = predict(p) s_real = (sum(s["spreads"]) / len(s["spreads"])) if s["spreads"] else float("nan") @@ -494,13 +494,26 @@ def main(): "s_real": s_real, "d_real": d_real, "n_mid": len(s["mids"]), "price": mm / 10000.0, "dp_log": dp_log, "tick_ratio": tick_ratio}) + rows.sort(key=lambda r: r["tick_ratio"]) + return rows + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--symbols", default="AAPL,AMZN,GOOG,INTC,MSFT") + ap.add_argument("--skip-open", type=float, default=0.0, + help="seconds of the session to skip; the paper excludes the " + "opening auction, which LOBSTER samples already omit") + ap.add_argument("--markdown", action="store_true") + args = ap.parse_args() + + rows = measure_all([x.strip().upper() for x in args.symbols.split(",") if x.strip()], + args.skip_open) if not rows: print(f"no LOBSTER data under {DATA_ROOT}", file=sys.stderr) return 1 - rows.sort(key=lambda r: r["tick_ratio"]) - print("\nmeasured model parameters (event time, log price, shares)") hdr = (f"{'sym':<6} {'events':>8} {'mu':>9} {'sigma':>8} {'alpha':>11} " f"{'delta':>9} {'eps':>8}") diff --git a/analysis/validate_scaling.py b/analysis/validate_scaling.py new file mode 100644 index 0000000..04c0cb1 --- /dev/null +++ b/analysis/validate_scaling.py @@ -0,0 +1,166 @@ +#!/usr/bin/env python3 +"""Is the tick constraint a mechanism, or just a correlate? + +analysis/farmer2005.py found that the Farmer/Patelli/Zovko spread law fails +across five LOBSTER stocks, and that the size of the failure is rank-ordered by +the model's own nondimensional tick size dp/p_c -- the parameter Equation 1 +assumes away by taking dp -> 0. + +That is a correlation on five points, and it has an obvious alternative +explanation. dp/p_c is large exactly for INTC and MSFT, which are also the two +cheapest, highest-volume, most heavily quoted names in the sample. Any of a +dozen things that distinguish a $30 stock from a $500 one would produce the +same ordering. + +SIMULATION SEPARATES THOSE, because it can hold everything else fixed. + +Run the paper's own model -- tools/zi_paper, the austere specification, not the +richer zi_sim -- at each stock's measured (alpha, mu, delta, sigma) and its real +tick size. Nothing about a $30 stock is present except four flow parameters and +dp. So if simulated INTC reproduces the inflated spread ratio that real INTC +shows, the tick is doing the work. + +The prediction, made before running it: + + small dp/p_c simulated ratio near 1 (the mean field result holds) + large dp/p_c simulated ratio near the EMPIRICAL ratio for that stock + +The second half is the risky one. It is easy to get an inflated spread out of a +coarse grid; matching the size of the real inflation is not automatic, and a +mismatch would say the tick explains the direction but not the magnitude. + +Parameters come from farmer2005.measure_all rather than being re-measured here, +so a difference in measurement cannot masquerade as a difference in the model. + +Usage: + python3 analysis/validate_scaling.py + python3 analysis/validate_scaling.py --events 2000000 --seeds 5 +""" + +import argparse +import math +import os +import statistics +import subprocess +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import farmer2005 as f # noqa: E402 + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +ZI_PAPER = os.path.join(ROOT, "build", "zi_paper") + + +def simulate(row, events, seed, width_pc): + """One run of the paper's model at this stock's measured parameters.""" + cmd = [ZI_PAPER, + "--alpha", f"{row['alpha']:.10g}", + "--mu", f"{row['mu']:.10g}", + "--delta", f"{row['delta']:.10g}", + "--sigma", f"{row['sigma']:.10g}", + "--dp", f"{row['dp_log']:.10g}", + "--events", str(events), + "--width-pc", str(width_pc), + "--seed", str(seed), + "--no-header"] + r = subprocess.run(cmd, capture_output=True, text=True) + if r.returncode != 0: + print(f" {row['symbol']} seed {seed} failed: {r.stderr.strip()}", file=sys.stderr) + return None + parts = r.stdout.split() + if len(parts) < 6: + return None + # eps dp/p_c half_tk s_hat s_sim ratio resting wide% + return {"s_sim": float(parts[4]), "ratio": float(parts[5]), + "resting": float(parts[6])} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--symbols", default="AAPL,AMZN,GOOG,INTC,MSFT") + ap.add_argument("--events", type=int, default=1000000) + ap.add_argument("--seeds", type=int, default=5) + ap.add_argument("--width-pc", type=float, default=50.0) + args = ap.parse_args() + + if not os.path.exists(ZI_PAPER): + print(f"{ZI_PAPER} not built. cmake --build build --target zi_paper", + file=sys.stderr) + return 1 + + syms = [s.strip().upper() for s in args.symbols.split(",") if s.strip()] + rows = f.measure_all(syms) + if not rows: + return 1 + + print(f"\nsimulating the paper's model at each stock's measured parameters") + print(f"{args.seeds} seeds x {args.events:,} events, width = {args.width_pc} p_c\n") + + out = [] + for r in rows: + sims = [simulate(r, args.events, s, args.width_pc) + for s in range(1, args.seeds + 1)] + sims = [s for s in sims if s] + if not sims: + continue + ratios = [s["ratio"] for s in sims] + emp = r["s_real"] / r["s_hat"] if r["s_hat"] else float("nan") + out.append({"symbol": r["symbol"], "tick_ratio": r["tick_ratio"], + "sim_mean": statistics.mean(ratios), + "sim_sd": statistics.stdev(ratios) if len(ratios) > 1 else 0.0, + "emp": emp, + "s_sim": statistics.mean(s["s_sim"] for s in sims), + "s_real": r["s_real"], "s_hat": r["s_hat"]}) + print(f" {r['symbol']} done", file=sys.stderr) + + hdr = (f"{'sym':<6} {'dp/p_c':>8} {'sim ratio':>16} {'real ratio':>11} " + f"{'sim/real':>9}") + print() + print(hdr) + print("-" * len(hdr)) + for o in out: + agree = o["sim_mean"] / o["emp"] if o["emp"] else float("nan") + print(f"{o['symbol']:<6} {o['tick_ratio']:>8.2f} " + f"{o['sim_mean']:>10.2f} +/-{o['sim_sd']:>4.2f} {o['emp']:>11.2f} " + f"{agree:>9.2f}") + + print("\n sim ratio simulated spread / s_hat, the paper's model at these parameters") + print(" real ratio actual LOBSTER spread / s_hat, from farmer2005.py") + print(" sim/real 1.00 means the tick constraint alone accounts for the") + print(" whole empirical departure from the scaling law") + + # Does the simulated inflation track the real one? With five points a rank + # test is the honest statistic, for the same reason as in farmer2005.py. + sp = f.spearman_exact([o["sim_mean"] for o in out], [o["emp"] for o in out]) + if sp: + print(f"\n simulated vs real inflation: rho = {sp['rho']:+.3f}, " + f"exact p = {sp['p']:.4f} (n = {sp['n']})") + # The cleaner claim: inside the simulation, dp/p_c is the ONLY thing that + # varies besides the four flow parameters, so if the simulated inflation is + # ordered by it there is no confounder left to appeal to. + sp2 = f.spearman_exact([o["tick_ratio"] for o in out], + [o["sim_mean"] for o in out]) + if sp2: + print(f" simulated inflation vs dp/p_c: rho = {sp2['rho']:+.3f}, " + f"exact p = {sp2['p']:.4f} (n = {sp2['n']})") + + small = [o for o in out if o["tick_ratio"] < 1.0] + large = [o for o in out if o["tick_ratio"] >= 1.0] + if small: + m = statistics.mean(o["sim_mean"] for o in small) + print(f"\n dp/p_c < 1 ({', '.join(o['symbol'] for o in small)}): " + f"simulated ratio {m:.2f}") + print(" near 1 means the mean field result describes the simulation,") + print(" so the engine and the law agree where the law claims to apply.") + if large: + m = statistics.mean(o["sim_mean"] for o in large) + me = statistics.mean(o["emp"] for o in large) + print(f"\n dp/p_c >= 1 ({', '.join(o['symbol'] for o in large)}): " + f"simulated {m:.1f} vs real {me:.1f}") + print(" the tick alone, with no other property of a cheap stock present,") + print(f" reproduces {100.0*m/me:.0f}% of the observed inflation.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/FARMER_2005.md b/docs/FARMER_2005.md index 456c7c6..844fac6 100644 --- a/docs/FARMER_2005.md +++ b/docs/FARMER_2005.md @@ -171,6 +171,97 @@ nondimensional tick size predicts the size of its error. --- +--- + +## Is the tick a mechanism, or just a correlate? + +Everything above is a correlation on five points, and it has an obvious +alternative explanation. `dp/p_c` is large for exactly INTC and MSFT, which are +also the two cheapest, highest-volume, most heavily quoted names in the sample. +A dozen things that distinguish a $30 stock from a $500 stock would produce the +same ordering. + +Simulation separates them, because it can hold everything else fixed. +[`tools/zi_paper.cpp`](../tools/zi_paper.cpp) implements the paper's model — +the austere one: single order size, uniform deposition, equal rates, constant +cancellation — on this project's own matching engine. Run it at each stock's +measured `(α, μ, δ, σ)` and its **real tick size**, and nothing about a cheap +stock is present except four flow parameters and `dp`. + +```bash +cmake --build build --target zi_paper +python3 analysis/validate_scaling.py --events 1000000 --seeds 5 +``` + +The prediction, stated before running it: small `dp/p_c` should give a +simulated ratio near 1, and large `dp/p_c` should give a ratio near the +*empirical* ratio for that stock. The second half is the risky one — a coarse +grid inflating the spread is easy, matching the *size* of the real inflation is +not. + +| | dp/p_c | sim ratio | real ratio | sim/real | +|---|---:|---:|---:|---:| +| GOOG | 0.21 | 0.66 ± 0.01 | 4.36 | 0.15 | +| AAPL | 0.35 | 0.72 ± 0.01 | 3.70 | 0.19 | +| AMZN | 0.76 | 0.83 ± 0.01 | 5.66 | 0.15 | +| INTC | 17.30 | **32.23** ± 0.00 | **39.75** | **0.81** | +| MSFT | 22.03 | **40.87** ± 0.00 | **50.35** | **0.81** | + +**The tick alone reproduces 81% of the observed inflation** for both +tick-constrained stocks — the same fraction for each, which is not something +the run was tuned to produce. + +The simulated inflation is **perfectly rank-ordered by `dp/p_c`** (ρ = 1.000, +exact p = 0.017). Inside the simulation that is a much stronger statement than +the same number was on real data: `dp/p_c` and the four flow parameters are the +*only* things that vary, so there is no confounder left to appeal to. + +Two things this does **not** show: + +* **The remaining 19% is real.** The tick is the dominant term, not the whole + story. What is left is presumably the things the model deletes — strategic + quoting, hidden liquidity, heterogeneous order sizes. +* **The small-tick ratio is 0.66–0.83, not 1.0.** The simulation runs about 25% + *below* the mean field prediction, consistently. `f(ε)` is itself an + approximation, so a systematic gap of this size is unremarkable, but it is a + gap and it is not being rounded to "agreement". + +The useful by-product: this is an independent exercise of the matching engine — +a continuous double auction driven for millions of events with an answer known +analytically — and it agrees with that answer to within a constant wherever the +analytic result claims to apply. + +### The width scan is the negative control + +The paper's deposition intervals are semi-infinite and a simulation must +truncate them. If the answer moves when the truncation moves, the boundary is +setting the spread rather than the order flow, and the whole result is an +artifact: + +```bash +./build/zi_paper --alpha ... --dp ... --width-scan +``` + +GOOG returns 0.66, 0.68, 0.65, 0.69, 0.70 across a 16× range of widths; INTC +returns 32.23 at every width. + +This was not a formality. The first implementation truncated to a **fixed price +box** centred at zero, and the scan produced 32.23, 32.23, 32.23, **0.00**, +32.23 — non-monotonic in the width, which is the signature of a bug rather than +a boundary effect. The book could pin its best bid against the top of the box, +at which point the sell interval `[b+1, box_top]` was empty and no sell order +could ever arrive again: an absorbing one-sided state that silently reported a +spread of zero. The same fixed box also made the buy and sell arrival rates +depend on where the price sat inside it, quietly breaking the model's +equal-rates assumption. + +Anchoring each interval to the opposing best quote — buys on `[a(t)−W, a(t)−1]`, +sells on `[b(t)+1, b(t)+W]` — fixes both: constant equal widths, and prices free +to wander. Without the scan, the fixed-box version would have produced a +plausible-looking table. + +--- + ## Limits — read before quoting any number here * **5 stocks, not 11. One day, not 434.** The paper averages parameters over diff --git a/tools/zi_paper.cpp b/tools/zi_paper.cpp new file mode 100644 index 0000000..cc99872 --- /dev/null +++ b/tools/zi_paper.cpp @@ -0,0 +1,349 @@ +// The Farmer/Patelli/Zovko (2005) zero-intelligence model, simulated exactly as +// specified, on the project's real matching engine. +// +// J. D. Farmer, P. Patelli, I. I. Zovko, "The predictive power of zero +// intelligence in financial markets", PNAS 102(6):2254-2259, 2005. +// +// WHY THIS EXISTS SEPARATELY FROM zi_sim +// +// tools/zi_sim.cpp is a RICHER model: empirical order sizes, an empirical +// placement histogram fitted to LOBSTER. That makes it a better imitation of a +// real market and a worse test of this paper, whose scaling laws are derived +// for a specific and much more austere specification: +// +// * every order is the same size, sigma +// * buy limit orders deposit UNIFORMLY on (-inf, a(t)), sells on (b(t), +inf) +// * equal rates for buying and selling +// * every process Poisson and independent except through the boundaries +// * resting orders cancel at a constant rate delta, regardless of position +// +// Mixing the two would let a difference in the model masquerade as a difference +// in the law. So this file implements the paper's model and nothing else. +// +// WHAT IT TESTS +// +// analysis/farmer2005.py found that the spread law fails across five LOBSTER +// stocks, and that the error is rank-ordered by the model's own nondimensional +// tick size dp/p_c -- the parameter Equation 1 assumes away by taking dp -> 0. +// That is a correlation on five points. It is consistent with the tick +// constraint causing the failure, and equally consistent with dp/p_c being a +// proxy for something else that separates $30 stocks from $500 stocks. +// +// Simulation separates those. Run the paper's own model at a stock's measured +// (alpha, mu, delta, sigma) and its REAL tick size, and the only thing that can +// inflate the spread is the tick, because nothing else about a $30 stock is +// present. If simulated INTC reproduces the inflated ratio that real INTC +// shows, the tick constraint is a mechanism rather than a correlate. +// +// This is also an independent exercise of the matching engine: a continuous +// double auction driven for millions of events, where the answer is known +// analytically for small dp. +// +// Prices are log-price tick indices: index i is a log price of i*dp. The mean +// field result is in log price, and this keeps the engine's exact integer +// comparisons intact. + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "matching_engine/matching_engine.hpp" + +namespace { + +struct Config { + // Measured per stock by analysis/farmer2005.py. Units are shares, + // log-price and events; see that script for how each is estimated. + double alpha = 0.0; // shares per unit log-price per unit time + double mu = 0.0; // shares per unit time + double delta = 0.0; // 1/time, per resting order + double sigma = 0.0; // shares + double dp = 0.0; // tick size, in log price + + std::uint64_t events = 2000000; + double warmup_frac = 0.2; + // Depth of the simulated deposition interval, in characteristic prices p_c. + // + // The paper's intervals are semi-infinite and a simulation needs them + // finite, so each is truncated at this distance BEHIND the opposing best + // quote -- buys on [a(t)-W, a(t)-1], sells on [b(t)+1, b(t)+W]. + // + // Anchoring to the opposing quote rather than to a fixed price box matters + // for two reasons, both learned the hard way. A fixed box lets the book + // pin its best bid against the top wall, at which point the sell interval + // [b+1, box_top] is empty and no sell can ever arrive again: an absorbing + // one-sided state that silently produced a spread of zero. A fixed box + // also makes the buy and sell arrival rates depend on where the price sits + // inside it, which breaks the model's equal-rates-for-buying-and-selling + // assumption. Anchored intervals have constant, equal width by + // construction, and prices are free to wander. + // + // --width-scan checks the answer does not depend on W rather than assuming + // it. If it does, the truncation is setting the spread instead of the + // order flow. + double width_pc = 50.0; + std::uint32_t seed = 1; +}; + +// f(eps) = 0.28 + 1.86 eps^(3/4), Equation 1. +double f_eps(double eps) { return 0.28 + 1.86 * std::pow(eps, 0.75); } + +struct Result { + double mean_spread_log = 0.0; // measured, in log price + double mean_spread_ticks = 0.0; + std::uint64_t measured = 0; + std::uint64_t trades = 0; + std::uint64_t boundary_hits = 0; // events where the range bound was reached + double mean_resting = 0.0; +}; + +class PaperSim { +public: + PaperSim(const Config& c, std::int64_t half_ticks) + : c_(c), half_(half_ticks), rng_(c.seed), + size_(static_cast(std::max(1, std::llround(c.sigma)))) {} + + Result run() { + Result r; + const std::uint64_t warm = + static_cast(static_cast(c_.events) * c_.warmup_frac); + double spread_sum = 0.0; + double resting_sum = 0.0; + std::uint64_t ts = 0; + + for (std::uint64_t ev = 0; ev < c_.events; ++ev) { + std::int64_t bid = 0, ask = 0; + const bool has_bid = eng_.book().best_bid_ticks(bid); + const bool has_ask = eng_.book().best_ask_ticks(ask); + + // Deposition intervals, per the model: buy limit orders arrive + // with constant density below the best ask, sells above the best + // bid. Each is truncated half_ ticks behind the quote it is + // anchored to, so both are the same constant width and neither can + // be squeezed to nothing. + // + // With one side empty there is no opposing quote to anchor to, so + // the other side's best stands in and the book is one tick wide. + const std::int64_t ref_ask = has_ask ? ask : (has_bid ? bid + 1 : 1); + const std::int64_t ref_bid = has_bid ? bid : (has_ask ? ask - 1 : 0); + const std::int64_t buy_hi = ref_ask - 1; + const std::int64_t buy_lo = buy_hi - half_ + 1; + const std::int64_t sell_lo = ref_bid + 1; + const std::int64_t sell_hi = sell_lo + half_ - 1; + if (has_bid && has_ask && (ask - bid) >= half_) ++r.boundary_hits; + + // Rates. alpha is a density in shares per unit log price, so the + // arrival rate over an interval is alpha * width, and the ORDER + // rate is that divided by the order size. Both intervals are + // half_ ticks wide, which is the model's equal rates for buying + // and selling. + const double w = static_cast(half_) * c_.dp; + const double r_lb = c_.alpha * w / c_.sigma; + const double r_ls = r_lb; + const double r_mkt = c_.mu / c_.sigma; + const double n_live = static_cast(live_.size()); + const double r_can = c_.delta * n_live; + const double total = r_lb + r_ls + r_mkt + r_can; + if (total <= 0.0) break; + + double u = uni_(rng_) * total; + ++ts; + + if (u < r_lb) { + place(Order::BID, pick(buy_lo, buy_hi), ts); + } else if ((u -= r_lb) < r_ls) { + place(Order::ASK, pick(sell_lo, sell_hi), ts); + } else if ((u -= r_ls) < r_mkt) { + r.trades += market(uni_(rng_) < 0.5 ? Order::BID : Order::ASK, ts); + } else { + cancel(); + } + + if (ev >= warm) { + std::int64_t b2 = 0, a2 = 0; + if (eng_.book().best_bid_ticks(b2) && eng_.book().best_ask_ticks(a2) && a2 > b2) { + // The paper measures the spread after every event, each + // with equal weight. + spread_sum += static_cast(a2 - b2); + ++r.measured; + } + resting_sum += n_live; + } + } + + if (r.measured > 0) { + r.mean_spread_ticks = spread_sum / static_cast(r.measured); + r.mean_spread_log = r.mean_spread_ticks * c_.dp; + } + const auto denom = static_cast(c_.events) * (1.0 - c_.warmup_frac); + r.mean_resting = denom > 0 ? resting_sum / denom : 0.0; + return r; + } + +private: + std::int64_t pick(std::int64_t lo, std::int64_t hi) { + if (hi < lo) return lo; + const auto span = static_cast(hi - lo); + std::uniform_int_distribution d(0, span); + return lo + static_cast(d(rng_)); + } + + void place(Order::Side side, std::int64_t px, std::uint64_t ts) { + Order o = Order::make(px, size_, side, Order::LIMIT, ts); + const ApplyResult res = eng_.processOrder(o); + if (res.accepted) live_.push_back(o.id); + } + + std::uint64_t market(Order::Side side, std::uint64_t ts) { + const std::size_t before = eng_.trade_log_size(); + Order o = Order::make(0, size_, side, Order::MARKET, ts); + (void)eng_.processOrder(o); + const std::size_t after = eng_.trade_log_size(); + + // Every order in this model is the same size, so a market order + // consumes exactly one resting order in full. The passive side of each + // new trade is therefore gone from the book, and dropping it keeps the + // cancellation rate delta*N honest -- N must be the number of orders + // actually resting, not the number ever placed. + for (std::size_t i = before; i < after; ++i) { + const Trade& t = eng_.trade_log()[i]; + const std::uint64_t passive = (t.buyer_id == o.id) ? t.seller_id : t.buyer_id; + forget(passive); + } + eng_.clear_trade_log(); + return static_cast(after - before); + } + + void cancel() { + // "Queued limit orders are canceled at a constant rate", independent + // of price and of age, so the victim is drawn uniformly at random. + while (!live_.empty()) { + std::uniform_int_distribution d(0, live_.size() - 1); + const std::size_t i = d(rng_); + const std::uint64_t id = live_[i]; + live_[i] = live_.back(); + live_.pop_back(); + if (eng_.book().cancelOrder(id)) return; + // Stale entry: already executed. Not a cancellation event, so keep + // drawing rather than consuming the event on a no-op. + } + } + + void forget(std::uint64_t id) { + const auto it = std::find(live_.begin(), live_.end(), id); + if (it != live_.end()) { + *it = live_.back(); + live_.pop_back(); + } + } + + Config c_; + std::int64_t half_; + MatchingEngine eng_; + std::mt19937 rng_; + std::uniform_real_distribution uni_{0.0, 1.0}; + std::vector live_; + std::uint32_t size_; +}; + +int run_one(const Config& c, bool header) { + if (c.alpha <= 0 || c.mu <= 0 || c.delta <= 0 || c.sigma <= 0 || c.dp <= 0) { + std::fprintf(stderr, "alpha, mu, delta, sigma and dp must all be positive\n"); + return 2; + } + const double eps = c.delta * c.sigma / c.mu; + const double p_c = c.mu / c.alpha; + const double s_hat = p_c * f_eps(eps); + const double tick_ratio = c.dp / p_c; + + // The range is expressed in characteristic prices so it means the same + // thing for a large-tick and a small-tick stock. At least a few ticks + // either way, or there is no room for a book at all. + const auto half = std::max( + 3, static_cast(std::llround(c.width_pc * p_c / c.dp))); + + PaperSim sim(c, half); + const Result r = sim.run(); + + if (header) { + std::printf("%9s %9s %9s %12s %12s %8s %9s %8s\n", + "eps", "dp/p_c", "half_tk", "s_hat", "s_sim", "ratio", + "resting", "wide%"); + } + const double ratio = s_hat > 0 ? r.mean_spread_log / s_hat : 0.0; + const double bnd = c.events ? 100.0 * static_cast(r.boundary_hits) / + static_cast(c.events) + : 0.0; + std::printf("%9.4f %9.3f %9lld %12.3e %12.3e %8.2f %9.1f %8.2f\n", + eps, tick_ratio, static_cast(half), s_hat, + r.mean_spread_log, ratio, r.mean_resting, bnd); + return 0; +} + +} // namespace + +int main(int argc, char** argv) { + Config c; + bool scan = false; + bool header = true; + + for (int i = 1; i < argc; ++i) { + const std::string a = argv[i]; + auto next = [&]() -> std::string { return (i + 1 < argc) ? argv[++i] : ""; }; + if (a == "--alpha") c.alpha = std::atof(next().c_str()); + else if (a == "--mu") c.mu = std::atof(next().c_str()); + else if (a == "--delta") c.delta = std::atof(next().c_str()); + else if (a == "--sigma") c.sigma = std::atof(next().c_str()); + else if (a == "--dp") c.dp = std::atof(next().c_str()); + else if (a == "--events") c.events = std::strtoull(next().c_str(), nullptr, 10); + else if (a == "--width-pc") c.width_pc = std::atof(next().c_str()); + else if (a == "--seed") c.seed = static_cast(std::atoi(next().c_str())); + else if (a == "--no-header") header = false; + else if (a == "--width-scan") scan = true; + else if (a == "--help" || a == "-h") { + std::printf( + "usage: zi_paper --alpha A --mu M --delta D --sigma S --dp T [options]\n" + "\n" + "Simulates the Farmer/Patelli/Zovko (2005) zero-intelligence model on the\n" + "project's matching engine and compares the realised spread to Equation 1,\n" + " s_hat = (mu/alpha) * (0.28 + 1.86*eps^0.75), eps = delta*sigma/mu\n" + "\n" + "Parameters come from analysis/farmer2005.py, which measures them from\n" + "LOBSTER the way the paper measures them from the LSE. All of alpha, mu and\n" + "delta are per unit time; the spread prediction is invariant to the time\n" + "unit, since scaling all three leaves both eps and mu/alpha unchanged.\n" + "\n" + " --dp tick size IN LOG PRICE. This is the parameter Equation 1\n" + " assumes away by taking dp -> 0, so it is the point of the\n" + " whole exercise; dp/p_c is reported.\n" + " --events simulated events (default 2000000)\n" + " --width-pc half-width of the price range, in characteristic prices\n" + " --width-scan rerun at several widths. The semi-infinite intervals have\n" + " to be truncated somewhere, and if the answer moves with\n" + " the truncation then the boundary is setting the spread\n" + " rather than the order flow.\n"); + return 0; + } else { + std::fprintf(stderr, "unknown argument: %s\n", a.c_str()); + return 2; + } + } + + if (!scan) return run_one(c, header); + + std::printf("width scan -- the answer must not depend on where the interval is cut\n"); + bool first = true; + for (const double w : {12.5, 25.0, 50.0, 100.0, 200.0}) { + Config c2 = c; + c2.width_pc = w; + std::printf("width_pc = %6.1f ", w); + if (run_one(c2, first) != 0) return 2; + first = false; + } + return 0; +} From b6e0e216b80817ffb53669601196e998c61c7884 Mon Sep 17 00:00:00 2001 From: brianmmaina Date: Sun, 16 Aug 2026 12:22:44 -0700 Subject: [PATCH 2/2] readme: cut it down 368 lines to 315. The detail that got cut is in docs/ already, so the readme was repeating itself. Merged the benchmark commentary into two points instead of five, folded the crash-safety walkthrough into the paragraph that explains why the independent verifier is the part that matters, and collapsed the research section now that FARMER_2005.md carries the caveats. Test count was stale, 186 to 191. Gitignore __pycache__ and pdfs; the paper is not mine to redistribute. --- .gitignore | 7 ++ README.md | 281 ++++++++++++++++++++++------------------------------- 2 files changed, 121 insertions(+), 167 deletions(-) diff --git a/.gitignore b/.gitignore index cd242f8..d41b2f5 100644 --- a/.gitignore +++ b/.gitignore @@ -22,3 +22,10 @@ data/lobster/**/*.7z OME_*_PLAN.md *_PLAN.md notes/ + +# python bytecode +__pycache__/ +*.pyc + +# papers are not mine to redistribute +*.pdf diff --git a/README.md b/README.md index fbad8c6..3e5dd03 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ # Order Matching Machine A limit order book and matching engine in **C++17**, wrapped in a **networked -order gateway** with crash recovery. +order gateway** with crash recovery — then used to replicate a +[2005 PNAS paper](https://arxiv.org/abs/cond-mat/0309233) on market +microstructure. Clients connect over TCP, submit orders, and receive acks, fills and market data. A single matching thread owns the book; a write-ahead log makes the state @@ -12,7 +14,7 @@ survive being killed. No dependencies beyond GoogleTest. reimplementation of the book - Matching validated by differential replay against **269K NASDAQ LOBSTER messages** -- **186 tests**, green on clang and gcc, under ASan/UBSan and ThreadSanitizer +- **191 tests**, green on clang and gcc, under ASan/UBSan and ThreadSanitizer > **First C++ project.** I'm still learning the idioms and the tradeoffs below; > this is a learning sandbox, not battle-tested infrastructure. Feedback welcome. @@ -30,11 +32,10 @@ clients --TCP--> [network thread] --SPSC--> [matching thread] --> WAL ``` **The book is single-writer.** One thread owns every mutation, so there is no -lock anywhere near it. Concurrency lives entirely at the edges — network I/O and -market-data fanout — and the two SPSC queues are the seam. That is the whole -design in one sentence, and the rest follows from it: +lock anywhere near it. Concurrency lives entirely at the edges, and the two SPSC +queues are the seam. The rest follows from it: -- The network thread **never touches the book**. Not to check a price band, not +- The network thread **never touches the book** — not to check a price band, not to answer a cancel. Anything needing book state crosses the queue. - **Cancel-on-disconnect** travels the same queue as every other command, so a dropped connection needs no special path and no lock. @@ -49,7 +50,7 @@ design in one sentence, and the rest follows from it: ```bash cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build -ctest --test-dir build --output-on-failure # 186 tests +ctest --test-dir build --output-on-failure # 191 tests ./build/gateway --wal /tmp/ome.wal --port 9001 & python3 tools/smoke_client.py --port 9001 # send an order, print the ack @@ -69,23 +70,8 @@ Kill it and watch it come back: kill -9 $(pgrep -f 'build/gateway') ./build/gateway --wal /tmp/ome.wal --recover --port 9001 # recovered N commands, last seq N -# book digest after recovery: ... ``` -With periodic snapshots, recovery reads state rather than history: - -```bash -./build/gateway --wal /tmp/ome.wal --snapshot /tmp/ome.snap \ - --snapshot-every 100000 --port 9001 -# ... after a kill: -./build/gateway --wal /tmp/ome.wal --snapshot /tmp/ome.snap --recover --port 9001 -# restored 8020 orders from snapshot at seq 8020 -# recovered 1580 commands, last seq 9600 -``` - -The WAL is compacted behind each snapshot, so it holds the tail rather than the -whole history — 1,580 records instead of 9,600 above. - Needs network once on first configure (GoogleTest via FetchContent). Build type defaults to Release. Builds with `-Wall -Wextra -Werror`. @@ -93,9 +79,8 @@ defaults to Release. Builds with `-Wall -Wextra -Werror`. ## Measured results -Every number here comes from [`docs/BENCHMARK.md`](docs/BENCHMARK.md), which -states its own methodology and boundaries. Apple M4 Pro, Release, median of -repeated runs. +Apple M4 Pro, Release, median of repeated runs. Methodology and boundaries in +[`docs/BENCHMARK.md`](docs/BENCHMARK.md). | | | |---|---| @@ -103,14 +88,14 @@ repeated runs. | Order-to-ack p50 / p99 | 89 µs / 149 µs at 50,000 orders/sec, 10 clients | | Sustained throughput | **250,000 orders/sec** at 50 clients, nothing rejected | | WAL cost (`fsync`) | +11 µs p50, +52 µs p99 | -| WAL cost (`F_FULLFSYNC`) | 25× at the tail — see below | | Snapshot pause | 28 ms at 100,000 resting orders | -| Crash recovery | **50/50** WAL replay, **25/25** snapshot+tail, independent verifier | -| LOBSTER replay | ~270K messages parsed, applied and checked in ~0.5 s | +| Crash recovery | **50/50** WAL replay, **25/25** snapshot + tail | + +Two results worth stating outright: **Latency falls as load rises** — 185 µs at 100 orders/sec versus 45 µs at 5,000. Under load the matching thread's spin finds work and never parks; when -nearly idle every order pays a park-and-wake round trip. So "45 µs" is only +nearly idle every order pays a park-and-wake round trip. "45 µs" is only meaningful with its offered rate attached. **Media-level durability is expensive.** On macOS `fsync` returns once data @@ -125,143 +110,110 @@ quietly pick the weaker guarantee to look faster. ## Crash safety `tools/kill_test.sh` runs the gateway under load, `kill -9`s it at a random -moment, restarts with recovery, and checks the rebuilt book. **50/50 passing** -on WAL replay, and **25/25** on the snapshot+tail path -(`SNAPSHOT_EVERY=1500 tools/kill_test.sh 25`). - -The check that matters is not that recovery succeeds — it is what it is checked -*against*. `tools/wal_verify` rebuilds the book from the same log using a -separate implementation: `std::map` price levels, its own matching loop, no -shared code with `OrderBook`. Both must produce the same digest. Without that, -the test would be the engine agreeing with itself — a matching bug would produce -the same wrong book on both sides and pass. - -The harness was weaker than it looked at first. A deliberately broken verifier -passed 1/1, because the load generator quoted bids below asks and nothing ever -matched: both implementations agreed only that they could accumulate a book. It -now quotes crossing prices with varying sizes so orders partially fill and leave -real depth. With that fixed the broken verifier is caught immediately. +moment, restarts with recovery, and checks the rebuilt book. ```bash -tools/kill_test.sh 50 # WAL replay -SNAPSHOT_EVERY=1500 tools/kill_test.sh 25 # snapshot + tail +tools/kill_test.sh 50 # WAL replay 50/50 +SNAPSHOT_EVERY=1500 tools/kill_test.sh 25 # snapshot + tail 25/25 ``` +What matters is not that recovery succeeds but what it is checked *against*. +`tools/wal_verify` rebuilds the book from the same log using a separate +implementation — `std::map` price levels, its own matching loop, no shared code +with `OrderBook`. Both must produce the same digest. Otherwise the test is the +engine agreeing with itself, and a matching bug would produce the same wrong +book on both sides and pass. + +The harness was weaker than it looked. A deliberately broken verifier passed +1/1, because the load generator quoted bids below asks and nothing ever matched +— both implementations agreed only that they could accumulate a book. It now +quotes crossing prices with varying sizes, and the broken verifier is caught +immediately. + The snapshot path found a bug the unit tests could not: `truncate_before()` derived sequence numbers by counting from 1, which is right for a log that still -starts at 1 and wrong for one already truncated. The second truncation -renumbered every surviving record, producing a genuine sequence gap and a -gateway that correctly refused to start. It takes *two* truncations to see it, -so no single-truncation test would have. +starts at 1 and wrong for one already truncated. It takes *two* truncations to +see it. -**Scope, stated honestly.** `kill -9` destroys the process without letting it -flush, but it does not destroy the page cache, so everything already handed to -`write()` survives. This proves the recovery path, torn-tail handling, and the -append-before-apply ordering. It does **not** prove behavior under power loss, -which would additionally lose the group-commit window. +**Scope, stated honestly.** `kill -9` does not destroy the page cache, so +everything already handed to `write()` survives. This proves the recovery path, +torn-tail handling and append-before-apply ordering. It does **not** prove +behavior under power loss. --- -## How it works - -**Wire protocol** ([`docs/PROTOCOL.md`](docs/PROTOCOL.md)) — length-prefixed -binary, little-endian, `int64` tick prices, no floating point anywhere. -Serialization is field by field, never a struct `memcpy`: `sizeof(NewOrder)` is -24 on this build while its wire encoding is 22, and those two padding bytes are -uninitialized memory a struct copy would put on the wire. - -**Prices are integer ticks.** Not a style choice — durability asserts that a -recovered book's digest *equals* the original's, and a digest over doubles would -make that a float-equality claim dressed up as a guarantee. +## Research: replicating Farmer, Patelli & Zovko (2005) -**Write-ahead log** — the record is written before the command touches the book. -That ordering picks which failure mode a mid-command crash produces: -append-first leaves a record for a command that never reached the book, and -recovery replays it. Apply-first mutates the book for a command whose record -never landed, and nothing afterwards can detect it. +Full writeup and every caveat in [`docs/FARMER_2005.md`](docs/FARMER_2005.md). -**Recovery uses the live apply path** — the same function live traffic uses, not -a separate replay path. Otherwise "the recovered book is identical" would only -mean two implementations happen to agree. Verified by a 100-seed property test -over random command streams. - -**Risk checks live on the thread that owns the state they need.** Rate limiting -is on the network thread, so an over-limit client is refused before consuming -queue capacity. The price band is on the matching thread, because it is relative -to the last trade or the mid and only that thread may read the book. +[The paper](https://arxiv.org/abs/cond-mat/0309233) (PNAS 102(6):2254–2259) +predicts a stock's mean spread from its order flow alone, `ŝ = (μ/α)·f(ε)`, and +tests it **cross-sectionally** — across stocks, regressing `log s = A·log ŝ + B` +and asking whether A = 1. On 11 LSE stocks it gets A = 0.99 ± 0.10, R² = 0.96. ---- +[`analysis/farmer2005.py`](analysis/farmer2005.py) reruns that test on the five +LOBSTER symbols. **Both laws fail**, diffusion by four orders of magnitude — but +the failure is ordered by `dp/p_c`, the model's own nondimensional tick size, +which Equation 1 assumes away by taking `dp → 0`. A penny on a $27 stock is +seventeen times the characteristic price scale of its own order flow. -## Market data analysis - -[`analysis/stylized_facts.py`](analysis/stylized_facts.py) measures classic -microstructure facts directly from the LOBSTER message stream — no book -reconstruction, so none of it is affected by the replay accuracy gap. Results in -[`docs/STYLIZED_FACTS.md`](docs/STYLIZED_FACTS.md). - -Across five symbols: **91–96% of submitted orders are cancelled rather than -executed**, trade-sign autocorrelation runs **0.72–0.91 at lag 1** and decays -slowly, and trade-level returns show excess kurtosis of 12–24. - -Volatility clustering splits by tick constraint: clearly present in AAPL, AMZN -and GOOG at a 10-second horizon, and undetectable in INTC and MSFT — the two -low-priced names, pinned at a one-cent spread, where discrete bounce drowns the -signal. - -## Replicating Farmer, Patelli & Zovko (2005) - -[`analysis/farmer2005.py`](analysis/farmer2005.py) tests the two scaling laws -from [*The predictive power of zero intelligence in financial -markets*](https://arxiv.org/abs/cond-mat/0309233) (PNAS 102(6):2254–2259) -against the five LOBSTER symbols. Full writeup in -[`docs/FARMER_2005.md`](docs/FARMER_2005.md). - -The paper predicts a stock's mean spread and price diffusion rate from its -order flow alone — `ŝ = (μ/α)·f(ε)` — and tests it **cross-sectionally**, by -regressing `log s = A·log ŝ + B` and asking whether A = 1. - -Across all five symbols both laws fail, the diffusion rate by up to four orders -of magnitude. But the failure is not random: **the error is perfectly -rank-ordered by `dp/p_c`**, the model's own nondimensional tick size (ρ = 1.000, -exact p = 0.017 by enumerating all 120 permutations — a rank test rather than a -regression, because five points cannot support a regression). Equation 1 is -derived in the limit `dp → 0`, and INTC and MSFT sit at `dp/p_c` of 17 and 22 — -a penny on a $27 stock is seventeen times the characteristic price scale of its -own order flow. - -Inside the model's stated domain (`dp/p_c < 1`: GOOG, AAPL, AMZN) the spread -ratio is constant within a factor of 1.5, which is what the law predicts. So -the paper is not refuted here; it is confirmed in the weak sense five stocks -allow, and the tick constraint is what a naive replication would have -misreported as a refutation. - -That much is a correlation on five points, and `dp/p_c` is large for exactly -the two cheapest stocks — so plenty of other things could produce the same -ordering. [`tools/zi_paper.cpp`](tools/zi_paper.cpp) separates them by -simulating the paper's model on this project's matching engine at each stock's -measured parameters and its **real tick size**, where nothing about a cheap -stock is present except four flow parameters and `dp`: +That is a correlation on five points, and `dp/p_c` is large for exactly the two +cheapest stocks. [`tools/zi_paper.cpp`](tools/zi_paper.cpp) separates cause from +coincidence by simulating the paper's model on this project's matching engine at +each stock's measured parameters and its **real tick**, where nothing about a +cheap stock is present except four flow numbers and `dp`: | | dp/p_c | simulated ratio | real ratio | sim/real | |---|---:|---:|---:|---:| | GOOG | 0.21 | 0.66 | 4.36 | 0.15 | +| AAPL | 0.35 | 0.72 | 3.70 | 0.19 | | AMZN | 0.76 | 0.83 | 5.66 | 0.15 | | INTC | 17.30 | **32.23** | **39.75** | **0.81** | | MSFT | 22.03 | **40.87** | **50.35** | **0.81** | -**The tick alone reproduces 81% of the observed inflation**, the same fraction -for both constrained stocks, and the simulated inflation is perfectly -rank-ordered by `dp/p_c` (ρ = 1.000, p = 0.017). The remaining 19% is real and -is presumably what the model deletes — strategic quoting, hidden liquidity, -heterogeneous sizes. +**The tick alone reproduces 81% of the observed departure from the law** — the +same fraction for both constrained stocks — and the simulated inflation is +perfectly rank-ordered by `dp/p_c` (ρ = 1.000, exact p = 0.017). So the paper is +not refuted; ignoring its scope condition is what makes it look refuted. + +Two things this does not show, both kept in the writeup: the remaining 19% is +real, and the small-tick simulation runs ~25% *below* the mean-field prediction +rather than matching it. + +Earlier and narrower: [`docs/STYLIZED_FACTS.md`](docs/STYLIZED_FACTS.md) +measures cancel ratios, trade-sign autocorrelation and volatility clustering +straight from the message stream, and +[`docs/ZI_COMPARISON.md`](docs/ZI_COMPARISON.md) is a calibrated ZI baseline +that tests properties this paper never claims — and says so at the top. + +--- + +## Design decisions + +**Wire protocol** ([`docs/PROTOCOL.md`](docs/PROTOCOL.md)) — length-prefixed +binary, little-endian, `int64` tick prices, no floating point anywhere. +Serialization is field by field, never a struct `memcpy`: `sizeof(NewOrder)` is +24 on this build while its wire encoding is 22, and those two padding bytes are +uninitialized memory a struct copy would put on the wire. + +**Prices are integer ticks.** Durability asserts that a recovered book's digest +*equals* the original's, and a digest over doubles would make that a +float-equality claim dressed up as a guarantee. -It doubles as an independent exercise of the matching engine: a continuous -double auction driven for millions of events against an analytically known -answer, agreeing with it to within a constant wherever that answer applies. +**The WAL record is written before the command touches the book.** That ordering +picks which failure mode a mid-command crash produces: append-first leaves a +record for a command that never reached the book, and recovery replays it. +Apply-first mutates the book for a command whose record never landed, and +nothing afterwards can detect it. -The earlier stylized-facts comparison in -[`docs/ZI_COMPARISON.md`](docs/ZI_COMPARISON.md) tests properties this paper -never claims, and says so at the top. +**Recovery uses the live apply path**, not a separate replay path. Otherwise +"the recovered book is identical" would only mean two implementations happen to +agree. + +**Risk checks live on the thread that owns the state they need.** Rate limiting +is on the network thread, so an over-limit client is refused before consuming +queue capacity. The price band is on the matching thread, because it is relative +to the last trade or the mid and only that thread may read the book. --- @@ -278,14 +230,10 @@ loopback only. would get a second thread and a second book; a second thread on *one* book would need the locking this architecture exists to avoid. -**Benchmarks are loopback on one machine.** Not a network. The load generator -shares cores with the gateway, and at high client counts some of the tail is its -own scheduling. At 100 clients × 5,000 orders/sec the generator — not the -gateway — stops making progress, so that cell is blank rather than guessed. - -**The benchmark workload does not cross.** It measures the accept-and-acknowledge -path, not matching under contention. Published numbers were also taken with the -price band disabled and no subscriber attached. +**Benchmarks are loopback on one machine**, and the workload does not cross — it +measures the accept-and-acknowledge path, not matching under contention. At 100 +clients × 5,000 orders/sec the load generator, not the gateway, stops making +progress, so that cell is blank rather than guessed. **No self-trade prevention.** A session holding both sides will match itself. @@ -297,8 +245,7 @@ must not read an ack as "did not trade". **`BookUpdate` sequence numbers are not contiguous.** Conflation skips values by design; a client treating a gap as loss would flag healthy behavior as an error. - -**No delta encoding for market data** — every update is a full top-N snapshot. +No delta encoding either — every update is a full top-N snapshot. **LOBSTER replay accuracy is 70% at 1K events and 50% beyond.** Faithful reconstruction needs order-level state across the file window, and 21% of @@ -306,15 +253,16 @@ executions in this sample are against hidden liquidity that never appears in the book at all. **Snapshots pause the matching thread** — 28 ms at 100K resting orders. The fix -is measured and known (the copy is 2.2 ms while the write is 26 ms, so -serializing a copy on a background thread would remove most of it) and not yet +is measured and known (the copy is 2.2 ms while the write is 26 ms) and not yet implemented. Snapshots are off unless `--snapshot` is given. -**Cancel-on-disconnect is O(orders held).** A session disconnecting with a very -large book blocks the matching thread for the sweep. +**Cancel-on-disconnect is O(orders held)**, and **`poll()` is O(connections)** — +p50 goes 185 µs → 1,010 µs from 1 to 100 clients. `epoll`/`kqueue` are O(ready), +not yet worth the portability cost. -**`poll()` is O(connections).** p50 goes 185 µs → 1,010 µs from 1 to 100 -clients. `epoll`/`kqueue` are O(ready); not yet worth the portability cost. +**The replication is 5 stocks and 1 day**, against the paper's 11 stocks and 434 +days. At n = 5 the smallest attainable exact p-value *is* 0.017, so that result +is suggestive rather than established. --- @@ -327,9 +275,9 @@ clients. `epoll`/`kqueue` are O(ready); not yet worth the portability cost. | `include/ome/` | protocol, framing, sessions, WAL, snapshots, queues | | `src/net/tcp_server.cpp` | `poll()` event loop | | `bench/` | load generator and latency sweep | -| `tools/` | visualizer, live feed, WAL verifier, kill-test harness | -| `analysis/` | stylized facts from LOBSTER messages | -| `docs/` | protocol spec, benchmarks, measured market data facts | +| `tools/` | visualizer, live feed, WAL verifier, kill-test harness, ZI simulators | +| `analysis/` | LOBSTER measurement, calibration, the paper replication | +| `docs/` | protocol spec, benchmarks, market data facts, replication writeup | Sanitizer builds — `address`, `undefined`, `address,undefined`, or `thread` (thread and address are mutually exclusive): @@ -343,9 +291,10 @@ cmake --build build-asan && ctest --test-dir build-asan --output-on-failure ## LOBSTER data (optional) -Sample CSVs are **not** in git. Download a message + orderbook pair from +Sample CSVs are **not** in git. Download message + orderbook pairs from [LOBSTER samples](https://data.lobsterdata.com/info/DataSamples.php), place under `data/lobster/`, then see [`data/lobster/README.md`](data/lobster/README.md). +The replication needs all five symbols; everything else works with one. ```bash ./build/lobster_replay --messages --orderbook \ @@ -358,11 +307,9 @@ open tools/book_replay.html # load /tmp/replay.jsonl ## What I want to improve next - **LOBSTER parity** — order-level state across the file window, hidden - liquidity, cross and auction messages. Per-event golden checks rather than an - end snapshot. + liquidity, cross and auction messages. - **Copy-then-write snapshots**, to remove the matching-thread pause. - **A crossing benchmark**, to measure matching under contention rather than the accept path alone. -- **Synthetic order flow** calibrated to the LOBSTER data and run through the - same engine, to see which stylized facts survive when strategic behavior is - removed. +- **More trading days**, so the replication has error bars on the real side + rather than a single observation per stock.