R6 — simulate the paper's model to test whether the tick is the mechanism

CMakeLists.txt

@@ -76,6 +76,9 @@ target_link_libraries(wal_verify PRIVATE order_book ome_warnings)
 add_executable(zi_sim tools/zi_sim.cpp)
 target_link_libraries(zi_sim PRIVATE order_book ome_warnings)
 
+add_executable(zi_paper tools/zi_paper.cpp)
+target_link_libraries(zi_paper PRIVATE order_book ome_warnings)
+
 add_executable(loadgen bench/loadgen.cpp)
 target_link_libraries(loadgen PRIVATE order_book ome_warnings Threads::Threads)
 

README.md

@@ -235,6 +235,30 @@ the paper is not refuted here; it is confirmed in the weak sense five stocks
 allow, and the tick constraint is what a naive replication would have
 misreported as a refutation.
 
+That much is a correlation on five points, and `dp/p_c` is large for exactly
+the two cheapest stocks — so plenty of other things could produce the same
+ordering. [`tools/zi_paper.cpp`](tools/zi_paper.cpp) separates them by
+simulating the paper's model on this project's matching engine at each stock's
+measured parameters and its **real tick size**, where nothing about a cheap
+stock is present except four flow parameters and `dp`:
+
+| | dp/p_c | simulated ratio | real ratio | sim/real |
+|---|---:|---:|---:|---:|
+| GOOG | 0.21 | 0.66 | 4.36 | 0.15 |
+| AMZN | 0.76 | 0.83 | 5.66 | 0.15 |
+| INTC | 17.30 | **32.23** | **39.75** | **0.81** |
+| MSFT | 22.03 | **40.87** | **50.35** | **0.81** |
+
+**The tick alone reproduces 81% of the observed inflation**, the same fraction
+for both constrained stocks, and the simulated inflation is perfectly
+rank-ordered by `dp/p_c` (ρ = 1.000, p = 0.017). The remaining 19% is real and
+is presumably what the model deletes — strategic quoting, hidden liquidity,
+heterogeneous sizes.
+
+It doubles as an independent exercise of the matching engine: a continuous
+double auction driven for millions of events against an analytically known
+answer, agreeing with it to within a constant wherever that answer applies.
+
 The earlier stylized-facts comparison in
 [`docs/ZI_COMPARISON.md`](docs/ZI_COMPARISON.md) tests properties this paper
 never claims, and says so at the top.

analysis/farmer2005.py

@@ -454,23 +454,23 @@ def report_regression(name, r, markdown=False):
         print("  confirmation.")
 
 
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--symbols", default="AAPL,AMZN,GOOG,INTC,MSFT")
-    ap.add_argument("--skip-open", type=float, default=0.0,
-                    help="seconds of the session to skip; the paper excludes the "
-                         "opening auction, which LOBSTER samples already omit")
-    ap.add_argument("--markdown", action="store_true")
-    args = ap.parse_args()
+def measure_all(symbols, skip_open=0.0, verbose=True):
+    """Measure every symbol. Shared with analysis/validate_scaling.py.
 
+    The simulator must run on exactly the parameters the empirical test used,
+    or a difference in the measurement would look like a difference in the
+    model. Same reason analysis/compare.py imports stylized_facts rather than
+    reimplementing it.
+    """
     rows = []
-    for sym in [s.strip().upper() for s in args.symbols.split(",") if s.strip()]:
+    for sym in symbols:
         m, b = find_pair(sym)
         if not m or not b:
             print(f"skipping {sym}: no message/orderbook pair", file=sys.stderr)
             continue
-        print(f"scanning {sym} ...", file=sys.stderr)
-        s = scan(sym, m, b, args.skip_open)
+        if verbose:
+            print(f"scanning {sym} ...", file=sys.stderr)
+        s = scan(sym, m, b, skip_open)
         p = parameters(s)
         eps, p_c, s_hat, d_hat = predict(p)
         s_real = (sum(s["spreads"]) / len(s["spreads"])) if s["spreads"] else float("nan")
@@ -494,13 +494,26 @@ def main():
                      "s_real": s_real, "d_real": d_real, "n_mid": len(s["mids"]),
                      "price": mm / 10000.0, "dp_log": dp_log,
                      "tick_ratio": tick_ratio})
+    rows.sort(key=lambda r: r["tick_ratio"])
+    return rows
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--symbols", default="AAPL,AMZN,GOOG,INTC,MSFT")
+    ap.add_argument("--skip-open", type=float, default=0.0,
+                    help="seconds of the session to skip; the paper excludes the "
+                         "opening auction, which LOBSTER samples already omit")
+    ap.add_argument("--markdown", action="store_true")
+    args = ap.parse_args()
+
+    rows = measure_all([x.strip().upper() for x in args.symbols.split(",") if x.strip()],
+                       args.skip_open)
 
     if not rows:
         print(f"no LOBSTER data under {DATA_ROOT}", file=sys.stderr)
         return 1
 
-    rows.sort(key=lambda r: r["tick_ratio"])
-
     print("\nmeasured model parameters (event time, log price, shares)")
     hdr = (f"{'sym':<6} {'events':>8} {'mu':>9} {'sigma':>8} {'alpha':>11} "
            f"{'delta':>9} {'eps':>8}")

analysis/validate_scaling.py

@@ -0,0 +1,166 @@
+#!/usr/bin/env python3
+"""Is the tick constraint a mechanism, or just a correlate?
+
+analysis/farmer2005.py found that the Farmer/Patelli/Zovko spread law fails
+across five LOBSTER stocks, and that the size of the failure is rank-ordered by
+the model's own nondimensional tick size dp/p_c -- the parameter Equation 1
+assumes away by taking dp -> 0.
+
+That is a correlation on five points, and it has an obvious alternative
+explanation. dp/p_c is large exactly for INTC and MSFT, which are also the two
+cheapest, highest-volume, most heavily quoted names in the sample. Any of a
+dozen things that distinguish a $30 stock from a $500 one would produce the
+same ordering.
+
+SIMULATION SEPARATES THOSE, because it can hold everything else fixed.
+
+Run the paper's own model -- tools/zi_paper, the austere specification, not the
+richer zi_sim -- at each stock's measured (alpha, mu, delta, sigma) and its real
+tick size. Nothing about a $30 stock is present except four flow parameters and
+dp. So if simulated INTC reproduces the inflated spread ratio that real INTC
+shows, the tick is doing the work.
+
+The prediction, made before running it:
+
+    small dp/p_c   simulated ratio near 1        (the mean field result holds)
+    large dp/p_c   simulated ratio near the EMPIRICAL ratio for that stock
+
+The second half is the risky one. It is easy to get an inflated spread out of a
+coarse grid; matching the size of the real inflation is not automatic, and a
+mismatch would say the tick explains the direction but not the magnitude.
+
+Parameters come from farmer2005.measure_all rather than being re-measured here,
+so a difference in measurement cannot masquerade as a difference in the model.
+
+Usage:
+    python3 analysis/validate_scaling.py
+    python3 analysis/validate_scaling.py --events 2000000 --seeds 5
+"""
+
+import argparse
+import math
+import os
+import statistics
+import subprocess
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import farmer2005 as f  # noqa: E402
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ZI_PAPER = os.path.join(ROOT, "build", "zi_paper")
+
+
+def simulate(row, events, seed, width_pc):
+    """One run of the paper's model at this stock's measured parameters."""
+    cmd = [ZI_PAPER,
+           "--alpha", f"{row['alpha']:.10g}",
+           "--mu", f"{row['mu']:.10g}",
+           "--delta", f"{row['delta']:.10g}",
+           "--sigma", f"{row['sigma']:.10g}",
+           "--dp", f"{row['dp_log']:.10g}",
+           "--events", str(events),
+           "--width-pc", str(width_pc),
+           "--seed", str(seed),
+           "--no-header"]
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    if r.returncode != 0:
+        print(f"  {row['symbol']} seed {seed} failed: {r.stderr.strip()}", file=sys.stderr)
+        return None
+    parts = r.stdout.split()
+    if len(parts) < 6:
+        return None
+    # eps dp/p_c half_tk s_hat s_sim ratio resting wide%
+    return {"s_sim": float(parts[4]), "ratio": float(parts[5]),
+            "resting": float(parts[6])}
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--symbols", default="AAPL,AMZN,GOOG,INTC,MSFT")
+    ap.add_argument("--events", type=int, default=1000000)
+    ap.add_argument("--seeds", type=int, default=5)
+    ap.add_argument("--width-pc", type=float, default=50.0)
+    args = ap.parse_args()
+
+    if not os.path.exists(ZI_PAPER):
+        print(f"{ZI_PAPER} not built. cmake --build build --target zi_paper",
+              file=sys.stderr)
+        return 1
+
+    syms = [s.strip().upper() for s in args.symbols.split(",") if s.strip()]
+    rows = f.measure_all(syms)
+    if not rows:
+        return 1
+
+    print(f"\nsimulating the paper's model at each stock's measured parameters")
+    print(f"{args.seeds} seeds x {args.events:,} events, width = {args.width_pc} p_c\n")
+
+    out = []
+    for r in rows:
+        sims = [simulate(r, args.events, s, args.width_pc)
+                for s in range(1, args.seeds + 1)]
+        sims = [s for s in sims if s]
+        if not sims:
+            continue
+        ratios = [s["ratio"] for s in sims]
+        emp = r["s_real"] / r["s_hat"] if r["s_hat"] else float("nan")
+        out.append({"symbol": r["symbol"], "tick_ratio": r["tick_ratio"],
+                    "sim_mean": statistics.mean(ratios),
+                    "sim_sd": statistics.stdev(ratios) if len(ratios) > 1 else 0.0,
+                    "emp": emp,
+                    "s_sim": statistics.mean(s["s_sim"] for s in sims),
+                    "s_real": r["s_real"], "s_hat": r["s_hat"]})
+        print(f"  {r['symbol']} done", file=sys.stderr)
+
+    hdr = (f"{'sym':<6} {'dp/p_c':>8} {'sim ratio':>16} {'real ratio':>11} "
+           f"{'sim/real':>9}")
+    print()
+    print(hdr)
+    print("-" * len(hdr))
+    for o in out:
+        agree = o["sim_mean"] / o["emp"] if o["emp"] else float("nan")
+        print(f"{o['symbol']:<6} {o['tick_ratio']:>8.2f} "
+              f"{o['sim_mean']:>10.2f} +/-{o['sim_sd']:>4.2f} {o['emp']:>11.2f} "
+              f"{agree:>9.2f}")
+
+    print("\n  sim ratio   simulated spread / s_hat, the paper's model at these parameters")
+    print("  real ratio  actual LOBSTER spread / s_hat, from farmer2005.py")
+    print("  sim/real    1.00 means the tick constraint alone accounts for the")
+    print("              whole empirical departure from the scaling law")
+
+    # Does the simulated inflation track the real one? With five points a rank
+    # test is the honest statistic, for the same reason as in farmer2005.py.
+    sp = f.spearman_exact([o["sim_mean"] for o in out], [o["emp"] for o in out])
+    if sp:
+        print(f"\n  simulated vs real inflation:  rho = {sp['rho']:+.3f}, "
+              f"exact p = {sp['p']:.4f} (n = {sp['n']})")
+    # The cleaner claim: inside the simulation, dp/p_c is the ONLY thing that
+    # varies besides the four flow parameters, so if the simulated inflation is
+    # ordered by it there is no confounder left to appeal to.
+    sp2 = f.spearman_exact([o["tick_ratio"] for o in out],
+                           [o["sim_mean"] for o in out])
+    if sp2:
+        print(f"  simulated inflation vs dp/p_c: rho = {sp2['rho']:+.3f}, "
+              f"exact p = {sp2['p']:.4f} (n = {sp2['n']})")
+
+    small = [o for o in out if o["tick_ratio"] < 1.0]
+    large = [o for o in out if o["tick_ratio"] >= 1.0]
+    if small:
+        m = statistics.mean(o["sim_mean"] for o in small)
+        print(f"\n  dp/p_c < 1  ({', '.join(o['symbol'] for o in small)}): "
+              f"simulated ratio {m:.2f}")
+        print("      near 1 means the mean field result describes the simulation,")
+        print("      so the engine and the law agree where the law claims to apply.")
+    if large:
+        m = statistics.mean(o["sim_mean"] for o in large)
+        me = statistics.mean(o["emp"] for o in large)
+        print(f"\n  dp/p_c >= 1 ({', '.join(o['symbol'] for o in large)}): "
+              f"simulated {m:.1f} vs real {me:.1f}")
+        print("      the tick alone, with no other property of a cheap stock present,")
+        print(f"      reproduces {100.0*m/me:.0f}% of the observed inflation.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

docs/FARMER_2005.md

@@ -171,6 +171,97 @@ nondimensional tick size predicts the size of its error.
 
 ---
 
+---
+
+## Is the tick a mechanism, or just a correlate?
+
+Everything above is a correlation on five points, and it has an obvious
+alternative explanation. `dp/p_c` is large for exactly INTC and MSFT, which are
+also the two cheapest, highest-volume, most heavily quoted names in the sample.
+A dozen things that distinguish a $30 stock from a $500 stock would produce the
+same ordering.
+
+Simulation separates them, because it can hold everything else fixed.
+[`tools/zi_paper.cpp`](../tools/zi_paper.cpp) implements the paper's model —
+the austere one: single order size, uniform deposition, equal rates, constant
+cancellation — on this project's own matching engine. Run it at each stock's
+measured `(α, μ, δ, σ)` and its **real tick size**, and nothing about a cheap
+stock is present except four flow parameters and `dp`.
+
+```bash
+cmake --build build --target zi_paper
+python3 analysis/validate_scaling.py --events 1000000 --seeds 5
+```
+
+The prediction, stated before running it: small `dp/p_c` should give a
+simulated ratio near 1, and large `dp/p_c` should give a ratio near the
+*empirical* ratio for that stock. The second half is the risky one — a coarse
+grid inflating the spread is easy, matching the *size* of the real inflation is
+not.
+
+| | dp/p_c | sim ratio | real ratio | sim/real |
+|---|---:|---:|---:|---:|
+| GOOG | 0.21 | 0.66 ± 0.01 | 4.36 | 0.15 |
+| AAPL | 0.35 | 0.72 ± 0.01 | 3.70 | 0.19 |
+| AMZN | 0.76 | 0.83 ± 0.01 | 5.66 | 0.15 |
+| INTC | 17.30 | **32.23** ± 0.00 | **39.75** | **0.81** |
+| MSFT | 22.03 | **40.87** ± 0.00 | **50.35** | **0.81** |
+
+**The tick alone reproduces 81% of the observed inflation** for both
+tick-constrained stocks — the same fraction for each, which is not something
+the run was tuned to produce.
+
+The simulated inflation is **perfectly rank-ordered by `dp/p_c`** (ρ = 1.000,
+exact p = 0.017). Inside the simulation that is a much stronger statement than
+the same number was on real data: `dp/p_c` and the four flow parameters are the
+*only* things that vary, so there is no confounder left to appeal to.
+
+Two things this does **not** show:
+
+* **The remaining 19% is real.** The tick is the dominant term, not the whole
+  story. What is left is presumably the things the model deletes — strategic
+  quoting, hidden liquidity, heterogeneous order sizes.
+* **The small-tick ratio is 0.66–0.83, not 1.0.** The simulation runs about 25%
+  *below* the mean field prediction, consistently. `f(ε)` is itself an
+  approximation, so a systematic gap of this size is unremarkable, but it is a
+  gap and it is not being rounded to "agreement".
+
+The useful by-product: this is an independent exercise of the matching engine —
+a continuous double auction driven for millions of events with an answer known
+analytically — and it agrees with that answer to within a constant wherever the
+analytic result claims to apply.
+
+### The width scan is the negative control
+
+The paper's deposition intervals are semi-infinite and a simulation must
+truncate them. If the answer moves when the truncation moves, the boundary is
+setting the spread rather than the order flow, and the whole result is an
+artifact:
+
+```bash
+./build/zi_paper --alpha ... --dp ... --width-scan
+```
+
+GOOG returns 0.66, 0.68, 0.65, 0.69, 0.70 across a 16× range of widths; INTC
+returns 32.23 at every width.
+
+This was not a formality. The first implementation truncated to a **fixed price
+box** centred at zero, and the scan produced 32.23, 32.23, 32.23, **0.00**,
+32.23 — non-monotonic in the width, which is the signature of a bug rather than
+a boundary effect. The book could pin its best bid against the top of the box,
+at which point the sell interval `[b+1, box_top]` was empty and no sell order
+could ever arrive again: an absorbing one-sided state that silently reported a
+spread of zero. The same fixed box also made the buy and sell arrival rates
+depend on where the price sat inside it, quietly breaking the model's
+equal-rates assumption.
+
+Anchoring each interval to the opposing best quote — buys on `[a(t)−W, a(t)−1]`,
+sells on `[b(t)+1, b(t)+W]` — fixes both: constant equal widths, and prices free
+to wander. Without the scan, the fixed-box version would have produced a
+plausible-looking table.
+
+---
+
 ## Limits — read before quoting any number here
 
 * **5 stocks, not 11. One day, not 434.** The paper averages parameters over