research: intraday error bars and charts

[brianmmaina]

Two things: error bars on the real side, and charts.

LOBSTER's free sample is one trading day

The paper averages parameters over 434 days per stock, which is what gives its real side an error bar. LOBSTER's free samples are a single day (2012-06-21) per ticker, so that design is not reachable without a paid or institutional subscription.

Intraday blocks are the substitute that is reachable. --blocks 13 cuts the session into 30-minute blocks and re-measures everything inside each — scan() now accumulates per block in one pass rather than needing a file read per block.

The two regimes never overlap

	dp/p_c	ratio (full day)	block mean ± sd	block range
GOOG	0.21	4.36	4.39 ± 0.88	2.70 – 6.00
AAPL	0.35	3.70	3.60 ± 0.44	2.84 – 4.60
AMZN	0.76	5.66	6.13 ± 1.77	3.45 – 9.49
INTC	17.30	39.75	40.43 ± 5.54	28.48 – 49.89
MSFT	22.03	50.35	36.52 ± 10.96	25.32 – 55.25

Small-tick tops out at 9.49; tick-constrained bottoms out at 25.32. No overlap across any of the 65 block measurements — a stronger statement than the full-day ratios alone, because it survives being re-measured 13 times.

The ordering is positive in 13/13 blocks. The persistent ρ = 0.9 rather than 1.0 is the same GOOG/AAPL swap seen over the full day, which makes that swap a real feature of the sample rather than noise.

What blocks do not buy, stated where the numbers are

• They measure sampling variability, not day-to-day variability. Overnight gaps, news and regime shifts are invisible inside one session, so these error bars are a lower bound on the true ones.
• Blocks from one day are correlated, so "13 of 13" is a consistency check, not thirteen replications. It answers is the ordering an artifact of one measurement window? — not how many sigma is the effect?
• MSFT's block mean (36.52) sits well below its full-day ratio (50.35). Per-block δ is more heavily censored, since an order outliving its block is dropped rather than credited to a clock it never ran on. The full-session row stays the headline estimate and the doc says why.

Charts

analysis/plot_farmer2005.py → docs/farmer2005.html. Hand-built SVG in a self-contained page: no JavaScript, no external assets, matching the dependency rule and the book_replay.html precedent. Theme-aware, and the wide charts scroll inside their own container.

Three panels:

1. Spread ratio against dp/p_c, with the intraday error bars. The dp/p_c > 1 region is shaded so the scope condition is on the chart rather than in a caption, and the caption states that a correct law gives a horizontal line at any height — the prediction only holds up to a constant, so flatness is the claim, not proximity to 1.
2. The simulated ratio overlaid, from the paper's own model at each stock's measured parameters and real tick.
3. The width scan as the negative control, with the fixed-price-box bug written into the caption.

Verification

The refactor merged measure_all and measure_blocks onto one row builder so the two cannot drift. I diffed full-session output before and after: byte-identical, so the refactor is behaviour-preserving and the block work is purely additive.

Also verified the generated page has zero external resource references and every plotted point lands inside its viewBox.

[brianmmaina]

Added: a correction to my own strongest result

Auditing the diffusion finding before it went in a writeup turned up a units problem.

The paper measures diffusion on a different clock from the one its parameters live on. §A3 defines event time as order placements and cancellations, and μ, α, δ are all per event — so D̂ is per event. §A4 then says "here an event is anything that changes the midpoint price m" — so D is per midpoint change.

The ratio carries a hidden factor of events-per-midpoint-change. Harmless if that factor is roughly constant across the sample, which is presumably true of the paper's 11 LSE stocks. It runs 6 to 186 here:

	dp/p_c	ev/midchg	raw ratio	matched
GOOG	0.21	5.9	11.6	1.96
AAPL	0.35	6.0	12.6	2.11
AMZN	0.76	9.6	93.2	9.70
INTC	17.30	186.3	9,386	50.37
MSFT	22.03	158.5	20,629	130.15

A spread pinned at one tick is a spread whose midpoint rarely moves — so the mismatch inflates exactly the stocks the tick already inflates, and a raw comparison double counts the same effect.

Two claims in this repo were wrong

1. "Diffusion fails by four orders of magnitude" — matched, the spread of ratios is 66× rather than 1,777×, so closer to two. Corrected in README and FARMER_2005.md.
2. The dramatic negative slope was substantially an artifact — A moves from −0.612 (R² = 0.80) to +0.171 (R² = 0.39). The narrative holds, there was a missing control and it was dp/p_c, but that particular statistic was inflated.

The rank result does not move

ρ = 1.000, p = 0.0167 on both clocks.

That is the whole argument for having built on a rank test rather than the regression: it survived a factor-of-27 error in its own input. The regression did not.

Both clocks are now reported side by side rather than one silently replacing the other, since the raw column is the literal §A4 quantity and someone comparing against the paper needs to see it.