From 86c09e271e6f1f52851a5f7abf9f54866f8da27a Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Sat, 5 Sep 2026 12:56:12 +0000 Subject: [PATCH] docs: archive completed market-data planning docs The root CLAUDE.md documents that market data details live in planning/archive/, but the finished docs (MARKET_DATA_DESIGN, MARKET_DATA_REVIEW, MARKET_INTERFACE, MARKET_SIMULATOR, MASSIVE_API) were duplicated at the planning/ top level while archive/ held stale pre-implementation drafts. Move the final versions into archive/, replacing the drafts, so planning/ only surfaces PLAN.md and MARKET_DATA_SUMMARY.md as living docs, matching the documented layout. Fixes #7 Co-authored-by: GBRCenter <225887058+GBRCenter@users.noreply.github.com> --- planning/MARKET_DATA_DESIGN.md | 1478 ---------------- planning/MARKET_DATA_REVIEW.md | 242 --- planning/MARKET_DATA_SUMMARY.md | 2 +- planning/MARKET_INTERFACE.md | 439 ----- planning/MARKET_SIMULATOR.md | 456 ----- planning/MASSIVE_API.md | 525 ------ planning/archive/MARKET_DATA_DESIGN.md | 2218 ++++++++++++------------ planning/archive/MARKET_DATA_REVIEW.md | 313 ++-- planning/archive/MARKET_INTERFACE.md | 596 ++++--- planning/archive/MARKET_SIMULATOR.md | 611 ++++--- planning/archive/MASSIVE_API.md | 636 +++++-- 11 files changed, 2542 insertions(+), 4974 deletions(-) delete mode 100644 planning/MARKET_DATA_DESIGN.md delete mode 100644 planning/MARKET_DATA_REVIEW.md delete mode 100644 planning/MARKET_INTERFACE.md delete mode 100644 planning/MARKET_SIMULATOR.md delete mode 100644 planning/MASSIVE_API.md diff --git a/planning/MARKET_DATA_DESIGN.md b/planning/MARKET_DATA_DESIGN.md deleted file mode 100644 index d7c962e..0000000 --- a/planning/MARKET_DATA_DESIGN.md +++ /dev/null @@ -1,1478 +0,0 @@ -# MARKET_DATA_DESIGN.md — Market Data Backend, Detailed Design - -The implementation-level design for FinAlly's market data subsystem: one unified API, two -interchangeable sources (GBM simulator and the Massive REST API), a shared in-memory cache, -and the SSE stream that carries prices to the browser. - -**Audience:** the agent (or human) implementing or extending `backend/app/market/`. -This document is meant to be read once, top to bottom, and then implemented from — every -snippet below is either the code that ships today or the code that should ship. - -**Companion documents.** `PLAN.md` §6 is the frozen contract; `MARKET_INTERFACE.md`, -`MARKET_SIMULATOR.md`, and `MASSIVE_API.md` are the reference material this design draws on. -Where they disagree with this document, this document is the design and they are the background. - ---- - -## 0. Status — what exists, what is missing - -Verified by running the suite in `backend/` on 2026-09-01: - -``` -73 passed in 1.77s TOTAL coverage 91% -app/market/cache.py 100% -app/market/models.py 100% -app/market/simulator.py 98% -app/market/massive_client.py 94% <- high coverage, two live defects (§8.4) -app/market/stream.py 33% <- the SSE generator is effectively untested -``` - -| Piece | State | Section | -|---|---|---| -| `PriceUpdate` wire model | Ships, frozen contract | §4 | -| `PriceCache` (latest price + version) | Ships | §5.1 | -| `PriceCache` rolling history | **Missing** | §5.2 | -| `MarketDataSource` ABC | Ships | §6 | -| `SimulatorDataSource` + `GBMSimulator` | Ships | §7 | -| `MassiveDataSource` | Ships but **writes nothing to the cache** | §8.4 | -| `create_market_data_source` | Ships | §9 | -| SSE `/api/stream/prices` | Ships | §10.1 | -| SSE keepalive | **Missing** | §10.2 | -| `GET /api/prices/{ticker}/history` | **Missing** | §11 | -| Lifespan wiring + tracked-set reconciliation | **Missing** | §12 | - -Four gaps, all backend, all small. §15 orders them. - ---- - -## 1. The shape of the design - -Two sources with nothing in common — a 500ms in-process computation and a 15-second blocking -HTTP poll — must be interchangeable to everything downstream. The design achieves that with -**one indirection and one shared buffer**: - -``` - writes reads - ┌──────────────────┐ ┌────────────┐ ┌──────────────────────┐ - │ SimulatorSource │───┐ │ │───────────────│ SSE /api/stream │ - │ (500ms step) │ ├───▶│ PriceCache │───────────────│ Portfolio valuation │ - ├──────────────────┤ │ │ (in-mem, │───────────────│ Trade execution │ - │ MassiveSource │───┘ │thread-safe)│───────────────│ Snapshot task │ - │ (15s poll) │ │ │───────────────│ /api/prices/history │ - └──────────────────┘ └────────────┘ └──────────────────────┘ - MarketDataSource - (abstract interface) -``` - -**The one invariant that makes this work: nothing downstream ever asks a source for a price.** -Sources are write-only from the application's point of view; readers only ever touch the cache. -That is why a 30× difference in update cadence is invisible to the rest of the app, and why a -`get_price()` on the interface would be a design error — under Massive it would turn every -portfolio valuation into a billed HTTP request. - -### File structure - -``` -backend/app/market/ -├── __init__.py # public exports -├── models.py # PriceUpdate — the unit of data -├── cache.py # PriceCache — latest price + version + rolling history -├── interface.py # MarketDataSource — the ABC -├── seed_prices.py # simulator constants, no logic -├── simulator.py # GBMSimulator (pure) + SimulatorDataSource (async) -├── massive_client.py # MassiveDataSource -├── factory.py # create_market_data_source -└── stream.py # SSE router + history router -``` - -Public surface, unchanged by this design: - -```python -from app.market import ( - PriceUpdate, - PriceCache, - MarketDataSource, - create_market_data_source, - create_stream_router, -) -``` - ---- - -## 2. Vocabulary - -| Term | Meaning | -|---|---| -| **tick** | One simulator step (500ms) or one Massive poll (15s) | -| **tracked set** | `watchlist ∪ {tickers with a non-zero position}` — §12.2 | -| **version** | Monotonic counter on `PriceCache`, bumped on every write; the SSE change signal | -| **seeding** | Writing an initial price into the cache so a ticker never renders as `—` unnecessarily | - ---- - -## 3. Non-negotiable contracts - -These are frozen because the frontend and the shipped module already depend on them. Everything -else in this document is open to reasonable change. - -1. **SSE payload is a map keyed by ticker, one event carries every ticker.** Not one event per ticker. -2. **`timestamp` is Unix epoch seconds as a float.** Never ISO, never milliseconds. The frontend - multiplies by 1000 for `Date`. -3. **`change_percent` is already in percent units.** `0.021` means 0.021%. This deliberately - differs from REST responses elsewhere in the API, where percentages are fractions - (`PLAN.md` §8). The inconsistency is real and preserved. -4. **A connecting client gets a full snapshot immediately**, including after a reconnect, - because a fresh generator starts at `last_version = -1`. -5. **Tickers are uppercase everywhere**, normalized at the API boundary. - ---- - -## 4. `PriceUpdate` — the unit of data - -`backend/app/market/models.py`. Immutable, frozen, slotted. Both sources produce it; every -reader consumes it. - -```python -@dataclass(frozen=True, slots=True) -class PriceUpdate: - """Immutable snapshot of a single ticker's price at a point in time.""" - - ticker: str - price: float - previous_price: float - timestamp: float = field(default_factory=time.time) # Unix epoch SECONDS - - @property - def change(self) -> float: - return round(self.price - self.previous_price, 4) - - @property - def change_percent(self) -> float: - if self.previous_price == 0: - return 0.0 - return round((self.price - self.previous_price) / self.previous_price * 100, 4) - - @property - def direction(self) -> str: - if self.price > self.previous_price: - return "up" - elif self.price < self.previous_price: - return "down" - return "flat" - - def to_dict(self) -> dict: - return { - "ticker": self.ticker, - "price": self.price, - "previous_price": self.previous_price, - "timestamp": self.timestamp, - "change": self.change, - "change_percent": self.change_percent, - "direction": self.direction, - } -``` - -**`change`, `change_percent`, and `direction` are computed properties, not stored fields.** -They cannot drift out of sync with the prices they describe, and `to_dict()` cannot emit a -`direction` that contradicts its own `price`/`previous_price` pair. - -**`previous_price` means the price at the previous update**, not the previous session's close. -On the first update for a ticker it equals `price`, so `direction` is `"flat"` and `change` is -`0.0` — a newly added ticker never flashes green or red on its first tick. - -Example of the exact wire shape a client sees: - -```json -{ - "ticker": "AAPL", - "price": 190.52, - "previous_price": 190.48, - "timestamp": 1755873791.482, - "change": 0.04, - "change_percent": 0.021, - "direction": "up" -} -``` - ---- - -## 5. `PriceCache` — the shared buffer - -`backend/app/market/cache.py`. - -### 5.1 What ships today - -```python -class PriceCache: - def update(self, ticker: str, price: float, timestamp: float | None = None) -> PriceUpdate - def get(self, ticker: str) -> PriceUpdate | None - def get_all(self) -> dict[str, PriceUpdate] # shallow copy - def get_price(self, ticker: str) -> float | None - def remove(self, ticker: str) -> None - @property - def version(self) -> int - def __len__(self) -> int - def __contains__(self, ticker: str) -> bool -``` - -Three design points that matter: - -**`update()` derives `previous_price` itself.** Callers pass only the new price; the cache looks -up what it held and constructs the `PriceUpdate`. Neither source tracks prior state for the -purpose of computing a delta, so the two cannot implement it differently. - -```python -def update(self, ticker: str, price: float, timestamp: float | None = None) -> PriceUpdate: - with self._lock: - ts = timestamp or time.time() - prev = self._prices.get(ticker) - previous_price = prev.price if prev else price - - update = PriceUpdate( - ticker=ticker, - price=round(price, 2), - previous_price=round(previous_price, 2), - timestamp=ts, - ) - self._prices[ticker] = update - self._version += 1 - return update -``` - -**A `threading.Lock`, not an `asyncio.Lock`.** `MassiveDataSource` writes from an -`asyncio.to_thread` worker, so a genuine cross-thread lock is required. The critical sections -are a few dict operations; contention is irrelevant. - -**`version` is the SSE change-detection mechanism.** The stream compares an integer every 500ms -rather than diffing price maps. `get_all()` returns a shallow copy, and since `PriceUpdate` is -frozen, that copy is effectively deep and safe to iterate outside the lock. - -### 5.2 Rolling price history — to implement - -`PLAN.md` §6 requires the main chart to be populated the instant a ticker is clicked, rather -than drawing itself from scratch over the following minute. `PriceCache` gains a bounded -per-ticker deque of `(timestamp, price)`. - -```python -from collections import deque - -HISTORY_MAXLEN = 600 # ~5 minutes at the 500ms simulator cadence -``` - -Constructor: - -```python -def __init__(self, history_maxlen: int = HISTORY_MAXLEN) -> None: - self._prices: dict[str, PriceUpdate] = {} - self._history: dict[str, deque[tuple[float, float]]] = {} - self._history_maxlen = history_maxlen - self._lock = Lock() - self._version: int = 0 -``` - -Appended inside `update()`, under the same lock, immediately after the price is stored: - -```python - self._prices[ticker] = update - history = self._history.get(ticker) - if history is None: - history = deque(maxlen=self._history_maxlen) - self._history[ticker] = history - history.append((ts, update.price)) - self._version += 1 - return update -``` - -`remove()` must drop the deque too, or removed tickers leak memory and a re-added ticker -resurrects a stale chart: - -```python -def remove(self, ticker: str) -> None: - with self._lock: - self._prices.pop(ticker, None) - self._history.pop(ticker, None) -``` - -The reader, backing `GET /api/prices/{ticker}/history`: - -```python -def get_history(self, ticker: str, limit: int = HISTORY_MAXLEN) -> list[tuple[float, float]]: - """Oldest-first (timestamp, price) points. Empty list for an untracked ticker.""" - with self._lock: - points = self._history.get(ticker) - if not points: - return [] - return list(points)[-limit:] -``` - -Four properties worth stating explicitly: - -- **`deque(maxlen=600)` evicts the oldest point automatically** — there is no pruning logic to - write, and no unbounded growth to worry about. -- **An untracked ticker returns `[]`, not a 404.** The chart draws nothing rather than erroring - (`PLAN.md` §8). -- **Deliberately not persisted.** A restart clears it, which is the honest behavior for a - simulator whose prices also reset to seed on restart. -- **Memory is negligible**: 600 points × 50 tickers × ~16 bytes ≈ 500KB. - -Under Massive the deque fills at one point per 15-second poll, so five minutes of wall time is -20 points rather than 600. The chart is sparse but correct. Backfilling from `get_aggs` -(`MASSIVE_API.md` §5) is the eventual upgrade and is out of scope here. - ---- - -## 6. `MarketDataSource` — the abstract contract - -`backend/app/market/interface.py`. - -```python -class MarketDataSource(ABC): - @abstractmethod - async def start(self, tickers: list[str]) -> None: ... - @abstractmethod - async def stop(self) -> None: ... - @abstractmethod - async def add_ticker(self, ticker: str) -> None: ... - @abstractmethod - async def remove_ticker(self, ticker: str) -> None: ... - @abstractmethod - def get_tickers(self) -> list[str]: ... -``` - -Five methods, and every one is about **lifecycle and membership** — none returns a price. That -absence is the whole design (§1). - -### Behavioral contract - -Binding on both implementations. A test suite that passes against one should pass against the other. - -| Method | Guarantee | -|---|---| -| `start(tickers)` | Begins a background task writing to the cache. **Seeds the cache before returning**, so the first SSE event is never empty. Called exactly once; calling twice is undefined. | -| `stop()` | Cancels the task and releases resources. **Idempotent.** No writes to the cache afterwards. | -| `add_ticker(t)` | Adds to the tracked set. No-op if present. Simulator seeds a price immediately; Massive picks it up on the next poll. | -| `remove_ticker(t)` | Removes from the tracked set **and from the cache** (price and history). No-op if absent. | -| `get_tickers()` | Current tracked set. Synchronous — reads local state only. | - -Two asymmetries are permitted and must not be papered over: - -- **Seeding latency.** `add_ticker` on the simulator makes a price available immediately; on - Massive it takes up to one poll interval. The API contract already accommodates this — - `GET /api/watchlist` returns `price: null` until the first tick, and the UI shows `—`. -- **Cadence.** 500ms versus 15s. Readers must never assume a minimum update rate. This is - exactly what the SSE keepalive in §10.2 exists to handle. - -### `remove_ticker` is destructive — and that is the trap - -Both implementations call `self._cache.remove(ticker)`. Correct for the interface, but it means -removing a ticker whose position is still held silently freezes that position's valuation, P&L, -heatmap tile, and snapshot contribution. §12.2 is the rule that prevents it, and it is the single -most important piece of integration logic in this module because the failure mode is a wrong -number, not an error. - -### Adding a third source - -1. Subclass `MarketDataSource` and implement all five methods. -2. `start()` must **seed the cache before returning**. -3. Never write to the cache after `stop()`; make `stop()` idempotent. -4. `remove_ticker()` must call `cache.remove(ticker)`. -5. Convert timestamps to **Unix epoch seconds as a float** at the boundary. -6. Never let a fetch error kill the background loop — log and retry next cycle. -7. If the underlying client is synchronous, wrap **every** call in `asyncio.to_thread`. -8. Add a branch to `create_market_data_source` and a value to `market_source` in `/api/health`. - -Point 7 is not optional: a blocking HTTP call inside `async def` stalls the event loop for the -whole round trip, which stops the SSE stream and every in-flight request. - ---- - -## 7. The simulator — default source - -`backend/app/market/simulator.py` and `seed_prices.py`. Two classes with a clean split: -**`GBMSimulator` is pure and synchronous; `SimulatorDataSource` owns the async lifecycle and -the cache.** - -``` -┌──────────────────────────────────────────────────────────┐ -│ SimulatorDataSource(MarketDataSource) │ -│ owns the asyncio task, writes to PriceCache │ -│ start / stop / add_ticker / remove_ticker / get_tickers│ -│ │ │ -│ ▼ │ -│ GBMSimulator │ -│ pure math, no I/O, no async, no cache reference │ -│ step() -> {ticker: price} │ -└──────────────────────────────────────────────────────────┘ - │ - ▼ - seed_prices.py (constants only) -``` - -The separation pays off in testing: `GBMSimulator` needs no event loop, no cache, and no mocks. - -### 7.1 The model - -``` -S(t + dt) = S(t) · exp( (μ − σ²/2)·dt + σ·√dt·Z ) -``` - -Three properties earn GBM its place: - -**Prices cannot go negative.** The update is multiplicative — `exp(...)` is always positive. -No clamping, no `max(price, 0.01)` guard, no special case. An additive random walk needs all three. - -**Returns scale correctly with time.** σ is annualized; `√dt` converts it to the tick. The 500ms -cadence is a display choice, not a modelling parameter. - -**The `−σ²/2` term keeps the drift honest.** Without it, μ is not the expected return of the -price — a log-normal artefact. It costs one subtraction and makes the parameters mean what they say. - -### 7.2 Sizing `dt` - -`dt` is expressed against a **trading** year, not a calendar year. Markets are closed most of the -time; using 365×24h would understate per-tick moves by ~4.5×. - -```python -TRADING_SECONDS_PER_YEAR = 252 * 6.5 * 3600 # 5,896,800 -DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR # ~8.479e-8, sqrt(dt) = 2.912e-4 -``` - -What that produces per tick at the seed prices: - -| Ticker | σ | Per-tick σ | Per-tick $ | Per-minute $ (120 ticks) | -|---|---|---|---|---| -| AAPL | 0.22 | 0.0064% | $0.012 | $0.13 | -| JPM | 0.18 | 0.0052% | $0.010 | $0.11 | -| NVDA | 0.40 | 0.0116% | $0.093 | $1.02 | -| TSLA | 0.50 | 0.0146% | $0.036 | $0.40 | - -This is the number that decides whether the simulation looks right. A cent or two per tick on a -$200 stock means the price **rounds to a genuinely different value most ticks**, so the UI flashes -constantly, while a minute of drift stays in the tens of cents — what a real quote screen looks -like. Larger reads as a crash; smaller looks frozen. - -### 7.3 Correlation via Cholesky - -Independent draws would show tech stocks moving in opposite directions half the time. The eye -notices immediately. Standard fix: draw `n` independent normals, multiply by the Cholesky factor -`L` of the correlation matrix `C = L·Lᵀ`. - -Constants live in `seed_prices.py`, not in the simulator: - -```python -CORRELATION_GROUPS = { - "tech": {"AAPL", "GOOGL", "MSFT", "AMZN", "META", "NVDA", "NFLX"}, - "finance": {"JPM", "V"}, -} - -INTRA_TECH_CORR = 0.6 # tech stocks move together -INTRA_FINANCE_CORR = 0.5 # finance stocks move together -CROSS_GROUP_CORR = 0.3 # between sectors, and for unknown tickers -TSLA_CORR = 0.3 # TSLA does its own thing -``` - -Resolved pairwise, first match winning: - -```python -@staticmethod -def _pairwise_correlation(t1: str, t2: str) -> float: - tech = CORRELATION_GROUPS["tech"] - finance = CORRELATION_GROUPS["finance"] - - # TSLA is in the tech set but behaves independently - if t1 == "TSLA" or t2 == "TSLA": - return TSLA_CORR - - if t1 in tech and t2 in tech: - return INTRA_TECH_CORR - if t1 in finance and t2 in finance: - return INTRA_FINANCE_CORR - - return CROSS_GROUP_CORR -``` - -The TSLA clause is checked first on purpose: TSLA is a tech-set member for every other purpose, -but a demo where TSLA visibly decouples from the pack is more convincing than one where everything -moves in lockstep. `CROSS_GROUP_CORR` doubles as the default for any unknown symbol, which is what -makes §7.5 work. - -```python -def _rebuild_cholesky(self) -> None: - n = len(self._tickers) - if n <= 1: - self._cholesky = None # a single ticker needs no correlation - return - - corr = np.eye(n) - for i in range(n): - for j in range(i + 1, n): - rho = self._pairwise_correlation(self._tickers[i], self._tickers[j]) - corr[i, j] = rho - corr[j, i] = rho - - self._cholesky = np.linalg.cholesky(corr) -``` - -Rebuilt on every add and remove — `O(n²)` to build plus `O(n³)` to factor, on `n < 50`. That is -microseconds, and watchlist edits are human-speed, so caching it would be complexity without benefit. - -> **Known risk.** `np.linalg.cholesky` raises `LinAlgError` on a matrix that is not positive -> definite, and the call is unguarded. The current block structure (0.6 / 0.5 / 0.3) was verified -> positive definite at 7, 20, and 40 tickers — but raising `INTRA_TECH_CORR` toward 1.0, or adding -> a group whose intra-group correlation is *below* the cross-group value, can break -> positive-definiteness and take down `add_ticker`. Anyone editing these constants must re-run the -> test in §14.2. - -### 7.4 The tick - -`step()` is the hot path — every 500ms, for every ticker. - -```python -def step(self) -> dict[str, float]: - """Advance all tickers by one time step. Returns {ticker: new_price}.""" - n = len(self._tickers) - if n == 0: - return {} - - z_independent = np.random.standard_normal(n) - if self._cholesky is not None: - z_correlated = self._cholesky @ z_independent - else: - z_correlated = z_independent - - result: dict[str, float] = {} - for i, ticker in enumerate(self._tickers): - params = self._params[ticker] - mu, sigma = params["mu"], params["sigma"] - - drift = (mu - 0.5 * sigma**2) * self._dt - diffusion = sigma * math.sqrt(self._dt) * z_correlated[i] - self._prices[ticker] *= math.exp(drift + diffusion) - - if random.random() < self._event_prob: - shock_magnitude = random.uniform(0.02, 0.05) - shock_sign = random.choice([-1, 1]) - self._prices[ticker] *= 1 + shock_magnitude * shock_sign - - result[ticker] = round(self._prices[ticker], 2) - - return result -``` - -Two details worth pointing out: - -**Full precision is kept internally; only the returned value is rounded.** Rounding the stored -state would accumulate quantization error into a slow systematic drift over thousands of ticks. - -**One `standard_normal(n)` call per tick, not `n` calls.** A single vectorized draw feeding one -matrix multiply is why this stays negligible at 500ms. - -**Random events** fire at `event_probability = 0.001` per ticker per tick. With 10 tickers at -2 ticks/second the expected wait is `1 / (10 × 2 × 0.001) = 50 seconds` — frequent enough that -something happens during a demo, rare enough that the series is not pure noise. The shock -multiplies the price directly rather than feeding through GBM, so it is a genuine discontinuity — -a gap, which is what real news does to a stock. - -`_tickers` is an **ordered list** that indexes into the Cholesky matrix: row `i` corresponds to -`_tickers[i]`. That is why add and remove must both rebuild. `__init__` adds every ticker via -`_add_ticker_internal` and rebuilds **once** at the end — `O(n³)` instead of `O(n⁴)` on startup. - -### 7.5 Unknown tickers - -Any symbol passing the API-level pattern `^[A-Z][A-Z.]{0,5}$` works, with no allowlist. The AI -chat can add anything the user names, and it behaves plausibly. - -```python -def _add_ticker_internal(self, ticker: str) -> None: - if ticker in self._prices: - return - self._tickers.append(ticker) - self._prices[ticker] = SEED_PRICES.get(ticker, random.uniform(50.0, 300.0)) - self._params[ticker] = TICKER_PARAMS.get(ticker, dict(DEFAULT_PARAMS)) -``` - -- **Price**: `SEED_PRICES`, else uniform $50–$300 — where most large-cap US equities trade. -- **Parameters**: `TICKER_PARAMS`, else `DEFAULT_PARAMS` (σ=0.25, μ=0.05) — a mid-range large cap. -- **Correlation**: no sector membership, so `CROSS_GROUP_CORR` (0.3) against everything. - -`dict(DEFAULT_PARAMS)` **copies** rather than sharing the module-level dict. Without the copy, -tuning one unknown ticker's σ would mutate the default for every unknown ticker at once. - -This is a real advantage over the Massive path, where an unknown symbol never produces a price -and sits at `—` forever (§8.5). - -### 7.6 `SimulatorDataSource` — the async wrapper - -```python -class SimulatorDataSource(MarketDataSource): - def __init__(self, price_cache, update_interval=0.5, event_probability=0.001): ... - - async def start(self, tickers: list[str]) -> None: - self._sim = GBMSimulator(tickers=tickers, event_probability=self._event_prob) - # Seed the cache so the first SSE event carries real prices - for ticker in tickers: - price = self._sim.get_price(ticker) - if price is not None: - self._cache.update(ticker=ticker, price=price) - self._task = asyncio.create_task(self._run_loop(), name="simulator-loop") - - async def _run_loop(self) -> None: - while True: - try: - if self._sim: - for ticker, price in self._sim.step().items(): - self._cache.update(ticker=ticker, price=price) - except Exception: - logger.exception("Simulator step failed") - await asyncio.sleep(self._interval) -``` - -Three deliberate choices: - -**Seed the cache in `start()` before creating the task.** The first SSE event then carries real -prices rather than an empty object, so the watchlist never renders as ten dashes on load. - -**`add_ticker` seeds immediately.** The new ticker has a price on the very next SSE event, with no -wait for the following step — the reason adding a ticker feels instant. - -**The `try` is inside the loop, around the step.** An exception logs and the loop continues on the -next interval. Wrapping the loop instead would let one bad tick kill the feed permanently. This is -the one place defensive handling is warranted: a background task has no caller to propagate to, -and a dead price feed is a dead app. - -`stop()` cancels the task, awaits it, and swallows `CancelledError` — the normal shutdown path, -not an error. - -### 7.7 Parameters - -`seed_prices.py` holds constants only. Prices are realistic as of project creation; σ and μ are annualized. - -| Ticker | Seed | σ | μ | Note | -|---|---|---|---|---| -| AAPL | $190 | 0.22 | 0.05 | | -| GOOGL | $175 | 0.25 | 0.05 | | -| MSFT | $420 | 0.20 | 0.05 | | -| AMZN | $185 | 0.28 | 0.05 | | -| TSLA | $250 | 0.50 | 0.03 | High volatility, decorrelated | -| NVDA | $800 | 0.40 | 0.08 | High volatility, strong drift | -| META | $500 | 0.30 | 0.05 | | -| JPM | $195 | 0.18 | 0.04 | Low volatility (bank) | -| V | $280 | 0.17 | 0.04 | Low volatility (payments) | -| NFLX | $600 | 0.35 | 0.05 | | -| *unknown* | $50–300 | 0.25 | 0.05 | `DEFAULT_PARAMS` | - -The σ spread is what makes the watchlist readable at a glance: V and JPM barely move while NVDA -and TSLA jump, so the grid has texture instead of ten tickers twitching identically. - -There is **no mean reversion and no session boundary.** Prices random-walk from their seed for as -long as the container runs. Over a demo that looks like a trading day; over a week of uptime a -ticker may wander far. That is correct GBM behavior and not worth correcting — state is in memory -only, so a restart returns everything to seed. - ---- - -## 8. The Massive client — optional real data - -`backend/app/market/massive_client.py`. Verified against the `massive` SDK **2.2.0** installed in -`backend/.venv`. - -### 8.1 Why one snapshot endpoint, polled - -The free tier allows **5 requests/minute** — one request every 12 seconds at best. Per-ticker -endpoints are therefore unusable: 10 watchlist tickers via `get_last_trade` would be 10 requests -per cycle, blowing the entire budget in one poll. - -**The design must fetch all tickers in a single request.** That is -`GET /v2/snapshot/locale/us/markets/stocks/tickers`, one request returning the current state of -every ticker named: - -```python -from massive import RESTClient -from massive.rest.models import SnapshotMarketType - -client = RESTClient(api_key="YOUR_KEY") - -snapshots = client.get_snapshot_all( - market_type=SnapshotMarketType.STOCKS, - tickers=["AAPL", "GOOGL", "MSFT", "AMZN", "TSLA"], -) - -for snap in snapshots: - print(snap.ticker, snap.last_trade.price, snap.todays_change_percent) -``` - -The SDK joins a list into a comma-separated string, so passing `list[str]` is correct. -Default poll interval is **15 seconds**, which leaves headroom under the free tier even if a poll -overruns. Paid tiers can drop to 2–5 seconds via `poll_interval`. - -The v3 unified snapshot (`list_universal_snapshots`) is the alternative; it reports unknown -tickers explicitly with an `error` field instead of silently omitting them. FinAlly stays on v2: -a 10-ticker watchlist never approaches v3's 250-ticker limit, v2 is a single non-paginated -request, and per-ticker validation feedback is marginal when the simulator is the default path. -v3 is the right upgrade if that feedback is ever wanted. - -### 8.2 `RESTClient` is synchronous — wrap every call - -It is `urllib3`-based. Calling it from `async def` blocks the event loop for the whole HTTP round -trip, which in this app means visibly stuttering prices on the SSE stream. - -```python -snapshots = await asyncio.to_thread(self._fetch_snapshots) -``` - -It also **retries 429 internally** (3 attempts, honoring `Retry-After`), so a rate-limited poll -blocks its worker thread rather than failing fast. That is fine — the worker is not the event loop. - -### 8.3 Timestamp units — the trap - -Massive uses three different time units across endpoints and the SDK passes them through unchanged. - -| Source | Attribute | Unit | To Unix seconds | -|---|---|---|---| -| Snapshot `lastTrade` | `sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | -| Snapshot `lastQuote` | `sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | -| Snapshot top level | `updated` | **nanoseconds** | `/ 1_000_000_000` | -| Snapshot `min` | `timestamp` | **milliseconds** | `/ 1_000` | -| Aggregates (`Agg`, `PreviousCloseAgg`, grouped) | `timestamp` | **milliseconds** | `/ 1_000` | - -And **attribute names never match JSON keys.** The wire format is single-letter (`p`, `s`, `t`, -`x`); `from_dict` maps those to readable attributes. `@modelclass` builds a plain dataclass with -no `__getattr__` fallback, so reading a key name raises `AttributeError`. - -| `LastTrade` attribute | JSON key | Units | -|---|---|---| -| `price` | `p` | dollars | -| `size` | `s` | shares | -| `sip_timestamp` | `t` | **nanoseconds** | -| `exchange` | `x` | exchange ID | - -### 8.4 Two defects in the shipped client — reproduced, not inferred - -`_poll_once` currently reads: - -```python -price = snap.last_trade.price -timestamp = snap.last_trade.timestamp / 1000.0 # AttributeError, then wrong unit -``` - -Reproduction against the installed SDK, run on 2026-09-01: - -```python -from massive.rest.models.snapshot import TickerSnapshot - -snap = TickerSnapshot.from_dict({ - "ticker": "AAPL", - "lastTrade": {"p": 190.52, "s": 100, "t": 1755873791482000000, "x": 4}, -}) - -snap.last_trade.price # 190.52 -snap.last_trade.sip_timestamp # 1755873791482000000 -hasattr(snap.last_trade, "timestamp") # False -``` - -**Defect 1 — `last_trade.timestamp` does not exist, so the Massive path writes nothing at all.** -The loop wraps each snapshot in `except (AttributeError, TypeError)` and merely logs a warning, so -the exception is swallowed once per ticker on every poll. The symptom is not a crash: it is a -watchlist where every ticker shows `—` forever, with `Skipping snapshot for AAPL` in the logs. - -**Defect 2 — the divisor is wrong by 10⁶.** Even with the attribute corrected, `/ 1000.0` treats -nanoseconds as milliseconds: `1755873791482000000 / 1000` ≈ 1.76 × 10¹⁵ seconds, roughly 55 million -years in the future. Any chart keyed on that timestamp is unusable. - -**Why 94% coverage did not catch either.** `tests/market/test_massive.py` builds snapshots from -`MagicMock`, which answers to any attribute name: - -```python -snap.last_trade.timestamp = timestamp_ms # an attribute the real model does not have -``` - -`test_timestamp_conversion` then locks in the wrong unit as well. The lesson generalizes: -**mocking a third-party model tests your assumptions about the library, not the library.** -Parsing tests must go through the real `TickerSnapshot.from_dict` with a documented payload -(§14.4). That test needs no network and would have failed on its first run. - -### 8.5 The corrected parse - -```python -NANOS_PER_SECOND = 1_000_000_000 - -for snap in snapshots: - trade = snap.last_trade - if trade is None or trade.price is None: - continue # no print yet today; leave the ticker showing "—" - self._cache.update( - ticker=snap.ticker, - price=trade.price, - timestamp=( - trade.sip_timestamp / NANOS_PER_SECOND - if trade.sip_timestamp - else time.time() - ), - ) - processed += 1 -``` - -**Guarding on `is None` rather than catching `AttributeError` is what makes the difference.** -A genuinely absent field is a normal condition to handle; a misspelled attribute is a bug that -should be loud. The existing blanket `except AttributeError` is precisely what hid defect 1. - -### 8.6 Error handling in the poll loop - -The SDK raises only two exception types (`massive/exceptions.py`): `AuthError` (empty or missing -key, raised at construction) and `BadResponse` (any non-200 surviving the retry policy). -`urllib3` raises its own for connection failures and timeouts. - -```python -from massive.exceptions import AuthError, BadResponse - -async def _poll_once(self) -> None: - if not self._tickers or not self._client: - return - try: - snapshots = await asyncio.to_thread(self._fetch_snapshots) - except AuthError: - logger.error("Massive API key rejected — the source does not fall back automatically") - raise # unrecoverable: do not retry on a loop - except BadResponse as e: - logger.warning("Massive returned an error response: %s", e) - return # transient: retry next interval - except Exception: - logger.exception("Massive poll failed") - return - ... # the §8.5 parse -``` - -`start()` performs one poll synchronously **before** creating the task, so the cache is warm -before the first client connects: - -```python -async def _poll_loop(self) -> None: - """Poll on interval. The first poll already happened in start().""" - while True: - await asyncio.sleep(self._interval) - await self._poll_once() -``` - -### 8.7 Behaviors to surface in the README - -Properties of the data source, not bugs — users will otherwise report them as bugs: - -- **Unknown symbols vanish silently.** The v2 snapshot omits tickers it does not recognize; there - is no error entry. The ticker sits in the watchlist showing `—` indefinitely. -- **Prices freeze outside market hours.** Overnight, at weekends, and on holidays the snapshot - returns the previous session's last trade. The UI looks broken but is correct. **This is the - main reason the simulator is the default.** -- **Free-tier data is 15 minutes delayed**, so prices will not match any other quote source the - user has open. -- **Snapshot data is cleared at midnight ET** and repopulates from about 4am ET. Between those - times `last_trade` may be absent entirely — exactly the `None` case §8.5 guards. - -`client.get_market_status()` is worth one call to explain a frozen feed rather than leaving the -user guessing. - -### 8.8 Live verification - -Run once a real key exists — it confirms auth, the multi-ticker snapshot, and unit conversion in -one pass: - -```python -# backend/scripts/verify_massive.py -"""Smoke-test the Massive REST API against a live key.""" - -import os -from datetime import UTC, datetime - -from massive import RESTClient -from massive.rest.models import SnapshotMarketType - -NANOS_PER_SECOND = 1_000_000_000 -TICKERS = ["AAPL", "GOOGL", "MSFT", "NVDA", "TSLA"] - - -def main() -> None: - client = RESTClient(api_key=os.environ["MASSIVE_API_KEY"]) - - print(f"market: {client.get_market_status().market}") - - snapshots = client.get_snapshot_all(SnapshotMarketType.STOCKS, TICKERS) - print(f"requested {len(TICKERS)}, received {len(snapshots)}") - - for snap in snapshots: - trade = snap.last_trade - if trade is None or trade.price is None: - print(f"{snap.ticker}: no trade data") - continue - when = datetime.fromtimestamp(trade.sip_timestamp / NANOS_PER_SECOND, UTC) - print(f"{snap.ticker}: ${trade.price:.2f} at {when:%Y-%m-%d %H:%M:%S} UTC") - - missing = set(TICKERS) - {s.ticker for s in snapshots} - if missing: - print(f"absent from response (unknown or untraded): {sorted(missing)}") - - -if __name__ == "__main__": - main() -``` - -```bash -cd backend && uv run python scripts/verify_massive.py -``` - -Expected: a market status and five priced tickers with timestamps **in the recent past**. -Timestamps far in the future mean the divisor regressed; `AttributeError` means §8.4 regressed. - ---- - -## 9. Selection — `create_market_data_source` - -```python -def create_market_data_source(price_cache: PriceCache) -> MarketDataSource: - """Create the market data source indicated by the environment. - - MASSIVE_API_KEY set and non-empty -> MassiveDataSource (real data) - otherwise -> SimulatorDataSource (GBM simulation) - - Returns an unstarted source; the caller must await source.start(tickers). - """ - api_key = os.environ.get("MASSIVE_API_KEY", "").strip() - - if api_key: - logger.info("Market data source: Massive API (real data)") - return MassiveDataSource(api_key=api_key, price_cache=price_cache) - - logger.info("Market data source: GBM Simulator") - return SimulatorDataSource(price_cache=price_cache) -``` - -**`.strip()` before the truth test is deliberate.** `.env` files routinely contain -`MASSIVE_API_KEY=` or a stray space, and a whitespace-only key would otherwise select the Massive -path and then fail every poll with a 401. Empty means empty. - -**The choice is made once at startup and never at runtime.** A source that silently switched to -the simulator after a Massive outage would show users invented prices while they believed they -were seeing the market. Rejected keys and failed polls are logged; they do not change the source. -`GET /api/health` reports which one is live: - -```json -{"status": "ok", "market_source": "simulator", "llm_mock": false} -``` - -Returning an **unstarted** source keeps construction synchronous and lets the caller decide the -ticker set from the database — the factory has no business reading tables. - ---- - -## 10. The SSE stream - -### 10.1 What ships - -`GET /api/stream/prices`, `Content-Type: text/event-stream`. The generator opens with -`retry: 1000`, then pushes the **entire cache as one JSON object** whenever `version` changes, -polled every 500ms: - -``` -retry: 1000 - -data: {"AAPL": {"ticker": "AAPL", "price": 190.52, "previous_price": 190.48, "timestamp": 1755873791.482, "change": 0.04, "change_percent": 0.021, "direction": "up"}, "GOOGL": {...}} -``` - -One event carries every ticker. The client replaces its price map wholesale — no merge logic, no -missed-update reconciliation. Because a fresh generator starts at `last_version = -1`, the first -comparison always differs, so **every connecting client immediately receives a full snapshot**, -including after a reconnect. That is why no separate snapshot endpoint exists. - -Response headers matter as much as the payload: - -```python -return StreamingResponse( - _generate_events(price_cache, request), - media_type="text/event-stream", - headers={ - "Cache-Control": "no-cache", - "Connection": "keep-alive", - "X-Accel-Buffering": "no", # disable nginx buffering if proxied - }, -) -``` - -**Why poll-and-push instead of event-driven?** A 500ms integer comparison is cheaper to reason -about than a pub/sub fan-out across an arbitrary number of generators, and it naturally coalesces: -if the cache updated ten tickers since the last check, the client gets one event, not ten. - -### 10.2 Keepalive — to implement - -When the version has not changed for 15 seconds, emit an SSE comment line. The complete generator: - -```python -KEEPALIVE_SECONDS = 15.0 - - -async def _generate_events( - price_cache: PriceCache, - request: Request, - interval: float = 0.5, -) -> AsyncGenerator[str, None]: - """Yield SSE events whenever the cache version changes; ping when it does not.""" - yield "retry: 1000\n\n" - - last_version = -1 - last_sent = time.monotonic() - client_ip = request.client.host if request.client else "unknown" - logger.info("SSE client connected: %s", client_ip) - - try: - while True: - if await request.is_disconnected(): - logger.info("SSE client disconnected: %s", client_ip) - break - - current_version = price_cache.version - if current_version != last_version: - last_version = current_version - prices = price_cache.get_all() - if prices: - payload = json.dumps({t: u.to_dict() for t, u in prices.items()}) - yield f"data: {payload}\n\n" - last_sent = time.monotonic() - elif time.monotonic() - last_sent >= KEEPALIVE_SECONDS: - yield ": ping\n\n" - last_sent = time.monotonic() - - await asyncio.sleep(interval) - except asyncio.CancelledError: - logger.info("SSE stream cancelled for: %s", client_ip) -``` - -Without this, a Massive-backed feed sends no bytes between 15-second polls. That idle-times-out -through proxies and leaves the frontend unable to distinguish a quiet market from a dead -connection. The frontend indicator — green on `onopen`, yellow on `onerror`, red after a gap -beyond ~40 seconds — depends on it. - -A line beginning with `:` is a comment in the SSE grammar: `EventSource` ignores it entirely, so -it costs the client nothing while keeping the socket warm. - ---- - -## 11. `GET /api/prices/{ticker}/history` — to implement - -Backed by `PriceCache.get_history` (§5.2). It belongs in `stream.py` next to the SSE endpoint, -since both are pure cache readers with no database involvement. - -```python -history_router = APIRouter(prefix="/api/prices", tags=["prices"]) - - -def create_history_router(price_cache: PriceCache) -> APIRouter: - @history_router.get("/{ticker}/history") - async def get_price_history(ticker: str, limit: int = 600) -> dict: - """Rolling in-memory price history for the main chart. - - Returns an empty `points` list for an untracked ticker — not a 404, - so the chart draws nothing rather than erroring. - """ - ticker = ticker.strip().upper() - limit = max(1, min(limit, HISTORY_MAXLEN)) - points = price_cache.get_history(ticker, limit=limit) - return { - "ticker": ticker, - "points": [{"timestamp": ts, "price": price} for ts, price in points], - } - - return history_router -``` - -Response: - -```json -{"ticker": "AAPL", "points": [{"timestamp": 1755873791.482, "price": 190.52}]} -``` - -Oldest-first, matching what Recharts wants for a left-to-right time axis. `limit` is clamped -rather than validated with a 400 — a chart asking for 10,000 points should get 600, not an error. - -This endpoint reads only in-memory state, so `async def` is correct here; there is no SQLite call -to keep off the event loop. - ---- - -## 12. Wiring - -### 12.1 Lifespan - -One `PriceCache` and one source per process, owned by the FastAPI lifespan. - -```python -from contextlib import asynccontextmanager - -from fastapi import FastAPI - -from app.market import PriceCache, create_market_data_source, create_stream_router - - -@asynccontextmanager -async def lifespan(app: FastAPI): - init_db() # lazy schema creation + seed (PLAN.md §7) - - cache = PriceCache() - source = create_market_data_source(cache) - - # Reconciliation: watchlist ∪ held positions, not just the watchlist - tickers = sorted(set(get_watchlist_tickers()) | set(get_position_tickers())) - await source.start(tickers) - - app.state.price_cache = cache - app.state.market_source = source - try: - yield - finally: - await source.stop() - - -app = FastAPI(lifespan=lifespan) - -# 1. API routers FIRST -app.include_router(create_stream_router(cache)) -app.include_router(create_history_router(cache)) -# ... portfolio, watchlist, chat routers ... -# 2. static assets -# 3. catch-all -> index.html -``` - -Three things this gets right and are easy to get wrong: - -**Reading both tables at startup**, not just the watchlist, is what makes a position held across -a restart come back with a live price. Without it, an off-watchlist holding valuates at `avg_cost` -forever and the snapshot task stalls under the "skip if any held ticker has no price" rule. - -**The cache and source are passed explicitly** (via router factories or `app.state`) rather than -held in module globals, which is what keeps tests able to construct an isolated cache per test. - -**Mount all `/api/*` routers before the static file mount.** A `StaticFiles(html=True)` mount at -`/` registered first shadows every endpoint, including the SSE stream (`PLAN.md` §11). - -Route handlers reach the cache through `app.state` or a dependency: - -```python -def get_price_cache(request: Request) -> PriceCache: - return request.app.state.price_cache - - -def get_market_source(request: Request) -> MarketDataSource: - return request.app.state.market_source -``` - -### 12.2 The tracked ticker set - -**The tracked set is `watchlist ∪ {tickers with a non-zero position}`.** - -The two sets diverge the moment a user buys TSLA and then removes it from the watchlist. The -position still needs a live price for valuation, P&L, the heatmap, and snapshots. - -| Trigger | Action | -|---|---| -| `POST /api/watchlist` | always `await source.add_ticker(t)` | -| `DELETE /api/watchlist/{t}` | `await source.remove_ticker(t)` **only if no position in `t` is held** | -| Buy a ticker not currently tracked | `await source.add_ticker(t)` as part of trade execution | -| Sell a position to zero | if `t` is not on the watchlist, `await source.remove_ticker(t)` | -| `POST /api/reset` | re-sync the tracked set to exactly the ten default tickers | - -One helper keeps the rule in one place rather than at four call sites: - -```python -async def untrack_if_unused(source: MarketDataSource, ticker: str) -> None: - """Stop tracking a ticker only if it is neither watched nor held.""" - if is_on_watchlist(ticker) or has_position(ticker): - return - await source.remove_ticker(ticker) -``` - -### 12.3 Ticker validation at the boundary - -Applied at `POST /api/watchlist`, `POST /api/portfolio/trade`, and every LLM-proposed action, so -the market layer only ever sees canonical symbols: - -```python -TICKER_PATTERN = re.compile(r"^[A-Z][A-Z.]{0,5}$") - - -def normalize_ticker(raw: str) -> str: - """Uppercase and validate. Raises ValueError with the user-facing message.""" - ticker = raw.strip().upper() - if not TICKER_PATTERN.match(ticker): - raise ValueError("Invalid ticker symbol") - return ticker -``` - -No allowlist. Any symbol matching the pattern is accepted; the simulator invents plausible -behavior for it, and under Massive an unknown symbol shows `—`. Rejecting unknown symbols would -make the LLM's `watchlist_changes` feature feel broken. - -Uppercasing is not cosmetic: the `UNIQUE(user_id, ticker)` constraints would otherwise happily -hold both `AAPL` and `aapl`. - ---- - -## 13. Failure modes - -| Situation | Behavior | Where | -|---|---|---| -| Empty ticker list at startup | `step()` returns `{}`, SSE sends nothing until a ticker is added | §7.4 | -| One bad simulator tick | Logged, loop continues next interval | §7.6 | -| Massive poll fails (429, network) | Logged, cache keeps last prices, retry next interval | §8.6 | -| Massive key rejected | `AuthError` re-raised; **no automatic fallback to the simulator** | §8.6, §9 | -| Ticker has no `last_trade` yet | Skipped; ticker shows `—` | §8.5 | -| Held ticker has no cached price | Portfolio values it at `avg_cost`; snapshot task skips the write entirely | `PLAN.md` §7 | -| Ticker removed while held | Prevented by `untrack_if_unused` | §12.2 | -| Client disconnects mid-stream | `request.is_disconnected()` breaks the generator | §10.2 | -| Quiet feed (Massive, 15s polls) | `: ping` every 15s keeps the connection and the indicator alive | §10.2 | -| History requested for untracked ticker | `{"ticker": "X", "points": []}` | §11 | - ---- - -## 14. Testing - -Current state: **73 tests, 91% coverage** on the market module. `stream.py` sits at 33% — the SSE -generator is the least-tested code in the subsystem and the keepalive change is a good moment to -fix that. - -```bash -cd backend -uv run --extra dev pytest -v -uv run --extra dev pytest --cov=app --cov-report=term-missing -``` - -### 14.1 A stub source - -The cache and the tracked-set rules can be tested without either real source: - -```python -class StubDataSource(MarketDataSource): - """Records lifecycle calls; writes nothing on its own.""" - - def __init__(self, cache: PriceCache) -> None: - self._cache = cache - self._tickers: list[str] = [] - self.started = False - - async def start(self, tickers): self._tickers = list(tickers); self.started = True - async def stop(self): self.started = False - async def add_ticker(self, t): - if t not in self._tickers: - self._tickers.append(t) - async def remove_ticker(self, t): - self._tickers = [x for x in self._tickers if x != t] - self._cache.remove(t) - def get_tickers(self): return list(self._tickers) -``` - -### 14.2 Simulator - -Seed **both** RNGs — the simulator uses `numpy.random` for the normal draws and stdlib `random` -for events: - -```python -def test_step_is_reproducible(): - np.random.seed(42) - random.seed(42) - sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) - first = sim.step() - - np.random.seed(42) - random.seed(42) - sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) - assert sim.step() == first -``` - -Statistical properties need wide tolerances and events disabled — a 5% jump is a massive outlier -at this `dt` and would dominate the sample variance: - -```python -def test_realised_volatility_is_close_to_sigma(): - sim = GBMSimulator(tickers=["AAPL"], event_probability=0.0) - prices = [sim.get_price("AAPL")] - for _ in range(20_000): - prices.append(sim.step()["AAPL"]) - - log_returns = np.diff(np.log(prices)) - realised = log_returns.std() / np.sqrt(GBMSimulator.DEFAULT_DT) - assert 0.15 < realised < 0.35 # nominal sigma is 0.22 - - -def test_tech_tickers_are_positively_correlated(): - sim = GBMSimulator(tickers=["AAPL", "MSFT"], event_probability=0.0) - a, m = [], [] - for _ in range(10_000): - p = sim.step() - a.append(p["AAPL"]) - m.append(p["MSFT"]) - - rho = np.corrcoef(np.diff(np.log(a)), np.diff(np.log(m)))[0, 1] - assert rho > 0.4 # nominal 0.6 - - -def test_correlation_matrix_stays_positive_definite(): - """Run after ANY change to the correlation constants in seed_prices.py.""" - tickers = list(SEED_PRICES) + [f"UNK{i}" for i in range(40)] - GBMSimulator(tickers=tickers) # raises LinAlgError if not PD -``` - -Also cover: prices stay strictly positive over thousands of steps; `step()` returns exactly the -current ticker set; add/remove keeps `_tickers`/`_prices`/`_params` consistent and the Cholesky -shape matching; unknown tickers seed within $50–$300 with `DEFAULT_PARAMS`; `remove_ticker` on an -untracked symbol is a no-op. - -### 14.3 Cache and history - -```python -def test_history_is_bounded_and_oldest_first(): - cache = PriceCache(history_maxlen=5) - for i in range(10): - cache.update("AAPL", 100.0 + i, timestamp=float(i)) - - points = cache.get_history("AAPL") - assert len(points) == 5 - assert [ts for ts, _ in points] == [5.0, 6.0, 7.0, 8.0, 9.0] - - -def test_history_is_empty_for_untracked_ticker(): - assert PriceCache().get_history("NOPE") == [] - - -def test_remove_clears_price_and_history(): - cache = PriceCache() - cache.update("AAPL", 190.0) - cache.remove("AAPL") - assert cache.get("AAPL") is None - assert cache.get_history("AAPL") == [] -``` - -Plus: `previous_price` derivation, first-update `flat`, `version` monotonicity, and thread safety -under concurrent writers. - -### 14.4 Massive — through the real model, never `MagicMock` - -This is the test that would have caught both defects in §8.4, and it needs no network: - -```python -from massive.rest.models.snapshot import TickerSnapshot - -def test_snapshot_parse_produces_a_present_day_timestamp(): - snap = TickerSnapshot.from_dict({ - "ticker": "AAPL", - "lastTrade": {"p": 190.52, "s": 100, "t": 1755873791482000000, "x": 4}, - }) - - cache = PriceCache() - source = MassiveDataSource(api_key="x", price_cache=cache) - source._apply_snapshots([snap]) # extract the parse into a testable method - - update = cache.get("AAPL") - assert update.price == 190.52 - assert 1_600_000_000 < update.timestamp < 2_000_000_000 # plausible present, in SECONDS - - -def test_snapshot_without_a_last_trade_is_skipped(): - snap = TickerSnapshot.from_dict({"ticker": "AAPL"}) - cache = PriceCache() - source = MassiveDataSource(api_key="x", price_cache=cache) - source._apply_snapshots([snap]) - assert cache.get("AAPL") is None -``` - -Extracting the parse loop into `_apply_snapshots(snapshots)` is worth the small refactor: it makes -the parse testable without touching HTTP, which is the only part that actually broke. - -### 14.5 Factory and SSE - -- **Factory** — unset, empty, and whitespace-only `MASSIVE_API_KEY` all select the simulator; a - real value selects Massive. -- **SSE** — map-shaped payload, float timestamp, percent-unit `change_percent`, full snapshot on - connect, and a `: ping` after 15 idle seconds. Drive the generator directly with a fake request - object rather than through a live server; the keepalive test is far easier with an injected - `interval` and a monkeypatched clock than with 15 seconds of real waiting. - -### 14.6 Tracked set - -The two regressions that silently produce a frozen position: - -- Removing a watchlist ticker with an open position **keeps** it in the feed. -- Selling to zero while off-watchlist **removes** it. - -### 14.7 Eyeballing it - -```bash -cd backend && uv run market_data_demo.py -``` - -A Rich terminal dashboard of the live simulator — the fastest way to check whether a parameter -change still looks right. Statistical tests confirm σ; only the eye confirms "looks like a -trading terminal". - ---- - -## 15. Implementation order - -Small increments, each independently verifiable. Run `uv run --extra dev pytest` after every step. - -1. **Fix the Massive parse** (§8.5). Extract `_apply_snapshots`, correct the attribute and the - divisor, replace the `MagicMock` tests with `TickerSnapshot.from_dict` tests (§14.4). This is - first because the current code silently produces nothing, and because the fix is provable - offline. -2. **Add rolling history to `PriceCache`** (§5.2). Deque, `get_history`, `remove` clearing both. - Tests in §14.3. -3. **Add `GET /api/prices/{ticker}/history`** (§11). Depends on step 2. -4. **Add the SSE keepalive** (§10.2) and raise `stream.py` coverage off 33% (§14.5). -5. **Wire the lifespan** (§12.1) with startup reconciliation over `watchlist ∪ positions`, and - add `untrack_if_unused` (§12.2) where the watchlist and trade routes are built. - -Steps 1–4 are self-contained in `app/market/`. Step 5 is the seam with the rest of the backend and -should land alongside the portfolio and watchlist routes, not before them. - ---- - -## 16. Configuration reference - -| Setting | Default | Where | Effect | -|---|---|---|---| -| `MASSIVE_API_KEY` | unset | env | Non-empty selects Massive; otherwise simulator | -| `update_interval` | `0.5` | `SimulatorDataSource` | Simulator tick rate — **change `dt` with it** | -| `event_probability` | `0.001` | `SimulatorDataSource` | Shock chance per ticker per tick | -| `poll_interval` | `15.0` | `MassiveDataSource` | Seconds between snapshot requests | -| `HISTORY_MAXLEN` | `600` | `cache.py` | Rolling history depth (~5 min at 500ms) | -| `KEEPALIVE_SECONDS` | `15.0` | `stream.py` | Idle gap before a `: ping` | -| SSE poll `interval` | `0.5` | `stream.py` | How often the version is checked | - -### Tuning the simulator - -| Want | Change | Watch for | -|---|---|---| -| More visible motion | Raise σ in `TICKER_PARAMS` | Above ~0.8 it stops looking like equity | -| Faster updates | `update_interval` | **`DEFAULT_DT` hard-codes the 500ms tick** — see below | -| More drama | Raise `event_probability` | Above ~0.005 the series becomes jumps, not prices | -| Bigger shocks | Widen `random.uniform(0.02, 0.05)` | Beyond ~10% the P&L chart loses all detail | -| Different sectors | Edit `CORRELATION_GROUPS` and coefficients | Re-run the positive-definiteness test (§14.2) | -| A trending market | Raise μ | μ is annualized; even 0.5 is barely visible over a demo | - -**The `DEFAULT_DT` coupling is the one that catches people.** `DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR` -hard-codes the 500ms tick. Passing `update_interval=0.1` without also passing a matching `dt` runs -the simulation five times faster in model time, and annualized volatility silently becomes 5× what -`TICKER_PARAMS` claims. - ---- - -## 17. Summary - -| Concern | Resolution | -|---|---| -| Two sources, one consumer | `MarketDataSource` ABC + shared `PriceCache` | -| Which source | `create_market_data_source`, decided once at startup from `MASSIVE_API_KEY` | -| Default | Simulator — always alive, no key, no rate limit, any ticker | -| How prices are read | Only from the cache, never from the source | -| Which tickers are live | `watchlist ∪ positions`, reconciled at startup | -| Price model | GBM, per-ticker μ and σ, Cholesky-correlated by sector | -| Timestamp format | Unix epoch seconds (float), converted at each source boundary | -| Update delivery | SSE, full cache per event, on `version` change, `: ping` when idle | -| Chart backfill | 600-point in-memory deque per ticker, never persisted | -| Blocking I/O | `asyncio.to_thread` at the source, always | -| Failure handling | Per-cycle `try` inside the loop; the feed never dies from one bad tick | -| Outstanding work | The five steps in §15 | diff --git a/planning/MARKET_DATA_REVIEW.md b/planning/MARKET_DATA_REVIEW.md deleted file mode 100644 index 3ecb1f1..0000000 --- a/planning/MARKET_DATA_REVIEW.md +++ /dev/null @@ -1,242 +0,0 @@ -# Market Data Backend — Code Review - -**Date:** 2026-09-02 -**Scope:** `backend/app/market/` (8 source files, 730 LOC) and `backend/tests/market/` (7 test files, 1,161 LOC) -**Reviewer:** Claude, in response to issue #5 - ---- - -## 1. Test Execution — Could Not Run - -This review's environment does not have permission to execute shell commands that -run the Python interpreter or install dependencies (`uv sync`, `uv run pytest`, even -`python3 -c ...` are all blocked pending approval, and this run has no human -available to approve them). This is the same limitation `planning/MARKET_DATA_SUMMARY.md` -recorded on the previous pass. **No test was executed as part of this review.** - -To get a real pass/fail signal, re-run this task with `Bash(uv sync:*)` and -`Bash(uv run:*)` added to the allowed tools, or run locally: - -```bash -cd backend -uv sync --extra dev -uv run --extra dev pytest -v --cov=app --cov-report=term-missing -uv run --extra dev ruff check app/ tests/ -``` - -In place of execution, every test file was read in full and traced by hand against -the source it exercises (see §4). All 96 tests found in the suite exercise real -code paths correctly as far as static reading can confirm — no test asserts on -behavior the source doesn't actually implement, and no test's mocking hides a -divergence between the mock's shape and the real one (the concern that let a -prior bug through at 94% coverage, per `test_massive.py`'s own docstring). - -**Test count:** 96 across 7 files (`test_models.py` 11, `test_cache.py` 24, -`test_simulator.py` 17, `test_simulator_source.py` 10, `test_factory.py` 7, -`test_massive.py` 17, `test_stream.py` 15 by count of `def test_`/`async def test_` -— slightly higher than the 94 recorded in `MARKET_DATA_SUMMARY.md` §"Test Suite", -consistent with incremental additions since that doc was last updated). - ---- - -## 2. Architecture Assessment - -The module is well-factored and matches `planning/MARKET_DATA_DESIGN.md` and -`planning/MARKET_DATA_SUMMARY.md` closely: - -``` -MarketDataSource (ABC) -├── SimulatorDataSource → GBMSimulator (Cholesky-correlated GBM) -└── MassiveDataSource → Polygon.io REST poller - │ - ▼ - PriceCache (thread-safe, latest price + 600-point rolling history) - │ - ├──→ create_stream_router() → GET /api/stream/prices (SSE, with keepalive) - └──→ create_history_router() → GET /api/prices/{ticker}/history -``` - -**Strengths confirmed by this pass:** - -- Strategy pattern cleanly isolates the two data sources behind `MarketDataSource`; nothing downstream needs to know which is active. -- `PriceUpdate` is `frozen=True, slots=True` — correct choice for a value object shared across threads/tasks. -- `PriceCache` centralizes all locking (`threading.Lock`) around the one mutable structure producers and consumers touch; the API surface (`update`, `get`, `get_all`, `get_price`, `remove`, `get_history`) is small and each method acquires the lock exactly once. -- The GBM math is textbook-correct log-normal price evolution, and the `dt` sizing (`0.5s / (252 * 6.5h * 3600s)`) is derived, not guessed, with the derivation left in a comment. -- Cholesky-based correlated draws (`simulator.py:84-90`) are a genuinely nice touch for a simulator whose only job is to look convincing on a chart. -- The three TODOs recorded as open in `PLAN.md` §13 (SSE keepalive, rolling history, `/history` endpoint) are all implemented and each has direct test coverage (`test_stream.py`). -- The two defects `MARKET_DATA_DESIGN.md` §8.4 recorded against the Massive client (wrong attribute name, nanoseconds-as-milliseconds) are fixed in `massive_client.py:130-136`, and `test_massive.py` deliberately builds real `TickerSnapshot` objects via `TickerSnapshot.from_dict(...)` rather than `MagicMock`, which is exactly the right defense against that class of bug recurring silently. -- `pyproject.toml` already has `[tool.hatch.build.targets.wheel] packages = ["app"]` — the "High" build-breaking bug from the archived 2026-02-10 review (`planning/archive/MARKET_DATA_REVIEW.md` §3.1) is fixed. -- `massive` is a top-level import now (`massive_client.py:9-11`), not a lazy one — the archived review's §3.2 concern about tests being fragile without the package installed no longer applies; `pyproject.toml` lists it as a core dependency. - ---- - -## 3. Issues Found - -### 3.1 `create_stream_router` / `create_history_router` mutate a shared module-level router (Severity: Medium) - -`stream.py:18-19` defines `router` and `history_router` at module scope. Both -factory functions register their route via a closure on these **same shared -objects** rather than creating a fresh `APIRouter()` per call: - -```python -router = APIRouter(prefix="/api/stream", tags=["streaming"]) -history_router = APIRouter(prefix="/api/prices", tags=["prices"]) - -def create_stream_router(price_cache: PriceCache) -> APIRouter: - @router.get("/prices") - async def stream_prices(request: Request) -> StreamingResponse: - ... - return router -``` - -Calling either factory more than once appends another route to the same -underlying router rather than returning an independent one. This was flagged -as a "latent footgun for testing" in the archived review (§3.6) when there -were no tests exercising it; now there are, and it is no longer latent: -`test_stream.py`'s `_history_endpoint()` helper calls `create_history_router(cache)` -fresh in **six different tests**, so `history_router` in the running test -process accumulates six duplicate `/{ticker}/history` routes by the end of -the file. The tests still pass because they grab `router.routes[-1].endpoint` -— the most recently registered one — but this only works by coincidence of -ordering, not because the router is actually being rebuilt. - -The real risk is downstream: once this module is wired into the FastAPI app -(the next piece of work per `PLAN.md` §13 "Still open"), any test that builds -the app more than once per process — a very common pytest pattern (an `app` -fixture instantiated per test, or per module) — will silently accumulate -duplicate routes on every rebuild, since `router`/`history_router` are shared -mutable module state that outlives any single app instance. - -**Fix:** construct a new `APIRouter()` inside each factory function instead of -reusing a module-level instance: - -```python -def create_stream_router(price_cache: PriceCache) -> APIRouter: - router = APIRouter(prefix="/api/stream", tags=["streaming"]) - - @router.get("/prices") - async def stream_prices(request: Request) -> StreamingResponse: - ... - return router -``` - -### 3.2 `PriceCache.update()` treats a falsy timestamp as "no timestamp given" (Severity: Low) - -```python -ts = timestamp or time.time() -``` - -(`cache.py:40`) A caller that explicitly passes `timestamp=0.0` (Unix epoch, -1970-01-01) gets `time.time()` substituted instead, because `0.0` is falsy. -No current caller does this — `massive_client.py` only reaches this path with -`time.time()` already substituted upstream when `sip_timestamp` is falsy — so -this is not exploitable today, but it is a latent correctness gap for any -future caller (e.g., a test replaying historical data from epoch-adjacent -timestamps, or a backfill script). Prefer `timestamp if timestamp is not None -else time.time()`. - -### 3.3 `MassiveDataSource`'s poller task dies silently on `AuthError` (Severity: Low) - -`_poll_once()` deliberately re-raises `AuthError` (`massive_client.py:103-105`) -with the comment "unrecoverable: do not retry on a loop" — a reasonable -choice. But the only place that awaits `self._task` is `stop()` -(`massive_client.py:60-69`), which nothing calls until shutdown. If the key -is revoked *after* `start()` succeeds (rather than being bad from the first -poll), the background task raised inside `_poll_loop()` simply stops running; -asyncio logs "Task exception was never retrieved" at some later point (often -at garbage collection, easy to miss in container logs), and the app has no -other signal that live prices have silently frozen. `test_auth_error_propagates` -confirms the exception propagates out of `_poll_once()`, but there is no test -for what happens to `_poll_loop()` or the app once that happens. - -This is fine as coded for now since nothing outside the market module reads -task health yet, but whoever wires this into the app (`PLAN.md` §13, item 3) -should either attach a `Task.add_done_callback` that logs loudly / flips a -health flag, or have `GET /api/health` report `market_source` as degraded -when the task is dead. Worth a one-line note in `MARKET_DATA_SUMMARY.md` so -it isn't forgotten during integration. - -### 3.4 `PriceCache.version` property reads outside the lock (Severity: Trivial) - -Unchanged from the archived review's §3.4: `cache.py:94-97` reads `self._version` -without acquiring `self._lock`. Safe under CPython's GIL for a single `int` -read, inconsistent with the rest of the class, and only a real concern on a -no-GIL build. Not worth blocking on, but a two-line fix if anyone is passing -through this file for another reason. - -### 3.5 `market_data_demo.py` and `backend/README.md` are outside the reviewed test scope but were not separately verified - -The demo script (`market_data_demo.py`, 205 lines) is referenced by -`MARKET_DATA_SUMMARY.md` as a manual verification tool and has no automated -test coverage, which is appropriate for a Rich terminal demo — flagging only -so it's clear this review's "all tests pass" scope is `backend/tests/market/`, -not the demo script. - ---- - -## 4. Test Suite Assessment (by module) - -| Module | File | Assessment | -|---|---|---| -| `models.py` | `test_models.py` (11 tests) | Complete: creation, `change`/`change_percent`/`direction` in both directions, zero-previous-price edge case, `to_dict()` shape, and frozen-dataclass immutability. No gaps. | -| `cache.py` | `test_cache.py` (24 tests) | Thorough. Covers direction transitions, `version` monotonicity, `__len__`/`__contains__`, price rounding, custom timestamps, and a dedicated `TestPriceHistory` class covering bounding, ordering, per-ticker isolation, limit-narrower-than-stored, and that `remove()` clears history without touching other tickers. No test for concurrent multi-thread writes (the lock is exercised only single-threaded) — the archived review flagged this as missing in §4.2 and it remains missing; low priority since the logic is simple enough to verify by inspection. | -| `interface.py` | (no dedicated file; exercised transitively via simulator/massive tests) | Reasonable — it's an ABC with no logic of its own. | -| `seed_prices.py` | `test_simulator.py`, `test_factory.py` (transitively) | No dedicated test file, but every constant is exercised indirectly through `GBMSimulator` tests (`_pairwise_correlation` tests cover tech/finance/TSLA/cross-sector explicitly). Fine given it's pure data. | -| `simulator.py` | `test_simulator.py` (17), `test_simulator_source.py` (10) | Strong. `GBMSimulator`: positivity over 10,000 steps, seed matching, add/remove (including duplicate/nonexistent no-ops), unknown-ticker random seeding, Cholesky construction/teardown on ticker count crossing 1↔2, all four correlation branches, `dt` sanity, and rounding. `SimulatorDataSource`: cache population on start, periodic updates via real `asyncio.sleep`, idempotent stop, dynamic add/remove, empty-start, and exception resilience. The timing-based assertions (`asyncio.sleep(0.3)` then assert version advanced) are inherently a little flaky under CI load, but the margins used (3-6x the interval) are generous enough to be low-risk. | -| `massive_client.py` | `test_massive.py` (17) | Strong, and specifically hardened against the exact bug class that shipped previously — `_apply_snapshots` is tested against real `TickerSnapshot.from_dict(...)` objects, not mocks, for timestamp conversion, missing-trade skipping, mixed valid/invalid batches, and multi-ticker updates. Polling lifecycle covers success, `BadResponse` (swallowed), `AuthError` (re-raised, see §3.3), generic exceptions (swallowed), ticker add/remove with normalization, and start/stop idempotency. No gap of consequence. | -| `stream.py` | `test_stream.py` (15) | Was 31% covered and untested in the archived review; now has direct coverage of the async generator via a hand-rolled `FakeRequest`, including the retry directive, snapshot-on-connect (and thus reconnect), the frozen payload field set, keepalive timing (via `monkeypatch` on `KEEPALIVE_SECONDS` rather than a real 15s wait — good practice), a fresh data event following a ping, disconnect handling, and the empty-cache case. `create_history_router`'s endpoint is tested for ordering, unknown-ticker empty response, normalization, and limit clamping in both directions. The one real gap is architectural, not a missing test: see §3.1 — the tests would catch a *regression* in behavior but not the router-reuse issue itself, since grabbing `routes[-1]` happens to paper over it. | -| `factory.py` | `test_factory.py` (7) | Complete for its size: unset/empty/whitespace-only key → simulator, set key → Massive, and that both branches thread the cache reference through correctly. | - -**Net assessment:** the suite is comprehensive and, importantly, methodologically -careful — the deliberate choice to build real `TickerSnapshot` objects instead of -`MagicMock` in `test_massive.py` is the single best thing about this test suite, -since it's precisely what would have caught the `last_trade.timestamp` / -`sip_timestamp` bug the archived review found. No test was found asserting -something the source doesn't do, and no source behavior of consequence lacks a -test, with the caveats above (concurrency, and the router-reuse issue masked -by test ordering). - ---- - -## 5. Comparison Against the Prior Review - -`planning/archive/MARKET_DATA_REVIEW.md` (2026-02-10) recorded 7 issues. Status now: - -| # | Issue | Status | -|---|---|---| -| 3.1 | Missing hatchling wheel config | **Fixed** | -| 3.2 | Massive tests fragile without the `massive` package | **Fixed** (now a core dependency, imported at module level) | -| 3.3 | `_generate_events` return type `-> None` instead of `AsyncGenerator` | **Fixed** (`stream.py:87`) | -| 3.4 | `PriceCache.version` reads outside the lock | **Still open** (§3.4 above, trivial) | -| 3.5 | `SimulatorDataSource.get_tickers` reached into `GBMSimulator._tickers` | **Fixed** — `GBMSimulator.get_tickers()` now exists (`simulator.py:140-142`) and is used | -| 3.6 | Module-level router registered on repeated calls | **Still open, and now demonstrated by the test suite itself** (§3.1 above, upgraded to Medium given it will bite during app integration) | -| 3.7 | Unused imports in tests | **Fixed** — no unused `pytest`/`math`/`asyncio` imports found in any current test file | - -Also confirmed fixed: the two Massive parsing defects `MARKET_DATA_DESIGN.md` -§8.4 described (wrong attribute name, nanosecond/millisecond confusion), and -all three items `PLAN.md` §13 listed as open TODOs (rolling history, `/history` -endpoint, SSE keepalive). - ---- - -## 6. Verdict - -The market data backend is in good shape and ready to be built on. Of the two -open items: - -- **§3.1 (shared module-level router)** should be fixed before the FastAPI - `lifespan` wiring work begins (`PLAN.md` §13, item 3) — it's a small, - mechanical fix (stop reusing module-level `router`/`history_router`; build - one per call) and doing it now avoids a confusing bug later when the app - factory is instantiated more than once, which is standard practice for - backend test fixtures. -- **§3.2/§3.4 (falsy-timestamp substitution, unlocked version read)** are - low-risk and can be picked up opportunistically. -- **§3.3 (silent poller death on revoked key)** is a design note for whoever - adds the `GET /api/health` endpoint — surface poller liveness there. - -None of these block downstream work. **Tests were not executed in this pass** -due to environment permissions (§1) — that is the one action item this review -could not complete, and it should be re-run with `uv`/`python3` execution -permitted to get an authoritative pass/fail/coverage number rather than the -static analysis this document is based on. diff --git a/planning/MARKET_DATA_SUMMARY.md b/planning/MARKET_DATA_SUMMARY.md index 3f55110..d28d7b6 100644 --- a/planning/MARKET_DATA_SUMMARY.md +++ b/planning/MARKET_DATA_SUMMARY.md @@ -49,7 +49,7 @@ MarketDataSource (ABC) ## Gaps Closed -`planning/MARKET_DATA_DESIGN.md` §0 recorded four outstanding gaps against the code as it stood +`planning/archive/MARKET_DATA_DESIGN.md` §0 recorded four outstanding gaps against the code as it stood on 2026-09-01. All four are now closed: 1. **Massive client wrote nothing to the cache.** `last_trade.timestamp` does not exist on the diff --git a/planning/MARKET_INTERFACE.md b/planning/MARKET_INTERFACE.md deleted file mode 100644 index 7df6fa9..0000000 --- a/planning/MARKET_INTERFACE.md +++ /dev/null @@ -1,439 +0,0 @@ -# MARKET_INTERFACE.md — The Unified Market Data API - -How FinAlly retrieves stock prices from either the Massive API or the built-in simulator through one interface, selected by whether `MASSIVE_API_KEY` is set. - -Companion documents: `MASSIVE_API.md` (the real data provider) and `MARKET_SIMULATOR.md` (the fallback). This document is the contract between them and the rest of the backend. - -**Status:** the core of this design is implemented in `backend/app/market/`. Sections marked **TODO** are specified but not yet built. - ---- - -## 1. The problem this solves - -Two data sources with nothing in common: - -- **Massive** — a synchronous HTTP client, polled every 15 seconds, returning whatever the exchanges last reported, with gaps for unknown tickers and frozen values overnight. -- **The simulator** — a pure in-process computation, stepping every 500ms, always alive, and able to invent a plausible price for any symbol. - -Everything downstream — SSE streaming, portfolio valuation, trade execution, the P&L snapshot task — must not care which one is running. A trade fills at "the current price of AAPL" whether that price came from NASDAQ or from a random number generator. - -The design achieves that with **one indirection and one shared buffer**: - -``` - writes reads - ┌──────────────────┐ ┌────────────┐ ┌─────────────────────┐ - │ SimulatorSource │───┐ │ │──────────────│ SSE /api/stream │ - │ (500ms step) │ ├───▶│ PriceCache │──────────────│ Portfolio valuation │ - ├──────────────────┤ │ │ (in-mem, │──────────────│ Trade execution │ - │ MassiveSource │───┘ │thread-safe)│──────────────│ Snapshot task │ - │ (15s poll) │ │ │──────────────│ /api/prices/history │ - └──────────────────┘ └────────────┘ └─────────────────────┘ - MarketDataSource - (abstract interface) -``` - -The critical property: **nothing downstream ever calls the data source to get a price.** Sources are write-only from the application's point of view; readers only ever touch the cache. That is what makes the two implementations substitutable despite a 30× difference in update cadence. - -### Module map — `backend/app/market/` - -| File | Contents | -|---|---| -| `models.py` | `PriceUpdate` — the single price record | -| `cache.py` | `PriceCache` — the shared buffer | -| `interface.py` | `MarketDataSource` — the abstract contract | -| `simulator.py` | `GBMSimulator`, `SimulatorDataSource` | -| `massive_client.py` | `MassiveDataSource` | -| `factory.py` | `create_market_data_source` — the selection rule | -| `seed_prices.py` | Simulator constants | -| `stream.py` | The SSE endpoint | - ---- - -## 2. `PriceUpdate` — the unit of data - -An immutable, frozen dataclass. Both sources produce it; every reader consumes it. - -```python -@dataclass(frozen=True, slots=True) -class PriceUpdate: - ticker: str - price: float - previous_price: float - timestamp: float = field(default_factory=time.time) # Unix epoch SECONDS -``` - -`change`, `change_percent`, and `direction` are computed properties, not stored fields — they cannot drift out of sync with the prices they describe. - -```python -@property -def change(self) -> float: - return round(self.price - self.previous_price, 4) - -@property -def change_percent(self) -> float: - if self.previous_price == 0: - return 0.0 - return round((self.price - self.previous_price) / self.previous_price * 100, 4) - -@property -def direction(self) -> str: - if self.price > self.previous_price: - return "up" - elif self.price < self.previous_price: - return "down" - return "flat" -``` - -### Two frozen conventions - -`to_dict()` is the SSE wire format and is **frozen** — the shipped frontend contract depends on it (`PLAN.md` §6): - -- **`timestamp` is Unix epoch seconds as a float**, never ISO. The frontend multiplies by 1000 for `Date`. Massive's nanosecond and millisecond timestamps are converted at the boundary — see `MASSIVE_API.md` §7. -- **`change_percent` is already in percent units.** `0.021` means 0.021%, not 2.1%. Note this deliberately differs from REST responses elsewhere in the API, where percentages are fractions (`PLAN.md` §8). The inconsistency is real; it is preserved because the market module shipped first and the frontend was written against it. - -`previous_price` means *the price at the previous update*, not the previous session's close. On the first update for a ticker it equals `price`, so `direction` is `"flat"` and `change` is `0.0` — a new ticker never flashes green or red on its first tick. - ---- - -## 3. `PriceCache` — the shared buffer - -An in-memory `dict` behind a `threading.Lock`, plus a monotonic version counter. - -```python -class PriceCache: - def update(self, ticker: str, price: float, timestamp: float | None = None) -> PriceUpdate - def get(self, ticker: str) -> PriceUpdate | None - def get_all(self) -> dict[str, PriceUpdate] # shallow copy - def get_price(self, ticker: str) -> float | None - def remove(self, ticker: str) -> None - @property - def version(self) -> int - def __len__(self) -> int - def __contains__(self, ticker: str) -> bool -``` - -Three design points that matter: - -**`update()` derives `previous_price` itself.** Callers pass only the new price; the cache looks up what it had and constructs the `PriceUpdate`. Neither source needs to track prior state for the purpose of computing a delta, and the two cannot implement it differently. - -**A `threading.Lock`, not an `asyncio.Lock`.** `MassiveDataSource` writes from an `asyncio.to_thread` worker, so a genuine cross-thread lock is required. The critical sections are a few dict operations, so contention is irrelevant. - -**`version` increments on every write and is the SSE change-detection mechanism.** The stream compares it every 500ms rather than diffing prices. Because a fresh generator starts at `last_version = -1`, the first comparison always differs, so **every connecting client immediately receives a full snapshot** — including after a reconnect. This is why no separate snapshot endpoint exists. - -`get_all()` returns a shallow copy; since `PriceUpdate` is frozen, the copy is effectively deep and safe to iterate outside the lock. - -### Rolling price history — **TODO** - -`PLAN.md` §6 requires the main chart to be populated the instant a ticker is clicked. `PriceCache` gains a bounded per-ticker deque of `(timestamp, price)`: - -```python -from collections import deque - -HISTORY_MAXLEN = 600 # ~5 minutes at the 500ms simulator cadence - -self._history: dict[str, deque[tuple[float, float]]] = {} -``` - -- Appended inside `update()`, under the same lock. -- `deque(maxlen=600)` evicts the oldest point automatically — no pruning logic. -- `remove()` must drop the ticker's deque too, or removed tickers leak. -- Deliberately **not persisted**. A restart clears it, which is the honest behaviour for a simulator with no real history. - -Memory is negligible: 600 points × 50 tickers × ~16 bytes ≈ 500KB. - -New reader, backing `GET /api/prices/{ticker}/history?limit=600`: - -```python -def get_history(self, ticker: str, limit: int = 600) -> list[tuple[float, float]]: - """Oldest-first (timestamp, price) points. Empty list for an untracked ticker.""" - with self._lock: - points = self._history.get(ticker) - if not points: - return [] - return list(points)[-limit:] -``` - -An untracked ticker returns `[]`, not a 404 — the chart draws nothing rather than erroring (`PLAN.md` §8). - -Under Massive the deque fills at one point per 15-second poll, so five minutes of wall time is 20 points rather than 600. The chart is sparse but correct. Backfilling from `get_aggs` (`MASSIVE_API.md` §5) is the eventual upgrade. - ---- - -## 4. `MarketDataSource` — the abstract contract - -```python -class MarketDataSource(ABC): - @abstractmethod - async def start(self, tickers: list[str]) -> None: ... - @abstractmethod - async def stop(self) -> None: ... - @abstractmethod - async def add_ticker(self, ticker: str) -> None: ... - @abstractmethod - async def remove_ticker(self, ticker: str) -> None: ... - @abstractmethod - def get_tickers(self) -> list[str]: ... -``` - -Five methods, and every one is about *lifecycle and membership* — none of them returns a price. That absence is the whole design. A `get_price()` on this interface would tempt callers into a per-request API hit under Massive and would make the two implementations behave differently under load. - -### Behavioural contract - -Binding on both implementations. A test suite that passes against one should pass against the other. - -| Method | Guarantee | -|---|---| -| `start(tickers)` | Begins a background task writing to the cache. **Seeds the cache before returning**, so the first SSE event is never empty. Called exactly once; calling twice is undefined. | -| `stop()` | Cancels the task and releases resources. **Idempotent.** No writes to the cache afterwards. | -| `add_ticker(t)` | Adds to the tracked set. No-op if present. Simulator seeds a price immediately; Massive picks it up on the next poll. | -| `remove_ticker(t)` | Removes from the tracked set **and from the cache**. No-op if absent. | -| `get_tickers()` | Current tracked set. Synchronous — it reads local state only. | - -Two asymmetries are permitted and must not be papered over: - -- **Seeding latency.** `add_ticker` on the simulator makes a price available immediately; on Massive it takes up to one poll interval. The API contract already accommodates this — `GET /api/watchlist` returns `price: null` until the first tick, and the UI shows `—`. -- **Cadence.** 500ms versus 15s. Readers must never assume a minimum update rate. This is exactly what the SSE keepalive in §7 exists to handle. - -### `remove_ticker` also clears the cache — and why that is dangerous - -Both implementations call `self._cache.remove(ticker)`. That is correct for the interface but makes the method destructive: a held position whose ticker is removed loses its price, and with it its valuation, its P&L, its heatmap tile, and its snapshot contribution. §5 is the rule that prevents it. - ---- - -## 5. Which tickers are tracked - -**The tracked set is `watchlist ∪ {tickers with a non-zero position}`.** - -The two sets diverge the moment a user buys TSLA and then removes it from the watchlist. The position still needs a live price. This rule is the single most important piece of integration logic in the module, because getting it wrong produces a silently frozen position rather than an error. - -| Trigger | Action | -|---|---| -| `POST /api/watchlist` | always `await source.add_ticker(t)` | -| `DELETE /api/watchlist/{t}` | `await source.remove_ticker(t)` **only if no position in `t` is held** | -| Buy a ticker not currently tracked | `await source.add_ticker(t)` as part of trade execution | -| Sell a position to zero | if `t` is not on the watchlist, `await source.remove_ticker(t)` | -| `POST /api/reset` | re-sync the tracked set to exactly the ten default tickers | - -A single helper keeps the rule in one place rather than at four call sites: - -```python -async def untrack_if_unused(source: MarketDataSource, ticker: str) -> None: - """Stop tracking a ticker only if it is neither watched nor held.""" - if is_on_watchlist(ticker) or has_position(ticker): - return - await source.remove_ticker(ticker) -``` - -### Startup - -```python -tickers = sorted(set(get_watchlist_tickers()) | set(get_position_tickers())) -await source.start(tickers) -``` - -Reading both tables at startup — not just the watchlist — is what makes a position held across a restart come back with a live price. - ---- - -## 6. Selection — `create_market_data_source` - -```python -def create_market_data_source(price_cache: PriceCache) -> MarketDataSource: - """Create the market data source indicated by the environment. - - MASSIVE_API_KEY set and non-empty -> MassiveDataSource (real data) - otherwise -> SimulatorDataSource (GBM simulation) - - Returns an unstarted source; the caller must await source.start(tickers). - """ - api_key = os.environ.get("MASSIVE_API_KEY", "").strip() - - if api_key: - logger.info("Market data source: Massive API (real data)") - return MassiveDataSource(api_key=api_key, price_cache=price_cache) - - logger.info("Market data source: GBM Simulator") - return SimulatorDataSource(price_cache=price_cache) -``` - -`.strip()` before the truth test is deliberate: `.env` files routinely contain `MASSIVE_API_KEY=` or a stray space, and a whitespace-only key would otherwise select the Massive path and then fail every poll with a 401. Empty means empty. - -**The simulator is the default, and the fallback is decided once at startup — never at runtime.** A source that silently switched to the simulator after a Massive outage would show users invented prices while they believed they were seeing the market. Rejected keys and failed polls are logged; they do not change the source. `GET /api/health` reports which one is live: - -```json -{"status": "ok", "market_source": "simulator", "llm_mock": false} -``` - -Returning an **unstarted** source keeps construction synchronous and lets the caller decide the ticker set from the database — the factory has no business reading tables. - ---- - -## 7. Lifecycle and wiring - -One `PriceCache` and one source per process, owned by the FastAPI lifespan. - -```python -from contextlib import asynccontextmanager -from fastapi import FastAPI - -from app.market import PriceCache, create_market_data_source, create_stream_router - - -@asynccontextmanager -async def lifespan(app: FastAPI): - cache = PriceCache() - source = create_market_data_source(cache) - - tickers = sorted(set(get_watchlist_tickers()) | set(get_position_tickers())) - await source.start(tickers) - - app.state.price_cache = cache - app.state.market_source = source - try: - yield - finally: - await source.stop() - - -app = FastAPI(lifespan=lifespan) -app.include_router(create_stream_router(app.state.price_cache)) -``` - -The cache and source are passed explicitly (via router factories or `app.state`) rather than held in module globals, which is what keeps tests able to construct an isolated cache per test. - -**Ordering, from `PLAN.md` §11:** mount all `/api/*` routers *before* the static file mount. A `StaticFiles(html=True)` mount at `/` registered first shadows every endpoint, including the SSE stream. - -### The SSE stream - -`GET /api/stream/prices`, `Content-Type: text/event-stream`. The generator opens with `retry: 1000`, then pushes the **entire cache as one JSON object** whenever `version` changes, polled every 500ms: - -``` -retry: 1000 - -data: {"AAPL": {"ticker": "AAPL", "price": 190.52, "previous_price": 190.48, "timestamp": 1755873791.482, "change": 0.04, "change_percent": 0.021, "direction": "up"}, "GOOGL": {...}} -``` - -One event carries every ticker — not one event per ticker. The client replaces its price map wholesale, so there is no merge logic and no missed-update reconciliation. - -### Keepalive — **TODO** - -When the version has not changed for 15 seconds, emit an SSE comment: - -```python -KEEPALIVE_SECONDS = 15.0 - -last_sent = time.monotonic() -while True: - if await request.is_disconnected(): - break - - current_version = price_cache.version - if current_version != last_version: - last_version = current_version - prices = price_cache.get_all() - if prices: - payload = json.dumps({t: u.to_dict() for t, u in prices.items()}) - yield f"data: {payload}\n\n" - last_sent = time.monotonic() - elif time.monotonic() - last_sent >= KEEPALIVE_SECONDS: - yield ": ping\n\n" - last_sent = time.monotonic() - - await asyncio.sleep(interval) -``` - -Without this, a Massive-backed feed sends no bytes between 15-second polls. That idle-times-out through proxies and leaves the frontend unable to distinguish a quiet market from a dead connection. The connection indicator — green on `onopen`, yellow on `onerror`, red after a ping gap beyond ~40 seconds — depends on it. - ---- - -## 8. Implementing a new source - -The interface is small enough that a third source is a contained piece of work. The checklist: - -1. Subclass `MarketDataSource` and implement all five methods. -2. `start()` must **seed the cache before returning**. -3. Never write to the cache after `stop()`; make `stop()` idempotent. -4. `remove_ticker()` must call `cache.remove(ticker)`. -5. Convert timestamps to **Unix epoch seconds as a float** at the boundary. -6. Never let a fetch error kill the background loop — log and retry on the next cycle. -7. If the source is synchronous, wrap every call in `asyncio.to_thread`. -8. Add a branch to `create_market_data_source` and a value to `market_source` in `/api/health`. - -Point 7 is not optional. `massive.RESTClient` is `urllib3`-based and blocking; calling it directly from `async def` stalls the event loop for the duration of the HTTP round trip, which stops the SSE stream and every in-flight request. `MassiveDataSource` gets this right: - -```python -snapshots = await asyncio.to_thread(self._fetch_snapshots) -``` - -### Massive polling, in outline - -```python -async def _poll_loop(self) -> None: - """Poll on interval. The first poll already happened in start().""" - while True: - await asyncio.sleep(self._interval) - await self._poll_once() -``` - -`start()` performs one poll synchronously before creating the task, so the cache is warm before the first client connects. `_poll_interval` defaults to 15 seconds to stay inside the free tier's 5 requests/minute (`MASSIVE_API.md` §2); paid tiers can drop to 2–5 seconds. - -> The `_poll_once` parsing in `massive_client.py` currently reads a non-existent attribute and uses the wrong unit divisor, which means the Massive path writes nothing to the cache at all. Both defects are reproduced and the corrected parse is given in `MASSIVE_API.md` §8. Fixing them is a prerequisite to the Massive path working. - ---- - -## 9. Testing - -Existing coverage: **73 tests passing at 91%** for the market module, measured by running the suite while writing this document. (`MARKET_DATA_SUMMARY.md` still quotes 84%, which is stale.) - -**The cache and the interface can be tested without either real source.** A stub is a few lines, and it is the right tool for testing the tracked-ticker rules: - -```python -class StubDataSource(MarketDataSource): - """Records lifecycle calls; writes nothing on its own.""" - - def __init__(self, cache: PriceCache) -> None: - self._cache = cache - self._tickers: list[str] = [] - self.started = False - - async def start(self, tickers): self._tickers = list(tickers); self.started = True - async def stop(self): self.started = False - async def add_ticker(self, t): - if t not in self._tickers: - self._tickers.append(t) - async def remove_ticker(self, t): - self._tickers = [x for x in self._tickers if x != t] - self._cache.remove(t) - def get_tickers(self): return list(self._tickers) -``` - -What to cover: - -- **Cache** — `previous_price` derivation, first-update `flat`, `version` monotonicity, `remove` clearing both price and history, thread safety under concurrent writers. -- **Factory** — unset, empty, and whitespace-only `MASSIVE_API_KEY` all select the simulator; a real value selects Massive. -- **Tracked set** — removing a watchlist ticker with an open position keeps it in the feed; selling to zero off-watchlist removes it. These are the two regressions that produce a frozen position. -- **Massive parsing** — feed the real `TickerSnapshot.from_dict` a documented payload and assert the cached timestamp lands in the plausible present. Never build these snapshots from `MagicMock`: the existing tests do, which is precisely why 94% coverage of `massive_client.py` still missed both defects (`MASSIVE_API.md` §8.3). -- **SSE** — map-shaped payload, float timestamp, percent-unit `change_percent`, full snapshot on connect, keepalive after 15 idle seconds. -- **History** — deque bounded at 600, oldest-first ordering, `[]` for an untracked ticker. - -```bash -cd backend -uv run pytest -uv run pytest --cov=app --cov-report=term-missing -``` - ---- - -## 10. Summary - -| Concern | Resolution | -|---|---| -| Two sources, one consumer | `MarketDataSource` ABC + shared `PriceCache` | -| Which source | `create_market_data_source`, decided once at startup from `MASSIVE_API_KEY` | -| Default | Simulator — always alive, no key, no rate limit | -| How prices are read | Only from the cache, never from the source | -| Which tickers are live | `watchlist ∪ positions` | -| Timestamp format | Unix epoch seconds (float), converted at each source boundary | -| Update delivery | SSE, full cache per event, on `version` change | -| Blocking I/O | `asyncio.to_thread` at the source, always | -| Outstanding | Rolling history + endpoint, SSE keepalive, the two `massive_client.py` defects | diff --git a/planning/MARKET_SIMULATOR.md b/planning/MARKET_SIMULATOR.md deleted file mode 100644 index 25c60ef..0000000 --- a/planning/MARKET_SIMULATOR.md +++ /dev/null @@ -1,456 +0,0 @@ -# MARKET_SIMULATOR.md — The Market Simulator - -The approach and code structure for simulating stock prices when no `MASSIVE_API_KEY` is configured. This is FinAlly's **default** data source, so it is what almost every user will see. - -Companion documents: `MARKET_INTERFACE.md` (the abstraction it implements) and `MASSIVE_API.md` (the alternative). Implemented in `backend/app/market/simulator.py` and `backend/app/market/seed_prices.py`. - ---- - -## 1. What it must achieve - -The simulator is not a research tool. It exists so that a student who clones the repo and runs one Docker command sees a trading terminal that looks alive, at any hour, on any day, with no account and no API key. - -That sets the bar precisely: - -| Requirement | Why | -|---|---| -| Visible motion every 500ms | The watchlist flashes green and red; a static grid looks broken | -| Motion at a *plausible* scale | AAPL moving $12 per tick destroys the illusion instantly | -| Prices that stay positive | A stock at −$4 is not a rendering bug the user will forgive | -| Correlated moves | Real tech stocks rise together; independent random walks look obviously fake | -| Occasional drama | A flat five minutes is boring; a sudden 3% drop gives the demo a story | -| Any ticker works | The AI chat can add any symbol; "we don't have that one" would feel broken | -| No external dependency | It must run offline, at 3am, on a weekend | - -And explicitly **not** required: predictive value, real historical data, order books, bid-ask spreads, or volume modelling. Nothing in the app consumes them. - ---- - -## 2. The model — Geometric Brownian Motion - -GBM is the standard model for equity prices and the one that satisfies the requirements above almost incidentally. - -``` -S(t + dt) = S(t) · exp( (μ − σ²/2)·dt + σ·√dt·Z ) -``` - -| Symbol | Meaning | -|---|---| -| `S(t)` | Current price | -| `μ` | Annualised drift — expected return | -| `σ` | Annualised volatility | -| `dt` | Time step, as a fraction of a trading year | -| `Z` | Standard normal draw, correlated across tickers | - -Three properties earn its place here: - -**Prices cannot go negative.** The update is multiplicative — `exp(...)` is always positive, so `S` never crosses zero. No clamping, no `max(price, 0.01)` guard, no special case. An additive random walk would need all three. - -**Returns scale correctly with time.** Volatility is expressed per *year*, and `√dt` converts it to the tick. Change the tick rate and the price series keeps the same annualised character. The 500ms cadence is a display choice, not a modelling parameter. - -**The `−σ²/2` term keeps the drift honest.** Without it, `μ` would not be the expected return of the price — a well-known artefact of the log-normal distribution. It costs one subtraction and makes the parameters mean what they say. - -### Sizing `dt` - -`dt` is expressed against a **trading** year, not a calendar year — markets are closed most of the time, and using 365×24h would understate per-tick moves by a factor of about 4.5. - -```python -TRADING_SECONDS_PER_YEAR = 252 * 6.5 * 3600 # 5,896,800 -DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR # ~8.479e-8 -``` - -252 trading days × 6.5 hours × 3600 seconds. A 500ms tick is therefore `8.479e-8` of a year, and `√dt = 2.912e-4`. - -What that produces per tick, for `σ·√dt` at the seed prices: - -| Ticker | σ | Per-tick σ | Per-tick $ | Per-minute $ (120 ticks) | -|---|---|---|---|---| -| AAPL | 0.22 | 0.0064% | $0.012 | $0.13 | -| JPM | 0.18 | 0.0052% | $0.010 | $0.11 | -| NVDA | 0.40 | 0.0116% | $0.093 | $1.02 | -| TSLA | 0.50 | 0.0146% | $0.036 | $0.40 | - -This is the number that decides whether the simulation looks right. Around a cent or two per tick on a $200 stock means prices are **rounded to 2 decimals into a genuinely different value most ticks** — so the UI flashes constantly — while a minute of drift stays in the tens of cents, which is what a real quote screen looks like. Larger and it reads as a crash; smaller and the grid appears frozen. - ---- - -## 3. Correlation via Cholesky decomposition - -Independent draws per ticker would show tech stocks moving in opposite directions half the time. Real markets do not do that, and the eye notices immediately. - -The fix is standard: draw `n` independent standard normals, then multiply by the Cholesky factor `L` of the desired correlation matrix `C`, where `C = L·Lᵀ`. The resulting vector has exactly the correlation structure of `C`. - -```python -z_independent = np.random.standard_normal(n) -z_correlated = self._cholesky @ z_independent -``` - -### The correlation structure - -Sector membership and coefficients live in `seed_prices.py`, not in the simulator: - -```python -CORRELATION_GROUPS = { - "tech": {"AAPL", "GOOGL", "MSFT", "AMZN", "META", "NVDA", "NFLX"}, - "finance": {"JPM", "V"}, -} - -INTRA_TECH_CORR = 0.6 # tech stocks move together -INTRA_FINANCE_CORR = 0.5 # finance stocks move together -CROSS_GROUP_CORR = 0.3 # between sectors, and for unknown tickers -TSLA_CORR = 0.3 # TSLA does its own thing -``` - -Resolved pairwise, first match winning: - -```python -@staticmethod -def _pairwise_correlation(t1: str, t2: str) -> float: - tech = CORRELATION_GROUPS["tech"] - finance = CORRELATION_GROUPS["finance"] - - # TSLA is in the tech set but behaves independently - if t1 == "TSLA" or t2 == "TSLA": - return TSLA_CORR - - if t1 in tech and t2 in tech: - return INTRA_TECH_CORR - if t1 in finance and t2 in finance: - return INTRA_FINANCE_CORR - - return CROSS_GROUP_CORR -``` - -The TSLA clause is checked first on purpose: TSLA is a member of the tech set for every other purpose, but empirically trades on its own news, and a demo where TSLA visibly decouples from the pack is more convincing than one where everything moves in lockstep. - -`CROSS_GROUP_CORR` doubles as the default for any symbol the simulator has never heard of, which is what makes §5 work. - -### Rebuilding - -`_rebuild_cholesky()` runs on every add and remove — `O(n²)` to build the matrix plus `O(n³)` to factor it, on `n < 50`. That is microseconds, and watchlist edits are a human-speed operation, so caching it would be complexity without benefit. - -```python -def _rebuild_cholesky(self) -> None: - n = len(self._tickers) - if n <= 1: - self._cholesky = None # a single ticker needs no correlation - return - - corr = np.eye(n) - for i in range(n): - for j in range(i + 1, n): - rho = self._pairwise_correlation(self._tickers[i], self._tickers[j]) - corr[i, j] = rho - corr[j, i] = rho - - self._cholesky = np.linalg.cholesky(corr) -``` - -`step()` falls back to uncorrelated draws when `_cholesky is None`, which covers both the single-ticker and empty cases. - -> **Known risk.** `np.linalg.cholesky` raises `LinAlgError` on a matrix that is not positive definite, and this call is unguarded. The current block structure (0.6 / 0.5 / 0.3) was verified positive definite at 7, 20, and 40 tickers, so it is safe as configured — but raising `INTRA_TECH_CORR` toward 1.0, or adding a group whose intra-group correlation is below the cross-group value, can break positive-definiteness and take down `add_ticker`. Anyone editing these constants should re-run the check in §8. - ---- - -## 4. The tick - -`step()` is the hot path — every 500ms, for every ticker. - -```python -def step(self) -> dict[str, float]: - """Advance all tickers by one time step. Returns {ticker: new_price}.""" - n = len(self._tickers) - if n == 0: - return {} - - z_independent = np.random.standard_normal(n) - if self._cholesky is not None: - z_correlated = self._cholesky @ z_independent - else: - z_correlated = z_independent - - result: dict[str, float] = {} - for i, ticker in enumerate(self._tickers): - params = self._params[ticker] - mu, sigma = params["mu"], params["sigma"] - - drift = (mu - 0.5 * sigma**2) * self._dt - diffusion = sigma * math.sqrt(self._dt) * z_correlated[i] - self._prices[ticker] *= math.exp(drift + diffusion) - - if random.random() < self._event_prob: - shock_magnitude = random.uniform(0.02, 0.05) - shock_sign = random.choice([-1, 1]) - self._prices[ticker] *= 1 + shock_magnitude * shock_sign - - result[ticker] = round(self._prices[ticker], 2) - - return result -``` - -Two details worth pointing out: - -**Full precision is kept internally; only the returned value is rounded.** `self._prices[ticker]` stays a full float. Rounding the stored state would accumulate quantisation error into a slow, systematic drift over thousands of ticks. - -**One `standard_normal(n)` call per tick, not `n` calls.** A single vectorised draw feeding one matrix multiply is the reason this stays negligible at 500ms. - -### Random events - -```python -event_probability: float = 0.001 # per ticker, per tick -``` - -A 2–5% jump in either direction. With 10 tickers at 2 ticks/second, the expected wait is `1 / (10 × 2 × 0.001) = 50 seconds` — frequent enough that something interesting happens during a demo, rare enough that the price series is not pure noise. - -The shock multiplies the price directly rather than feeding through GBM, so it is a genuine discontinuity — a gap, which is what real news does to a stock. - ---- - -## 5. Unknown tickers - -Any symbol passing the API-level pattern `^[A-Z][A-Z.]{0,5}$` works, with no allowlist. The AI chat can add anything the user names, and it behaves plausibly. - -```python -def _add_ticker_internal(self, ticker: str) -> None: - if ticker in self._prices: - return - self._tickers.append(ticker) - self._prices[ticker] = SEED_PRICES.get(ticker, random.uniform(50.0, 300.0)) - self._params[ticker] = TICKER_PARAMS.get(ticker, dict(DEFAULT_PARAMS)) -``` - -- **Price**: seeded from `SEED_PRICES`, else uniform in $50–$300 — the range where most large-cap US equities actually trade. -- **Parameters**: `TICKER_PARAMS`, else `DEFAULT_PARAMS` (`σ=0.25`, `μ=0.05`) — a mid-range large cap. -- **Correlation**: no sector membership, so `CROSS_GROUP_CORR` (0.3) against everything. - -`dict(DEFAULT_PARAMS)` copies rather than sharing the module-level dict — without the copy, per-ticker parameter tuning would mutate the default for every unknown ticker at once. - -This is a real advantage over the Massive path, where an unknown symbol simply never produces a price and sits at `—` forever (`MASSIVE_API.md` §9). - ---- - -## 6. Code structure - -Two classes with a clean split: **`GBMSimulator` is pure and synchronous; `SimulatorDataSource` handles async lifecycle and the cache.** - -``` -┌──────────────────────────────────────────────────────────┐ -│ SimulatorDataSource(MarketDataSource) │ -│ owns the asyncio task, writes to PriceCache │ -│ start / stop / add_ticker / remove_ticker / get_tickers│ -│ │ │ -│ ▼ │ -│ GBMSimulator │ -│ pure math, no I/O, no async, no cache reference │ -│ step() -> {ticker: price} │ -└──────────────────────────────────────────────────────────┘ - │ - ▼ - seed_prices.py - constants only, no logic -``` - -The separation pays off in testing: `GBMSimulator` needs no event loop, no cache, and no mocks. Statistical properties are asserted by calling `step()` in a loop. - -### `GBMSimulator` — pure math - -```python -class GBMSimulator: - TRADING_SECONDS_PER_YEAR = 252 * 6.5 * 3600 - DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR - - def __init__(self, tickers, dt=DEFAULT_DT, event_probability=0.001): ... - - def step(self) -> dict[str, float]: ... - def add_ticker(self, ticker: str) -> None: ... - def remove_ticker(self, ticker: str) -> None: ... - def get_price(self, ticker: str) -> float | None: ... - def get_tickers(self) -> list[str]: ... -``` - -State is three parallel dicts keyed by ticker (`_prices`, `_params`) plus `_tickers` as the **ordered** list that indexes into the Cholesky matrix. Order matters: row `i` of `_cholesky` corresponds to `_tickers[i]`, which is why add and remove must both rebuild. - -`__init__` adds every ticker via `_add_ticker_internal` and rebuilds Cholesky **once** at the end, rather than rebuilding per ticker — `O(n³)` instead of `O(n⁴)` on startup. - -### `SimulatorDataSource` — the async wrapper - -```python -class SimulatorDataSource(MarketDataSource): - def __init__(self, price_cache, update_interval=0.5, event_probability=0.001): ... - - async def start(self, tickers: list[str]) -> None: - self._sim = GBMSimulator(tickers=tickers, event_probability=self._event_prob) - # Seed the cache with initial prices so SSE has data immediately - for ticker in tickers: - price = self._sim.get_price(ticker) - if price is not None: - self._cache.update(ticker=ticker, price=price) - self._task = asyncio.create_task(self._run_loop(), name="simulator-loop") - - async def _run_loop(self) -> None: - while True: - try: - if self._sim: - for ticker, price in self._sim.step().items(): - self._cache.update(ticker=ticker, price=price) - except Exception: - logger.exception("Simulator step failed") - await asyncio.sleep(self._interval) -``` - -Three deliberate choices: - -**Seed the cache in `start()` before creating the task.** The first SSE event then carries real prices rather than an empty object, so the watchlist never renders as ten dashes on load. - -**`add_ticker` seeds immediately.** The new ticker has a price on the very next SSE event, with no wait for the following step — the reason adding a ticker feels instant. - -**The `try` is inside the loop, around the step.** An exception logs and the loop continues on the next interval. Wrapping the loop instead would let one bad tick kill the feed permanently. This is the one place defensive handling is warranted: the background task has no caller to propagate to, and a dead price feed is a dead app. - -`stop()` cancels the task and awaits it, swallowing `CancelledError` — the normal shutdown path, not an error. - -### Parameters - -`seed_prices.py` holds constants only. Prices are realistic as of project creation; `σ` and `μ` are annualised. - -| Ticker | Seed | σ | μ | Note | -|---|---|---|---|---| -| AAPL | $190 | 0.22 | 0.05 | | -| GOOGL | $175 | 0.25 | 0.05 | | -| MSFT | $420 | 0.20 | 0.05 | | -| AMZN | $185 | 0.28 | 0.05 | | -| TSLA | $250 | 0.50 | 0.03 | High volatility, decorrelated | -| NVDA | $800 | 0.40 | 0.08 | High volatility, strong drift | -| META | $500 | 0.30 | 0.05 | | -| JPM | $195 | 0.18 | 0.04 | Low volatility (bank) | -| V | $280 | 0.17 | 0.04 | Low volatility (payments) | -| NFLX | $600 | 0.35 | 0.05 | | -| *unknown* | $50–300 | 0.25 | 0.05 | `DEFAULT_PARAMS` | - -The σ spread is what makes the watchlist readable at a glance: V and JPM barely move while NVDA and TSLA jump, so the grid has visible texture instead of ten tickers twitching identically. - ---- - -## 7. Behaviour over time - -There is **no mean reversion and no session boundary.** Prices random-walk from their seed for as long as the container runs. Over a demo — minutes to hours — drift is small and the series looks like a trading day. Over a week of uptime, a ticker may wander far from its seed. That is correct GBM behaviour and not worth correcting: the state is in memory only, so a restart returns everything to the seed prices. - -That in turn is why the rolling price history is not persisted (`MARKET_INTERFACE.md` §3). A restart legitimately resets the world. - ---- - -## 8. Testing - -Existing coverage is **73 tests passing at 91%** across the market module (`simulator.py` itself is at 98%). `GBMSimulator` is pure, so its tests are fast and deterministic under a seeded RNG. - -**Deterministic tests** — seed both RNGs, since the simulator uses `numpy.random` for the normal draws and the stdlib `random` for events: - -```python -def test_step_is_reproducible(): - np.random.seed(42) - random.seed(42) - sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) - first = sim.step() - - np.random.seed(42) - random.seed(42) - sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) - assert sim.step() == first -``` - -**Structural properties:** - -- Prices stay strictly positive over many thousands of steps. -- `step()` returns exactly the current ticker set. -- Add/remove keeps `_tickers`, `_prices`, and `_params` consistent, and the Cholesky shape matches `len(_tickers)`. -- Unknown tickers seed within $50–$300 and get `DEFAULT_PARAMS`. -- `remove_ticker` on an untracked symbol is a no-op, not an error. - -**Statistical properties** — over enough steps, with a tolerance: - -```python -def test_realised_volatility_is_close_to_sigma(): - sim = GBMSimulator(tickers=["AAPL"], event_probability=0.0) - prices = [sim.get_price("AAPL")] - for _ in range(20_000): - prices.append(sim.step()["AAPL"]) - - log_returns = np.diff(np.log(prices)) - realised = log_returns.std() / np.sqrt(GBMSimulator.DEFAULT_DT) - assert 0.15 < realised < 0.35 # nominal sigma is 0.22 -``` - -Disable events (`event_probability=0.0`) for statistical tests — a 5% jump is a massive outlier at this dt and will dominate the sample variance. Keep tolerances wide; these are sampling estimates, and a tight bound produces a test that fails a few times a year for no reason. - -**Correlation:** - -```python -def test_tech_tickers_are_positively_correlated(): - sim = GBMSimulator(tickers=["AAPL", "MSFT"], event_probability=0.0) - a, m = [], [] - for _ in range(10_000): - p = sim.step() - a.append(p["AAPL"]) - m.append(p["MSFT"]) - - rho = np.corrcoef(np.diff(np.log(a)), np.diff(np.log(m)))[0, 1] - assert rho > 0.4 # nominal 0.6 -``` - -**Cholesky positive-definiteness** — run after any change to the correlation constants: - -```python -def test_correlation_matrix_stays_positive_definite(): - tickers = list(SEED_PRICES) + [f"UNK{i}" for i in range(40)] - GBMSimulator(tickers=tickers) # raises LinAlgError if not PD -``` - -**`SimulatorDataSource`** needs an event loop and a real `PriceCache`, but no mocks: - -- `start()` populates the cache before returning. -- The cache updates after roughly one interval. -- `add_ticker` seeds a price immediately. -- `remove_ticker` clears the ticker from the cache. -- `stop()` is idempotent and halts writes. - -```bash -cd backend -uv run pytest tests/market/ -uv run pytest --cov=app --cov-report=term-missing -``` - -`backend/market_data_demo.py` is a Rich terminal demo of the live simulator — the fastest way to eyeball whether a parameter change still looks right. - ---- - -## 9. Tuning guide - -Everything worth adjusting, and what it costs: - -| Want | Change | Watch for | -|---|---|---| -| More visible price motion | Raise `σ` in `TICKER_PARAMS` | Above ~0.8 it stops looking like equity | -| Faster or slower updates | `update_interval` on `SimulatorDataSource` | `DEFAULT_DT` is derived from 0.5s; change both together or annualised σ shifts | -| More frequent drama | Raise `event_probability` | Above ~0.005 the series becomes jumps, not prices | -| Bigger shocks | Widen `random.uniform(0.02, 0.05)` | Beyond ~10% the P&L chart loses all detail | -| Different sector behaviour | Edit `CORRELATION_GROUPS` and the coefficients | Re-run the positive-definiteness test in §8 | -| Different starting prices | `SEED_PRICES` | Unlisted tickers still land in $50–$300 | -| A trending market | Raise `μ` | `μ` is annualised; even 0.5 is barely visible over a demo | - -The `DEFAULT_DT` coupling is the one that catches people. `DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR` hard-codes the 500ms tick. Passing `update_interval=0.1` to `SimulatorDataSource` without also passing a matching `dt` to `GBMSimulator` makes the simulation run five times faster in model time — annualised volatility silently becomes 5× what `TICKER_PARAMS` claims. - ---- - -## 10. Summary - -| Concern | Approach | -|---|---| -| Price model | Geometric Brownian Motion, per-ticker `μ` and `σ` | -| Positivity | Guaranteed by the multiplicative `exp` form — no clamping | -| Time step | `0.5s / (252 × 6.5 × 3600)` ≈ `8.479e-8` of a trading year | -| Correlation | Cholesky factor of a sector-block matrix, rebuilt on add/remove | -| Sectors | tech 0.6, finance 0.5, cross-sector and unknown 0.3, TSLA 0.3 | -| Drama | 0.1% chance per ticker per tick of a 2–5% jump — about one per 50s | -| Unknown tickers | Random $50–$300 seed, `DEFAULT_PARAMS`, cross-sector correlation | -| Structure | Pure `GBMSimulator` + async `SimulatorDataSource`, constants in `seed_prices.py` | -| Persistence | None — a restart returns to seed prices | -| Failure handling | Per-step `try` inside the loop; the feed never dies from one bad tick | diff --git a/planning/MASSIVE_API.md b/planning/MASSIVE_API.md deleted file mode 100644 index d7f94ef..0000000 --- a/planning/MASSIVE_API.md +++ /dev/null @@ -1,525 +0,0 @@ -# MASSIVE_API.md — Massive (formerly Polygon.io) REST API - -Reference for retrieving real-time and end-of-day prices for multiple tickers. - -**Verified against `massive` Python SDK `2.2.0`** (installed in `backend/.venv`) and the official docs at . Every signature, field name, and unit below was checked against the installed package source or the live documentation — not from memory. - -> No `MASSIVE_API_KEY` was available in this repo when this document was written, so responses could not be exercised against the live service. Field shapes come from the SDK's `from_dict` parsers and the published response schemas, which is authoritative for how the client will deserialize. The verification script in §10 closes the loop once a key exists. - ---- - -## 1. Orientation - -Polygon.io rebranded to **Massive** in 2026. The API surface, the API keys, and the endpoint paths are unchanged; the hostname and the Python package are new. - -| | Value | -|---|---| -| Base URL | `https://api.massive.com` | -| Python SDK | `massive` (PyPI), currently `2.2.0` | -| Repository | | -| Auth | `Authorization: Bearer ` header | -| Env var read by the SDK | `MASSIVE_API_KEY` | - -The SDK reads the same environment variable name this project already uses, so `RESTClient()` with no arguments works when `MASSIVE_API_KEY` is exported. FinAlly passes the key explicitly instead, because the factory has already read and validated it. - -### Install - -```bash -uv add massive -``` - -### Authentication - -The SDK sets the header for you (`massive/rest/base.py`): - -```python -self.headers = { - "Authorization": "Bearer " + self.API_KEY, - "Accept-Encoding": "gzip", - "User-Agent": f"Massive.com PythonClient/{version_number}", -} -``` - -Constructing a client with no key and no env var raises `massive.exceptions.AuthError` immediately — it does not wait for the first request. - -```python -from massive import RESTClient - -client = RESTClient(api_key="YOUR_KEY") # or RESTClient() to read MASSIVE_API_KEY -``` - -Full constructor defaults, from the installed SDK: - -```python -RESTClient( - api_key: str | None = None, - connect_timeout: float = 10.0, - read_timeout: float = 10.0, - num_pools: int = 10, - retries: int = 3, # urllib3 Retry on 413/429/499/500/502/503... - base: str = "https://api.massive.com", - pagination: bool = True, - verbose: bool = False, - trace: bool = False, - custom_json: Any | None = None, -) -``` - -Two consequences worth knowing: - -- **`RESTClient` is synchronous.** It uses `urllib3.PoolManager`. Calling it from an `async def` blocks the event loop, which in this app means visibly stuttering prices on the SSE stream. Always wrap it in `asyncio.to_thread`. -- **It retries 429 internally** (3 attempts, honouring `Retry-After`). A poll that hits the rate limit therefore blocks its worker thread rather than failing fast. - ---- - -## 2. Plans and rate limits - -| Plan | Requests/min | Data freshness | -|---|---|---| -| Basic (free) | **5** | End-of-day, and 15-minute-delayed intraday | -| Paid (Starter and above) | Unlimited | Real-time (15-min delayed on Starter) | - -This single number drives the whole polling design: **5 requests/minute means one request every 12 seconds at best.** FinAlly polls every 15 seconds by default, which leaves headroom and stays under the limit even if a poll overruns. - -The corollary is that per-ticker endpoints are unusable on the free tier — 10 watchlist tickers via `get_last_trade` would be 10 requests per cycle, blowing the budget in one poll. **The design must fetch all tickers in a single request**, which is what §3 is about. - ---- - -## 3. Real-time prices for multiple tickers - -### 3.1 Full Market Snapshot (v2) — the primary endpoint - -`GET /v2/snapshot/locale/us/markets/stocks/tickers` - -One request returns the current state of every ticker you name. This is the endpoint FinAlly uses. - -| Query param | Meaning | -|---|---| -| `tickers` | Case-insensitive comma-separated list. Omit to get the entire US market. | -| `include_otc` | Include OTC securities. Default `false`. | - -SDK signature: - -```python -client.get_snapshot_all( - market_type: str | SnapshotMarketType, - tickers: str | list[str] | None = None, - include_otc: bool | None = False, - params: dict | None = None, - raw: bool = False, -) -> list[TickerSnapshot] -``` - -The SDK joins a list into a comma-separated string for you (`",".join(tickers)`), so passing a `list[str]` is correct and idiomatic. - -```python -from massive import RESTClient -from massive.rest.models import SnapshotMarketType - -client = RESTClient(api_key="YOUR_KEY") - -snapshots = client.get_snapshot_all( - market_type=SnapshotMarketType.STOCKS, - tickers=["AAPL", "GOOGL", "MSFT", "AMZN", "TSLA"], -) - -for snap in snapshots: - print(snap.ticker, snap.last_trade.price, snap.todays_change_percent) -``` - -**Response shape** (`TickerSnapshot`, from `massive/rest/models/snapshot.py`): - -| Attribute | JSON key | Type | Notes | -|---|---|---|---| -| `ticker` | `ticker` | `str` | | -| `todays_change` | `todaysChange` | `float` | Absolute change vs. prior close | -| `todays_change_percent` | `todaysChangePerc` | `float` | **Already in percent units** (`0.39` = 0.39%) | -| `updated` | `updated` | `int` | **Nanoseconds** | -| `day` | `day` | `Agg` | Today's bar so far | -| `prev_day` | `prevDay` | `Agg` | Previous session's bar | -| `min` | `min` | `MinuteSnapshot` | Most recent minute bar | -| `last_trade` | `lastTrade` | `LastTrade` | Most recent execution | -| `last_quote` | `lastQuote` | `LastQuote` | Most recent NBBO | -| `fair_market_value` | `fmv` | `float` | Business plans only; `None` otherwise | - -`LastTrade` — note the attribute names, they do **not** match the JSON keys: - -| Attribute | JSON key | Units | -|---|---|---| -| `price` | `p` | dollars | -| `size` | `s` | shares | -| `sip_timestamp` | `t` | **nanoseconds** | -| `exchange` | `x` | exchange ID | -| `conditions` | `c` | `list[int]` | -| `id` | `i` | trade ID | -| `ticker` | `T` | usually `None` inside a snapshot | - -`Agg` (used for `day` and `prev_day`): `open`/`o`, `high`/`h`, `low`/`l`, `close`/`c`, `volume`/`v`, `vwap`/`vw`, `timestamp`/`t`, `transactions`/`n`. - -### 3.2 Unified Snapshot (v3) — the alternative - -`GET /v3/snapshot` - -Multi-asset-class, paginated, and — the reason it is worth mentioning — it **reports unknown tickers explicitly** instead of silently omitting them. - -```python -snaps = client.list_universal_snapshots( - type="stocks", - ticker_any_of=["AAPL", "NVDA", "NOTAREALTICKER"], - limit=250, -) - -for s in snaps: - if s.error: - print(f"{s.ticker}: {s.error} — {s.message}") # e.g. NOT_FOUND - else: - print(s.ticker, s.session.close, s.last_trade.price) -``` - -- `ticker_any_of` accepts **up to 250** tickers. -- `limit` defaults to 10 and maxes at 250 — **leave it at the default and you will silently get only 10 results.** Always set it explicitly. -- Returns an *iterator* and auto-paginates (`pagination=True`), so a careless call can fan out into many billed requests. With `ticker_any_of` bounded at 250 and `limit=250` there is exactly one page. - -FinAlly stays on v2 because a 10-ticker watchlist never approaches the 250 limit, v2 is a single non-paginated request, and the `error` field is of marginal value when the simulator is the default path anyway. v3 is the right upgrade if per-ticker validation feedback is ever wanted. - -### 3.3 Single ticker - -Useful for a one-off lookup; unusable as a polling strategy on the free tier. - -```python -snap = client.get_snapshot_ticker(SnapshotMarketType.STOCKS, "AAPL") -trade = client.get_last_trade("AAPL") # LastTrade: .price, .size, .sip_timestamp (ns) -quote = client.get_last_quote("AAPL") # LastQuote: .bid_price, .ask_price, ... -``` - ---- - -## 4. End-of-day prices - -### 4.1 Previous close — per ticker - -`GET /v2/aggs/ticker/{ticker}/prev` - -```python -prev = client.get_previous_close_agg("AAPL", adjusted=True) -print(prev.ticker, prev.open, prev.high, prev.low, prev.close, prev.volume, prev.vwap) -``` - -`PreviousCloseAgg` fields: `ticker`, `open`, `high`, `low`, `close`, `volume`, `vwap`, `timestamp` (**milliseconds**, start of the aggregate window). - -Works on the free tier. One request per ticker, so 10 tickers = 10 requests = two minutes of free-tier budget. - -### 4.2 Daily market summary — the whole market in one request - -`GET /v2/aggs/grouped/locale/us/market/stocks/{date}` - -The efficient way to get EOD for many tickers: **one request returns every US ticker for that date.** - -```python -from datetime import date - -bars = client.get_grouped_daily_aggs(date="2026-08-28", adjusted=True) - -wanted = {"AAPL", "GOOGL", "MSFT"} -closes = {b.ticker: b.close for b in bars if b.ticker in wanted} -print(closes) -``` - -`GroupedDailyAgg` adds a `ticker` attribute (JSON key `T`) to the standard `Agg` fields. `timestamp`/`t` is **milliseconds**, marking the *end* of the aggregate window. - -Caveats: the date must be a **trading day** — a weekend or holiday returns an empty result set, not an error. And the response covers the entire market (thousands of rows), so filter client-side. - -### 4.3 Daily open/close for one ticker on one date - -`GET /v1/open-close/{ticker}/{date}` - -```python -oc = client.get_daily_open_close_agg("AAPL", date="2026-08-28", adjusted=True) -print(oc.open, oc.close, oc.pre_market, oc.after_hours, oc.status) -``` - -`DailyOpenCloseAgg` is the one model that carries pre-market and after-hours prints. Note it uses `symbol` (not `ticker`) and `from_` (not `from`, which is a Python keyword). - ---- - -## 5. Historical bars — for charts and backfill - -`GET /v2/aggs/ticker/{ticker}/range/{multiplier}/{timespan}/{from}/{to}` - -```python -# 1-minute bars for one session -bars = client.get_aggs( - ticker="AAPL", - multiplier=1, - timespan="minute", # second|minute|hour|day|week|month|quarter|year - from_="2026-08-28", # YYYY-MM-DD, date, datetime, or Unix ms - to="2026-08-28", - adjusted=True, - sort="asc", - limit=50000, -) - -for b in bars: - print(b.timestamp, b.open, b.high, b.low, b.close, b.volume) -``` - -`get_aggs` returns a **list** and is capped at 50,000 bars. `list_aggs` takes the same arguments but returns an **auto-paginating iterator** — convenient for long ranges, and a way to accidentally issue many billed requests. Prefer `get_aggs` with an explicit range unless you genuinely need more than 50k bars. - -`Agg.timestamp` is **milliseconds**, marking the start of the window. - -Relevance to FinAlly: this is the only way to seed a chart with real history under Massive. The rolling in-memory history in §6 of `PLAN.md` covers the simulator; a Massive-backed deployment could optionally backfill `GET /api/prices/{ticker}/history` from 1-minute aggregates instead. That is out of scope today, and noted here so the option is not rediscovered later. - ---- - -## 6. Market status - -Worth calling to explain a frozen feed to the user rather than leaving them guessing. - -```python -status = client.get_market_status() -print(status.market) # "open" | "closed" | "extended-hours" -print(status.exchanges) -print(status.after_hours, status.early_hours) -``` - -`client.get_market_holidays()` returns upcoming closures and early closes. - ---- - -## 7. Timestamp units — the trap - -Massive uses **three different time units across endpoints**, and the SDK passes them through unchanged. This is the single easiest thing to get wrong. - -| Source | Attribute | Unit | To Unix seconds | -|---|---|---|---| -| Snapshot `lastTrade` | `sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | -| Snapshot `lastQuote` | `sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | -| Snapshot top level | `updated` | **nanoseconds** | `/ 1_000_000_000` | -| Snapshot `min` | `timestamp` | **milliseconds** | `/ 1_000` | -| Aggregates (`Agg`, `PreviousCloseAgg`, grouped) | `timestamp` | **milliseconds** | `/ 1_000` | - -FinAlly's `PriceUpdate.timestamp` is **Unix epoch seconds as a float** (§7 of `PLAN.md`), so every value from this API needs converting, and the divisor depends on which endpoint it came from. - -```python -NANOS_PER_SECOND = 1_000_000_000 -MILLIS_PER_SECOND = 1_000 - -ts_seconds = snap.last_trade.sip_timestamp / NANOS_PER_SECOND # snapshot -ts_seconds = agg.timestamp / MILLIS_PER_SECOND # aggregates -``` - -### Attribute names never match JSON keys - -The wire format is single-letter (`p`, `s`, `t`, `x`); the SDK's `from_dict` maps those to readable attributes. You must use the **attribute** names. Reading `snap.last_trade.t` or `snap.last_trade.timestamp` raises `AttributeError`, because `@modelclass` builds a plain dataclass with no `__getattr__` fallback: - -```python -# massive/rest/models/trades.py -@staticmethod -def from_dict(d): - return LastTrade( - d.get("T"), d.get("f"), d.get("q"), d.get("t"), # "t" -> sip_timestamp - d.get("y"), d.get("c"), d.get("e"), d.get("i"), - d.get("p"), # "p" -> price - d.get("r"), d.get("s"), d.get("x"), d.get("z"), - ) -``` - ---- - -## 8. Two defects confirmed in `backend/app/market/massive_client.py` - -Both were reproduced against the installed SDK, not inferred. They are recorded here because this document is the reference the fix should be written from; the fix itself belongs to whoever next touches that module. - -### 8.1 `last_trade.timestamp` does not exist — the Massive path returns no prices at all - -`_poll_once` reads: - -```python -price = snap.last_trade.price -timestamp = snap.last_trade.timestamp / 1000.0 # AttributeError -``` - -Reproduction, using a payload shaped exactly as the v2 snapshot documentation specifies: - -```python -from massive.rest.models.snapshot import TickerSnapshot - -snap = TickerSnapshot.from_dict({ - "ticker": "AAPL", - "lastTrade": {"p": 190.52, "s": 100, "t": 1755873791482000000, "x": 4}, -}) - -snap.last_trade.price # 190.52 -snap.last_trade.sip_timestamp # 1755873791482000000 -snap.last_trade.timestamp # AttributeError: 'LastTrade' object has no attribute 'timestamp' -``` - -The loop wraps each snapshot in `except (AttributeError, TypeError)` and merely logs a warning, so the exception is swallowed **once per ticker, on every poll**. The cache is never written. The observable symptom is not a crash: it is a watchlist where every ticker shows `—` forever, with `Skipping snapshot for AAPL` in the logs. - -The correct attribute is `sip_timestamp`. - -### 8.2 The unit divisor is wrong by a factor of 10⁶ - -Even with the attribute corrected, `/ 1000.0` treats nanoseconds as milliseconds. `1755873791482000000 / 1000` is ≈ 1.76 × 10¹⁵ seconds — roughly 55 million years in the future. Charts keyed on that timestamp would be unusable. The divisor must be `1_000_000_000`. - -### 8.3 Why 94% test coverage did not catch either defect - -`massive_client.py` is 94% covered and all 73 tests pass. The tests nonetheless assert the buggy behaviour, because they build snapshots from `MagicMock` (`backend/tests/market/test_massive.py`): - -```python -def _make_snapshot(ticker: str, price: float, timestamp_ms: int) -> MagicMock: - snap = MagicMock() - snap.last_trade = MagicMock() - snap.last_trade.price = price - snap.last_trade.timestamp = timestamp_ms # attribute the real model does not have - return snap -``` - -A `MagicMock` answers to any attribute name, so `snap.last_trade.timestamp` resolves happily in the test and raises `AttributeError` in production. `test_timestamp_conversion` then locks in the wrong unit as well: - -```python -assert update.timestamp == 1707580800.0 # asserts milliseconds -> seconds -``` - -The lesson generalises: **mocking a third-party model tests your assumptions about the library, not the library.** Parsing tests must go through the real `TickerSnapshot.from_dict` with a documented payload, as in §10. That form of test needs no network and would have failed on the first run. - -### Corrected parse - -```python -NANOS_PER_SECOND = 1_000_000_000 - -for snap in snapshots: - trade = snap.last_trade - if trade is None or trade.price is None: - continue # no print yet today; leave the ticker showing "—" - self._cache.update( - ticker=snap.ticker, - price=trade.price, - timestamp=( - trade.sip_timestamp / NANOS_PER_SECOND - if trade.sip_timestamp - else time.time() - ), - ) -``` - -Guarding on `is None` rather than catching `AttributeError` is what makes the difference: a genuinely absent field is a normal condition to handle, whereas a misspelled attribute is a bug that should be loud. The existing blanket `except AttributeError` is precisely what hid this one. - ---- - -## 9. Errors and operational behaviour - -The SDK raises only two exception types (`massive/exceptions.py`): - -| Exception | Cause | -|---|---| -| `AuthError` | Empty or missing API key at construction time | -| `BadResponse` | Any non-200 response that survived the retry policy | - -`urllib3` raises its own errors for connection failures and timeouts. A poll loop should therefore catch broadly and keep going, since a failed poll is recoverable on the next cycle: - -```python -from massive.exceptions import AuthError, BadResponse - -try: - snapshots = await asyncio.to_thread(self._fetch_snapshots) -except AuthError: - logger.error("Massive API key rejected — falling back is not automatic") - raise # unrecoverable: do not retry on a loop -except BadResponse as e: - logger.warning("Massive returned an error response: %s", e) - return # transient: retry next interval -except Exception: - logger.exception("Massive poll failed") - return -``` - -### Behaviours to surface in the README - -These are properties of the data source, not bugs, and users will otherwise report them as bugs: - -- **Unknown symbols vanish silently.** The v2 snapshot omits tickers it does not recognise; there is no error entry. The ticker sits in the watchlist showing `—` indefinitely. (v3 would report `NOT_FOUND` — see §3.2.) -- **Prices freeze outside market hours.** Overnight, at weekends, and on holidays the snapshot returns the last trade of the previous session. The UI looks broken but is correct. This is the main reason the simulator is the default. -- **Free-tier data is 15 minutes delayed**, so prices will not match any other quote source the user has open. -- **Snapshot data is cleared at midnight ET** and repopulates from about 4am ET. Between those times `last_trade` may be absent entirely — which is exactly the `None` case §8 guards. - ---- - -## 10. Verification script - -Run this once a real `MASSIVE_API_KEY` is available. It confirms auth, the multi-ticker snapshot, unit conversion, and the EOD path in one pass. - -```python -# backend/scripts/verify_massive.py -"""Smoke-test the Massive REST API against a live key.""" - -import os -from datetime import UTC, datetime - -from massive import RESTClient -from massive.rest.models import SnapshotMarketType - -NANOS_PER_SECOND = 1_000_000_000 -TICKERS = ["AAPL", "GOOGL", "MSFT", "NVDA", "TSLA"] - - -def main() -> None: - key = os.environ["MASSIVE_API_KEY"] - client = RESTClient(api_key=key) - - status = client.get_market_status() - print(f"market: {status.market}") - - snapshots = client.get_snapshot_all(SnapshotMarketType.STOCKS, TICKERS) - print(f"requested {len(TICKERS)}, received {len(snapshots)}") - - for snap in snapshots: - trade = snap.last_trade - if trade is None or trade.price is None: - print(f"{snap.ticker}: no trade data") - continue - seconds = trade.sip_timestamp / NANOS_PER_SECOND - when = datetime.fromtimestamp(seconds, UTC) - print(f"{snap.ticker}: ${trade.price:.2f} at {when:%Y-%m-%d %H:%M:%S} UTC") - - missing = set(TICKERS) - {s.ticker for s in snapshots} - if missing: - print(f"absent from response (unknown or untraded): {sorted(missing)}") - - prev = client.get_previous_close_agg("AAPL") - print(f"AAPL previous close: ${prev.close:.2f}") - - -if __name__ == "__main__": - main() -``` - -```bash -uv run python scripts/verify_massive.py -``` - -Expected: a market status, five priced tickers with timestamps in the recent past (not 55 million years hence), and a previous close. Timestamps far in the future mean the unit divisor is wrong; `AttributeError` means §8.1 has regressed. - ---- - -## 11. Summary — what FinAlly uses - -| Need | Endpoint | SDK call | Cost | -|---|---|---|---| -| Live prices, all watched tickers | `/v2/snapshot/.../tickers` | `get_snapshot_all` | 1 request per poll | -| EOD close, one ticker | `/v2/aggs/ticker/{t}/prev` | `get_previous_close_agg` | 1 request per ticker | -| EOD close, many tickers | `/v2/aggs/grouped/...` | `get_grouped_daily_aggs` | 1 request total | -| Chart backfill | `/v2/aggs/ticker/{t}/range/...` | `get_aggs` | 1 request per ticker | -| Explain a frozen feed | `/v1/marketstatus/now` | `get_market_status` | 1 request | - -The polling design that follows from the 5 req/min free tier — one snapshot request covering the union of watchlist and held positions, every 15 seconds — is specified in `MARKET_INTERFACE.md`. - -## Sources - -- [Full Market Snapshot](https://massive.com/docs/rest/stocks/snapshots/full-market-snapshot) -- [Unified Snapshot](https://massive.com/docs/rest/stocks/snapshots/unified-snapshot) -- [Previous Day Bar](https://massive.com/docs/rest/stocks/aggregates/previous-day-bar) -- [Daily Market Summary](https://massive.com/docs/rest/stocks/aggregates/daily-market-summary) -- [Request limits for Massive's RESTful APIs](https://massive.com/knowledge-base/article/what-is-the-request-limit-for-massives-restful-apis) -- [massive-com/client-python](https://github.com/massive-com/client-python) -- Installed SDK source: `backend/.venv/lib/python3.13/site-packages/massive/` (v2.2.0) diff --git a/planning/archive/MARKET_DATA_DESIGN.md b/planning/archive/MARKET_DATA_DESIGN.md index 0d2cfd5..d7c962e 100644 --- a/planning/archive/MARKET_DATA_DESIGN.md +++ b/planning/archive/MARKET_DATA_DESIGN.md @@ -1,64 +1,138 @@ -# Market Data Backend — Detailed Design +# MARKET_DATA_DESIGN.md — Market Data Backend, Detailed Design -Implementation-ready design for the FinAlly market data subsystem. Covers the unified interface, in-memory price cache, GBM simulator, Massive API client, SSE streaming endpoint, and FastAPI lifecycle integration. +The implementation-level design for FinAlly's market data subsystem: one unified API, two +interchangeable sources (GBM simulator and the Massive REST API), a shared in-memory cache, +and the SSE stream that carries prices to the browser. -Everything in this document lives under `backend/app/market/`. +**Audience:** the agent (or human) implementing or extending `backend/app/market/`. +This document is meant to be read once, top to bottom, and then implemented from — every +snippet below is either the code that ships today or the code that should ship. + +**Companion documents.** `PLAN.md` §6 is the frozen contract; `MARKET_INTERFACE.md`, +`MARKET_SIMULATOR.md`, and `MASSIVE_API.md` are the reference material this design draws on. +Where they disagree with this document, this document is the design and they are the background. --- -## Table of Contents - -1. [File Structure](#1-file-structure) -2. [Data Model — `models.py`](#2-data-model) -3. [Price Cache — `cache.py`](#3-price-cache) -4. [Abstract Interface — `interface.py`](#4-abstract-interface) -5. [Seed Prices & Ticker Parameters — `seed_prices.py`](#5-seed-prices--ticker-parameters) -6. [GBM Simulator — `simulator.py`](#6-gbm-simulator) -7. [Massive API Client — `massive_client.py`](#7-massive-api-client) -8. [Factory — `factory.py`](#8-factory) -9. [SSE Streaming Endpoint — `stream.py`](#9-sse-streaming-endpoint) -10. [FastAPI Lifecycle Integration](#10-fastapi-lifecycle-integration) -11. [Watchlist Coordination](#11-watchlist-coordination) -12. [Testing Strategy](#12-testing-strategy) -13. [Error Handling & Edge Cases](#13-error-handling--edge-cases) -14. [Configuration Summary](#14-configuration-summary) +## 0. Status — what exists, what is missing + +Verified by running the suite in `backend/` on 2026-09-01: + +``` +73 passed in 1.77s TOTAL coverage 91% +app/market/cache.py 100% +app/market/models.py 100% +app/market/simulator.py 98% +app/market/massive_client.py 94% <- high coverage, two live defects (§8.4) +app/market/stream.py 33% <- the SSE generator is effectively untested +``` + +| Piece | State | Section | +|---|---|---| +| `PriceUpdate` wire model | Ships, frozen contract | §4 | +| `PriceCache` (latest price + version) | Ships | §5.1 | +| `PriceCache` rolling history | **Missing** | §5.2 | +| `MarketDataSource` ABC | Ships | §6 | +| `SimulatorDataSource` + `GBMSimulator` | Ships | §7 | +| `MassiveDataSource` | Ships but **writes nothing to the cache** | §8.4 | +| `create_market_data_source` | Ships | §9 | +| SSE `/api/stream/prices` | Ships | §10.1 | +| SSE keepalive | **Missing** | §10.2 | +| `GET /api/prices/{ticker}/history` | **Missing** | §11 | +| Lifespan wiring + tracked-set reconciliation | **Missing** | §12 | + +Four gaps, all backend, all small. §15 orders them. --- -## 1. File Structure +## 1. The shape of the design + +Two sources with nothing in common — a 500ms in-process computation and a 15-second blocking +HTTP poll — must be interchangeable to everything downstream. The design achieves that with +**one indirection and one shared buffer**: + +``` + writes reads + ┌──────────────────┐ ┌────────────┐ ┌──────────────────────┐ + │ SimulatorSource │───┐ │ │───────────────│ SSE /api/stream │ + │ (500ms step) │ ├───▶│ PriceCache │───────────────│ Portfolio valuation │ + ├──────────────────┤ │ │ (in-mem, │───────────────│ Trade execution │ + │ MassiveSource │───┘ │thread-safe)│───────────────│ Snapshot task │ + │ (15s poll) │ │ │───────────────│ /api/prices/history │ + └──────────────────┘ └────────────┘ └──────────────────────┘ + MarketDataSource + (abstract interface) +``` + +**The one invariant that makes this work: nothing downstream ever asks a source for a price.** +Sources are write-only from the application's point of view; readers only ever touch the cache. +That is why a 30× difference in update cadence is invisible to the rest of the app, and why a +`get_price()` on the interface would be a design error — under Massive it would turn every +portfolio valuation into a billed HTTP request. + +### File structure ``` -backend/ - app/ - market/ - __init__.py # Re-exports: PriceUpdate, PriceCache, MarketDataSource, create_market_data_source - models.py # PriceUpdate dataclass - cache.py # PriceCache (thread-safe in-memory store) - interface.py # MarketDataSource ABC - seed_prices.py # SEED_PRICES, TICKER_PARAMS, DEFAULT_PARAMS, CORRELATION_GROUPS - simulator.py # GBMSimulator + SimulatorDataSource - massive_client.py # MassiveDataSource - factory.py # create_market_data_source() - stream.py # SSE endpoint (FastAPI router) +backend/app/market/ +├── __init__.py # public exports +├── models.py # PriceUpdate — the unit of data +├── cache.py # PriceCache — latest price + version + rolling history +├── interface.py # MarketDataSource — the ABC +├── seed_prices.py # simulator constants, no logic +├── simulator.py # GBMSimulator (pure) + SimulatorDataSource (async) +├── massive_client.py # MassiveDataSource +├── factory.py # create_market_data_source +└── stream.py # SSE router + history router ``` -Each file has a single responsibility. The `__init__.py` re-exports the public API so that the rest of the backend imports from `app.market` without reaching into submodules. +Public surface, unchanged by this design: + +```python +from app.market import ( + PriceUpdate, + PriceCache, + MarketDataSource, + create_market_data_source, + create_stream_router, +) +``` --- -## 2. Data Model +## 2. Vocabulary -**File: `backend/app/market/models.py`** +| Term | Meaning | +|---|---| +| **tick** | One simulator step (500ms) or one Massive poll (15s) | +| **tracked set** | `watchlist ∪ {tickers with a non-zero position}` — §12.2 | +| **version** | Monotonic counter on `PriceCache`, bumped on every write; the SSE change signal | +| **seeding** | Writing an initial price into the cache so a ticker never renders as `—` unnecessarily | -`PriceUpdate` is the only data structure that leaves the market data layer. Every downstream consumer — SSE streaming, portfolio valuation, trade execution — works exclusively with this type. +--- -```python -from __future__ import annotations +## 3. Non-negotiable contracts -import time -from dataclasses import dataclass, field +These are frozen because the frontend and the shipped module already depend on them. Everything +else in this document is open to reasonable change. +1. **SSE payload is a map keyed by ticker, one event carries every ticker.** Not one event per ticker. +2. **`timestamp` is Unix epoch seconds as a float.** Never ISO, never milliseconds. The frontend + multiplies by 1000 for `Date`. +3. **`change_percent` is already in percent units.** `0.021` means 0.021%. This deliberately + differs from REST responses elsewhere in the API, where percentages are fractions + (`PLAN.md` §8). The inconsistency is real and preserved. +4. **A connecting client gets a full snapshot immediately**, including after a reconnect, + because a fresh generator starts at `last_version = -1`. +5. **Tickers are uppercase everywhere**, normalized at the API boundary. +--- + +## 4. `PriceUpdate` — the unit of data + +`backend/app/market/models.py`. Immutable, frozen, slotted. Both sources produce it; every +reader consumes it. + +```python @dataclass(frozen=True, slots=True) class PriceUpdate: """Immutable snapshot of a single ticker's price at a point in time.""" @@ -66,23 +140,20 @@ class PriceUpdate: ticker: str price: float previous_price: float - timestamp: float = field(default_factory=time.time) # Unix seconds + timestamp: float = field(default_factory=time.time) # Unix epoch SECONDS @property def change(self) -> float: - """Absolute price change from previous update.""" return round(self.price - self.previous_price, 4) @property def change_percent(self) -> float: - """Percentage change from previous update.""" if self.previous_price == 0: return 0.0 return round((self.price - self.previous_price) / self.previous_price * 100, 4) @property def direction(self) -> str: - """'up', 'down', or 'flat'.""" if self.price > self.previous_price: return "up" elif self.price < self.previous_price: @@ -90,7 +161,6 @@ class PriceUpdate: return "flat" def to_dict(self) -> dict: - """Serialize for JSON / SSE transmission.""" return { "ticker": self.ticker, "price": self.price, @@ -102,820 +172,856 @@ class PriceUpdate: } ``` -### Design decisions - -- **`frozen=True`**: Price updates are immutable value objects. Once created they never change, which makes them safe to share across async tasks without copying. -- **`slots=True`**: Minor memory optimization — we create many of these per second. -- **Computed properties** (`change`, `direction`, `change_percent`): Derived from `price` and `previous_price` so they can never be inconsistent. No risk of a stale `direction` field. -- **`to_dict()`**: Single serialization point used by both the SSE endpoint and REST API responses. +**`change`, `change_percent`, and `direction` are computed properties, not stored fields.** +They cannot drift out of sync with the prices they describe, and `to_dict()` cannot emit a +`direction` that contradicts its own `price`/`previous_price` pair. + +**`previous_price` means the price at the previous update**, not the previous session's close. +On the first update for a ticker it equals `price`, so `direction` is `"flat"` and `change` is +`0.0` — a newly added ticker never flashes green or red on its first tick. + +Example of the exact wire shape a client sees: + +```json +{ + "ticker": "AAPL", + "price": 190.52, + "previous_price": 190.48, + "timestamp": 1755873791.482, + "change": 0.04, + "change_percent": 0.021, + "direction": "up" +} +``` --- -## 3. Price Cache +## 5. `PriceCache` — the shared buffer -**File: `backend/app/market/cache.py`** +`backend/app/market/cache.py`. -The price cache is the central data hub. Data sources write to it; SSE streaming and portfolio valuation read from it. It must be thread-safe because the simulator/poller may run in a thread pool executor while SSE reads happen on the async event loop. +### 5.1 What ships today ```python -from __future__ import annotations +class PriceCache: + def update(self, ticker: str, price: float, timestamp: float | None = None) -> PriceUpdate + def get(self, ticker: str) -> PriceUpdate | None + def get_all(self) -> dict[str, PriceUpdate] # shallow copy + def get_price(self, ticker: str) -> float | None + def remove(self, ticker: str) -> None + @property + def version(self) -> int + def __len__(self) -> int + def __contains__(self, ticker: str) -> bool +``` -import asyncio -import time -from threading import Lock -from typing import Callable +Three design points that matter: -from .models import PriceUpdate +**`update()` derives `previous_price` itself.** Callers pass only the new price; the cache looks +up what it held and constructs the `PriceUpdate`. Neither source tracks prior state for the +purpose of computing a delta, so the two cannot implement it differently. +```python +def update(self, ticker: str, price: float, timestamp: float | None = None) -> PriceUpdate: + with self._lock: + ts = timestamp or time.time() + prev = self._prices.get(ticker) + previous_price = prev.price if prev else price + + update = PriceUpdate( + ticker=ticker, + price=round(price, 2), + previous_price=round(previous_price, 2), + timestamp=ts, + ) + self._prices[ticker] = update + self._version += 1 + return update +``` -class PriceCache: - """Thread-safe in-memory cache of the latest price for each ticker. +**A `threading.Lock`, not an `asyncio.Lock`.** `MassiveDataSource` writes from an +`asyncio.to_thread` worker, so a genuine cross-thread lock is required. The critical sections +are a few dict operations; contention is irrelevant. - Writers: SimulatorDataSource or MassiveDataSource (one at a time). - Readers: SSE streaming endpoint, portfolio valuation, trade execution. - """ +**`version` is the SSE change-detection mechanism.** The stream compares an integer every 500ms +rather than diffing price maps. `get_all()` returns a shallow copy, and since `PriceUpdate` is +frozen, that copy is effectively deep and safe to iterate outside the lock. - def __init__(self) -> None: - self._prices: dict[str, PriceUpdate] = {} - self._lock = Lock() - self._version: int = 0 # Monotonically increasing; bumped on every update +### 5.2 Rolling price history — to implement - def update(self, ticker: str, price: float, timestamp: float | None = None) -> PriceUpdate: - """Record a new price for a ticker. Returns the created PriceUpdate. +`PLAN.md` §6 requires the main chart to be populated the instant a ticker is clicked, rather +than drawing itself from scratch over the following minute. `PriceCache` gains a bounded +per-ticker deque of `(timestamp, price)`. - Automatically computes direction and change from the previous price. - If this is the first update for the ticker, previous_price == price (direction='flat'). - """ - with self._lock: - ts = timestamp or time.time() - prev = self._prices.get(ticker) - previous_price = prev.price if prev else price - - update = PriceUpdate( - ticker=ticker, - price=round(price, 2), - previous_price=round(previous_price, 2), - timestamp=ts, - ) - self._prices[ticker] = update - self._version += 1 - return update - - def get(self, ticker: str) -> PriceUpdate | None: - """Get the latest price for a single ticker, or None if unknown.""" - with self._lock: - return self._prices.get(ticker) - - def get_all(self) -> dict[str, PriceUpdate]: - """Snapshot of all current prices. Returns a shallow copy.""" - with self._lock: - return dict(self._prices) - - def get_price(self, ticker: str) -> float | None: - """Convenience: get just the price float, or None.""" - update = self.get(ticker) - return update.price if update else None - - def remove(self, ticker: str) -> None: - """Remove a ticker from the cache (e.g., when removed from watchlist).""" - with self._lock: - self._prices.pop(ticker, None) +```python +from collections import deque - @property - def version(self) -> int: - """Current version counter. Useful for SSE change detection.""" - return self._version +HISTORY_MAXLEN = 600 # ~5 minutes at the 500ms simulator cadence +``` - def __len__(self) -> int: - with self._lock: - return len(self._prices) +Constructor: - def __contains__(self, ticker: str) -> bool: - with self._lock: - return ticker in self._prices +```python +def __init__(self, history_maxlen: int = HISTORY_MAXLEN) -> None: + self._prices: dict[str, PriceUpdate] = {} + self._history: dict[str, deque[tuple[float, float]]] = {} + self._history_maxlen = history_maxlen + self._lock = Lock() + self._version: int = 0 ``` -### Why a version counter? +Appended inside `update()`, under the same lock, immediately after the price is stored: -The SSE streaming loop polls the cache every ~500ms. Without a version counter, it would serialize and send all prices every tick even if nothing changed (e.g., Massive API only updates every 15s). The version counter lets the SSE loop skip sends when nothing is new: +```python + self._prices[ticker] = update + history = self._history.get(ticker) + if history is None: + history = deque(maxlen=self._history_maxlen) + self._history[ticker] = history + history.append((ts, update.price)) + self._version += 1 + return update +``` + +`remove()` must drop the deque too, or removed tickers leak memory and a re-added ticker +resurrects a stale chart: ```python -last_version = -1 -while True: - if price_cache.version != last_version: - last_version = price_cache.version - yield format_sse(price_cache.get_all()) - await asyncio.sleep(0.5) +def remove(self, ticker: str) -> None: + with self._lock: + self._prices.pop(ticker, None) + self._history.pop(ticker, None) ``` -### Thread safety rationale +The reader, backing `GET /api/prices/{ticker}/history`: -The `threading.Lock` is used instead of `asyncio.Lock` because: -- The Massive client's synchronous `get_snapshot_all()` runs in `asyncio.to_thread()`, which operates in a real OS thread — `asyncio.Lock` would not protect against that. -- The GBM simulator's `step()` is CPU-bound and could also be offloaded to a thread for fairness. -- `threading.Lock` works correctly from both sync threads and the async event loop. +```python +def get_history(self, ticker: str, limit: int = HISTORY_MAXLEN) -> list[tuple[float, float]]: + """Oldest-first (timestamp, price) points. Empty list for an untracked ticker.""" + with self._lock: + points = self._history.get(ticker) + if not points: + return [] + return list(points)[-limit:] +``` + +Four properties worth stating explicitly: + +- **`deque(maxlen=600)` evicts the oldest point automatically** — there is no pruning logic to + write, and no unbounded growth to worry about. +- **An untracked ticker returns `[]`, not a 404.** The chart draws nothing rather than erroring + (`PLAN.md` §8). +- **Deliberately not persisted.** A restart clears it, which is the honest behavior for a + simulator whose prices also reset to seed on restart. +- **Memory is negligible**: 600 points × 50 tickers × ~16 bytes ≈ 500KB. + +Under Massive the deque fills at one point per 15-second poll, so five minutes of wall time is +20 points rather than 600. The chart is sparse but correct. Backfilling from `get_aggs` +(`MASSIVE_API.md` §5) is the eventual upgrade and is out of scope here. --- -## 4. Abstract Interface +## 6. `MarketDataSource` — the abstract contract -**File: `backend/app/market/interface.py`** +`backend/app/market/interface.py`. ```python -from __future__ import annotations +class MarketDataSource(ABC): + @abstractmethod + async def start(self, tickers: list[str]) -> None: ... + @abstractmethod + async def stop(self) -> None: ... + @abstractmethod + async def add_ticker(self, ticker: str) -> None: ... + @abstractmethod + async def remove_ticker(self, ticker: str) -> None: ... + @abstractmethod + def get_tickers(self) -> list[str]: ... +``` -from abc import ABC, abstractmethod +Five methods, and every one is about **lifecycle and membership** — none returns a price. That +absence is the whole design (§1). +### Behavioral contract -class MarketDataSource(ABC): - """Contract for market data providers. - - Implementations push price updates into a shared PriceCache on their own - schedule. Downstream code never calls the data source directly for prices — - it reads from the cache. - - Lifecycle: - source = create_market_data_source(cache) - await source.start(["AAPL", "GOOGL", ...]) - # ... app runs ... - await source.add_ticker("TSLA") - await source.remove_ticker("GOOGL") - # ... app shutting down ... - await source.stop() - """ +Binding on both implementations. A test suite that passes against one should pass against the other. - @abstractmethod - async def start(self, tickers: list[str]) -> None: - """Begin producing price updates for the given tickers. +| Method | Guarantee | +|---|---| +| `start(tickers)` | Begins a background task writing to the cache. **Seeds the cache before returning**, so the first SSE event is never empty. Called exactly once; calling twice is undefined. | +| `stop()` | Cancels the task and releases resources. **Idempotent.** No writes to the cache afterwards. | +| `add_ticker(t)` | Adds to the tracked set. No-op if present. Simulator seeds a price immediately; Massive picks it up on the next poll. | +| `remove_ticker(t)` | Removes from the tracked set **and from the cache** (price and history). No-op if absent. | +| `get_tickers()` | Current tracked set. Synchronous — reads local state only. | - Starts a background task that periodically writes to the PriceCache. - Must be called exactly once. Calling start() twice is undefined behavior. - """ +Two asymmetries are permitted and must not be papered over: - @abstractmethod - async def stop(self) -> None: - """Stop the background task and release resources. +- **Seeding latency.** `add_ticker` on the simulator makes a price available immediately; on + Massive it takes up to one poll interval. The API contract already accommodates this — + `GET /api/watchlist` returns `price: null` until the first tick, and the UI shows `—`. +- **Cadence.** 500ms versus 15s. Readers must never assume a minimum update rate. This is + exactly what the SSE keepalive in §10.2 exists to handle. - Safe to call multiple times. After stop(), the source will not write - to the cache again. - """ +### `remove_ticker` is destructive — and that is the trap - @abstractmethod - async def add_ticker(self, ticker: str) -> None: - """Add a ticker to the active set. No-op if already present. +Both implementations call `self._cache.remove(ticker)`. Correct for the interface, but it means +removing a ticker whose position is still held silently freezes that position's valuation, P&L, +heatmap tile, and snapshot contribution. §12.2 is the rule that prevents it, and it is the single +most important piece of integration logic in this module because the failure mode is a wrong +number, not an error. - The next update cycle will include this ticker. - """ +### Adding a third source - @abstractmethod - async def remove_ticker(self, ticker: str) -> None: - """Remove a ticker from the active set. No-op if not present. +1. Subclass `MarketDataSource` and implement all five methods. +2. `start()` must **seed the cache before returning**. +3. Never write to the cache after `stop()`; make `stop()` idempotent. +4. `remove_ticker()` must call `cache.remove(ticker)`. +5. Convert timestamps to **Unix epoch seconds as a float** at the boundary. +6. Never let a fetch error kill the background loop — log and retry next cycle. +7. If the underlying client is synchronous, wrap **every** call in `asyncio.to_thread`. +8. Add a branch to `create_market_data_source` and a value to `market_source` in `/api/health`. - Also removes the ticker from the PriceCache. - """ +Point 7 is not optional: a blocking HTTP call inside `async def` stalls the event loop for the +whole round trip, which stops the SSE stream and every in-flight request. - @abstractmethod - def get_tickers(self) -> list[str]: - """Return the current list of actively tracked tickers.""" +--- + +## 7. The simulator — default source + +`backend/app/market/simulator.py` and `seed_prices.py`. Two classes with a clean split: +**`GBMSimulator` is pure and synchronous; `SimulatorDataSource` owns the async lifecycle and +the cache.** + +``` +┌──────────────────────────────────────────────────────────┐ +│ SimulatorDataSource(MarketDataSource) │ +│ owns the asyncio task, writes to PriceCache │ +│ start / stop / add_ticker / remove_ticker / get_tickers│ +│ │ │ +│ ▼ │ +│ GBMSimulator │ +│ pure math, no I/O, no async, no cache reference │ +│ step() -> {ticker: price} │ +└──────────────────────────────────────────────────────────┘ + │ + ▼ + seed_prices.py (constants only) ``` -### Why the source writes to the cache instead of returning prices +The separation pays off in testing: `GBMSimulator` needs no event loop, no cache, and no mocks. -This push model decouples timing. The simulator ticks at 500ms, Massive polls at 15s, but SSE always reads from the cache at its own 500ms cadence. There is no need for the SSE layer to know which data source is active or what its update interval is. +### 7.1 The model ---- +``` +S(t + dt) = S(t) · exp( (μ − σ²/2)·dt + σ·√dt·Z ) +``` + +Three properties earn GBM its place: + +**Prices cannot go negative.** The update is multiplicative — `exp(...)` is always positive. +No clamping, no `max(price, 0.01)` guard, no special case. An additive random walk needs all three. + +**Returns scale correctly with time.** σ is annualized; `√dt` converts it to the tick. The 500ms +cadence is a display choice, not a modelling parameter. -## 5. Seed Prices & Ticker Parameters +**The `−σ²/2` term keeps the drift honest.** Without it, μ is not the expected return of the +price — a log-normal artefact. It costs one subtraction and makes the parameters mean what they say. -**File: `backend/app/market/seed_prices.py`** +### 7.2 Sizing `dt` -Constants only — no logic, no imports beyond stdlib. This file is shared by both the simulator (for initial prices and GBM parameters) and potentially by the Massive client (as fallback prices if the API hasn't responded yet). +`dt` is expressed against a **trading** year, not a calendar year. Markets are closed most of the +time; using 365×24h would understate per-tick moves by ~4.5×. ```python -"""Seed prices and per-ticker parameters for the market simulator.""" - -# Realistic starting prices for the default watchlist (as of project creation) -SEED_PRICES: dict[str, float] = { - "AAPL": 190.00, - "GOOGL": 175.00, - "MSFT": 420.00, - "AMZN": 185.00, - "TSLA": 250.00, - "NVDA": 800.00, - "META": 500.00, - "JPM": 195.00, - "V": 280.00, - "NFLX": 600.00, -} +TRADING_SECONDS_PER_YEAR = 252 * 6.5 * 3600 # 5,896,800 +DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR # ~8.479e-8, sqrt(dt) = 2.912e-4 +``` -# Per-ticker GBM parameters -# sigma: annualized volatility (higher = more price movement) -# mu: annualized drift / expected return -TICKER_PARAMS: dict[str, dict[str, float]] = { - "AAPL": {"sigma": 0.22, "mu": 0.05}, - "GOOGL": {"sigma": 0.25, "mu": 0.05}, - "MSFT": {"sigma": 0.20, "mu": 0.05}, - "AMZN": {"sigma": 0.28, "mu": 0.05}, - "TSLA": {"sigma": 0.50, "mu": 0.03}, # High volatility - "NVDA": {"sigma": 0.40, "mu": 0.08}, # High volatility, strong drift - "META": {"sigma": 0.30, "mu": 0.05}, - "JPM": {"sigma": 0.18, "mu": 0.04}, # Low volatility (bank) - "V": {"sigma": 0.17, "mu": 0.04}, # Low volatility (payments) - "NFLX": {"sigma": 0.35, "mu": 0.05}, -} +What that produces per tick at the seed prices: + +| Ticker | σ | Per-tick σ | Per-tick $ | Per-minute $ (120 ticks) | +|---|---|---|---|---| +| AAPL | 0.22 | 0.0064% | $0.012 | $0.13 | +| JPM | 0.18 | 0.0052% | $0.010 | $0.11 | +| NVDA | 0.40 | 0.0116% | $0.093 | $1.02 | +| TSLA | 0.50 | 0.0146% | $0.036 | $0.40 | + +This is the number that decides whether the simulation looks right. A cent or two per tick on a +$200 stock means the price **rounds to a genuinely different value most ticks**, so the UI flashes +constantly, while a minute of drift stays in the tens of cents — what a real quote screen looks +like. Larger reads as a crash; smaller looks frozen. -# Default parameters for tickers not in the list above (dynamically added) -DEFAULT_PARAMS: dict[str, float] = {"sigma": 0.25, "mu": 0.05} +### 7.3 Correlation via Cholesky -# Correlation groups for the simulator's Cholesky decomposition -# Tickers in the same group have higher intra-group correlation -CORRELATION_GROUPS: dict[str, set[str]] = { - "tech": {"AAPL", "GOOGL", "MSFT", "AMZN", "META", "NVDA", "NFLX"}, +Independent draws would show tech stocks moving in opposite directions half the time. The eye +notices immediately. Standard fix: draw `n` independent normals, multiply by the Cholesky factor +`L` of the correlation matrix `C = L·Lᵀ`. + +Constants live in `seed_prices.py`, not in the simulator: + +```python +CORRELATION_GROUPS = { + "tech": {"AAPL", "GOOGL", "MSFT", "AMZN", "META", "NVDA", "NFLX"}, "finance": {"JPM", "V"}, } -# Correlation coefficients -INTRA_TECH_CORR = 0.6 # Tech stocks move together -INTRA_FINANCE_CORR = 0.5 # Finance stocks move together -CROSS_GROUP_CORR = 0.3 # Between sectors -TSLA_CORR = 0.3 # TSLA does its own thing -DEFAULT_CORR = 0.3 # Unknown tickers +INTRA_TECH_CORR = 0.6 # tech stocks move together +INTRA_FINANCE_CORR = 0.5 # finance stocks move together +CROSS_GROUP_CORR = 0.3 # between sectors, and for unknown tickers +TSLA_CORR = 0.3 # TSLA does its own thing ``` ---- +Resolved pairwise, first match winning: -## 6. GBM Simulator +```python +@staticmethod +def _pairwise_correlation(t1: str, t2: str) -> float: + tech = CORRELATION_GROUPS["tech"] + finance = CORRELATION_GROUPS["finance"] + + # TSLA is in the tech set but behaves independently + if t1 == "TSLA" or t2 == "TSLA": + return TSLA_CORR -**File: `backend/app/market/simulator.py`** + if t1 in tech and t2 in tech: + return INTRA_TECH_CORR + if t1 in finance and t2 in finance: + return INTRA_FINANCE_CORR -This file contains two classes: -- `GBMSimulator`: Pure math engine. Stateful — holds current prices and advances them one step at a time. -- `SimulatorDataSource`: The `MarketDataSource` implementation that wraps `GBMSimulator` in an async loop and writes to the `PriceCache`. + return CROSS_GROUP_CORR +``` -### 6.1 GBMSimulator — The Math Engine +The TSLA clause is checked first on purpose: TSLA is a tech-set member for every other purpose, +but a demo where TSLA visibly decouples from the pack is more convincing than one where everything +moves in lockstep. `CROSS_GROUP_CORR` doubles as the default for any unknown symbol, which is what +makes §7.5 work. ```python -from __future__ import annotations - -import asyncio -import logging -import math -import random - -import numpy as np - -from .cache import PriceCache -from .interface import MarketDataSource -from .seed_prices import ( - CORRELATION_GROUPS, - CROSS_GROUP_CORR, - DEFAULT_CORR, - DEFAULT_PARAMS, - INTRA_FINANCE_CORR, - INTRA_TECH_CORR, - SEED_PRICES, - TICKER_PARAMS, - TSLA_CORR, -) +def _rebuild_cholesky(self) -> None: + n = len(self._tickers) + if n <= 1: + self._cholesky = None # a single ticker needs no correlation + return + + corr = np.eye(n) + for i in range(n): + for j in range(i + 1, n): + rho = self._pairwise_correlation(self._tickers[i], self._tickers[j]) + corr[i, j] = rho + corr[j, i] = rho + + self._cholesky = np.linalg.cholesky(corr) +``` -logger = logging.getLogger(__name__) +Rebuilt on every add and remove — `O(n²)` to build plus `O(n³)` to factor, on `n < 50`. That is +microseconds, and watchlist edits are human-speed, so caching it would be complexity without benefit. +> **Known risk.** `np.linalg.cholesky` raises `LinAlgError` on a matrix that is not positive +> definite, and the call is unguarded. The current block structure (0.6 / 0.5 / 0.3) was verified +> positive definite at 7, 20, and 40 tickers — but raising `INTRA_TECH_CORR` toward 1.0, or adding +> a group whose intra-group correlation is *below* the cross-group value, can break +> positive-definiteness and take down `add_ticker`. Anyone editing these constants must re-run the +> test in §14.2. -class GBMSimulator: - """Geometric Brownian Motion simulator for correlated stock prices. +### 7.4 The tick - Math: - S(t+dt) = S(t) * exp((mu - sigma^2/2) * dt + sigma * sqrt(dt) * Z) +`step()` is the hot path — every 500ms, for every ticker. - Where: - S(t) = current price - mu = annualized drift (expected return) - sigma = annualized volatility - dt = time step as fraction of a trading year - Z = correlated standard normal random variable +```python +def step(self) -> dict[str, float]: + """Advance all tickers by one time step. Returns {ticker: new_price}.""" + n = len(self._tickers) + if n == 0: + return {} + + z_independent = np.random.standard_normal(n) + if self._cholesky is not None: + z_correlated = self._cholesky @ z_independent + else: + z_correlated = z_independent - The tiny dt (~8.5e-8 for 500ms ticks over 252 trading days * 6.5h/day) - produces sub-cent moves per tick that accumulate naturally over time. - """ + result: dict[str, float] = {} + for i, ticker in enumerate(self._tickers): + params = self._params[ticker] + mu, sigma = params["mu"], params["sigma"] - # 500ms expressed as a fraction of a trading year - # 252 trading days * 6.5 hours/day * 3600 seconds/hour = 5,896,800 seconds - TRADING_SECONDS_PER_YEAR = 252 * 6.5 * 3600 # 5,896,800 - DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR # ~8.48e-8 - - def __init__( - self, - tickers: list[str], - dt: float = DEFAULT_DT, - event_probability: float = 0.001, - ) -> None: - self._dt = dt - self._event_prob = event_probability - - # Per-ticker state - self._tickers: list[str] = [] - self._prices: dict[str, float] = {} - self._params: dict[str, dict[str, float]] = {} + drift = (mu - 0.5 * sigma**2) * self._dt + diffusion = sigma * math.sqrt(self._dt) * z_correlated[i] + self._prices[ticker] *= math.exp(drift + diffusion) - # Cholesky decomposition of the correlation matrix (for correlated moves) - self._cholesky: np.ndarray | None = None + if random.random() < self._event_prob: + shock_magnitude = random.uniform(0.02, 0.05) + shock_sign = random.choice([-1, 1]) + self._prices[ticker] *= 1 + shock_magnitude * shock_sign - # Initialize all starting tickers - for ticker in tickers: - self._add_ticker_internal(ticker) - self._rebuild_cholesky() + result[ticker] = round(self._prices[ticker], 2) - # --- Public API --- + return result +``` - def step(self) -> dict[str, float]: - """Advance all tickers by one time step. Returns {ticker: new_price}. +Two details worth pointing out: - This is the hot path — called every 500ms. Keep it fast. - """ - n = len(self._tickers) - if n == 0: - return {} - - # Generate n independent standard normal draws - z_independent = np.random.standard_normal(n) - - # Apply Cholesky to get correlated draws - if self._cholesky is not None: - z_correlated = self._cholesky @ z_independent - else: - z_correlated = z_independent - - result: dict[str, float] = {} - for i, ticker in enumerate(self._tickers): - params = self._params[ticker] - mu = params["mu"] - sigma = params["sigma"] - - # GBM: S(t+dt) = S(t) * exp((mu - 0.5*sigma^2)*dt + sigma*sqrt(dt)*Z) - drift = (mu - 0.5 * sigma ** 2) * self._dt - diffusion = sigma * math.sqrt(self._dt) * z_correlated[i] - self._prices[ticker] *= math.exp(drift + diffusion) - - # Random event: ~0.1% chance per tick per ticker - # With 10 tickers at 2 ticks/sec, expect an event ~every 50 seconds - if random.random() < self._event_prob: - shock_magnitude = random.uniform(0.02, 0.05) - shock_sign = random.choice([-1, 1]) - self._prices[ticker] *= 1 + shock_magnitude * shock_sign - logger.debug( - "Random event on %s: %.1f%% %s", - ticker, - shock_magnitude * 100, - "up" if shock_sign > 0 else "down", - ) - - result[ticker] = round(self._prices[ticker], 2) - - return result - - def add_ticker(self, ticker: str) -> None: - """Add a ticker to the simulation. Rebuilds the correlation matrix.""" - if ticker in self._prices: - return - self._add_ticker_internal(ticker) - self._rebuild_cholesky() - - def remove_ticker(self, ticker: str) -> None: - """Remove a ticker from the simulation. Rebuilds the correlation matrix.""" - if ticker not in self._prices: - return - self._tickers.remove(ticker) - del self._prices[ticker] - del self._params[ticker] - self._rebuild_cholesky() - - def get_price(self, ticker: str) -> float | None: - """Current price for a ticker, or None if not tracked.""" - return self._prices.get(ticker) - - # --- Internals --- - - def _add_ticker_internal(self, ticker: str) -> None: - """Add a ticker without rebuilding Cholesky (for batch initialization).""" - if ticker in self._prices: - return - self._tickers.append(ticker) - self._prices[ticker] = SEED_PRICES.get(ticker, random.uniform(50.0, 300.0)) - self._params[ticker] = TICKER_PARAMS.get(ticker, dict(DEFAULT_PARAMS)) - - def _rebuild_cholesky(self) -> None: - """Rebuild the Cholesky decomposition of the ticker correlation matrix. - - Called whenever tickers are added or removed. O(n^2) but n < 50. - """ - n = len(self._tickers) - if n <= 1: - self._cholesky = None - return - - # Build the correlation matrix - corr = np.eye(n) - for i in range(n): - for j in range(i + 1, n): - rho = self._pairwise_correlation(self._tickers[i], self._tickers[j]) - corr[i, j] = rho - corr[j, i] = rho - - self._cholesky = np.linalg.cholesky(corr) - - @staticmethod - def _pairwise_correlation(t1: str, t2: str) -> float: - """Determine correlation between two tickers based on sector grouping. - - Correlation structure: - - Same tech sector: 0.6 - - Same finance sector: 0.5 - - TSLA with anything: 0.3 (it does its own thing) - - Cross-sector: 0.3 - - Unknown tickers: 0.3 - """ - tech = CORRELATION_GROUPS["tech"] - finance = CORRELATION_GROUPS["finance"] +**Full precision is kept internally; only the returned value is rounded.** Rounding the stored +state would accumulate quantization error into a slow systematic drift over thousands of ticks. - # TSLA is in tech set but behaves independently - if t1 == "TSLA" or t2 == "TSLA": - return TSLA_CORR +**One `standard_normal(n)` call per tick, not `n` calls.** A single vectorized draw feeding one +matrix multiply is why this stays negligible at 500ms. - if t1 in tech and t2 in tech: - return INTRA_TECH_CORR - if t1 in finance and t2 in finance: - return INTRA_FINANCE_CORR +**Random events** fire at `event_probability = 0.001` per ticker per tick. With 10 tickers at +2 ticks/second the expected wait is `1 / (10 × 2 × 0.001) = 50 seconds` — frequent enough that +something happens during a demo, rare enough that the series is not pure noise. The shock +multiplies the price directly rather than feeding through GBM, so it is a genuine discontinuity — +a gap, which is what real news does to a stock. - return CROSS_GROUP_CORR -``` +`_tickers` is an **ordered list** that indexes into the Cholesky matrix: row `i` corresponds to +`_tickers[i]`. That is why add and remove must both rebuild. `__init__` adds every ticker via +`_add_ticker_internal` and rebuilds **once** at the end — `O(n³)` instead of `O(n⁴)` on startup. -### 6.2 SimulatorDataSource — Async Wrapper +### 7.5 Unknown tickers + +Any symbol passing the API-level pattern `^[A-Z][A-Z.]{0,5}$` works, with no allowlist. The AI +chat can add anything the user names, and it behaves plausibly. ```python -class SimulatorDataSource(MarketDataSource): - """MarketDataSource backed by the GBM simulator. +def _add_ticker_internal(self, ticker: str) -> None: + if ticker in self._prices: + return + self._tickers.append(ticker) + self._prices[ticker] = SEED_PRICES.get(ticker, random.uniform(50.0, 300.0)) + self._params[ticker] = TICKER_PARAMS.get(ticker, dict(DEFAULT_PARAMS)) +``` - Runs a background asyncio task that calls GBMSimulator.step() every - `update_interval` seconds and writes results to the PriceCache. - """ +- **Price**: `SEED_PRICES`, else uniform $50–$300 — where most large-cap US equities trade. +- **Parameters**: `TICKER_PARAMS`, else `DEFAULT_PARAMS` (σ=0.25, μ=0.05) — a mid-range large cap. +- **Correlation**: no sector membership, so `CROSS_GROUP_CORR` (0.3) against everything. + +`dict(DEFAULT_PARAMS)` **copies** rather than sharing the module-level dict. Without the copy, +tuning one unknown ticker's σ would mutate the default for every unknown ticker at once. - def __init__( - self, - price_cache: PriceCache, - update_interval: float = 0.5, - event_probability: float = 0.001, - ) -> None: - self._cache = price_cache - self._interval = update_interval - self._event_prob = event_probability - self._sim: GBMSimulator | None = None - self._task: asyncio.Task | None = None +This is a real advantage over the Massive path, where an unknown symbol never produces a price +and sits at `—` forever (§8.5). + +### 7.6 `SimulatorDataSource` — the async wrapper + +```python +class SimulatorDataSource(MarketDataSource): + def __init__(self, price_cache, update_interval=0.5, event_probability=0.001): ... async def start(self, tickers: list[str]) -> None: - self._sim = GBMSimulator( - tickers=tickers, - event_probability=self._event_prob, - ) - # Seed the cache with initial prices so SSE has data immediately + self._sim = GBMSimulator(tickers=tickers, event_probability=self._event_prob) + # Seed the cache so the first SSE event carries real prices for ticker in tickers: price = self._sim.get_price(ticker) if price is not None: self._cache.update(ticker=ticker, price=price) self._task = asyncio.create_task(self._run_loop(), name="simulator-loop") - logger.info("Simulator started with %d tickers", len(tickers)) - - async def stop(self) -> None: - if self._task and not self._task.done(): - self._task.cancel() - try: - await self._task - except asyncio.CancelledError: - pass - self._task = None - logger.info("Simulator stopped") - - async def add_ticker(self, ticker: str) -> None: - if self._sim: - self._sim.add_ticker(ticker) - # Seed cache immediately so the ticker has a price right away - price = self._sim.get_price(ticker) - if price is not None: - self._cache.update(ticker=ticker, price=price) - logger.info("Simulator: added ticker %s", ticker) - - async def remove_ticker(self, ticker: str) -> None: - if self._sim: - self._sim.remove_ticker(ticker) - self._cache.remove(ticker) - logger.info("Simulator: removed ticker %s", ticker) - - def get_tickers(self) -> list[str]: - return list(self._sim._tickers) if self._sim else [] async def _run_loop(self) -> None: - """Core loop: step the simulation, write to cache, sleep.""" while True: try: if self._sim: - prices = self._sim.step() - for ticker, price in prices.items(): + for ticker, price in self._sim.step().items(): self._cache.update(ticker=ticker, price=price) except Exception: logger.exception("Simulator step failed") await asyncio.sleep(self._interval) ``` -### Key behaviors +Three deliberate choices: + +**Seed the cache in `start()` before creating the task.** The first SSE event then carries real +prices rather than an empty object, so the watchlist never renders as ten dashes on load. + +**`add_ticker` seeds immediately.** The new ticker has a price on the very next SSE event, with no +wait for the following step — the reason adding a ticker feels instant. -- **Immediate seeding**: When `start()` is called, the cache is populated with seed prices *before* the loop begins. This means the SSE endpoint has data to send on its very first tick, with no blank-screen delay. -- **Graceful cancellation**: `stop()` cancels the task and awaits it, catching `CancelledError`. This ensures clean shutdown during FastAPI lifespan teardown. -- **Exception resilience**: The loop catches exceptions per-step so a single bad tick doesn't kill the entire data feed. +**The `try` is inside the loop, around the step.** An exception logs and the loop continues on the +next interval. Wrapping the loop instead would let one bad tick kill the feed permanently. This is +the one place defensive handling is warranted: a background task has no caller to propagate to, +and a dead price feed is a dead app. + +`stop()` cancels the task, awaits it, and swallows `CancelledError` — the normal shutdown path, +not an error. + +### 7.7 Parameters + +`seed_prices.py` holds constants only. Prices are realistic as of project creation; σ and μ are annualized. + +| Ticker | Seed | σ | μ | Note | +|---|---|---|---|---| +| AAPL | $190 | 0.22 | 0.05 | | +| GOOGL | $175 | 0.25 | 0.05 | | +| MSFT | $420 | 0.20 | 0.05 | | +| AMZN | $185 | 0.28 | 0.05 | | +| TSLA | $250 | 0.50 | 0.03 | High volatility, decorrelated | +| NVDA | $800 | 0.40 | 0.08 | High volatility, strong drift | +| META | $500 | 0.30 | 0.05 | | +| JPM | $195 | 0.18 | 0.04 | Low volatility (bank) | +| V | $280 | 0.17 | 0.04 | Low volatility (payments) | +| NFLX | $600 | 0.35 | 0.05 | | +| *unknown* | $50–300 | 0.25 | 0.05 | `DEFAULT_PARAMS` | + +The σ spread is what makes the watchlist readable at a glance: V and JPM barely move while NVDA +and TSLA jump, so the grid has texture instead of ten tickers twitching identically. + +There is **no mean reversion and no session boundary.** Prices random-walk from their seed for as +long as the container runs. Over a demo that looks like a trading day; over a week of uptime a +ticker may wander far. That is correct GBM behavior and not worth correcting — state is in memory +only, so a restart returns everything to seed. --- -## 7. Massive API Client +## 8. The Massive client — optional real data + +`backend/app/market/massive_client.py`. Verified against the `massive` SDK **2.2.0** installed in +`backend/.venv`. -**File: `backend/app/market/massive_client.py`** +### 8.1 Why one snapshot endpoint, polled -Polls the Massive (formerly Polygon.io) REST API snapshot endpoint on a configurable interval. The synchronous Massive client runs in `asyncio.to_thread()` to avoid blocking the event loop. +The free tier allows **5 requests/minute** — one request every 12 seconds at best. Per-ticker +endpoints are therefore unusable: 10 watchlist tickers via `get_last_trade` would be 10 requests +per cycle, blowing the entire budget in one poll. + +**The design must fetch all tickers in a single request.** That is +`GET /v2/snapshot/locale/us/markets/stocks/tickers`, one request returning the current state of +every ticker named: ```python -from __future__ import annotations +from massive import RESTClient +from massive.rest.models import SnapshotMarketType -import asyncio -import logging -from typing import Any +client = RESTClient(api_key="YOUR_KEY") -from .cache import PriceCache -from .interface import MarketDataSource +snapshots = client.get_snapshot_all( + market_type=SnapshotMarketType.STOCKS, + tickers=["AAPL", "GOOGL", "MSFT", "AMZN", "TSLA"], +) -logger = logging.getLogger(__name__) +for snap in snapshots: + print(snap.ticker, snap.last_trade.price, snap.todays_change_percent) +``` +The SDK joins a list into a comma-separated string, so passing `list[str]` is correct. +Default poll interval is **15 seconds**, which leaves headroom under the free tier even if a poll +overruns. Paid tiers can drop to 2–5 seconds via `poll_interval`. -class MassiveDataSource(MarketDataSource): - """MarketDataSource backed by the Massive (Polygon.io) REST API. +The v3 unified snapshot (`list_universal_snapshots`) is the alternative; it reports unknown +tickers explicitly with an `error` field instead of silently omitting them. FinAlly stays on v2: +a 10-ticker watchlist never approaches v3's 250-ticker limit, v2 is a single non-paginated +request, and per-ticker validation feedback is marginal when the simulator is the default path. +v3 is the right upgrade if that feedback is ever wanted. - Polls GET /v2/snapshot/locale/us/markets/stocks/tickers for all watched - tickers in a single API call, then writes results to the PriceCache. +### 8.2 `RESTClient` is synchronous — wrap every call - Rate limits: - - Free tier: 5 req/min → poll every 15s (default) - - Paid tiers: higher limits → poll every 2-5s - """ +It is `urllib3`-based. Calling it from `async def` blocks the event loop for the whole HTTP round +trip, which in this app means visibly stuttering prices on the SSE stream. - def __init__( - self, - api_key: str, - price_cache: PriceCache, - poll_interval: float = 15.0, - ) -> None: - self._api_key = api_key - self._cache = price_cache - self._interval = poll_interval - self._tickers: list[str] = [] - self._task: asyncio.Task | None = None - self._client: Any = None # Lazy import to avoid hard dependency +```python +snapshots = await asyncio.to_thread(self._fetch_snapshots) +``` - async def start(self, tickers: list[str]) -> None: - # Lazy import: only import massive when actually using real market data. - # This means the massive package is not required when using the simulator. - from massive import RESTClient +It also **retries 429 internally** (3 attempts, honoring `Retry-After`), so a rate-limited poll +blocks its worker thread rather than failing fast. That is fine — the worker is not the event loop. - self._client = RESTClient(api_key=self._api_key) - self._tickers = list(tickers) +### 8.3 Timestamp units — the trap - # Do an immediate first poll so the cache has data right away - await self._poll_once() +Massive uses three different time units across endpoints and the SDK passes them through unchanged. - self._task = asyncio.create_task(self._poll_loop(), name="massive-poller") - logger.info( - "Massive poller started: %d tickers, %.1fs interval", - len(tickers), - self._interval, - ) +| Source | Attribute | Unit | To Unix seconds | +|---|---|---|---| +| Snapshot `lastTrade` | `sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | +| Snapshot `lastQuote` | `sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | +| Snapshot top level | `updated` | **nanoseconds** | `/ 1_000_000_000` | +| Snapshot `min` | `timestamp` | **milliseconds** | `/ 1_000` | +| Aggregates (`Agg`, `PreviousCloseAgg`, grouped) | `timestamp` | **milliseconds** | `/ 1_000` | - async def stop(self) -> None: - if self._task and not self._task.done(): - self._task.cancel() - try: - await self._task - except asyncio.CancelledError: - pass - self._task = None - self._client = None - logger.info("Massive poller stopped") - - async def add_ticker(self, ticker: str) -> None: - ticker = ticker.upper().strip() - if ticker not in self._tickers: - self._tickers.append(ticker) - logger.info("Massive: added ticker %s (will appear on next poll)", ticker) - - async def remove_ticker(self, ticker: str) -> None: - ticker = ticker.upper().strip() - self._tickers = [t for t in self._tickers if t != ticker] - self._cache.remove(ticker) - logger.info("Massive: removed ticker %s", ticker) - - def get_tickers(self) -> list[str]: - return list(self._tickers) - - # --- Internal --- - - async def _poll_loop(self) -> None: - """Poll on interval. First poll already happened in start().""" - while True: - await asyncio.sleep(self._interval) - await self._poll_once() - - async def _poll_once(self) -> None: - """Execute one poll cycle: fetch snapshots, update cache.""" - if not self._tickers or not self._client: - return - - try: - # The Massive RESTClient is synchronous — run in a thread to - # avoid blocking the event loop. - snapshots = await asyncio.to_thread(self._fetch_snapshots) - processed = 0 - for snap in snapshots: - try: - price = snap.last_trade.price - # Massive timestamps are Unix milliseconds → convert to seconds - timestamp = snap.last_trade.timestamp / 1000.0 - self._cache.update( - ticker=snap.ticker, - price=price, - timestamp=timestamp, - ) - processed += 1 - except (AttributeError, TypeError) as e: - logger.warning( - "Skipping snapshot for %s: %s", - getattr(snap, "ticker", "???"), - e, - ) - logger.debug("Massive poll: updated %d/%d tickers", processed, len(self._tickers)) - - except Exception as e: - logger.error("Massive poll failed: %s", e) - # Don't re-raise — the loop will retry on the next interval. - # Common failures: 401 (bad key), 429 (rate limit), network errors. - - def _fetch_snapshots(self) -> list: - """Synchronous call to the Massive REST API. Runs in a thread.""" - from massive.rest.models import SnapshotMarketType - - return self._client.get_snapshot_all( - market_type=SnapshotMarketType.STOCKS, - tickers=self._tickers, - ) +And **attribute names never match JSON keys.** The wire format is single-letter (`p`, `s`, `t`, +`x`); `from_dict` maps those to readable attributes. `@modelclass` builds a plain dataclass with +no `__getattr__` fallback, so reading a key name raises `AttributeError`. + +| `LastTrade` attribute | JSON key | Units | +|---|---|---| +| `price` | `p` | dollars | +| `size` | `s` | shares | +| `sip_timestamp` | `t` | **nanoseconds** | +| `exchange` | `x` | exchange ID | + +### 8.4 Two defects in the shipped client — reproduced, not inferred + +`_poll_once` currently reads: + +```python +price = snap.last_trade.price +timestamp = snap.last_trade.timestamp / 1000.0 # AttributeError, then wrong unit ``` -### Error handling philosophy +Reproduction against the installed SDK, run on 2026-09-01: -The Massive poller is intentionally resilient: +```python +from massive.rest.models.snapshot import TickerSnapshot -| Error | Behavior | -|-------|----------| -| **401 Unauthorized** | Logged as error. Poller keeps running (user might fix `.env` and restart). | -| **429 Rate Limited** | Logged as error. Next poll retries after `poll_interval` seconds. | -| **Network timeout** | Logged as error. Retries automatically on next cycle. | -| **Malformed snapshot** | Individual ticker skipped with warning. Other tickers still processed. | -| **All tickers fail** | Cache retains last-known prices. SSE keeps streaming stale data (better than no data). | +snap = TickerSnapshot.from_dict({ + "ticker": "AAPL", + "lastTrade": {"p": 190.52, "s": 100, "t": 1755873791482000000, "x": 4}, +}) -### Lazy import strategy +snap.last_trade.price # 190.52 +snap.last_trade.sip_timestamp # 1755873791482000000 +hasattr(snap.last_trade, "timestamp") # False +``` -`from massive import RESTClient` happens inside `start()`, not at module import time. This means: -- The `massive` package is only required when `MASSIVE_API_KEY` is set. -- Students who don't have a Massive API key don't need the package installed at all. -- The simulator path has zero external dependencies beyond `numpy`. +**Defect 1 — `last_trade.timestamp` does not exist, so the Massive path writes nothing at all.** +The loop wraps each snapshot in `except (AttributeError, TypeError)` and merely logs a warning, so +the exception is swallowed once per ticker on every poll. The symptom is not a crash: it is a +watchlist where every ticker shows `—` forever, with `Skipping snapshot for AAPL` in the logs. ---- +**Defect 2 — the divisor is wrong by 10⁶.** Even with the attribute corrected, `/ 1000.0` treats +nanoseconds as milliseconds: `1755873791482000000 / 1000` ≈ 1.76 × 10¹⁵ seconds, roughly 55 million +years in the future. Any chart keyed on that timestamp is unusable. -## 8. Factory +**Why 94% coverage did not catch either.** `tests/market/test_massive.py` builds snapshots from +`MagicMock`, which answers to any attribute name: -**File: `backend/app/market/factory.py`** +```python +snap.last_trade.timestamp = timestamp_ms # an attribute the real model does not have +``` + +`test_timestamp_conversion` then locks in the wrong unit as well. The lesson generalizes: +**mocking a third-party model tests your assumptions about the library, not the library.** +Parsing tests must go through the real `TickerSnapshot.from_dict` with a documented payload +(§14.4). That test needs no network and would have failed on its first run. + +### 8.5 The corrected parse ```python -from __future__ import annotations +NANOS_PER_SECOND = 1_000_000_000 + +for snap in snapshots: + trade = snap.last_trade + if trade is None or trade.price is None: + continue # no print yet today; leave the ticker showing "—" + self._cache.update( + ticker=snap.ticker, + price=trade.price, + timestamp=( + trade.sip_timestamp / NANOS_PER_SECOND + if trade.sip_timestamp + else time.time() + ), + ) + processed += 1 +``` + +**Guarding on `is None` rather than catching `AttributeError` is what makes the difference.** +A genuinely absent field is a normal condition to handle; a misspelled attribute is a bug that +should be loud. The existing blanket `except AttributeError` is precisely what hid defect 1. + +### 8.6 Error handling in the poll loop + +The SDK raises only two exception types (`massive/exceptions.py`): `AuthError` (empty or missing +key, raised at construction) and `BadResponse` (any non-200 surviving the retry policy). +`urllib3` raises its own for connection failures and timeouts. + +```python +from massive.exceptions import AuthError, BadResponse + +async def _poll_once(self) -> None: + if not self._tickers or not self._client: + return + try: + snapshots = await asyncio.to_thread(self._fetch_snapshots) + except AuthError: + logger.error("Massive API key rejected — the source does not fall back automatically") + raise # unrecoverable: do not retry on a loop + except BadResponse as e: + logger.warning("Massive returned an error response: %s", e) + return # transient: retry next interval + except Exception: + logger.exception("Massive poll failed") + return + ... # the §8.5 parse +``` + +`start()` performs one poll synchronously **before** creating the task, so the cache is warm +before the first client connects: + +```python +async def _poll_loop(self) -> None: + """Poll on interval. The first poll already happened in start().""" + while True: + await asyncio.sleep(self._interval) + await self._poll_once() +``` + +### 8.7 Behaviors to surface in the README + +Properties of the data source, not bugs — users will otherwise report them as bugs: + +- **Unknown symbols vanish silently.** The v2 snapshot omits tickers it does not recognize; there + is no error entry. The ticker sits in the watchlist showing `—` indefinitely. +- **Prices freeze outside market hours.** Overnight, at weekends, and on holidays the snapshot + returns the previous session's last trade. The UI looks broken but is correct. **This is the + main reason the simulator is the default.** +- **Free-tier data is 15 minutes delayed**, so prices will not match any other quote source the + user has open. +- **Snapshot data is cleared at midnight ET** and repopulates from about 4am ET. Between those + times `last_trade` may be absent entirely — exactly the `None` case §8.5 guards. + +`client.get_market_status()` is worth one call to explain a frozen feed rather than leaving the +user guessing. + +### 8.8 Live verification + +Run once a real key exists — it confirms auth, the multi-ticker snapshot, and unit conversion in +one pass: + +```python +# backend/scripts/verify_massive.py +"""Smoke-test the Massive REST API against a live key.""" -import logging import os +from datetime import UTC, datetime + +from massive import RESTClient +from massive.rest.models import SnapshotMarketType + +NANOS_PER_SECOND = 1_000_000_000 +TICKERS = ["AAPL", "GOOGL", "MSFT", "NVDA", "TSLA"] + + +def main() -> None: + client = RESTClient(api_key=os.environ["MASSIVE_API_KEY"]) + + print(f"market: {client.get_market_status().market}") + + snapshots = client.get_snapshot_all(SnapshotMarketType.STOCKS, TICKERS) + print(f"requested {len(TICKERS)}, received {len(snapshots)}") + + for snap in snapshots: + trade = snap.last_trade + if trade is None or trade.price is None: + print(f"{snap.ticker}: no trade data") + continue + when = datetime.fromtimestamp(trade.sip_timestamp / NANOS_PER_SECOND, UTC) + print(f"{snap.ticker}: ${trade.price:.2f} at {when:%Y-%m-%d %H:%M:%S} UTC") + + missing = set(TICKERS) - {s.ticker for s in snapshots} + if missing: + print(f"absent from response (unknown or untraded): {sorted(missing)}") + + +if __name__ == "__main__": + main() +``` + +```bash +cd backend && uv run python scripts/verify_massive.py +``` -from .cache import PriceCache -from .interface import MarketDataSource +Expected: a market status and five priced tickers with timestamps **in the recent past**. +Timestamps far in the future mean the divisor regressed; `AttributeError` means §8.4 regressed. -logger = logging.getLogger(__name__) +--- +## 9. Selection — `create_market_data_source` +```python def create_market_data_source(price_cache: PriceCache) -> MarketDataSource: - """Create the appropriate market data source based on environment variables. + """Create the market data source indicated by the environment. - - MASSIVE_API_KEY set and non-empty → MassiveDataSource (real market data) - - Otherwise → SimulatorDataSource (GBM simulation) + MASSIVE_API_KEY set and non-empty -> MassiveDataSource (real data) + otherwise -> SimulatorDataSource (GBM simulation) - Returns an unstarted source. Caller must await source.start(tickers). + Returns an unstarted source; the caller must await source.start(tickers). """ api_key = os.environ.get("MASSIVE_API_KEY", "").strip() if api_key: - from .massive_client import MassiveDataSource - logger.info("Market data source: Massive API (real data)") return MassiveDataSource(api_key=api_key, price_cache=price_cache) - else: - from .simulator import SimulatorDataSource - logger.info("Market data source: GBM Simulator") - return SimulatorDataSource(price_cache=price_cache) + logger.info("Market data source: GBM Simulator") + return SimulatorDataSource(price_cache=price_cache) ``` -### Usage at app startup - -```python -price_cache = PriceCache() -source = create_market_data_source(price_cache) -await source.start(initial_tickers) # e.g., ["AAPL", "GOOGL", ...] -``` - ---- +**`.strip()` before the truth test is deliberate.** `.env` files routinely contain +`MASSIVE_API_KEY=` or a stray space, and a whitespace-only key would otherwise select the Massive +path and then fail every poll with a 401. Empty means empty. -## 9. SSE Streaming Endpoint +**The choice is made once at startup and never at runtime.** A source that silently switched to +the simulator after a Massive outage would show users invented prices while they believed they +were seeing the market. Rejected keys and failed polls are logged; they do not change the source. +`GET /api/health` reports which one is live: -**File: `backend/app/market/stream.py`** - -The SSE endpoint is a FastAPI route that holds open a long-lived HTTP connection and pushes price updates to the client as `text/event-stream`. +```json +{"status": "ok", "market_source": "simulator", "llm_mock": false} +``` -```python -from __future__ import annotations +Returning an **unstarted** source keeps construction synchronous and lets the caller decide the +ticker set from the database — the factory has no business reading tables. -import asyncio -import json -import logging -import time +--- -from fastapi import APIRouter, Request -from fastapi.responses import StreamingResponse +## 10. The SSE stream -from .cache import PriceCache +### 10.1 What ships -logger = logging.getLogger(__name__) +`GET /api/stream/prices`, `Content-Type: text/event-stream`. The generator opens with +`retry: 1000`, then pushes the **entire cache as one JSON object** whenever `version` changes, +polled every 500ms: -router = APIRouter(prefix="/api/stream", tags=["streaming"]) +``` +retry: 1000 +data: {"AAPL": {"ticker": "AAPL", "price": 190.52, "previous_price": 190.48, "timestamp": 1755873791.482, "change": 0.04, "change_percent": 0.021, "direction": "up"}, "GOOGL": {...}} +``` -def create_stream_router(price_cache: PriceCache) -> APIRouter: - """Create the SSE streaming router with a reference to the price cache. +One event carries every ticker. The client replaces its price map wholesale — no merge logic, no +missed-update reconciliation. Because a fresh generator starts at `last_version = -1`, the first +comparison always differs, so **every connecting client immediately receives a full snapshot**, +including after a reconnect. That is why no separate snapshot endpoint exists. - This factory pattern lets us inject the PriceCache without globals. - """ +Response headers matter as much as the payload: - @router.get("/prices") - async def stream_prices(request: Request) -> StreamingResponse: - """SSE endpoint for live price updates. +```python +return StreamingResponse( + _generate_events(price_cache, request), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "Connection": "keep-alive", + "X-Accel-Buffering": "no", # disable nginx buffering if proxied + }, +) +``` - Streams all tracked ticker prices every ~500ms. The client connects - with EventSource and receives events in the format: +**Why poll-and-push instead of event-driven?** A 500ms integer comparison is cheaper to reason +about than a pub/sub fan-out across an arbitrary number of generators, and it naturally coalesces: +if the cache updated ten tickers since the last check, the client gets one event, not ten. - data: {"AAPL": {"ticker": "AAPL", "price": 190.50, ...}, ...} +### 10.2 Keepalive — to implement - Includes a retry directive so the browser auto-reconnects on - disconnection (EventSource built-in behavior). - """ - return StreamingResponse( - _generate_events(price_cache, request), - media_type="text/event-stream", - headers={ - "Cache-Control": "no-cache", - "Connection": "keep-alive", - "X-Accel-Buffering": "no", # Disable nginx buffering if proxied - }, - ) +When the version has not changed for 15 seconds, emit an SSE comment line. The complete generator: - return router +```python +KEEPALIVE_SECONDS = 15.0 async def _generate_events( price_cache: PriceCache, request: Request, interval: float = 0.5, -) -> None: - """Async generator that yields SSE-formatted price events. - - Sends all prices every `interval` seconds. Stops when the client - disconnects (detected via request.is_disconnected()). - """ - # Tell the client to retry after 1 second if the connection drops +) -> AsyncGenerator[str, None]: + """Yield SSE events whenever the cache version changes; ping when it does not.""" yield "retry: 1000\n\n" last_version = -1 + last_sent = time.monotonic() client_ip = request.client.host if request.client else "unknown" logger.info("SSE client connected: %s", client_ip) try: while True: - # Check for client disconnect if await request.is_disconnected(): logger.info("SSE client disconnected: %s", client_ip) break @@ -924,567 +1030,449 @@ async def _generate_events( if current_version != last_version: last_version = current_version prices = price_cache.get_all() - if prices: - data = { - ticker: update.to_dict() - for ticker, update in prices.items() - } - payload = json.dumps(data) + payload = json.dumps({t: u.to_dict() for t, u in prices.items()}) yield f"data: {payload}\n\n" + last_sent = time.monotonic() + elif time.monotonic() - last_sent >= KEEPALIVE_SECONDS: + yield ": ping\n\n" + last_sent = time.monotonic() await asyncio.sleep(interval) except asyncio.CancelledError: logger.info("SSE stream cancelled for: %s", client_ip) ``` -### SSE wire format +Without this, a Massive-backed feed sends no bytes between 15-second polls. That idle-times-out +through proxies and leaves the frontend unable to distinguish a quiet market from a dead +connection. The frontend indicator — green on `onopen`, yellow on `onerror`, red after a gap +beyond ~40 seconds — depends on it. -Each event the client receives looks like this: +A line beginning with `:` is a comment in the SSE grammar: `EventSource` ignores it entirely, so +it costs the client nothing while keeping the socket warm. -``` -data: {"AAPL":{"ticker":"AAPL","price":190.50,"previous_price":190.42,"timestamp":1707580800.5,"change":0.08,"change_percent":0.042,"direction":"up"},"GOOGL":{"ticker":"GOOGL","price":175.12,...}} +--- + +## 11. `GET /api/prices/{ticker}/history` — to implement + +Backed by `PriceCache.get_history` (§5.2). It belongs in `stream.py` next to the SSE endpoint, +since both are pure cache readers with no database involvement. + +```python +history_router = APIRouter(prefix="/api/prices", tags=["prices"]) + + +def create_history_router(price_cache: PriceCache) -> APIRouter: + @history_router.get("/{ticker}/history") + async def get_price_history(ticker: str, limit: int = 600) -> dict: + """Rolling in-memory price history for the main chart. + + Returns an empty `points` list for an untracked ticker — not a 404, + so the chart draws nothing rather than erroring. + """ + ticker = ticker.strip().upper() + limit = max(1, min(limit, HISTORY_MAXLEN)) + points = price_cache.get_history(ticker, limit=limit) + return { + "ticker": ticker, + "points": [{"timestamp": ts, "price": price} for ts, price in points], + } + return history_router ``` -The client parses this with: +Response: -```javascript -const eventSource = new EventSource('/api/stream/prices'); -eventSource.onmessage = (event) => { - const prices = JSON.parse(event.data); - // prices is { "AAPL": { ticker, price, previous_price, ... }, ... } -}; +```json +{"ticker": "AAPL", "points": [{"timestamp": 1755873791.482, "price": 190.52}]} ``` -### Why poll-and-push instead of event-driven? +Oldest-first, matching what Recharts wants for a left-to-right time axis. `limit` is clamped +rather than validated with a 400 — a chart asking for 10,000 points should get 600, not an error. -The SSE endpoint polls the cache on a fixed interval rather than being notified by the data source. This is simpler and produces predictable, evenly-spaced updates for the frontend. The frontend accumulates these into sparkline charts — regular spacing is important for clean visualization. +This endpoint reads only in-memory state, so `async def` is correct here; there is no SQLite call +to keep off the event loop. --- -## 10. FastAPI Lifecycle Integration +## 12. Wiring -The market data system starts and stops with the FastAPI application using the `lifespan` context manager pattern. +### 12.1 Lifespan -**In `backend/app/main.py`:** +One `PriceCache` and one source per process, owned by the FastAPI lifespan. ```python from contextlib import asynccontextmanager from fastapi import FastAPI -from app.market.cache import PriceCache -from app.market.factory import create_market_data_source -from app.market.interface import MarketDataSource -from app.market.stream import create_stream_router +from app.market import PriceCache, create_market_data_source, create_stream_router @asynccontextmanager async def lifespan(app: FastAPI): - """Manage startup and shutdown of background services.""" + init_db() # lazy schema creation + seed (PLAN.md §7) - # --- STARTUP --- + cache = PriceCache() + source = create_market_data_source(cache) - # 1. Create the shared price cache - price_cache = PriceCache() - app.state.price_cache = price_cache + # Reconciliation: watchlist ∪ held positions, not just the watchlist + tickers = sorted(set(get_watchlist_tickers()) | set(get_position_tickers())) + await source.start(tickers) - # 2. Create and start the market data source - source = create_market_data_source(price_cache) + app.state.price_cache = cache app.state.market_source = source + try: + yield + finally: + await source.stop() - # 3. Load initial tickers from the database watchlist - initial_tickers = await load_watchlist_tickers() # reads from SQLite - await source.start(initial_tickers) - # 4. Register the SSE streaming router - stream_router = create_stream_router(price_cache) - app.include_router(stream_router) +app = FastAPI(lifespan=lifespan) - yield # App is running +# 1. API routers FIRST +app.include_router(create_stream_router(cache)) +app.include_router(create_history_router(cache)) +# ... portfolio, watchlist, chat routers ... +# 2. static assets +# 3. catch-all -> index.html +``` + +Three things this gets right and are easy to get wrong: - # --- SHUTDOWN --- - await source.stop() +**Reading both tables at startup**, not just the watchlist, is what makes a position held across +a restart come back with a live price. Without it, an off-watchlist holding valuates at `avg_cost` +forever and the snapshot task stalls under the "skip if any held ticker has no price" rule. +**The cache and source are passed explicitly** (via router factories or `app.state`) rather than +held in module globals, which is what keeps tests able to construct an isolated cache per test. -app = FastAPI(title="FinAlly", lifespan=lifespan) +**Mount all `/api/*` routers before the static file mount.** A `StaticFiles(html=True)` mount at +`/` registered first shadows every endpoint, including the SSE stream (`PLAN.md` §11). +Route handlers reach the cache through `app.state` or a dependency: -# Dependency for injecting the price cache into route handlers -def get_price_cache() -> PriceCache: - return app.state.price_cache +```python +def get_price_cache(request: Request) -> PriceCache: + return request.app.state.price_cache -def get_market_source() -> MarketDataSource: - return app.state.market_source +def get_market_source(request: Request) -> MarketDataSource: + return request.app.state.market_source ``` -### Accessing market data from other routes +### 12.2 The tracked ticker set + +**The tracked set is `watchlist ∪ {tickers with a non-zero position}`.** -Other parts of the backend (trade execution, portfolio valuation, watchlist management) access the price cache and data source via FastAPI's dependency injection: +The two sets diverge the moment a user buys TSLA and then removes it from the watchlist. The +position still needs a live price for valuation, P&L, the heatmap, and snapshots. + +| Trigger | Action | +|---|---| +| `POST /api/watchlist` | always `await source.add_ticker(t)` | +| `DELETE /api/watchlist/{t}` | `await source.remove_ticker(t)` **only if no position in `t` is held** | +| Buy a ticker not currently tracked | `await source.add_ticker(t)` as part of trade execution | +| Sell a position to zero | if `t` is not on the watchlist, `await source.remove_ticker(t)` | +| `POST /api/reset` | re-sync the tracked set to exactly the ten default tickers | + +One helper keeps the rule in one place rather than at four call sites: ```python -from fastapi import APIRouter, Depends - -router = APIRouter(prefix="/api") - -@router.post("/portfolio/trade") -async def execute_trade( - trade: TradeRequest, - price_cache: PriceCache = Depends(get_price_cache), -): - current_price = price_cache.get_price(trade.ticker) - if current_price is None: - raise HTTPException(404, f"No price available for {trade.ticker}") - # ... execute trade at current_price ... - - -@router.post("/watchlist") -async def add_to_watchlist( - payload: WatchlistAdd, - source: MarketDataSource = Depends(get_market_source), - price_cache: PriceCache = Depends(get_price_cache), -): - # Add to database ... - # Then tell the data source to start tracking it - await source.add_ticker(payload.ticker) - # ... - - -@router.delete("/watchlist/{ticker}") -async def remove_from_watchlist( - ticker: str, - source: MarketDataSource = Depends(get_market_source), -): - # Remove from database ... - # Then stop tracking +async def untrack_if_unused(source: MarketDataSource, ticker: str) -> None: + """Stop tracking a ticker only if it is neither watched nor held.""" + if is_on_watchlist(ticker) or has_position(ticker): + return await source.remove_ticker(ticker) - # ... ``` ---- +### 12.3 Ticker validation at the boundary -## 11. Watchlist Coordination +Applied at `POST /api/watchlist`, `POST /api/portfolio/trade`, and every LLM-proposed action, so +the market layer only ever sees canonical symbols: -When the watchlist changes (via REST API or LLM chat), the market data source must be notified so it tracks the right set of tickers. +```python +TICKER_PATTERN = re.compile(r"^[A-Z][A-Z.]{0,5}$") -### Flow: Adding a Ticker +def normalize_ticker(raw: str) -> str: + """Uppercase and validate. Raises ValueError with the user-facing message.""" + ticker = raw.strip().upper() + if not TICKER_PATTERN.match(ticker): + raise ValueError("Invalid ticker symbol") + return ticker ``` -User (or LLM) → POST /api/watchlist {ticker: "PYPL"} - → Insert into watchlist table (SQLite) - → await source.add_ticker("PYPL") - Simulator: adds to GBMSimulator, rebuilds Cholesky, seeds cache - Massive: appends to ticker list, appears on next poll - → Return success (ticker + current price if available) -``` - -### Flow: Removing a Ticker -``` -User (or LLM) → DELETE /api/watchlist/PYPL - → Delete from watchlist table (SQLite) - → await source.remove_ticker("PYPL") - Simulator: removes from GBMSimulator, rebuilds Cholesky, removes from cache - Massive: removes from ticker list, removes from cache - → Return success -``` +No allowlist. Any symbol matching the pattern is accepted; the simulator invents plausible +behavior for it, and under Massive an unknown symbol shows `—`. Rejecting unknown symbols would +make the LLM's `watchlist_changes` feature feel broken. -### Edge case: Ticker has an open position +Uppercasing is not cosmetic: the `UNIQUE(user_id, ticker)` constraints would otherwise happily +hold both `AAPL` and `aapl`. -If the user removes a ticker from the watchlist but still holds shares, the ticker should remain in the data source so portfolio valuation stays accurate. The watchlist route should check for this: +--- -```python -@router.delete("/watchlist/{ticker}") -async def remove_from_watchlist( - ticker: str, - source: MarketDataSource = Depends(get_market_source), -): - # Remove from watchlist table - await db.delete_watchlist_entry(ticker) - - # Only stop tracking if no open position - position = await db.get_position(ticker) - if position is None or position.quantity == 0: - await source.remove_ticker(ticker) - - return {"status": "ok"} -``` +## 13. Failure modes + +| Situation | Behavior | Where | +|---|---|---| +| Empty ticker list at startup | `step()` returns `{}`, SSE sends nothing until a ticker is added | §7.4 | +| One bad simulator tick | Logged, loop continues next interval | §7.6 | +| Massive poll fails (429, network) | Logged, cache keeps last prices, retry next interval | §8.6 | +| Massive key rejected | `AuthError` re-raised; **no automatic fallback to the simulator** | §8.6, §9 | +| Ticker has no `last_trade` yet | Skipped; ticker shows `—` | §8.5 | +| Held ticker has no cached price | Portfolio values it at `avg_cost`; snapshot task skips the write entirely | `PLAN.md` §7 | +| Ticker removed while held | Prevented by `untrack_if_unused` | §12.2 | +| Client disconnects mid-stream | `request.is_disconnected()` breaks the generator | §10.2 | +| Quiet feed (Massive, 15s polls) | `: ping` every 15s keeps the connection and the indicator alive | §10.2 | +| History requested for untracked ticker | `{"ticker": "X", "points": []}` | §11 | --- -## 12. Testing Strategy +## 14. Testing -### 12.1 Unit Tests for GBMSimulator +Current state: **73 tests, 91% coverage** on the market module. `stream.py` sits at 33% — the SSE +generator is the least-tested code in the subsystem and the keepalive change is a good moment to +fix that. + +```bash +cd backend +uv run --extra dev pytest -v +uv run --extra dev pytest --cov=app --cov-report=term-missing +``` -**File: `backend/tests/market/test_simulator.py`** +### 14.1 A stub source + +The cache and the tracked-set rules can be tested without either real source: ```python -import math -import pytest -from app.market.simulator import GBMSimulator -from app.market.seed_prices import SEED_PRICES - - -class TestGBMSimulator: - """Unit tests for the GBM price simulator.""" - - def test_step_returns_all_tickers(self): - sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) - result = sim.step() - assert set(result.keys()) == {"AAPL", "GOOGL"} - - def test_prices_are_positive(self): - """GBM prices can never go negative (exp() is always positive).""" - sim = GBMSimulator(tickers=["AAPL"]) - for _ in range(10_000): - prices = sim.step() - assert prices["AAPL"] > 0 - - def test_initial_prices_match_seeds(self): - sim = GBMSimulator(tickers=["AAPL"]) - # Before any step, price should be the seed price - assert sim.get_price("AAPL") == SEED_PRICES["AAPL"] - - def test_add_ticker(self): - sim = GBMSimulator(tickers=["AAPL"]) - sim.add_ticker("TSLA") - result = sim.step() - assert "TSLA" in result - - def test_remove_ticker(self): - sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) - sim.remove_ticker("GOOGL") - result = sim.step() - assert "GOOGL" not in result - assert "AAPL" in result - - def test_add_duplicate_is_noop(self): - sim = GBMSimulator(tickers=["AAPL"]) - sim.add_ticker("AAPL") - assert len(sim._tickers) == 1 - - def test_remove_nonexistent_is_noop(self): - sim = GBMSimulator(tickers=["AAPL"]) - sim.remove_ticker("NOPE") # Should not raise - - def test_unknown_ticker_gets_random_seed_price(self): - sim = GBMSimulator(tickers=["ZZZZ"]) - price = sim.get_price("ZZZZ") - assert 50.0 <= price <= 300.0 - - def test_empty_step(self): - sim = GBMSimulator(tickers=[]) - result = sim.step() - assert result == {} - - def test_prices_change_over_time(self): - """After many steps, prices should have drifted from their seeds.""" - sim = GBMSimulator(tickers=["AAPL"]) - for _ in range(1000): - sim.step() - # Price should have changed (extremely unlikely to be exactly the seed) - assert sim.get_price("AAPL") != SEED_PRICES["AAPL"] - - def test_cholesky_rebuilds_on_add(self): - sim = GBMSimulator(tickers=["AAPL"]) - assert sim._cholesky is None # Only 1 ticker, no correlation matrix - sim.add_ticker("GOOGL") - assert sim._cholesky is not None # Now 2 tickers, matrix exists +class StubDataSource(MarketDataSource): + """Records lifecycle calls; writes nothing on its own.""" + + def __init__(self, cache: PriceCache) -> None: + self._cache = cache + self._tickers: list[str] = [] + self.started = False + + async def start(self, tickers): self._tickers = list(tickers); self.started = True + async def stop(self): self.started = False + async def add_ticker(self, t): + if t not in self._tickers: + self._tickers.append(t) + async def remove_ticker(self, t): + self._tickers = [x for x in self._tickers if x != t] + self._cache.remove(t) + def get_tickers(self): return list(self._tickers) ``` -### 12.2 Unit Tests for PriceCache +### 14.2 Simulator -**File: `backend/tests/market/test_cache.py`** +Seed **both** RNGs — the simulator uses `numpy.random` for the normal draws and stdlib `random` +for events: ```python -import pytest -from app.market.cache import PriceCache - - -class TestPriceCache: - - def test_update_and_get(self): - cache = PriceCache() - update = cache.update("AAPL", 190.50) - assert update.ticker == "AAPL" - assert update.price == 190.50 - assert cache.get("AAPL") == update - - def test_first_update_is_flat(self): - cache = PriceCache() - update = cache.update("AAPL", 190.50) - assert update.direction == "flat" - assert update.previous_price == 190.50 - - def test_direction_up(self): - cache = PriceCache() - cache.update("AAPL", 190.00) - update = cache.update("AAPL", 191.00) - assert update.direction == "up" - assert update.change == 1.00 - - def test_direction_down(self): - cache = PriceCache() - cache.update("AAPL", 190.00) - update = cache.update("AAPL", 189.00) - assert update.direction == "down" - assert update.change == -1.00 - - def test_remove(self): - cache = PriceCache() - cache.update("AAPL", 190.00) - cache.remove("AAPL") - assert cache.get("AAPL") is None - - def test_get_all(self): - cache = PriceCache() - cache.update("AAPL", 190.00) - cache.update("GOOGL", 175.00) - all_prices = cache.get_all() - assert set(all_prices.keys()) == {"AAPL", "GOOGL"} - - def test_version_increments(self): - cache = PriceCache() - v0 = cache.version - cache.update("AAPL", 190.00) - assert cache.version == v0 + 1 - cache.update("AAPL", 191.00) - assert cache.version == v0 + 2 - - def test_get_price_convenience(self): - cache = PriceCache() - cache.update("AAPL", 190.50) - assert cache.get_price("AAPL") == 190.50 - assert cache.get_price("NOPE") is None +def test_step_is_reproducible(): + np.random.seed(42) + random.seed(42) + sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) + first = sim.step() + + np.random.seed(42) + random.seed(42) + sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) + assert sim.step() == first ``` -### 12.3 Integration Test: SimulatorDataSource - -**File: `backend/tests/market/test_simulator_source.py`** +Statistical properties need wide tolerances and events disabled — a 5% jump is a massive outlier +at this `dt` and would dominate the sample variance: ```python -import asyncio -import pytest -from app.market.cache import PriceCache -from app.market.simulator import SimulatorDataSource +def test_realised_volatility_is_close_to_sigma(): + sim = GBMSimulator(tickers=["AAPL"], event_probability=0.0) + prices = [sim.get_price("AAPL")] + for _ in range(20_000): + prices.append(sim.step()["AAPL"]) + log_returns = np.diff(np.log(prices)) + realised = log_returns.std() / np.sqrt(GBMSimulator.DEFAULT_DT) + assert 0.15 < realised < 0.35 # nominal sigma is 0.22 -@pytest.mark.asyncio -class TestSimulatorDataSource: - async def test_start_populates_cache(self): - cache = PriceCache() - source = SimulatorDataSource(price_cache=cache, update_interval=0.1) - await source.start(["AAPL", "GOOGL"]) +def test_tech_tickers_are_positively_correlated(): + sim = GBMSimulator(tickers=["AAPL", "MSFT"], event_probability=0.0) + a, m = [], [] + for _ in range(10_000): + p = sim.step() + a.append(p["AAPL"]) + m.append(p["MSFT"]) - # Cache should have seed prices immediately (before first loop tick) - assert cache.get("AAPL") is not None - assert cache.get("GOOGL") is not None + rho = np.corrcoef(np.diff(np.log(a)), np.diff(np.log(m)))[0, 1] + assert rho > 0.4 # nominal 0.6 - await source.stop() - async def test_prices_update_over_time(self): - cache = PriceCache() - source = SimulatorDataSource(price_cache=cache, update_interval=0.05) - await source.start(["AAPL"]) +def test_correlation_matrix_stays_positive_definite(): + """Run after ANY change to the correlation constants in seed_prices.py.""" + tickers = list(SEED_PRICES) + [f"UNK{i}" for i in range(40)] + GBMSimulator(tickers=tickers) # raises LinAlgError if not PD +``` - initial = cache.get("AAPL").price - await asyncio.sleep(0.3) # Several update cycles - current = cache.get("AAPL").price +Also cover: prices stay strictly positive over thousands of steps; `step()` returns exactly the +current ticker set; add/remove keeps `_tickers`/`_prices`/`_params` consistent and the Cholesky +shape matching; unknown tickers seed within $50–$300 with `DEFAULT_PARAMS`; `remove_ticker` on an +untracked symbol is a no-op. - # Extremely unlikely to be identical after many steps - # (but not impossible, so this is a probabilistic test) - assert current != initial or True # Soft assertion +### 14.3 Cache and history - await source.stop() +```python +def test_history_is_bounded_and_oldest_first(): + cache = PriceCache(history_maxlen=5) + for i in range(10): + cache.update("AAPL", 100.0 + i, timestamp=float(i)) - async def test_stop_is_clean(self): - cache = PriceCache() - source = SimulatorDataSource(price_cache=cache, update_interval=0.1) - await source.start(["AAPL"]) - await source.stop() - # Double stop should not raise - await source.stop() + points = cache.get_history("AAPL") + assert len(points) == 5 + assert [ts for ts, _ in points] == [5.0, 6.0, 7.0, 8.0, 9.0] - async def test_add_and_remove_ticker(self): - cache = PriceCache() - source = SimulatorDataSource(price_cache=cache, update_interval=0.1) - await source.start(["AAPL"]) - await source.add_ticker("TSLA") - assert "TSLA" in source.get_tickers() - assert cache.get("TSLA") is not None +def test_history_is_empty_for_untracked_ticker(): + assert PriceCache().get_history("NOPE") == [] - await source.remove_ticker("TSLA") - assert "TSLA" not in source.get_tickers() - assert cache.get("TSLA") is None - await source.stop() +def test_remove_clears_price_and_history(): + cache = PriceCache() + cache.update("AAPL", 190.0) + cache.remove("AAPL") + assert cache.get("AAPL") is None + assert cache.get_history("AAPL") == [] ``` -### 12.4 Unit Test: MassiveDataSource (Mocked) +Plus: `previous_price` derivation, first-update `flat`, `version` monotonicity, and thread safety +under concurrent writers. -**File: `backend/tests/market/test_massive.py`** +### 14.4 Massive — through the real model, never `MagicMock` -```python -import asyncio -from unittest.mock import MagicMock, patch -import pytest -from app.market.cache import PriceCache -from app.market.massive_client import MassiveDataSource - - -def _make_snapshot(ticker: str, price: float, timestamp_ms: int) -> MagicMock: - """Create a mock Massive snapshot object.""" - snap = MagicMock() - snap.ticker = ticker - snap.last_trade.price = price - snap.last_trade.timestamp = timestamp_ms - return snap - - -@pytest.mark.asyncio -class TestMassiveDataSource: - - async def test_poll_updates_cache(self): - cache = PriceCache() - source = MassiveDataSource( - api_key="test-key", - price_cache=cache, - poll_interval=60.0, # Long interval so the loop doesn't auto-poll - ) +This is the test that would have caught both defects in §8.4, and it needs no network: - mock_snapshots = [ - _make_snapshot("AAPL", 190.50, 1707580800000), - _make_snapshot("GOOGL", 175.25, 1707580800000), - ] +```python +from massive.rest.models.snapshot import TickerSnapshot - with patch.object(source, "_fetch_snapshots", return_value=mock_snapshots): - await source._poll_once() +def test_snapshot_parse_produces_a_present_day_timestamp(): + snap = TickerSnapshot.from_dict({ + "ticker": "AAPL", + "lastTrade": {"p": 190.52, "s": 100, "t": 1755873791482000000, "x": 4}, + }) - assert cache.get_price("AAPL") == 190.50 - assert cache.get_price("GOOGL") == 175.25 + cache = PriceCache() + source = MassiveDataSource(api_key="x", price_cache=cache) + source._apply_snapshots([snap]) # extract the parse into a testable method - async def test_malformed_snapshot_skipped(self): - cache = PriceCache() - source = MassiveDataSource( - api_key="test-key", - price_cache=cache, - poll_interval=60.0, - ) - source._tickers = ["AAPL", "BAD"] - - good_snap = _make_snapshot("AAPL", 190.50, 1707580800000) - bad_snap = MagicMock() - bad_snap.ticker = "BAD" - bad_snap.last_trade = None # Will cause AttributeError - - with patch.object(source, "_fetch_snapshots", return_value=[good_snap, bad_snap]): - await source._poll_once() - - # Good ticker processed, bad one skipped - assert cache.get_price("AAPL") == 190.50 - assert cache.get_price("BAD") is None - - async def test_api_error_does_not_crash(self): - cache = PriceCache() - source = MassiveDataSource( - api_key="test-key", - price_cache=cache, - poll_interval=60.0, - ) - source._tickers = ["AAPL"] + update = cache.get("AAPL") + assert update.price == 190.52 + assert 1_600_000_000 < update.timestamp < 2_000_000_000 # plausible present, in SECONDS - with patch.object(source, "_fetch_snapshots", side_effect=Exception("network error")): - await source._poll_once() # Should not raise - assert cache.get_price("AAPL") is None # No update happened +def test_snapshot_without_a_last_trade_is_skipped(): + snap = TickerSnapshot.from_dict({"ticker": "AAPL"}) + cache = PriceCache() + source = MassiveDataSource(api_key="x", price_cache=cache) + source._apply_snapshots([snap]) + assert cache.get("AAPL") is None ``` ---- +Extracting the parse loop into `_apply_snapshots(snapshots)` is worth the small refactor: it makes +the parse testable without touching HTTP, which is the only part that actually broke. -## 13. Error Handling & Edge Cases +### 14.5 Factory and SSE -### 13.1 Startup: Empty Watchlist +- **Factory** — unset, empty, and whitespace-only `MASSIVE_API_KEY` all select the simulator; a + real value selects Massive. +- **SSE** — map-shaped payload, float timestamp, percent-unit `change_percent`, full snapshot on + connect, and a `: ping` after 15 idle seconds. Drive the generator directly with a fake request + object rather than through a live server; the keepalive test is far easier with an injected + `interval` and a monkeypatched clock than with 15 seconds of real waiting. -If the database has no watchlist entries (user deleted everything), `start()` receives an empty list. Both data sources handle this gracefully — the simulator produces no prices, the Massive poller skips its API call. The SSE endpoint sends empty events. When the user adds a ticker, the source starts tracking it immediately. +### 14.6 Tracked set -### 13.2 Price Cache Miss During Trade +The two regressions that silently produce a frozen position: -If a user tries to trade a ticker that has no cached price (e.g., just added to watchlist, Massive hasn't polled yet): - -```python -price = price_cache.get_price(ticker) -if price is None: - raise HTTPException( - status_code=400, - detail=f"Price not yet available for {ticker}. Please wait a moment and try again.", - ) -``` +- Removing a watchlist ticker with an open position **keeps** it in the feed. +- Selling to zero while off-watchlist **removes** it. -The simulator avoids this by seeding the cache in `add_ticker()`. The Massive client may have a brief gap — the HTTP 400 with a clear message is the correct response. +### 14.7 Eyeballing it -### 13.3 Massive API Key Invalid +```bash +cd backend && uv run market_data_demo.py +``` -If the API key is set but invalid, the first poll will fail with a 401. The poller logs the error and keeps retrying. The SSE endpoint streams empty data. The user sees no prices and a connection status indicator showing "connected" (SSE is working, just no data). The fix is to correct the API key and restart. +A Rich terminal dashboard of the live simulator — the fastest way to check whether a parameter +change still looks right. Statistical tests confirm σ; only the eye confirms "looks like a +trading terminal". -### 13.4 Thread Safety Under Load +--- -The `PriceCache` uses `threading.Lock` which is a mutex — only one thread can hold it at a time. Under normal load (10 tickers, 2 updates/sec), lock contention is negligible. The critical section is tiny (dict lookup + assignment). +## 15. Implementation order -If this ever became a bottleneck (hundreds of tickers, many concurrent SSE readers), the fix would be a `ReadWriteLock` — but that level of optimization is unnecessary for this project. +Small increments, each independently verifiable. Run `uv run --extra dev pytest` after every step. -### 13.5 Simulator Precision +1. **Fix the Massive parse** (§8.5). Extract `_apply_snapshots`, correct the attribute and the + divisor, replace the `MagicMock` tests with `TickerSnapshot.from_dict` tests (§14.4). This is + first because the current code silently produces nothing, and because the fix is provable + offline. +2. **Add rolling history to `PriceCache`** (§5.2). Deque, `get_history`, `remove` clearing both. + Tests in §14.3. +3. **Add `GET /api/prices/{ticker}/history`** (§11). Depends on step 2. +4. **Add the SSE keepalive** (§10.2) and raise `stream.py` coverage off 33% (§14.5). +5. **Wire the lifespan** (§12.1) with startup reconciliation over `watchlist ∪ positions`, and + add `untrack_if_unused` (§12.2) where the watchlist and trade routes are built. -GBM with tiny `dt` produces very small per-tick moves. Floating-point precision is not a concern because: -- Prices are `round()`ed to 2 decimal places in `GBMSimulator.step()` -- The exponential formulation (`exp(drift + diffusion)`) is numerically stable -- Prices are always positive (exponential function) +Steps 1–4 are self-contained in `app/market/`. Step 5 is the seam with the rest of the backend and +should land alongside the portfolio and watchlist routes, not before them. --- -## 14. Configuration Summary - -All tunable parameters and their defaults: - -| Parameter | Location | Default | Description | -|-----------|----------|---------|-------------| -| `MASSIVE_API_KEY` | Environment variable | `""` (empty) | If set, use Massive API; otherwise use simulator | -| `update_interval` | `SimulatorDataSource.__init__` | `0.5` (seconds) | Time between simulator ticks | -| `poll_interval` | `MassiveDataSource.__init__` | `15.0` (seconds) | Time between Massive API polls | -| `event_probability` | `GBMSimulator.__init__` | `0.001` | Chance of a random shock event per ticker per tick | -| `dt` | `GBMSimulator.__init__` | `~8.5e-8` | GBM time step (fraction of a trading year) | -| SSE push interval | `_generate_events()` | `0.5` (seconds) | Time between SSE pushes to the client | -| SSE retry directive | `_generate_events()` | `1000` (ms) | Browser EventSource reconnection delay | +## 16. Configuration reference + +| Setting | Default | Where | Effect | +|---|---|---|---| +| `MASSIVE_API_KEY` | unset | env | Non-empty selects Massive; otherwise simulator | +| `update_interval` | `0.5` | `SimulatorDataSource` | Simulator tick rate — **change `dt` with it** | +| `event_probability` | `0.001` | `SimulatorDataSource` | Shock chance per ticker per tick | +| `poll_interval` | `15.0` | `MassiveDataSource` | Seconds between snapshot requests | +| `HISTORY_MAXLEN` | `600` | `cache.py` | Rolling history depth (~5 min at 500ms) | +| `KEEPALIVE_SECONDS` | `15.0` | `stream.py` | Idle gap before a `: ping` | +| SSE poll `interval` | `0.5` | `stream.py` | How often the version is checked | + +### Tuning the simulator + +| Want | Change | Watch for | +|---|---|---| +| More visible motion | Raise σ in `TICKER_PARAMS` | Above ~0.8 it stops looking like equity | +| Faster updates | `update_interval` | **`DEFAULT_DT` hard-codes the 500ms tick** — see below | +| More drama | Raise `event_probability` | Above ~0.005 the series becomes jumps, not prices | +| Bigger shocks | Widen `random.uniform(0.02, 0.05)` | Beyond ~10% the P&L chart loses all detail | +| Different sectors | Edit `CORRELATION_GROUPS` and coefficients | Re-run the positive-definiteness test (§14.2) | +| A trending market | Raise μ | μ is annualized; even 0.5 is barely visible over a demo | + +**The `DEFAULT_DT` coupling is the one that catches people.** `DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR` +hard-codes the 500ms tick. Passing `update_interval=0.1` without also passing a matching `dt` runs +the simulation five times faster in model time, and annualized volatility silently becomes 5× what +`TICKER_PARAMS` claims. -### Package `__init__.py` - -**File: `backend/app/market/__init__.py`** +--- -```python -"""Market data subsystem for FinAlly. - -Public API: - PriceUpdate - Immutable price snapshot dataclass - PriceCache - Thread-safe in-memory price store - MarketDataSource - Abstract interface for data providers - create_market_data_source - Factory that selects simulator or Massive - create_stream_router - FastAPI router factory for SSE endpoint -""" - -from .cache import PriceCache -from .factory import create_market_data_source -from .interface import MarketDataSource -from .models import PriceUpdate -from .stream import create_stream_router - -__all__ = [ - "PriceUpdate", - "PriceCache", - "MarketDataSource", - "create_market_data_source", - "create_stream_router", -] -``` +## 17. Summary + +| Concern | Resolution | +|---|---| +| Two sources, one consumer | `MarketDataSource` ABC + shared `PriceCache` | +| Which source | `create_market_data_source`, decided once at startup from `MASSIVE_API_KEY` | +| Default | Simulator — always alive, no key, no rate limit, any ticker | +| How prices are read | Only from the cache, never from the source | +| Which tickers are live | `watchlist ∪ positions`, reconciled at startup | +| Price model | GBM, per-ticker μ and σ, Cholesky-correlated by sector | +| Timestamp format | Unix epoch seconds (float), converted at each source boundary | +| Update delivery | SSE, full cache per event, on `version` change, `: ping` when idle | +| Chart backfill | 600-point in-memory deque per ticker, never persisted | +| Blocking I/O | `asyncio.to_thread` at the source, always | +| Failure handling | Per-cycle `try` inside the loop; the feed never dies from one bad tick | +| Outstanding work | The five steps in §15 | diff --git a/planning/archive/MARKET_DATA_REVIEW.md b/planning/archive/MARKET_DATA_REVIEW.md index 61b4d6b..3ecb1f1 100644 --- a/planning/archive/MARKET_DATA_REVIEW.md +++ b/planning/archive/MARKET_DATA_REVIEW.md @@ -1,173 +1,242 @@ # Market Data Backend — Code Review -**Date:** 2026-02-10 -**Scope:** `backend/app/market/` (8 source files) and `backend/tests/market/` (6 test files) +**Date:** 2026-09-02 +**Scope:** `backend/app/market/` (8 source files, 730 LOC) and `backend/tests/market/` (7 test files, 1,161 LOC) +**Reviewer:** Claude, in response to issue #5 --- -## 1. Test Results Summary +## 1. Test Execution — Could Not Run -**73 tests collected, 68 passed, 5 failed.** +This review's environment does not have permission to execute shell commands that +run the Python interpreter or install dependencies (`uv sync`, `uv run pytest`, even +`python3 -c ...` are all blocked pending approval, and this run has no human +available to approve them). This is the same limitation `planning/MARKET_DATA_SUMMARY.md` +recorded on the previous pass. **No test was executed as part of this review.** -All failures are in `test_massive.py` and stem from the same root cause: the `massive` package is not installed in the test environment, so `patch("app.market.massive_client.RESTClient")` fails with `AttributeError` because the module-level name `RESTClient` was never imported (it is lazy-imported inside methods). This is an environment issue, not a logic bug — the tests are correctly structured but require the `massive` package to be available (or `create=True` on the patch) so that the mock target exists. +To get a real pass/fail signal, re-run this task with `Bash(uv sync:*)` and +`Bash(uv run:*)` added to the allowed tools, or run locally: -Failing tests: -- `test_poll_updates_cache` — `asyncio.to_thread` fails because `_fetch_snapshots` is not properly mocked when `massive` is absent -- `test_malformed_snapshot_skipped` — same cause -- `test_timestamp_conversion` — same cause -- `test_stop_cancels_task` — `patch("app.market.massive_client.RESTClient")` fails because the name doesn't exist at module level -- `test_start_immediate_poll` — same as above - -The underlying `_poll_once()` logic itself is correct. The 3 tests that mock `source._fetch_snapshots` directly fail because `asyncio.to_thread(self._fetch_snapshots)` calls the real method which tries to import `massive`. The 2 tests that use `patch("app.market.massive_client.RESTClient")` fail because the name doesn't exist in the module's namespace (lazy import). Both issues resolve when the `massive` package is installed. +```bash +cd backend +uv sync --extra dev +uv run --extra dev pytest -v --cov=app --cov-report=term-missing +uv run --extra dev ruff check app/ tests/ +``` -**Lint (ruff):** Source code passes clean. Tests have 5 unused-import warnings (`pytest`, `math`, `asyncio` imported but not used in some test files). +In place of execution, every test file was read in full and traced by hand against +the source it exercises (see §4). All 96 tests found in the suite exercise real +code paths correctly as far as static reading can confirm — no test asserts on +behavior the source doesn't actually implement, and no test's mocking hides a +divergence between the mock's shape and the real one (the concern that let a +prior bug through at 94% coverage, per `test_massive.py`'s own docstring). -**Coverage:** 84% overall. -| Module | Coverage | Notes | -|---|---|---| -| models.py | 100% | | -| cache.py | 100% | | -| interface.py | 100% | | -| seed_prices.py | 100% | | -| factory.py | 100% | | -| simulator.py | 98% | Uncovered: `_add_ticker_internal` duplicate guard (L145), exception log in `_run_loop` (L264-265) | -| massive_client.py | 56% | Expected — real API methods can't run without the massive package | -| stream.py | 31% | Expected — SSE generator requires a running ASGI server to test | +**Test count:** 96 across 7 files (`test_models.py` 11, `test_cache.py` 24, +`test_simulator.py` 17, `test_simulator_source.py` 10, `test_factory.py` 7, +`test_massive.py` 17, `test_stream.py` 15 by count of `def test_`/`async def test_` +— slightly higher than the 94 recorded in `MARKET_DATA_SUMMARY.md` §"Test Suite", +consistent with incremental additions since that doc was last updated). --- ## 2. Architecture Assessment -The market data subsystem is well-designed. It follows a clean strategy pattern: +The module is well-factored and matches `planning/MARKET_DATA_DESIGN.md` and +`planning/MARKET_DATA_SUMMARY.md` closely: ``` MarketDataSource (ABC) -├── SimulatorDataSource (GBM simulator) -└── MassiveDataSource (Polygon.io REST poller) +├── SimulatorDataSource → GBMSimulator (Cholesky-correlated GBM) +└── MassiveDataSource → Polygon.io REST poller │ ▼ - PriceCache (shared, thread-safe) + PriceCache (thread-safe, latest price + 600-point rolling history) │ - ▼ - SSE stream → Frontend + ├──→ create_stream_router() → GET /api/stream/prices (SSE, with keepalive) + └──→ create_history_router() → GET /api/prices/{ticker}/history ``` -**Strengths:** -- Clear separation of concerns across 8 focused modules -- Factory pattern with lazy imports — the `massive` package is only needed when `MASSIVE_API_KEY` is set -- PriceCache as the single point of truth decouples producers from consumers -- Immutable `PriceUpdate` dataclass with `frozen=True, slots=True` is correct and efficient -- The GBM math is proper: log-normal price paths via `exp((mu - 0.5*sigma^2)*dt + sigma*sqrt(dt)*Z)` -- Correlated moves via Cholesky decomposition are a nice touch for realism -- All background tasks are properly cancellable and idempotent on stop() +**Strengths confirmed by this pass:** + +- Strategy pattern cleanly isolates the two data sources behind `MarketDataSource`; nothing downstream needs to know which is active. +- `PriceUpdate` is `frozen=True, slots=True` — correct choice for a value object shared across threads/tasks. +- `PriceCache` centralizes all locking (`threading.Lock`) around the one mutable structure producers and consumers touch; the API surface (`update`, `get`, `get_all`, `get_price`, `remove`, `get_history`) is small and each method acquires the lock exactly once. +- The GBM math is textbook-correct log-normal price evolution, and the `dt` sizing (`0.5s / (252 * 6.5h * 3600s)`) is derived, not guessed, with the derivation left in a comment. +- Cholesky-based correlated draws (`simulator.py:84-90`) are a genuinely nice touch for a simulator whose only job is to look convincing on a chart. +- The three TODOs recorded as open in `PLAN.md` §13 (SSE keepalive, rolling history, `/history` endpoint) are all implemented and each has direct test coverage (`test_stream.py`). +- The two defects `MARKET_DATA_DESIGN.md` §8.4 recorded against the Massive client (wrong attribute name, nanoseconds-as-milliseconds) are fixed in `massive_client.py:130-136`, and `test_massive.py` deliberately builds real `TickerSnapshot` objects via `TickerSnapshot.from_dict(...)` rather than `MagicMock`, which is exactly the right defense against that class of bug recurring silently. +- `pyproject.toml` already has `[tool.hatch.build.targets.wheel] packages = ["app"]` — the "High" build-breaking bug from the archived 2026-02-10 review (`planning/archive/MARKET_DATA_REVIEW.md` §3.1) is fixed. +- `massive` is a top-level import now (`massive_client.py:9-11`), not a lazy one — the archived review's §3.2 concern about tests being fragile without the package installed no longer applies; `pyproject.toml` lists it as a core dependency. --- ## 3. Issues Found -### 3.1 Build Configuration Bug (Severity: High) - -`pyproject.toml` is missing the hatchling package discovery configuration. Running `uv sync` fails: +### 3.1 `create_stream_router` / `create_history_router` mutate a shared module-level router (Severity: Medium) -``` -ValueError: Unable to determine which files to ship inside the wheel -``` +`stream.py:18-19` defines `router` and `history_router` at module scope. Both +factory functions register their route via a closure on these **same shared +objects** rather than creating a fresh `APIRouter()` per call: -**Fix:** Add to `pyproject.toml`: -```toml -[tool.hatch.build.targets.wheel] -packages = ["app"] +```python +router = APIRouter(prefix="/api/stream", tags=["streaming"]) +history_router = APIRouter(prefix="/api/prices", tags=["prices"]) + +def create_stream_router(price_cache: PriceCache) -> APIRouter: + @router.get("/prices") + async def stream_prices(request: Request) -> StreamingResponse: + ... + return router ``` -This will block Docker builds and any fresh `uv sync` until fixed. - -### 3.2 Massive Test Fragility (Severity: Medium) - -Five tests in `test_massive.py` fail when the `massive` package is not installed. The root cause is twofold: - -1. **`_poll_once` uses `asyncio.to_thread(self._fetch_snapshots)`** — even when `_fetch_snapshots` is patched on the instance, `to_thread` runs it in a thread executor. Three tests mock `_fetch_snapshots` as a `MagicMock` (synchronous), but `asyncio.to_thread` wraps it in `loop.run_in_executor`, which works... except that when `_fetch_snapshots` is NOT patched, the real method tries `from massive.rest.models import SnapshotMarketType` and fails. - -2. **`patch("app.market.massive_client.RESTClient")`** targets a name that doesn't exist at module level because `massive_client.py` uses a lazy import inside `start()`. The patch needs `create=True` or the import needs to be at module level behind a `TYPE_CHECKING` guard. - -These tests pass when `massive>=1.0.0` is installed (as `pyproject.toml` declares it as a core dependency), so this is technically a test-environment issue, not a code bug. However, since the whole point of lazy imports is to make `massive` optional for simulator-only use, the tests should also work without it. - -### 3.3 `_generate_events` Return Type Annotation (Severity: Low) - -`stream.py:54` declares the return type as `-> None` but the function is an async generator (it uses `yield`). The correct annotation would be `-> AsyncGenerator[str, None]` or simply removing the annotation. This doesn't cause runtime issues but is misleading for type checkers and developers. - -### 3.4 `version` Property Not Under Lock (Severity: Low) - -`PriceCache.version` reads `self._version` without acquiring `self._lock`: +Calling either factory more than once appends another route to the same +underlying router rather than returning an independent one. This was flagged +as a "latent footgun for testing" in the archived review (§3.6) when there +were no tests exercising it; now there are, and it is no longer latent: +`test_stream.py`'s `_history_endpoint()` helper calls `create_history_router(cache)` +fresh in **six different tests**, so `history_router` in the running test +process accumulates six duplicate `/{ticker}/history` routes by the end of +the file. The tests still pass because they grab `router.routes[-1].endpoint` +— the most recently registered one — but this only works by coincidence of +ordering, not because the router is actually being rebuilt. + +The real risk is downstream: once this module is wired into the FastAPI app +(the next piece of work per `PLAN.md` §13 "Still open"), any test that builds +the app more than once per process — a very common pytest pattern (an `app` +fixture instantiated per test, or per module) — will silently accumulate +duplicate routes on every rebuild, since `router`/`history_router` are shared +mutable module state that outlives any single app instance. + +**Fix:** construct a new `APIRouter()` inside each factory function instead of +reusing a module-level instance: ```python -@property -def version(self) -> int: - return self._version -``` +def create_stream_router(price_cache: PriceCache) -> APIRouter: + router = APIRouter(prefix="/api/stream", tags=["streaming"]) -On CPython with the GIL, reading a single `int` is atomic, so this won't cause corruption. However, it's inconsistent with the rest of the class, and if the project ever runs on a no-GIL Python build (PEP 703, Python 3.13t+), this could become a race. A minor concern given the current context. + @router.get("/prices") + async def stream_prices(request: Request) -> StreamingResponse: + ... + return router +``` -### 3.5 `SimulatorDataSource.get_tickers` Accesses Private State (Severity: Low) +### 3.2 `PriceCache.update()` treats a falsy timestamp as "no timestamp given" (Severity: Low) -`simulator.py:254`: ```python -def get_tickers(self) -> list[str]: - return list(self._sim._tickers) if self._sim else [] +ts = timestamp or time.time() ``` -This reaches into `GBMSimulator._tickers` (private attribute). `GBMSimulator` should expose a `get_tickers()` method or a `tickers` property to keep the boundary clean. - -### 3.6 Module-Level Router Instance (Severity: Low) - -`stream.py:16` creates a module-level `router` object, and `create_stream_router()` registers a route on it via closure. If `create_stream_router` were called twice (e.g., in tests), the `/prices` route would be registered twice on the same router. In practice this won't happen because the function is called once during app startup, but it's a latent footgun for testing. - -### 3.7 Unused Imports in Tests (Severity: Trivial) - -Five lint warnings from `ruff`: -- `test_cache.py`: unused `pytest` -- `test_factory.py`: unused `pytest` -- `test_massive.py`: unused `asyncio` -- `test_simulator.py`: unused `math`, unused `pytest` +(`cache.py:40`) A caller that explicitly passes `timestamp=0.0` (Unix epoch, +1970-01-01) gets `time.time()` substituted instead, because `0.0` is falsy. +No current caller does this — `massive_client.py` only reaches this path with +`time.time()` already substituted upstream when `sip_timestamp` is falsy — so +this is not exploitable today, but it is a latent correctness gap for any +future caller (e.g., a test replaying historical data from epoch-adjacent +timestamps, or a backfill script). Prefer `timestamp if timestamp is not None +else time.time()`. + +### 3.3 `MassiveDataSource`'s poller task dies silently on `AuthError` (Severity: Low) + +`_poll_once()` deliberately re-raises `AuthError` (`massive_client.py:103-105`) +with the comment "unrecoverable: do not retry on a loop" — a reasonable +choice. But the only place that awaits `self._task` is `stop()` +(`massive_client.py:60-69`), which nothing calls until shutdown. If the key +is revoked *after* `start()` succeeds (rather than being bad from the first +poll), the background task raised inside `_poll_loop()` simply stops running; +asyncio logs "Task exception was never retrieved" at some later point (often +at garbage collection, easy to miss in container logs), and the app has no +other signal that live prices have silently frozen. `test_auth_error_propagates` +confirms the exception propagates out of `_poll_once()`, but there is no test +for what happens to `_poll_loop()` or the app once that happens. + +This is fine as coded for now since nothing outside the market module reads +task health yet, but whoever wires this into the app (`PLAN.md` §13, item 3) +should either attach a `Task.add_done_callback` that logs loudly / flips a +health flag, or have `GET /api/health` report `market_source` as degraded +when the task is dead. Worth a one-line note in `MARKET_DATA_SUMMARY.md` so +it isn't forgotten during integration. + +### 3.4 `PriceCache.version` property reads outside the lock (Severity: Trivial) + +Unchanged from the archived review's §3.4: `cache.py:94-97` reads `self._version` +without acquiring `self._lock`. Safe under CPython's GIL for a single `int` +read, inconsistent with the rest of the class, and only a real concern on a +no-GIL build. Not worth blocking on, but a two-line fix if anyone is passing +through this file for another reason. + +### 3.5 `market_data_demo.py` and `backend/README.md` are outside the reviewed test scope but were not separately verified + +The demo script (`market_data_demo.py`, 205 lines) is referenced by +`MARKET_DATA_SUMMARY.md` as a manual verification tool and has no automated +test coverage, which is appropriate for a Rich terminal demo — flagging only +so it's clear this review's "all tests pass" scope is `backend/tests/market/`, +not the demo script. --- -## 4. Design Observations +## 4. Test Suite Assessment (by module) -### 4.1 Things Done Well - -- **GBM parameter tuning is thoughtful.** TSLA at sigma=0.50 vs V at 0.17 reflects real-world volatility differences. The shock event system (~0.1% per tick, producing visible moves every ~50s) adds visual drama without destabilizing prices. -- **Cholesky decomposition for correlated moves** is the mathematically correct approach. The sector-based correlation structure (tech 0.6, finance 0.5, cross 0.3) is reasonable. -- **Defensive error handling in both data sources.** Both `_run_loop` (simulator) and `_poll_once`/`_poll_loop` (massive) catch exceptions and continue, which is essential for a long-running background service. -- **SSE implementation is clean.** The version-based change detection avoids sending redundant payloads. The `retry: 1000\n\n` directive ensures browser auto-reconnect. Nginx buffering is proactively disabled. -- **Seed prices in the cache at start** means the frontend gets data on the first SSE poll, with no visible delay. -- **Thread-safe cache with Lock** is the right choice since the Massive client runs API calls via `asyncio.to_thread`. - -### 4.2 Missing Tests - -- **SSE streaming (`stream.py`)** at 31% coverage has no dedicated tests. Testing SSE requires an ASGI test client (e.g., `httpx.AsyncClient` with `app`). Given that this is the primary consumer of PriceCache, even a basic integration test would add confidence. -- **No concurrent/thread-safety test for PriceCache.** The lock usage looks correct from inspection, but a test with multiple threads writing simultaneously would verify it empirically. -- **No test for `GBMSimulator` with all 10 default tickers.** Tests use 1-2 tickers. A test confirming the Cholesky decomposition succeeds for the full 10-ticker default set would catch correlation matrix issues. - -### 4.3 Potential Future Considerations - -- The `PriceCache` doesn't cap history; it only stores the latest price per ticker, so memory is bounded at O(tickers). Good. -- The `DEFAULT_CORR` constant (0.3, `seed_prices.py:48`) is defined but never referenced in `_pairwise_correlation`. The static method returns `CROSS_GROUP_CORR` (also 0.3) as the fallback. This is semantically confusing — `DEFAULT_CORR` seems intended for tickers not in any group, but the code returns `CROSS_GROUP_CORR` for all non-matched pairs. Both happen to be 0.3, so behavior is correct, but the naming is misleading. +| Module | File | Assessment | +|---|---|---| +| `models.py` | `test_models.py` (11 tests) | Complete: creation, `change`/`change_percent`/`direction` in both directions, zero-previous-price edge case, `to_dict()` shape, and frozen-dataclass immutability. No gaps. | +| `cache.py` | `test_cache.py` (24 tests) | Thorough. Covers direction transitions, `version` monotonicity, `__len__`/`__contains__`, price rounding, custom timestamps, and a dedicated `TestPriceHistory` class covering bounding, ordering, per-ticker isolation, limit-narrower-than-stored, and that `remove()` clears history without touching other tickers. No test for concurrent multi-thread writes (the lock is exercised only single-threaded) — the archived review flagged this as missing in §4.2 and it remains missing; low priority since the logic is simple enough to verify by inspection. | +| `interface.py` | (no dedicated file; exercised transitively via simulator/massive tests) | Reasonable — it's an ABC with no logic of its own. | +| `seed_prices.py` | `test_simulator.py`, `test_factory.py` (transitively) | No dedicated test file, but every constant is exercised indirectly through `GBMSimulator` tests (`_pairwise_correlation` tests cover tech/finance/TSLA/cross-sector explicitly). Fine given it's pure data. | +| `simulator.py` | `test_simulator.py` (17), `test_simulator_source.py` (10) | Strong. `GBMSimulator`: positivity over 10,000 steps, seed matching, add/remove (including duplicate/nonexistent no-ops), unknown-ticker random seeding, Cholesky construction/teardown on ticker count crossing 1↔2, all four correlation branches, `dt` sanity, and rounding. `SimulatorDataSource`: cache population on start, periodic updates via real `asyncio.sleep`, idempotent stop, dynamic add/remove, empty-start, and exception resilience. The timing-based assertions (`asyncio.sleep(0.3)` then assert version advanced) are inherently a little flaky under CI load, but the margins used (3-6x the interval) are generous enough to be low-risk. | +| `massive_client.py` | `test_massive.py` (17) | Strong, and specifically hardened against the exact bug class that shipped previously — `_apply_snapshots` is tested against real `TickerSnapshot.from_dict(...)` objects, not mocks, for timestamp conversion, missing-trade skipping, mixed valid/invalid batches, and multi-ticker updates. Polling lifecycle covers success, `BadResponse` (swallowed), `AuthError` (re-raised, see §3.3), generic exceptions (swallowed), ticker add/remove with normalization, and start/stop idempotency. No gap of consequence. | +| `stream.py` | `test_stream.py` (15) | Was 31% covered and untested in the archived review; now has direct coverage of the async generator via a hand-rolled `FakeRequest`, including the retry directive, snapshot-on-connect (and thus reconnect), the frozen payload field set, keepalive timing (via `monkeypatch` on `KEEPALIVE_SECONDS` rather than a real 15s wait — good practice), a fresh data event following a ping, disconnect handling, and the empty-cache case. `create_history_router`'s endpoint is tested for ordering, unknown-ticker empty response, normalization, and limit clamping in both directions. The one real gap is architectural, not a missing test: see §3.1 — the tests would catch a *regression* in behavior but not the router-reuse issue itself, since grabbing `routes[-1]` happens to paper over it. | +| `factory.py` | `test_factory.py` (7) | Complete for its size: unset/empty/whitespace-only key → simulator, set key → Massive, and that both branches thread the cache reference through correctly. | + +**Net assessment:** the suite is comprehensive and, importantly, methodologically +careful — the deliberate choice to build real `TickerSnapshot` objects instead of +`MagicMock` in `test_massive.py` is the single best thing about this test suite, +since it's precisely what would have caught the `last_trade.timestamp` / +`sip_timestamp` bug the archived review found. No test was found asserting +something the source doesn't do, and no source behavior of consequence lacks a +test, with the caveats above (concurrency, and the router-reuse issue masked +by test ordering). --- -## 5. Verdict +## 5. Comparison Against the Prior Review -The market data backend is solid and well-structured. The GBM simulator, price cache, abstract interface, factory pattern, and SSE streaming all work correctly and follow good practices. The architecture will integrate cleanly with the rest of the application. +`planning/archive/MARKET_DATA_REVIEW.md` (2026-02-10) recorded 7 issues. Status now: -**Must fix before proceeding:** -1. Add `[tool.hatch.build.targets.wheel] packages = ["app"]` to `pyproject.toml` — without this, `uv sync` and Docker builds fail. +| # | Issue | Status | +|---|---|---| +| 3.1 | Missing hatchling wheel config | **Fixed** | +| 3.2 | Massive tests fragile without the `massive` package | **Fixed** (now a core dependency, imported at module level) | +| 3.3 | `_generate_events` return type `-> None` instead of `AsyncGenerator` | **Fixed** (`stream.py:87`) | +| 3.4 | `PriceCache.version` reads outside the lock | **Still open** (§3.4 above, trivial) | +| 3.5 | `SimulatorDataSource.get_tickers` reached into `GBMSimulator._tickers` | **Fixed** — `GBMSimulator.get_tickers()` now exists (`simulator.py:140-142`) and is used | +| 3.6 | Module-level router registered on repeated calls | **Still open, and now demonstrated by the test suite itself** (§3.1 above, upgraded to Medium given it will bite during app integration) | +| 3.7 | Unused imports in tests | **Fixed** — no unused `pytest`/`math`/`asyncio` imports found in any current test file | + +Also confirmed fixed: the two Massive parsing defects `MARKET_DATA_DESIGN.md` +§8.4 described (wrong attribute name, nanosecond/millisecond confusion), and +all three items `PLAN.md` §13 listed as open TODOs (rolling history, `/history` +endpoint, SSE keepalive). -**Should fix:** -2. Make the Massive tests resilient to the `massive` package being absent (use `create=True` on patches, or restructure mocks). -3. Fix the `_generate_events` return type annotation. -4. Remove unused imports in test files. +--- -**Nice to have:** -5. Add a `get_tickers()` public method to `GBMSimulator`. -6. Add at least one SSE integration test. -7. Clarify `DEFAULT_CORR` vs `CROSS_GROUP_CORR` naming. +## 6. Verdict + +The market data backend is in good shape and ready to be built on. Of the two +open items: + +- **§3.1 (shared module-level router)** should be fixed before the FastAPI + `lifespan` wiring work begins (`PLAN.md` §13, item 3) — it's a small, + mechanical fix (stop reusing module-level `router`/`history_router`; build + one per call) and doing it now avoids a confusing bug later when the app + factory is instantiated more than once, which is standard practice for + backend test fixtures. +- **§3.2/§3.4 (falsy-timestamp substitution, unlocked version read)** are + low-risk and can be picked up opportunistically. +- **§3.3 (silent poller death on revoked key)** is a design note for whoever + adds the `GET /api/health` endpoint — surface poller liveness there. + +None of these block downstream work. **Tests were not executed in this pass** +due to environment permissions (§1) — that is the one action item this review +could not complete, and it should be re-run with `uv`/`python3` execution +permitted to get an authoritative pass/fail/coverage number rather than the +static analysis this document is based on. diff --git a/planning/archive/MARKET_INTERFACE.md b/planning/archive/MARKET_INTERFACE.md index 156cad2..7df6fa9 100644 --- a/planning/archive/MARKET_INTERFACE.md +++ b/planning/archive/MARKET_INTERFACE.md @@ -1,273 +1,439 @@ -# Market Data Interface Design +# MARKET_INTERFACE.md — The Unified Market Data API -Unified Python interface for market data in FinAlly. Two implementations (simulator and Massive API) behind one abstract interface. All downstream code — SSE streaming, price cache, portfolio valuation — is source-agnostic. +How FinAlly retrieves stock prices from either the Massive API or the built-in simulator through one interface, selected by whether `MASSIVE_API_KEY` is set. -## Core Data Model +Companion documents: `MASSIVE_API.md` (the real data provider) and `MARKET_SIMULATOR.md` (the fallback). This document is the contract between them and the rest of the backend. -```python -from dataclasses import dataclass +**Status:** the core of this design is implemented in `backend/app/market/`. Sections marked **TODO** are specified but not yet built. + +--- + +## 1. The problem this solves + +Two data sources with nothing in common: + +- **Massive** — a synchronous HTTP client, polled every 15 seconds, returning whatever the exchanges last reported, with gaps for unknown tickers and frozen values overnight. +- **The simulator** — a pure in-process computation, stepping every 500ms, always alive, and able to invent a plausible price for any symbol. + +Everything downstream — SSE streaming, portfolio valuation, trade execution, the P&L snapshot task — must not care which one is running. A trade fills at "the current price of AAPL" whether that price came from NASDAQ or from a random number generator. + +The design achieves that with **one indirection and one shared buffer**: + +``` + writes reads + ┌──────────────────┐ ┌────────────┐ ┌─────────────────────┐ + │ SimulatorSource │───┐ │ │──────────────│ SSE /api/stream │ + │ (500ms step) │ ├───▶│ PriceCache │──────────────│ Portfolio valuation │ + ├──────────────────┤ │ │ (in-mem, │──────────────│ Trade execution │ + │ MassiveSource │───┘ │thread-safe)│──────────────│ Snapshot task │ + │ (15s poll) │ │ │──────────────│ /api/prices/history │ + └──────────────────┘ └────────────┘ └─────────────────────┘ + MarketDataSource + (abstract interface) +``` + +The critical property: **nothing downstream ever calls the data source to get a price.** Sources are write-only from the application's point of view; readers only ever touch the cache. That is what makes the two implementations substitutable despite a 30× difference in update cadence. + +### Module map — `backend/app/market/` + +| File | Contents | +|---|---| +| `models.py` | `PriceUpdate` — the single price record | +| `cache.py` | `PriceCache` — the shared buffer | +| `interface.py` | `MarketDataSource` — the abstract contract | +| `simulator.py` | `GBMSimulator`, `SimulatorDataSource` | +| `massive_client.py` | `MassiveDataSource` | +| `factory.py` | `create_market_data_source` — the selection rule | +| `seed_prices.py` | Simulator constants | +| `stream.py` | The SSE endpoint | + +--- + +## 2. `PriceUpdate` — the unit of data + +An immutable, frozen dataclass. Both sources produce it; every reader consumes it. -@dataclass +```python +@dataclass(frozen=True, slots=True) class PriceUpdate: - """A single price update for one ticker.""" ticker: str price: float previous_price: float - timestamp: float # Unix seconds - change: float # price - previous_price - direction: str # "up", "down", or "flat" + timestamp: float = field(default_factory=time.time) # Unix epoch SECONDS +``` + +`change`, `change_percent`, and `direction` are computed properties, not stored fields — they cannot drift out of sync with the prices they describe. + +```python +@property +def change(self) -> float: + return round(self.price - self.previous_price, 4) + +@property +def change_percent(self) -> float: + if self.previous_price == 0: + return 0.0 + return round((self.price - self.previous_price) / self.previous_price * 100, 4) + +@property +def direction(self) -> str: + if self.price > self.previous_price: + return "up" + elif self.price < self.previous_price: + return "down" + return "flat" ``` -This is the only data structure that leaves the market data layer. Everything downstream works with `PriceUpdate` objects. +### Two frozen conventions + +`to_dict()` is the SSE wire format and is **frozen** — the shipped frontend contract depends on it (`PLAN.md` §6): + +- **`timestamp` is Unix epoch seconds as a float**, never ISO. The frontend multiplies by 1000 for `Date`. Massive's nanosecond and millisecond timestamps are converted at the boundary — see `MASSIVE_API.md` §7. +- **`change_percent` is already in percent units.** `0.021` means 0.021%, not 2.1%. Note this deliberately differs from REST responses elsewhere in the API, where percentages are fractions (`PLAN.md` §8). The inconsistency is real; it is preserved because the market module shipped first and the frontend was written against it. + +`previous_price` means *the price at the previous update*, not the previous session's close. On the first update for a ticker it equals `price`, so `direction` is `"flat"` and `change` is `0.0` — a new ticker never flashes green or red on its first tick. -## Abstract Interface +--- + +## 3. `PriceCache` — the shared buffer + +An in-memory `dict` behind a `threading.Lock`, plus a monotonic version counter. ```python -from abc import ABC, abstractmethod +class PriceCache: + def update(self, ticker: str, price: float, timestamp: float | None = None) -> PriceUpdate + def get(self, ticker: str) -> PriceUpdate | None + def get_all(self) -> dict[str, PriceUpdate] # shallow copy + def get_price(self, ticker: str) -> float | None + def remove(self, ticker: str) -> None + @property + def version(self) -> int + def __len__(self) -> int + def __contains__(self, ticker: str) -> bool +``` -class MarketDataSource(ABC): - """Abstract interface for market data providers.""" +Three design points that matter: - @abstractmethod - async def start(self, tickers: list[str]) -> None: - """Begin producing price updates for the given tickers.""" +**`update()` derives `previous_price` itself.** Callers pass only the new price; the cache looks up what it had and constructs the `PriceUpdate`. Neither source needs to track prior state for the purpose of computing a delta, and the two cannot implement it differently. - @abstractmethod - async def stop(self) -> None: - """Stop producing price updates and clean up.""" +**A `threading.Lock`, not an `asyncio.Lock`.** `MassiveDataSource` writes from an `asyncio.to_thread` worker, so a genuine cross-thread lock is required. The critical sections are a few dict operations, so contention is irrelevant. - @abstractmethod - async def add_ticker(self, ticker: str) -> None: - """Add a ticker to the active set.""" +**`version` increments on every write and is the SSE change-detection mechanism.** The stream compares it every 500ms rather than diffing prices. Because a fresh generator starts at `last_version = -1`, the first comparison always differs, so **every connecting client immediately receives a full snapshot** — including after a reconnect. This is why no separate snapshot endpoint exists. - @abstractmethod - async def remove_ticker(self, ticker: str) -> None: - """Remove a ticker from the active set.""" +`get_all()` returns a shallow copy; since `PriceUpdate` is frozen, the copy is effectively deep and safe to iterate outside the lock. - @abstractmethod - def get_tickers(self) -> list[str]: - """Return the current list of active tickers.""" +### Rolling price history — **TODO** + +`PLAN.md` §6 requires the main chart to be populated the instant a ticker is clicked. `PriceCache` gains a bounded per-ticker deque of `(timestamp, price)`: + +```python +from collections import deque + +HISTORY_MAXLEN = 600 # ~5 minutes at the 500ms simulator cadence + +self._history: dict[str, deque[tuple[float, float]]] = {} ``` -Both implementations write to a shared `PriceCache` (see below). The interface does **not** return prices directly — it pushes updates into the cache on its own schedule. +- Appended inside `update()`, under the same lock. +- `deque(maxlen=600)` evicts the oldest point automatically — no pruning logic. +- `remove()` must drop the ticker's deque too, or removed tickers leak. +- Deliberately **not persisted**. A restart clears it, which is the honest behaviour for a simulator with no real history. -## Price Cache +Memory is negligible: 600 points × 50 tickers × ~16 bytes ≈ 500KB. -Shared in-memory store that both data sources write to and the SSE streamer reads from. +New reader, backing `GET /api/prices/{ticker}/history?limit=600`: ```python -import time -from threading import Lock +def get_history(self, ticker: str, limit: int = 600) -> list[tuple[float, float]]: + """Oldest-first (timestamp, price) points. Empty list for an untracked ticker.""" + with self._lock: + points = self._history.get(ticker) + if not points: + return [] + return list(points)[-limit:] +``` -class PriceCache: - """Thread-safe cache of latest prices per ticker.""" - - def __init__(self): - self._prices: dict[str, PriceUpdate] = {} - self._lock = Lock() - - def update(self, ticker: str, price: float, timestamp: float | None = None) -> PriceUpdate: - """Update price for a ticker. Returns the PriceUpdate.""" - with self._lock: - ts = timestamp or time.time() - previous = self._prices.get(ticker) - previous_price = previous.price if previous else price - - if price > previous_price: - direction = "up" - elif price < previous_price: - direction = "down" - else: - direction = "flat" - - update = PriceUpdate( - ticker=ticker, - price=price, - previous_price=previous_price, - timestamp=ts, - change=price - previous_price, - direction=direction, - ) - self._prices[ticker] = update - return update - - def get(self, ticker: str) -> PriceUpdate | None: - """Get latest price for a ticker.""" - with self._lock: - return self._prices.get(ticker) - - def get_all(self) -> dict[str, PriceUpdate]: - """Get all current prices.""" - with self._lock: - return dict(self._prices) - - def remove(self, ticker: str) -> None: - """Remove a ticker from the cache.""" - with self._lock: - self._prices.pop(ticker, None) +An untracked ticker returns `[]`, not a 404 — the chart draws nothing rather than erroring (`PLAN.md` §8). + +Under Massive the deque fills at one point per 15-second poll, so five minutes of wall time is 20 points rather than 600. The chart is sparse but correct. Backfilling from `get_aggs` (`MASSIVE_API.md` §5) is the eventual upgrade. + +--- + +## 4. `MarketDataSource` — the abstract contract + +```python +class MarketDataSource(ABC): + @abstractmethod + async def start(self, tickers: list[str]) -> None: ... + @abstractmethod + async def stop(self) -> None: ... + @abstractmethod + async def add_ticker(self, ticker: str) -> None: ... + @abstractmethod + async def remove_ticker(self, ticker: str) -> None: ... + @abstractmethod + def get_tickers(self) -> list[str]: ... ``` -## Factory Function +Five methods, and every one is about *lifecycle and membership* — none of them returns a price. That absence is the whole design. A `get_price()` on this interface would tempt callers into a per-request API hit under Massive and would make the two implementations behave differently under load. + +### Behavioural contract + +Binding on both implementations. A test suite that passes against one should pass against the other. + +| Method | Guarantee | +|---|---| +| `start(tickers)` | Begins a background task writing to the cache. **Seeds the cache before returning**, so the first SSE event is never empty. Called exactly once; calling twice is undefined. | +| `stop()` | Cancels the task and releases resources. **Idempotent.** No writes to the cache afterwards. | +| `add_ticker(t)` | Adds to the tracked set. No-op if present. Simulator seeds a price immediately; Massive picks it up on the next poll. | +| `remove_ticker(t)` | Removes from the tracked set **and from the cache**. No-op if absent. | +| `get_tickers()` | Current tracked set. Synchronous — it reads local state only. | + +Two asymmetries are permitted and must not be papered over: + +- **Seeding latency.** `add_ticker` on the simulator makes a price available immediately; on Massive it takes up to one poll interval. The API contract already accommodates this — `GET /api/watchlist` returns `price: null` until the first tick, and the UI shows `—`. +- **Cadence.** 500ms versus 15s. Readers must never assume a minimum update rate. This is exactly what the SSE keepalive in §7 exists to handle. + +### `remove_ticker` also clears the cache — and why that is dangerous + +Both implementations call `self._cache.remove(ticker)`. That is correct for the interface but makes the method destructive: a held position whose ticker is removed loses its price, and with it its valuation, its P&L, its heatmap tile, and its snapshot contribution. §5 is the rule that prevents it. + +--- + +## 5. Which tickers are tracked + +**The tracked set is `watchlist ∪ {tickers with a non-zero position}`.** + +The two sets diverge the moment a user buys TSLA and then removes it from the watchlist. The position still needs a live price. This rule is the single most important piece of integration logic in the module, because getting it wrong produces a silently frozen position rather than an error. -Select the data source at startup based on environment: +| Trigger | Action | +|---|---| +| `POST /api/watchlist` | always `await source.add_ticker(t)` | +| `DELETE /api/watchlist/{t}` | `await source.remove_ticker(t)` **only if no position in `t` is held** | +| Buy a ticker not currently tracked | `await source.add_ticker(t)` as part of trade execution | +| Sell a position to zero | if `t` is not on the watchlist, `await source.remove_ticker(t)` | +| `POST /api/reset` | re-sync the tracked set to exactly the ten default tickers | + +A single helper keeps the rule in one place rather than at four call sites: + +```python +async def untrack_if_unused(source: MarketDataSource, ticker: str) -> None: + """Stop tracking a ticker only if it is neither watched nor held.""" + if is_on_watchlist(ticker) or has_position(ticker): + return + await source.remove_ticker(ticker) +``` + +### Startup ```python -import os +tickers = sorted(set(get_watchlist_tickers()) | set(get_position_tickers())) +await source.start(tickers) +``` + +Reading both tables at startup — not just the watchlist — is what makes a position held across a restart come back with a live price. + +--- +## 6. Selection — `create_market_data_source` + +```python def create_market_data_source(price_cache: PriceCache) -> MarketDataSource: - """Create the appropriate market data source based on environment.""" + """Create the market data source indicated by the environment. + + MASSIVE_API_KEY set and non-empty -> MassiveDataSource (real data) + otherwise -> SimulatorDataSource (GBM simulation) + + Returns an unstarted source; the caller must await source.start(tickers). + """ api_key = os.environ.get("MASSIVE_API_KEY", "").strip() if api_key: - from .massive_client import MassiveDataSource + logger.info("Market data source: Massive API (real data)") return MassiveDataSource(api_key=api_key, price_cache=price_cache) - else: - from .simulator import SimulatorDataSource - return SimulatorDataSource(price_cache=price_cache) + + logger.info("Market data source: GBM Simulator") + return SimulatorDataSource(price_cache=price_cache) ``` -## Massive Implementation Sketch +`.strip()` before the truth test is deliberate: `.env` files routinely contain `MASSIVE_API_KEY=` or a stray space, and a whitespace-only key would otherwise select the Massive path and then fail every poll with a 401. Empty means empty. -```python -import asyncio -from massive import RESTClient -from massive.rest.models import SnapshotMarketType - -class MassiveDataSource(MarketDataSource): - def __init__(self, api_key: str, price_cache: PriceCache, poll_interval: float = 15.0): - self._client = RESTClient(api_key=api_key) - self._cache = price_cache - self._interval = poll_interval - self._tickers: list[str] = [] - self._task: asyncio.Task | None = None - - async def start(self, tickers: list[str]) -> None: - self._tickers = list(tickers) - self._task = asyncio.create_task(self._poll_loop()) - - async def stop(self) -> None: - if self._task: - self._task.cancel() - - async def add_ticker(self, ticker: str) -> None: - if ticker not in self._tickers: - self._tickers.append(ticker) - - async def remove_ticker(self, ticker: str) -> None: - self._tickers = [t for t in self._tickers if t != ticker] - self._cache.remove(ticker) - - def get_tickers(self) -> list[str]: - return list(self._tickers) - - async def _poll_loop(self) -> None: - while True: - await self._poll_once() - await asyncio.sleep(self._interval) - - async def _poll_once(self) -> None: - if not self._tickers: - return - # Run synchronous Massive client in thread pool - snapshots = await asyncio.to_thread( - self._client.get_snapshot_all, - market_type=SnapshotMarketType.STOCKS, - tickers=self._tickers, - ) - for snap in snapshots: - self._cache.update( - ticker=snap.ticker, - price=snap.last_trade.price, - timestamp=snap.last_trade.timestamp / 1000, # ms -> seconds - ) +**The simulator is the default, and the fallback is decided once at startup — never at runtime.** A source that silently switched to the simulator after a Massive outage would show users invented prices while they believed they were seeing the market. Rejected keys and failed polls are logged; they do not change the source. `GET /api/health` reports which one is live: + +```json +{"status": "ok", "market_source": "simulator", "llm_mock": false} ``` -## Simulator Implementation Sketch +Returning an **unstarted** source keeps construction synchronous and lets the caller decide the ticker set from the database — the factory has no business reading tables. + +--- + +## 7. Lifecycle and wiring + +One `PriceCache` and one source per process, owned by the FastAPI lifespan. ```python -import asyncio +from contextlib import asynccontextmanager +from fastapi import FastAPI -class SimulatorDataSource(MarketDataSource): - def __init__(self, price_cache: PriceCache, update_interval: float = 0.5): - self._cache = price_cache - self._interval = update_interval - self._tickers: list[str] = [] - self._task: asyncio.Task | None = None - self._sim: GBMSimulator | None = None # See MARKET_SIMULATOR.md - - async def start(self, tickers: list[str]) -> None: - self._tickers = list(tickers) - self._sim = GBMSimulator(tickers=self._tickers) - self._task = asyncio.create_task(self._run_loop()) - - async def stop(self) -> None: - if self._task: - self._task.cancel() - - async def add_ticker(self, ticker: str) -> None: - if ticker not in self._tickers: - self._tickers.append(ticker) - self._sim.add_ticker(ticker) - - async def remove_ticker(self, ticker: str) -> None: - self._tickers = [t for t in self._tickers if t != ticker] - self._sim.remove_ticker(ticker) - self._cache.remove(ticker) - - def get_tickers(self) -> list[str]: - return list(self._tickers) - - async def _run_loop(self) -> None: - while True: - prices = self._sim.step() # Returns dict[str, float] - for ticker, price in prices.items(): - self._cache.update(ticker=ticker, price=price) - await asyncio.sleep(self._interval) +from app.market import PriceCache, create_market_data_source, create_stream_router + + +@asynccontextmanager +async def lifespan(app: FastAPI): + cache = PriceCache() + source = create_market_data_source(cache) + + tickers = sorted(set(get_watchlist_tickers()) | set(get_position_tickers())) + await source.start(tickers) + + app.state.price_cache = cache + app.state.market_source = source + try: + yield + finally: + await source.stop() + + +app = FastAPI(lifespan=lifespan) +app.include_router(create_stream_router(app.state.price_cache)) +``` + +The cache and source are passed explicitly (via router factories or `app.state`) rather than held in module globals, which is what keeps tests able to construct an isolated cache per test. + +**Ordering, from `PLAN.md` §11:** mount all `/api/*` routers *before* the static file mount. A `StaticFiles(html=True)` mount at `/` registered first shadows every endpoint, including the SSE stream. + +### The SSE stream + +`GET /api/stream/prices`, `Content-Type: text/event-stream`. The generator opens with `retry: 1000`, then pushes the **entire cache as one JSON object** whenever `version` changes, polled every 500ms: + +``` +retry: 1000 + +data: {"AAPL": {"ticker": "AAPL", "price": 190.52, "previous_price": 190.48, "timestamp": 1755873791.482, "change": 0.04, "change_percent": 0.021, "direction": "up"}, "GOOGL": {...}} ``` -## Integration with SSE +One event carries every ticker — not one event per ticker. The client replaces its price map wholesale, so there is no merge logic and no missed-update reconciliation. + +### Keepalive — **TODO** -The SSE endpoint reads from the `PriceCache` and pushes to connected clients: +When the version has not changed for 15 seconds, emit an SSE comment: ```python -async def price_stream(price_cache: PriceCache): - """SSE generator that yields price updates.""" - while True: +KEEPALIVE_SECONDS = 15.0 + +last_sent = time.monotonic() +while True: + if await request.is_disconnected(): + break + + current_version = price_cache.version + if current_version != last_version: + last_version = current_version prices = price_cache.get_all() - data = { - ticker: { - "ticker": p.ticker, - "price": p.price, - "previous_price": p.previous_price, - "change": p.change, - "direction": p.direction, - "timestamp": p.timestamp, - } - for ticker, p in prices.items() - } - yield f"data: {json.dumps(data)}\n\n" - await asyncio.sleep(0.5) + if prices: + payload = json.dumps({t: u.to_dict() for t, u in prices.items()}) + yield f"data: {payload}\n\n" + last_sent = time.monotonic() + elif time.monotonic() - last_sent >= KEEPALIVE_SECONDS: + yield ": ping\n\n" + last_sent = time.monotonic() + + await asyncio.sleep(interval) ``` -## File Structure +Without this, a Massive-backed feed sends no bytes between 15-second polls. That idle-times-out through proxies and leaves the frontend unable to distinguish a quiet market from a dead connection. The connection indicator — green on `onopen`, yellow on `onerror`, red after a ping gap beyond ~40 seconds — depends on it. + +--- +## 8. Implementing a new source + +The interface is small enough that a third source is a contained piece of work. The checklist: + +1. Subclass `MarketDataSource` and implement all five methods. +2. `start()` must **seed the cache before returning**. +3. Never write to the cache after `stop()`; make `stop()` idempotent. +4. `remove_ticker()` must call `cache.remove(ticker)`. +5. Convert timestamps to **Unix epoch seconds as a float** at the boundary. +6. Never let a fetch error kill the background loop — log and retry on the next cycle. +7. If the source is synchronous, wrap every call in `asyncio.to_thread`. +8. Add a branch to `create_market_data_source` and a value to `market_source` in `/api/health`. + +Point 7 is not optional. `massive.RESTClient` is `urllib3`-based and blocking; calling it directly from `async def` stalls the event loop for the duration of the HTTP round trip, which stops the SSE stream and every in-flight request. `MassiveDataSource` gets this right: + +```python +snapshots = await asyncio.to_thread(self._fetch_snapshots) ``` -backend/ - app/ - market/ - __init__.py - models.py # PriceUpdate dataclass - interface.py # MarketDataSource ABC, PriceCache - factory.py # create_market_data_source() - massive_client.py # MassiveDataSource - simulator.py # SimulatorDataSource + GBMSimulator - seed_prices.py # Default ticker seed prices + +### Massive polling, in outline + +```python +async def _poll_loop(self) -> None: + """Poll on interval. The first poll already happened in start().""" + while True: + await asyncio.sleep(self._interval) + await self._poll_once() ``` -## Lifecycle +`start()` performs one poll synchronously before creating the task, so the cache is warm before the first client connects. `_poll_interval` defaults to 15 seconds to stay inside the free tier's 5 requests/minute (`MASSIVE_API.md` §2); paid tiers can drop to 2–5 seconds. + +> The `_poll_once` parsing in `massive_client.py` currently reads a non-existent attribute and uses the wrong unit divisor, which means the Massive path writes nothing to the cache at all. Both defects are reproduced and the corrected parse is given in `MASSIVE_API.md` §8. Fixing them is a prerequisite to the Massive path working. + +--- + +## 9. Testing + +Existing coverage: **73 tests passing at 91%** for the market module, measured by running the suite while writing this document. (`MARKET_DATA_SUMMARY.md` still quotes 84%, which is stale.) + +**The cache and the interface can be tested without either real source.** A stub is a few lines, and it is the right tool for testing the tracked-ticker rules: + +```python +class StubDataSource(MarketDataSource): + """Records lifecycle calls; writes nothing on its own.""" + + def __init__(self, cache: PriceCache) -> None: + self._cache = cache + self._tickers: list[str] = [] + self.started = False + + async def start(self, tickers): self._tickers = list(tickers); self.started = True + async def stop(self): self.started = False + async def add_ticker(self, t): + if t not in self._tickers: + self._tickers.append(t) + async def remove_ticker(self, t): + self._tickers = [x for x in self._tickers if x != t] + self._cache.remove(t) + def get_tickers(self): return list(self._tickers) +``` + +What to cover: + +- **Cache** — `previous_price` derivation, first-update `flat`, `version` monotonicity, `remove` clearing both price and history, thread safety under concurrent writers. +- **Factory** — unset, empty, and whitespace-only `MASSIVE_API_KEY` all select the simulator; a real value selects Massive. +- **Tracked set** — removing a watchlist ticker with an open position keeps it in the feed; selling to zero off-watchlist removes it. These are the two regressions that produce a frozen position. +- **Massive parsing** — feed the real `TickerSnapshot.from_dict` a documented payload and assert the cached timestamp lands in the plausible present. Never build these snapshots from `MagicMock`: the existing tests do, which is precisely why 94% coverage of `massive_client.py` still missed both defects (`MASSIVE_API.md` §8.3). +- **SSE** — map-shaped payload, float timestamp, percent-unit `change_percent`, full snapshot on connect, keepalive after 15 idle seconds. +- **History** — deque bounded at 600, oldest-first ordering, `[]` for an untracked ticker. + +```bash +cd backend +uv run pytest +uv run pytest --cov=app --cov-report=term-missing +``` -1. **App startup**: Create `PriceCache`, call `create_market_data_source(price_cache)`, then `await source.start(initial_tickers)` -2. **Watchlist changes**: Call `source.add_ticker()` or `source.remove_ticker()` -3. **SSE streaming**: Reads from `PriceCache.get_all()` every 500ms -4. **Trade execution**: Reads current price from `PriceCache.get(ticker)` -5. **App shutdown**: Call `await source.stop()` +--- + +## 10. Summary + +| Concern | Resolution | +|---|---| +| Two sources, one consumer | `MarketDataSource` ABC + shared `PriceCache` | +| Which source | `create_market_data_source`, decided once at startup from `MASSIVE_API_KEY` | +| Default | Simulator — always alive, no key, no rate limit | +| How prices are read | Only from the cache, never from the source | +| Which tickers are live | `watchlist ∪ positions` | +| Timestamp format | Unix epoch seconds (float), converted at each source boundary | +| Update delivery | SSE, full cache per event, on `version` change | +| Blocking I/O | `asyncio.to_thread` at the source, always | +| Outstanding | Rolling history + endpoint, SSE keepalive, the two `massive_client.py` defects | diff --git a/planning/archive/MARKET_SIMULATOR.md b/planning/archive/MARKET_SIMULATOR.md index e157b6e..25c60ef 100644 --- a/planning/archive/MARKET_SIMULATOR.md +++ b/planning/archive/MARKET_SIMULATOR.md @@ -1,245 +1,456 @@ -# Market Simulator Design +# MARKET_SIMULATOR.md — The Market Simulator -Approach and code structure for simulating realistic stock prices when no Massive API key is configured. +The approach and code structure for simulating stock prices when no `MASSIVE_API_KEY` is configured. This is FinAlly's **default** data source, so it is what almost every user will see. -## Overview +Companion documents: `MARKET_INTERFACE.md` (the abstraction it implements) and `MASSIVE_API.md` (the alternative). Implemented in `backend/app/market/simulator.py` and `backend/app/market/seed_prices.py`. -The simulator uses **Geometric Brownian Motion (GBM)** to generate realistic stock price paths. GBM is the standard model underlying Black-Scholes option pricing — prices evolve continuously with random noise, can't go negative, and exhibit the lognormal distribution seen in real markets. +--- -Updates run at ~500ms intervals, producing a continuous stream of price changes that feel alive. +## 1. What it must achieve -## GBM Math +The simulator is not a research tool. It exists so that a student who clones the repo and runs one Docker command sees a trading terminal that looks alive, at any hour, on any day, with no account and no API key. -At each time step, a stock price evolves as: +That sets the bar precisely: + +| Requirement | Why | +|---|---| +| Visible motion every 500ms | The watchlist flashes green and red; a static grid looks broken | +| Motion at a *plausible* scale | AAPL moving $12 per tick destroys the illusion instantly | +| Prices that stay positive | A stock at −$4 is not a rendering bug the user will forgive | +| Correlated moves | Real tech stocks rise together; independent random walks look obviously fake | +| Occasional drama | A flat five minutes is boring; a sudden 3% drop gives the demo a story | +| Any ticker works | The AI chat can add any symbol; "we don't have that one" would feel broken | +| No external dependency | It must run offline, at 3am, on a weekend | + +And explicitly **not** required: predictive value, real historical data, order books, bid-ask spreads, or volume modelling. Nothing in the app consumes them. + +--- + +## 2. The model — Geometric Brownian Motion + +GBM is the standard model for equity prices and the one that satisfies the requirements above almost incidentally. ``` -S(t+dt) = S(t) * exp((mu - sigma^2/2) * dt + sigma * sqrt(dt) * Z) +S(t + dt) = S(t) · exp( (μ − σ²/2)·dt + σ·√dt·Z ) ``` -Where: -- `S(t)` = current price -- `mu` = annualized drift (expected return), e.g. 0.05 (5%) -- `sigma` = annualized volatility, e.g. 0.20 (20%) -- `dt` = time step as fraction of a trading year -- `Z` = standard normal random variable (drawn from N(0,1)) +| Symbol | Meaning | +|---|---| +| `S(t)` | Current price | +| `μ` | Annualised drift — expected return | +| `σ` | Annualised volatility | +| `dt` | Time step, as a fraction of a trading year | +| `Z` | Standard normal draw, correlated across tickers | + +Three properties earn its place here: + +**Prices cannot go negative.** The update is multiplicative — `exp(...)` is always positive, so `S` never crosses zero. No clamping, no `max(price, 0.01)` guard, no special case. An additive random walk would need all three. -For our 500ms updates with ~252 trading days and ~6.5 hours per day: +**Returns scale correctly with time.** Volatility is expressed per *year*, and `√dt` converts it to the tick. Change the tick rate and the price series keeps the same annualised character. The 500ms cadence is a display choice, not a modelling parameter. + +**The `−σ²/2` term keeps the drift honest.** Without it, `μ` would not be the expected return of the price — a well-known artefact of the log-normal distribution. It costs one subtraction and makes the parameters mean what they say. + +### Sizing `dt` + +`dt` is expressed against a **trading** year, not a calendar year — markets are closed most of the time, and using 365×24h would understate per-tick moves by a factor of about 4.5. + +```python +TRADING_SECONDS_PER_YEAR = 252 * 6.5 * 3600 # 5,896,800 +DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR # ~8.479e-8 ``` -dt = 0.5 / (252 * 6.5 * 3600) = ~8.5e-8 + +252 trading days × 6.5 hours × 3600 seconds. A 500ms tick is therefore `8.479e-8` of a year, and `√dt = 2.912e-4`. + +What that produces per tick, for `σ·√dt` at the seed prices: + +| Ticker | σ | Per-tick σ | Per-tick $ | Per-minute $ (120 ticks) | +|---|---|---|---|---| +| AAPL | 0.22 | 0.0064% | $0.012 | $0.13 | +| JPM | 0.18 | 0.0052% | $0.010 | $0.11 | +| NVDA | 0.40 | 0.0116% | $0.093 | $1.02 | +| TSLA | 0.50 | 0.0146% | $0.036 | $0.40 | + +This is the number that decides whether the simulation looks right. Around a cent or two per tick on a $200 stock means prices are **rounded to 2 decimals into a genuinely different value most ticks** — so the UI flashes constantly — while a minute of drift stays in the tens of cents, which is what a real quote screen looks like. Larger and it reads as a crash; smaller and the grid appears frozen. + +--- + +## 3. Correlation via Cholesky decomposition + +Independent draws per ticker would show tech stocks moving in opposite directions half the time. Real markets do not do that, and the eye notices immediately. + +The fix is standard: draw `n` independent standard normals, then multiply by the Cholesky factor `L` of the desired correlation matrix `C`, where `C = L·Lᵀ`. The resulting vector has exactly the correlation structure of `C`. + +```python +z_independent = np.random.standard_normal(n) +z_correlated = self._cholesky @ z_independent ``` -This tiny `dt` produces small, realistic per-tick moves. +### The correlation structure -## Correlated Moves +Sector membership and coefficients live in `seed_prices.py`, not in the simulator: -Real stocks don't move independently — tech stocks tend to move together, etc. We use a **Cholesky decomposition** of a correlation matrix to generate correlated random draws. +```python +CORRELATION_GROUPS = { + "tech": {"AAPL", "GOOGL", "MSFT", "AMZN", "META", "NVDA", "NFLX"}, + "finance": {"JPM", "V"}, +} -Given a correlation matrix `C`, compute `L = cholesky(C)`. Then for independent standard normals `Z_independent`: +INTRA_TECH_CORR = 0.6 # tech stocks move together +INTRA_FINANCE_CORR = 0.5 # finance stocks move together +CROSS_GROUP_CORR = 0.3 # between sectors, and for unknown tickers +TSLA_CORR = 0.3 # TSLA does its own thing ``` -Z_correlated = L @ Z_independent + +Resolved pairwise, first match winning: + +```python +@staticmethod +def _pairwise_correlation(t1: str, t2: str) -> float: + tech = CORRELATION_GROUPS["tech"] + finance = CORRELATION_GROUPS["finance"] + + # TSLA is in the tech set but behaves independently + if t1 == "TSLA" or t2 == "TSLA": + return TSLA_CORR + + if t1 in tech and t2 in tech: + return INTRA_TECH_CORR + if t1 in finance and t2 in finance: + return INTRA_FINANCE_CORR + + return CROSS_GROUP_CORR ``` -Default correlation groups: -- **Tech**: AAPL, GOOGL, MSFT, AMZN, META, NVDA, NFLX — corr ~0.6 within group -- **Finance**: JPM, V — corr ~0.5 within group -- **Cross-group**: ~0.3 baseline correlation -- **TSLA**: lower correlation with everything (~0.3) — it does its own thing +The TSLA clause is checked first on purpose: TSLA is a member of the tech set for every other purpose, but empirically trades on its own news, and a demo where TSLA visibly decouples from the pack is more convincing than one where everything moves in lockstep. -## Random Events +`CROSS_GROUP_CORR` doubles as the default for any symbol the simulator has never heard of, which is what makes §5 work. -Every step, each ticker has a small probability (~0.001) of a random event — a sudden 2-5% move. This adds drama and makes the dashboard visually interesting. +### Rebuilding + +`_rebuild_cholesky()` runs on every add and remove — `O(n²)` to build the matrix plus `O(n³)` to factor it, on `n < 50`. That is microseconds, and watchlist edits are a human-speed operation, so caching it would be complexity without benefit. ```python -if random.random() < event_probability: - shock = random.uniform(0.02, 0.05) * random.choice([-1, 1]) - price *= (1 + shock) +def _rebuild_cholesky(self) -> None: + n = len(self._tickers) + if n <= 1: + self._cholesky = None # a single ticker needs no correlation + return + + corr = np.eye(n) + for i in range(n): + for j in range(i + 1, n): + rho = self._pairwise_correlation(self._tickers[i], self._tickers[j]) + corr[i, j] = rho + corr[j, i] = rho + + self._cholesky = np.linalg.cholesky(corr) ``` -## Seed Prices +`step()` falls back to uncorrelated draws when `_cholesky is None`, which covers both the single-ticker and empty cases. + +> **Known risk.** `np.linalg.cholesky` raises `LinAlgError` on a matrix that is not positive definite, and this call is unguarded. The current block structure (0.6 / 0.5 / 0.3) was verified positive definite at 7, 20, and 40 tickers, so it is safe as configured — but raising `INTRA_TECH_CORR` toward 1.0, or adding a group whose intra-group correlation is below the cross-group value, can break positive-definiteness and take down `add_ticker`. Anyone editing these constants should re-run the check in §8. -Realistic starting prices for the default watchlist: +--- + +## 4. The tick + +`step()` is the hot path — every 500ms, for every ticker. ```python -SEED_PRICES: dict[str, float] = { - "AAPL": 190.0, - "GOOGL": 175.0, - "MSFT": 420.0, - "AMZN": 185.0, - "TSLA": 250.0, - "NVDA": 800.0, - "META": 500.0, - "JPM": 195.0, - "V": 280.0, - "NFLX": 600.0, -} +def step(self) -> dict[str, float]: + """Advance all tickers by one time step. Returns {ticker: new_price}.""" + n = len(self._tickers) + if n == 0: + return {} + + z_independent = np.random.standard_normal(n) + if self._cholesky is not None: + z_correlated = self._cholesky @ z_independent + else: + z_correlated = z_independent + + result: dict[str, float] = {} + for i, ticker in enumerate(self._tickers): + params = self._params[ticker] + mu, sigma = params["mu"], params["sigma"] + + drift = (mu - 0.5 * sigma**2) * self._dt + diffusion = sigma * math.sqrt(self._dt) * z_correlated[i] + self._prices[ticker] *= math.exp(drift + diffusion) + + if random.random() < self._event_prob: + shock_magnitude = random.uniform(0.02, 0.05) + shock_sign = random.choice([-1, 1]) + self._prices[ticker] *= 1 + shock_magnitude * shock_sign + + result[ticker] = round(self._prices[ticker], 2) + + return result ``` -Tickers added dynamically (not in the seed list) start at a random price between $50-$300. +Two details worth pointing out: -## Per-Ticker Parameters +**Full precision is kept internally; only the returned value is rounded.** `self._prices[ticker]` stays a full float. Rounding the stored state would accumulate quantisation error into a slow, systematic drift over thousands of ticks. -Each ticker has its own volatility to reflect real-world behavior: +**One `standard_normal(n)` call per tick, not `n` calls.** A single vectorised draw feeding one matrix multiply is the reason this stays negligible at 500ms. -```python -TICKER_PARAMS: dict[str, dict] = { - "AAPL": {"sigma": 0.22, "mu": 0.05}, - "GOOGL": {"sigma": 0.25, "mu": 0.05}, - "MSFT": {"sigma": 0.20, "mu": 0.05}, - "AMZN": {"sigma": 0.28, "mu": 0.05}, - "TSLA": {"sigma": 0.50, "mu": 0.03}, # High vol - "NVDA": {"sigma": 0.40, "mu": 0.08}, # High vol, strong drift - "META": {"sigma": 0.30, "mu": 0.05}, - "JPM": {"sigma": 0.18, "mu": 0.04}, # Low vol (bank) - "V": {"sigma": 0.17, "mu": 0.04}, # Low vol (payments) - "NFLX": {"sigma": 0.35, "mu": 0.05}, -} +### Random events -# Default for unknown tickers -DEFAULT_PARAMS = {"sigma": 0.25, "mu": 0.05} +```python +event_probability: float = 0.001 # per ticker, per tick ``` -## Implementation +A 2–5% jump in either direction. With 10 tickers at 2 ticks/second, the expected wait is `1 / (10 × 2 × 0.001) = 50 seconds` — frequent enough that something interesting happens during a demo, rare enough that the price series is not pure noise. + +The shock multiplies the price directly rather than feeding through GBM, so it is a genuine discontinuity — a gap, which is what real news does to a stock. + +--- + +## 5. Unknown tickers + +Any symbol passing the API-level pattern `^[A-Z][A-Z.]{0,5}$` works, with no allowlist. The AI chat can add anything the user names, and it behaves plausibly. ```python -import math -import random -import time -import numpy as np +def _add_ticker_internal(self, ticker: str) -> None: + if ticker in self._prices: + return + self._tickers.append(ticker) + self._prices[ticker] = SEED_PRICES.get(ticker, random.uniform(50.0, 300.0)) + self._params[ticker] = TICKER_PARAMS.get(ticker, dict(DEFAULT_PARAMS)) +``` + +- **Price**: seeded from `SEED_PRICES`, else uniform in $50–$300 — the range where most large-cap US equities actually trade. +- **Parameters**: `TICKER_PARAMS`, else `DEFAULT_PARAMS` (`σ=0.25`, `μ=0.05`) — a mid-range large cap. +- **Correlation**: no sector membership, so `CROSS_GROUP_CORR` (0.3) against everything. + +`dict(DEFAULT_PARAMS)` copies rather than sharing the module-level dict — without the copy, per-ticker parameter tuning would mutate the default for every unknown ticker at once. + +This is a real advantage over the Massive path, where an unknown symbol simply never produces a price and sits at `—` forever (`MASSIVE_API.md` §9). + +--- + +## 6. Code structure + +Two classes with a clean split: **`GBMSimulator` is pure and synchronous; `SimulatorDataSource` handles async lifecycle and the cache.** +``` +┌──────────────────────────────────────────────────────────┐ +│ SimulatorDataSource(MarketDataSource) │ +│ owns the asyncio task, writes to PriceCache │ +│ start / stop / add_ticker / remove_ticker / get_tickers│ +│ │ │ +│ ▼ │ +│ GBMSimulator │ +│ pure math, no I/O, no async, no cache reference │ +│ step() -> {ticker: price} │ +└──────────────────────────────────────────────────────────┘ + │ + ▼ + seed_prices.py + constants only, no logic +``` + +The separation pays off in testing: `GBMSimulator` needs no event loop, no cache, and no mocks. Statistical properties are asserted by calling `step()` in a loop. + +### `GBMSimulator` — pure math + +```python class GBMSimulator: - """Generates correlated GBM price paths for multiple tickers.""" - - def __init__( - self, - tickers: list[str], - dt: float = 8.5e-8, - event_probability: float = 0.001, - ): - self._dt = dt - self._event_prob = event_probability - self._prices: dict[str, float] = {} - self._params: dict[str, dict] = {} - self._tickers: list[str] = [] - self._cholesky: np.ndarray | None = None + TRADING_SECONDS_PER_YEAR = 252 * 6.5 * 3600 + DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR + def __init__(self, tickers, dt=DEFAULT_DT, event_probability=0.001): ... + + def step(self) -> dict[str, float]: ... + def add_ticker(self, ticker: str) -> None: ... + def remove_ticker(self, ticker: str) -> None: ... + def get_price(self, ticker: str) -> float | None: ... + def get_tickers(self) -> list[str]: ... +``` + +State is three parallel dicts keyed by ticker (`_prices`, `_params`) plus `_tickers` as the **ordered** list that indexes into the Cholesky matrix. Order matters: row `i` of `_cholesky` corresponds to `_tickers[i]`, which is why add and remove must both rebuild. + +`__init__` adds every ticker via `_add_ticker_internal` and rebuilds Cholesky **once** at the end, rather than rebuilding per ticker — `O(n³)` instead of `O(n⁴)` on startup. + +### `SimulatorDataSource` — the async wrapper + +```python +class SimulatorDataSource(MarketDataSource): + def __init__(self, price_cache, update_interval=0.5, event_probability=0.001): ... + + async def start(self, tickers: list[str]) -> None: + self._sim = GBMSimulator(tickers=tickers, event_probability=self._event_prob) + # Seed the cache with initial prices so SSE has data immediately for ticker in tickers: - self.add_ticker(ticker) - - def add_ticker(self, ticker: str) -> None: - if ticker in self._prices: - return - self._tickers.append(ticker) - self._prices[ticker] = SEED_PRICES.get(ticker, random.uniform(50, 300)) - self._params[ticker] = TICKER_PARAMS.get(ticker, DEFAULT_PARAMS) - self._rebuild_cholesky() - - def remove_ticker(self, ticker: str) -> None: - if ticker not in self._prices: - return - self._tickers.remove(ticker) - del self._prices[ticker] - del self._params[ticker] - self._rebuild_cholesky() - - def step(self) -> dict[str, float]: - """Advance one time step. Returns {ticker: new_price}.""" - n = len(self._tickers) - if n == 0: - return {} - - # Generate correlated random normals - z_independent = np.random.standard_normal(n) - if self._cholesky is not None: - z = self._cholesky @ z_independent - else: - z = z_independent - - result = {} - for i, ticker in enumerate(self._tickers): - params = self._params[ticker] - mu = params["mu"] - sigma = params["sigma"] - - # GBM step - drift = (mu - 0.5 * sigma**2) * self._dt - diffusion = sigma * math.sqrt(self._dt) * z[i] - self._prices[ticker] *= math.exp(drift + diffusion) - - # Random event - if random.random() < self._event_prob: - shock = random.uniform(0.02, 0.05) * random.choice([-1, 1]) - self._prices[ticker] *= (1 + shock) - - result[ticker] = round(self._prices[ticker], 2) - - return result - - def get_price(self, ticker: str) -> float | None: - return self._prices.get(ticker) - - def _rebuild_cholesky(self) -> None: - """Rebuild the Cholesky decomposition of the correlation matrix.""" - n = len(self._tickers) - if n <= 1: - self._cholesky = None - return - - corr = np.eye(n) - for i in range(n): - for j in range(i + 1, n): - rho = self._get_correlation(self._tickers[i], self._tickers[j]) - corr[i, j] = rho - corr[j, i] = rho - - self._cholesky = np.linalg.cholesky(corr) - - def _get_correlation(self, t1: str, t2: str) -> float: - """Return pairwise correlation between two tickers.""" - tech = {"AAPL", "GOOGL", "MSFT", "AMZN", "META", "NVDA", "NFLX"} - finance = {"JPM", "V"} - - t1_tech = t1 in tech - t2_tech = t2 in tech - t1_fin = t1 in finance - t2_fin = t2 in finance - - # Same sector: higher correlation - if t1_tech and t2_tech: - return 0.6 - if t1_fin and t2_fin: - return 0.5 - - # TSLA is a loner - if t1 == "TSLA" or t2 == "TSLA": - return 0.3 - - # Cross-sector or unknown - if (t1_tech and t2_fin) or (t1_fin and t2_tech): - return 0.3 - - # Default - return 0.3 -``` - -## File Structure - -All simulator code lives in a single module: - -``` -backend/ - app/ - market/ - simulator.py # GBMSimulator class + seed data + SimulatorDataSource - seed_prices.py # SEED_PRICES, TICKER_PARAMS, DEFAULT_PARAMS (constants) -``` + price = self._sim.get_price(ticker) + if price is not None: + self._cache.update(ticker=ticker, price=price) + self._task = asyncio.create_task(self._run_loop(), name="simulator-loop") + + async def _run_loop(self) -> None: + while True: + try: + if self._sim: + for ticker, price in self._sim.step().items(): + self._cache.update(ticker=ticker, price=price) + except Exception: + logger.exception("Simulator step failed") + await asyncio.sleep(self._interval) +``` + +Three deliberate choices: + +**Seed the cache in `start()` before creating the task.** The first SSE event then carries real prices rather than an empty object, so the watchlist never renders as ten dashes on load. + +**`add_ticker` seeds immediately.** The new ticker has a price on the very next SSE event, with no wait for the following step — the reason adding a ticker feels instant. + +**The `try` is inside the loop, around the step.** An exception logs and the loop continues on the next interval. Wrapping the loop instead would let one bad tick kill the feed permanently. This is the one place defensive handling is warranted: the background task has no caller to propagate to, and a dead price feed is a dead app. + +`stop()` cancels the task and awaits it, swallowing `CancelledError` — the normal shutdown path, not an error. + +### Parameters + +`seed_prices.py` holds constants only. Prices are realistic as of project creation; `σ` and `μ` are annualised. + +| Ticker | Seed | σ | μ | Note | +|---|---|---|---|---| +| AAPL | $190 | 0.22 | 0.05 | | +| GOOGL | $175 | 0.25 | 0.05 | | +| MSFT | $420 | 0.20 | 0.05 | | +| AMZN | $185 | 0.28 | 0.05 | | +| TSLA | $250 | 0.50 | 0.03 | High volatility, decorrelated | +| NVDA | $800 | 0.40 | 0.08 | High volatility, strong drift | +| META | $500 | 0.30 | 0.05 | | +| JPM | $195 | 0.18 | 0.04 | Low volatility (bank) | +| V | $280 | 0.17 | 0.04 | Low volatility (payments) | +| NFLX | $600 | 0.35 | 0.05 | | +| *unknown* | $50–300 | 0.25 | 0.05 | `DEFAULT_PARAMS` | + +The σ spread is what makes the watchlist readable at a glance: V and JPM barely move while NVDA and TSLA jump, so the grid has visible texture instead of ten tickers twitching identically. + +--- + +## 7. Behaviour over time + +There is **no mean reversion and no session boundary.** Prices random-walk from their seed for as long as the container runs. Over a demo — minutes to hours — drift is small and the series looks like a trading day. Over a week of uptime, a ticker may wander far from its seed. That is correct GBM behaviour and not worth correcting: the state is in memory only, so a restart returns everything to the seed prices. + +That in turn is why the rolling price history is not persisted (`MARKET_INTERFACE.md` §3). A restart legitimately resets the world. + +--- + +## 8. Testing + +Existing coverage is **73 tests passing at 91%** across the market module (`simulator.py` itself is at 98%). `GBMSimulator` is pure, so its tests are fast and deterministic under a seeded RNG. + +**Deterministic tests** — seed both RNGs, since the simulator uses `numpy.random` for the normal draws and the stdlib `random` for events: + +```python +def test_step_is_reproducible(): + np.random.seed(42) + random.seed(42) + sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) + first = sim.step() + + np.random.seed(42) + random.seed(42) + sim = GBMSimulator(tickers=["AAPL", "GOOGL"]) + assert sim.step() == first +``` + +**Structural properties:** + +- Prices stay strictly positive over many thousands of steps. +- `step()` returns exactly the current ticker set. +- Add/remove keeps `_tickers`, `_prices`, and `_params` consistent, and the Cholesky shape matches `len(_tickers)`. +- Unknown tickers seed within $50–$300 and get `DEFAULT_PARAMS`. +- `remove_ticker` on an untracked symbol is a no-op, not an error. + +**Statistical properties** — over enough steps, with a tolerance: + +```python +def test_realised_volatility_is_close_to_sigma(): + sim = GBMSimulator(tickers=["AAPL"], event_probability=0.0) + prices = [sim.get_price("AAPL")] + for _ in range(20_000): + prices.append(sim.step()["AAPL"]) + + log_returns = np.diff(np.log(prices)) + realised = log_returns.std() / np.sqrt(GBMSimulator.DEFAULT_DT) + assert 0.15 < realised < 0.35 # nominal sigma is 0.22 +``` + +Disable events (`event_probability=0.0`) for statistical tests — a 5% jump is a massive outlier at this dt and will dominate the sample variance. Keep tolerances wide; these are sampling estimates, and a tight bound produces a test that fails a few times a year for no reason. + +**Correlation:** + +```python +def test_tech_tickers_are_positively_correlated(): + sim = GBMSimulator(tickers=["AAPL", "MSFT"], event_probability=0.0) + a, m = [], [] + for _ in range(10_000): + p = sim.step() + a.append(p["AAPL"]) + m.append(p["MSFT"]) + + rho = np.corrcoef(np.diff(np.log(a)), np.diff(np.log(m)))[0, 1] + assert rho > 0.4 # nominal 0.6 +``` + +**Cholesky positive-definiteness** — run after any change to the correlation constants: + +```python +def test_correlation_matrix_stays_positive_definite(): + tickers = list(SEED_PRICES) + [f"UNK{i}" for i in range(40)] + GBMSimulator(tickers=tickers) # raises LinAlgError if not PD +``` + +**`SimulatorDataSource`** needs an event loop and a real `PriceCache`, but no mocks: + +- `start()` populates the cache before returning. +- The cache updates after roughly one interval. +- `add_ticker` seeds a price immediately. +- `remove_ticker` clears the ticker from the cache. +- `stop()` is idempotent and halts writes. + +```bash +cd backend +uv run pytest tests/market/ +uv run pytest --cov=app --cov-report=term-missing +``` + +`backend/market_data_demo.py` is a Rich terminal demo of the live simulator — the fastest way to eyeball whether a parameter change still looks right. + +--- + +## 9. Tuning guide + +Everything worth adjusting, and what it costs: + +| Want | Change | Watch for | +|---|---|---| +| More visible price motion | Raise `σ` in `TICKER_PARAMS` | Above ~0.8 it stops looking like equity | +| Faster or slower updates | `update_interval` on `SimulatorDataSource` | `DEFAULT_DT` is derived from 0.5s; change both together or annualised σ shifts | +| More frequent drama | Raise `event_probability` | Above ~0.005 the series becomes jumps, not prices | +| Bigger shocks | Widen `random.uniform(0.02, 0.05)` | Beyond ~10% the P&L chart loses all detail | +| Different sector behaviour | Edit `CORRELATION_GROUPS` and the coefficients | Re-run the positive-definiteness test in §8 | +| Different starting prices | `SEED_PRICES` | Unlisted tickers still land in $50–$300 | +| A trending market | Raise `μ` | `μ` is annualised; even 0.5 is barely visible over a demo | + +The `DEFAULT_DT` coupling is the one that catches people. `DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR` hard-codes the 500ms tick. Passing `update_interval=0.1` to `SimulatorDataSource` without also passing a matching `dt` to `GBMSimulator` makes the simulation run five times faster in model time — annualised volatility silently becomes 5× what `TICKER_PARAMS` claims. + +--- -`seed_prices.py` contains just the constant dictionaries. `simulator.py` contains the `GBMSimulator` class and the `SimulatorDataSource` (the `MarketDataSource` implementation that wraps `GBMSimulator` in an async loop). - -## Behavior Notes +## 10. Summary -- Prices never go negative (GBM is multiplicative — `exp()` is always positive) -- The tiny `dt` produces sub-cent moves per tick, which accumulate naturally over time -- With `sigma=0.50` (TSLA), a day of simulated trading produces roughly the right intraday range -- The correlation matrix must be positive semi-definite — Cholesky decomposition guarantees this for valid correlation matrices -- Random events happen ~0.1% of steps = roughly once every 500 seconds per ticker. With 10 tickers, expect an event somewhere roughly every 50 seconds — enough to keep it interesting -- When a new ticker is added mid-session, the Cholesky matrix is rebuilt. This is O(n^2) but n is small (<50 tickers) +| Concern | Approach | +|---|---| +| Price model | Geometric Brownian Motion, per-ticker `μ` and `σ` | +| Positivity | Guaranteed by the multiplicative `exp` form — no clamping | +| Time step | `0.5s / (252 × 6.5 × 3600)` ≈ `8.479e-8` of a trading year | +| Correlation | Cholesky factor of a sector-block matrix, rebuilt on add/remove | +| Sectors | tech 0.6, finance 0.5, cross-sector and unknown 0.3, TSLA 0.3 | +| Drama | 0.1% chance per ticker per tick of a 2–5% jump — about one per 50s | +| Unknown tickers | Random $50–$300 seed, `DEFAULT_PARAMS`, cross-sector correlation | +| Structure | Pure `GBMSimulator` + async `SimulatorDataSource`, constants in `seed_prices.py` | +| Persistence | None — a restart returns to seed prices | +| Failure handling | Per-step `try` inside the loop; the feed never dies from one bad tick | diff --git a/planning/archive/MASSIVE_API.md b/planning/archive/MASSIVE_API.md index 3266bc6..d7f94ef 100644 --- a/planning/archive/MASSIVE_API.md +++ b/planning/archive/MASSIVE_API.md @@ -1,251 +1,525 @@ -# Massive API Reference (formerly Polygon.io) +# MASSIVE_API.md — Massive (formerly Polygon.io) REST API -Reference documentation for the Massive (formerly Polygon.io) REST API as used in FinAlly. +Reference for retrieving real-time and end-of-day prices for multiple tickers. -## Overview +**Verified against `massive` Python SDK `2.2.0`** (installed in `backend/.venv`) and the official docs at . Every signature, field name, and unit below was checked against the installed package source or the live documentation — not from memory. -- **Base URL**: `https://api.massive.com` (legacy `https://api.polygon.io` still supported) -- **Python package**: `massive` (install via `pip install -U massive` / `uv add massive`) -- **Min Python version**: 3.9+ -- **Auth**: API key via `MASSIVE_API_KEY` env var or passed to `RESTClient(api_key=...)` -- **Auth header**: `Authorization: Bearer ` (the client handles this automatically) +> No `MASSIVE_API_KEY` was available in this repo when this document was written, so responses could not be exercised against the live service. Field shapes come from the SDK's `from_dict` parsers and the published response schemas, which is authoritative for how the client will deserialize. The verification script in §10 closes the loop once a key exists. -## Rate Limits +--- -| Tier | Limit | -|------|-------| -| Free | 5 requests/minute | -| Paid (all tiers) | Unlimited (recommended: stay under 100 req/s) | +## 1. Orientation -For FinAlly, we poll on a timer. Free tier: poll every 15s. Paid: poll every 2-5s. +Polygon.io rebranded to **Massive** in 2026. The API surface, the API keys, and the endpoint paths are unchanged; the hostname and the Python package are new. -## Client Initialization +| | Value | +|---|---| +| Base URL | `https://api.massive.com` | +| Python SDK | `massive` (PyPI), currently `2.2.0` | +| Repository | | +| Auth | `Authorization: Bearer ` header | +| Env var read by the SDK | `MASSIVE_API_KEY` | + +The SDK reads the same environment variable name this project already uses, so `RESTClient()` with no arguments works when `MASSIVE_API_KEY` is exported. FinAlly passes the key explicitly instead, because the factory has already read and validated it. + +### Install + +```bash +uv add massive +``` + +### Authentication + +The SDK sets the header for you (`massive/rest/base.py`): + +```python +self.headers = { + "Authorization": "Bearer " + self.API_KEY, + "Accept-Encoding": "gzip", + "User-Agent": f"Massive.com PythonClient/{version_number}", +} +``` + +Constructing a client with no key and no env var raises `massive.exceptions.AuthError` immediately — it does not wait for the first request. ```python from massive import RESTClient -# Reads MASSIVE_API_KEY from environment automatically -client = RESTClient() +client = RESTClient(api_key="YOUR_KEY") # or RESTClient() to read MASSIVE_API_KEY +``` + +Full constructor defaults, from the installed SDK: -# Or pass explicitly -client = RESTClient(api_key="your_key_here") +```python +RESTClient( + api_key: str | None = None, + connect_timeout: float = 10.0, + read_timeout: float = 10.0, + num_pools: int = 10, + retries: int = 3, # urllib3 Retry on 413/429/499/500/502/503... + base: str = "https://api.massive.com", + pagination: bool = True, + verbose: bool = False, + trace: bool = False, + custom_json: Any | None = None, +) ``` -## Endpoints Used in FinAlly +Two consequences worth knowing: + +- **`RESTClient` is synchronous.** It uses `urllib3.PoolManager`. Calling it from an `async def` blocks the event loop, which in this app means visibly stuttering prices on the SSE stream. Always wrap it in `asyncio.to_thread`. +- **It retries 429 internally** (3 attempts, honouring `Retry-After`). A poll that hits the rate limit therefore blocks its worker thread rather than failing fast. + +--- + +## 2. Plans and rate limits + +| Plan | Requests/min | Data freshness | +|---|---|---| +| Basic (free) | **5** | End-of-day, and 15-minute-delayed intraday | +| Paid (Starter and above) | Unlimited | Real-time (15-min delayed on Starter) | -### 1. Snapshot — All Tickers (Primary Endpoint) +This single number drives the whole polling design: **5 requests/minute means one request every 12 seconds at best.** FinAlly polls every 15 seconds by default, which leaves headroom and stays under the limit even if a poll overruns. -Gets current prices for multiple tickers in a **single API call**. This is the main endpoint we use for polling. +The corollary is that per-ticker endpoints are unusable on the free tier — 10 watchlist tickers via `get_last_trade` would be 10 requests per cycle, blowing the budget in one poll. **The design must fetch all tickers in a single request**, which is what §3 is about. -**REST**: `GET /v2/snapshot/locale/us/markets/stocks/tickers?tickers=AAPL,GOOGL,MSFT` +--- + +## 3. Real-time prices for multiple tickers + +### 3.1 Full Market Snapshot (v2) — the primary endpoint + +`GET /v2/snapshot/locale/us/markets/stocks/tickers` + +One request returns the current state of every ticker you name. This is the endpoint FinAlly uses. + +| Query param | Meaning | +|---|---| +| `tickers` | Case-insensitive comma-separated list. Omit to get the entire US market. | +| `include_otc` | Include OTC securities. Default `false`. | + +SDK signature: + +```python +client.get_snapshot_all( + market_type: str | SnapshotMarketType, + tickers: str | list[str] | None = None, + include_otc: bool | None = False, + params: dict | None = None, + raw: bool = False, +) -> list[TickerSnapshot] +``` + +The SDK joins a list into a comma-separated string for you (`",".join(tickers)`), so passing a `list[str]` is correct and idiomatic. -**Python client**: ```python from massive import RESTClient from massive.rest.models import SnapshotMarketType -client = RESTClient() +client = RESTClient(api_key="YOUR_KEY") -# Get snapshots for specific tickers (one API call) snapshots = client.get_snapshot_all( market_type=SnapshotMarketType.STOCKS, tickers=["AAPL", "GOOGL", "MSFT", "AMZN", "TSLA"], ) for snap in snapshots: - print(f"{snap.ticker}: ${snap.last_trade.price}") - print(f" Day change: {snap.day.change_percent}%") - print(f" Day OHLC: O={snap.day.open} H={snap.day.high} L={snap.day.low} C={snap.day.close}") - print(f" Volume: {snap.day.volume}") -``` - -**Response structure** (per ticker): -```json -{ - "ticker": "AAPL", - "day": { - "open": 129.61, - "high": 130.15, - "low": 125.07, - "close": 125.07, - "volume": 111237700, - "volume_weighted_average_price": 127.35, - "previous_close": 129.61, - "change": -4.54, - "change_percent": -3.50 - }, - "last_trade": { - "price": 125.07, - "size": 100, - "exchange": "XNYS", - "timestamp": 1675190399000 - }, - "last_quote": { - "bid_price": 125.06, - "ask_price": 125.08, - "bid_size": 500, - "ask_size": 1000, - "spread": 0.02, - "timestamp": 1675190399500 - }, - "prev_daily_bar": { "...": "previous day OHLCV" }, - "minute_volume": { "...": "volume per minute" } -} + print(snap.ticker, snap.last_trade.price, snap.todays_change_percent) ``` -**Key fields we extract**: -- `last_trade.price` — current price for trading and display -- `day.previous_close` — for calculating day change -- `day.change_percent` — day change percentage -- `last_trade.timestamp` — when the price was recorded +**Response shape** (`TickerSnapshot`, from `massive/rest/models/snapshot.py`): + +| Attribute | JSON key | Type | Notes | +|---|---|---|---| +| `ticker` | `ticker` | `str` | | +| `todays_change` | `todaysChange` | `float` | Absolute change vs. prior close | +| `todays_change_percent` | `todaysChangePerc` | `float` | **Already in percent units** (`0.39` = 0.39%) | +| `updated` | `updated` | `int` | **Nanoseconds** | +| `day` | `day` | `Agg` | Today's bar so far | +| `prev_day` | `prevDay` | `Agg` | Previous session's bar | +| `min` | `min` | `MinuteSnapshot` | Most recent minute bar | +| `last_trade` | `lastTrade` | `LastTrade` | Most recent execution | +| `last_quote` | `lastQuote` | `LastQuote` | Most recent NBBO | +| `fair_market_value` | `fmv` | `float` | Business plans only; `None` otherwise | + +`LastTrade` — note the attribute names, they do **not** match the JSON keys: + +| Attribute | JSON key | Units | +|---|---|---| +| `price` | `p` | dollars | +| `size` | `s` | shares | +| `sip_timestamp` | `t` | **nanoseconds** | +| `exchange` | `x` | exchange ID | +| `conditions` | `c` | `list[int]` | +| `id` | `i` | trade ID | +| `ticker` | `T` | usually `None` inside a snapshot | + +`Agg` (used for `day` and `prev_day`): `open`/`o`, `high`/`h`, `low`/`l`, `close`/`c`, `volume`/`v`, `vwap`/`vw`, `timestamp`/`t`, `transactions`/`n`. -### 2. Single Ticker Snapshot +### 3.2 Unified Snapshot (v3) — the alternative -For getting detailed data on one ticker (e.g., when user clicks a ticker for the detail view). +`GET /v3/snapshot` + +Multi-asset-class, paginated, and — the reason it is worth mentioning — it **reports unknown tickers explicitly** instead of silently omitting them. -**Python client**: ```python -snapshot = client.get_snapshot_ticker( - market_type=SnapshotMarketType.STOCKS, - ticker="AAPL", +snaps = client.list_universal_snapshots( + type="stocks", + ticker_any_of=["AAPL", "NVDA", "NOTAREALTICKER"], + limit=250, ) -print(f"Price: ${snapshot.last_trade.price}") -print(f"Bid/Ask: ${snapshot.last_quote.bid_price} / ${snapshot.last_quote.ask_price}") -print(f"Day range: ${snapshot.day.low} - ${snapshot.day.high}") +for s in snaps: + if s.error: + print(f"{s.ticker}: {s.error} — {s.message}") # e.g. NOT_FOUND + else: + print(s.ticker, s.session.close, s.last_trade.price) ``` -### 3. Previous Close +- `ticker_any_of` accepts **up to 250** tickers. +- `limit` defaults to 10 and maxes at 250 — **leave it at the default and you will silently get only 10 results.** Always set it explicitly. +- Returns an *iterator* and auto-paginates (`pagination=True`), so a careless call can fan out into many billed requests. With `ticker_any_of` bounded at 250 and `limit=250` there is exactly one page. + +FinAlly stays on v2 because a 10-ticker watchlist never approaches the 250 limit, v2 is a single non-paginated request, and the `error` field is of marginal value when the simulator is the default path anyway. v3 is the right upgrade if per-ticker validation feedback is ever wanted. -Gets the previous day's OHLC for a ticker. Useful for seed prices. +### 3.3 Single ticker -**REST**: `GET /v2/aggs/ticker/{ticker}/prev` +Useful for a one-off lookup; unusable as a polling strategy on the free tier. -**Python client**: ```python -prev = client.get_previous_close_agg(ticker="AAPL") +snap = client.get_snapshot_ticker(SnapshotMarketType.STOCKS, "AAPL") +trade = client.get_last_trade("AAPL") # LastTrade: .price, .size, .sip_timestamp (ns) +quote = client.get_last_quote("AAPL") # LastQuote: .bid_price, .ask_price, ... +``` + +--- + +## 4. End-of-day prices + +### 4.1 Previous close — per ticker + +`GET /v2/aggs/ticker/{ticker}/prev` -for agg in prev: - print(f"Previous close: ${agg.close}") - print(f"OHLC: O={agg.open} H={agg.high} L={agg.low} C={agg.close}") - print(f"Volume: {agg.volume}") +```python +prev = client.get_previous_close_agg("AAPL", adjusted=True) +print(prev.ticker, prev.open, prev.high, prev.low, prev.close, prev.volume, prev.vwap) ``` -**Response**: -```json -{ - "ticker": "AAPL", - "results": [ - { - "o": 150.0, - "h": 155.0, - "l": 149.0, - "c": 154.5, - "v": 1000000, - "t": 1672531200000 - } - ] -} +`PreviousCloseAgg` fields: `ticker`, `open`, `high`, `low`, `close`, `volume`, `vwap`, `timestamp` (**milliseconds**, start of the aggregate window). + +Works on the free tier. One request per ticker, so 10 tickers = 10 requests = two minutes of free-tier budget. + +### 4.2 Daily market summary — the whole market in one request + +`GET /v2/aggs/grouped/locale/us/market/stocks/{date}` + +The efficient way to get EOD for many tickers: **one request returns every US ticker for that date.** + +```python +from datetime import date + +bars = client.get_grouped_daily_aggs(date="2026-08-28", adjusted=True) + +wanted = {"AAPL", "GOOGL", "MSFT"} +closes = {b.ticker: b.close for b in bars if b.ticker in wanted} +print(closes) ``` -### 4. Aggregates (Bars) +`GroupedDailyAgg` adds a `ticker` attribute (JSON key `T`) to the standard `Agg` fields. `timestamp`/`t` is **milliseconds**, marking the *end* of the aggregate window. -Historical OHLCV bars over a date range. Not needed for live polling but useful if we add historical charts. +Caveats: the date must be a **trading day** — a weekend or holiday returns an empty result set, not an error. And the response covers the entire market (thousands of rows), so filter client-side. -**REST**: `GET /v2/aggs/ticker/{ticker}/range/{multiplier}/{timespan}/{from}/{to}` +### 4.3 Daily open/close for one ticker on one date + +`GET /v1/open-close/{ticker}/{date}` -**Python client**: ```python -aggs = [] -for a in client.list_aggs( +oc = client.get_daily_open_close_agg("AAPL", date="2026-08-28", adjusted=True) +print(oc.open, oc.close, oc.pre_market, oc.after_hours, oc.status) +``` + +`DailyOpenCloseAgg` is the one model that carries pre-market and after-hours prints. Note it uses `symbol` (not `ticker`) and `from_` (not `from`, which is a Python keyword). + +--- + +## 5. Historical bars — for charts and backfill + +`GET /v2/aggs/ticker/{ticker}/range/{multiplier}/{timespan}/{from}/{to}` + +```python +# 1-minute bars for one session +bars = client.get_aggs( ticker="AAPL", multiplier=1, - timespan="day", - from_="2024-01-01", - to="2024-01-31", + timespan="minute", # second|minute|hour|day|week|month|quarter|year + from_="2026-08-28", # YYYY-MM-DD, date, datetime, or Unix ms + to="2026-08-28", + adjusted=True, + sort="asc", limit=50000, -): - aggs.append(a) - -for a in aggs: - print(f"Date: {a.timestamp}, O={a.open} H={a.high} L={a.low} C={a.close} V={a.volume}") -``` - -**Response** (each bar): -```json -{ - "o": 130.0, - "h": 132.5, - "l": 129.8, - "c": 131.2, - "v": 50000000, - "t": 1672531200000 -} +) + +for b in bars: + print(b.timestamp, b.open, b.high, b.low, b.close, b.volume) ``` -### 5. Last Trade / Last Quote +`get_aggs` returns a **list** and is capped at 50,000 bars. `list_aggs` takes the same arguments but returns an **auto-paginating iterator** — convenient for long ranges, and a way to accidentally issue many billed requests. Prefer `get_aggs` with an explicit range unless you genuinely need more than 50k bars. + +`Agg.timestamp` is **milliseconds**, marking the start of the window. + +Relevance to FinAlly: this is the only way to seed a chart with real history under Massive. The rolling in-memory history in §6 of `PLAN.md` covers the simulator; a Massive-backed deployment could optionally backfill `GET /api/prices/{ticker}/history` from 1-minute aggregates instead. That is out of scope today, and noted here so the option is not rediscovered later. -Individual endpoints for the most recent trade or NBBO quote. +--- + +## 6. Market status + +Worth calling to explain a frozen feed to the user rather than leaving them guessing. ```python -# Last trade -trade = client.get_last_trade(ticker="AAPL") -print(f"Last trade: ${trade.price} x {trade.size}") +status = client.get_market_status() +print(status.market) # "open" | "closed" | "extended-hours" +print(status.exchanges) +print(status.after_hours, status.early_hours) +``` + +`client.get_market_holidays()` returns upcoming closures and early closes. + +--- + +## 7. Timestamp units — the trap + +Massive uses **three different time units across endpoints**, and the SDK passes them through unchanged. This is the single easiest thing to get wrong. + +| Source | Attribute | Unit | To Unix seconds | +|---|---|---|---| +| Snapshot `lastTrade` | `sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | +| Snapshot `lastQuote` | `sip_timestamp` | **nanoseconds** | `/ 1_000_000_000` | +| Snapshot top level | `updated` | **nanoseconds** | `/ 1_000_000_000` | +| Snapshot `min` | `timestamp` | **milliseconds** | `/ 1_000` | +| Aggregates (`Agg`, `PreviousCloseAgg`, grouped) | `timestamp` | **milliseconds** | `/ 1_000` | + +FinAlly's `PriceUpdate.timestamp` is **Unix epoch seconds as a float** (§7 of `PLAN.md`), so every value from this API needs converting, and the divisor depends on which endpoint it came from. -# Last NBBO quote -quote = client.get_last_quote(ticker="AAPL") -print(f"Bid: ${quote.bid} x {quote.bid_size}") -print(f"Ask: ${quote.ask} x {quote.ask_size}") +```python +NANOS_PER_SECOND = 1_000_000_000 +MILLIS_PER_SECOND = 1_000 + +ts_seconds = snap.last_trade.sip_timestamp / NANOS_PER_SECOND # snapshot +ts_seconds = agg.timestamp / MILLIS_PER_SECOND # aggregates +``` + +### Attribute names never match JSON keys + +The wire format is single-letter (`p`, `s`, `t`, `x`); the SDK's `from_dict` maps those to readable attributes. You must use the **attribute** names. Reading `snap.last_trade.t` or `snap.last_trade.timestamp` raises `AttributeError`, because `@modelclass` builds a plain dataclass with no `__getattr__` fallback: + +```python +# massive/rest/models/trades.py +@staticmethod +def from_dict(d): + return LastTrade( + d.get("T"), d.get("f"), d.get("q"), d.get("t"), # "t" -> sip_timestamp + d.get("y"), d.get("c"), d.get("e"), d.get("i"), + d.get("p"), # "p" -> price + d.get("r"), d.get("s"), d.get("x"), d.get("z"), + ) +``` + +--- + +## 8. Two defects confirmed in `backend/app/market/massive_client.py` + +Both were reproduced against the installed SDK, not inferred. They are recorded here because this document is the reference the fix should be written from; the fix itself belongs to whoever next touches that module. + +### 8.1 `last_trade.timestamp` does not exist — the Massive path returns no prices at all + +`_poll_once` reads: + +```python +price = snap.last_trade.price +timestamp = snap.last_trade.timestamp / 1000.0 # AttributeError +``` + +Reproduction, using a payload shaped exactly as the v2 snapshot documentation specifies: + +```python +from massive.rest.models.snapshot import TickerSnapshot + +snap = TickerSnapshot.from_dict({ + "ticker": "AAPL", + "lastTrade": {"p": 190.52, "s": 100, "t": 1755873791482000000, "x": 4}, +}) + +snap.last_trade.price # 190.52 +snap.last_trade.sip_timestamp # 1755873791482000000 +snap.last_trade.timestamp # AttributeError: 'LastTrade' object has no attribute 'timestamp' +``` + +The loop wraps each snapshot in `except (AttributeError, TypeError)` and merely logs a warning, so the exception is swallowed **once per ticker, on every poll**. The cache is never written. The observable symptom is not a crash: it is a watchlist where every ticker shows `—` forever, with `Skipping snapshot for AAPL` in the logs. + +The correct attribute is `sip_timestamp`. + +### 8.2 The unit divisor is wrong by a factor of 10⁶ + +Even with the attribute corrected, `/ 1000.0` treats nanoseconds as milliseconds. `1755873791482000000 / 1000` is ≈ 1.76 × 10¹⁵ seconds — roughly 55 million years in the future. Charts keyed on that timestamp would be unusable. The divisor must be `1_000_000_000`. + +### 8.3 Why 94% test coverage did not catch either defect + +`massive_client.py` is 94% covered and all 73 tests pass. The tests nonetheless assert the buggy behaviour, because they build snapshots from `MagicMock` (`backend/tests/market/test_massive.py`): + +```python +def _make_snapshot(ticker: str, price: float, timestamp_ms: int) -> MagicMock: + snap = MagicMock() + snap.last_trade = MagicMock() + snap.last_trade.price = price + snap.last_trade.timestamp = timestamp_ms # attribute the real model does not have + return snap +``` + +A `MagicMock` answers to any attribute name, so `snap.last_trade.timestamp` resolves happily in the test and raises `AttributeError` in production. `test_timestamp_conversion` then locks in the wrong unit as well: + +```python +assert update.timestamp == 1707580800.0 # asserts milliseconds -> seconds +``` + +The lesson generalises: **mocking a third-party model tests your assumptions about the library, not the library.** Parsing tests must go through the real `TickerSnapshot.from_dict` with a documented payload, as in §10. That form of test needs no network and would have failed on the first run. + +### Corrected parse + +```python +NANOS_PER_SECOND = 1_000_000_000 + +for snap in snapshots: + trade = snap.last_trade + if trade is None or trade.price is None: + continue # no print yet today; leave the ticker showing "—" + self._cache.update( + ticker=snap.ticker, + price=trade.price, + timestamp=( + trade.sip_timestamp / NANOS_PER_SECOND + if trade.sip_timestamp + else time.time() + ), + ) ``` -## How FinAlly Uses the API +Guarding on `is None` rather than catching `AttributeError` is what makes the difference: a genuinely absent field is a normal condition to handle, whereas a misspelled attribute is a bug that should be loud. The existing blanket `except AttributeError` is precisely what hid this one. + +--- -The Massive poller runs as a background task: +## 9. Errors and operational behaviour -1. Collects all tickers from the watchlist -2. Calls `get_snapshot_all()` with those tickers (one API call) -3. Extracts `last_trade.price` and `day.previous_close` from each snapshot -4. Writes to the shared in-memory price cache -5. Sleeps for the poll interval, then repeats +The SDK raises only two exception types (`massive/exceptions.py`): + +| Exception | Cause | +|---|---| +| `AuthError` | Empty or missing API key at construction time | +| `BadResponse` | Any non-200 response that survived the retry policy | + +`urllib3` raises its own errors for connection failures and timeouts. A poll loop should therefore catch broadly and keep going, since a failed poll is recoverable on the next cycle: ```python -import asyncio +from massive.exceptions import AuthError, BadResponse + +try: + snapshots = await asyncio.to_thread(self._fetch_snapshots) +except AuthError: + logger.error("Massive API key rejected — falling back is not automatic") + raise # unrecoverable: do not retry on a loop +except BadResponse as e: + logger.warning("Massive returned an error response: %s", e) + return # transient: retry next interval +except Exception: + logger.exception("Massive poll failed") + return +``` + +### Behaviours to surface in the README + +These are properties of the data source, not bugs, and users will otherwise report them as bugs: + +- **Unknown symbols vanish silently.** The v2 snapshot omits tickers it does not recognise; there is no error entry. The ticker sits in the watchlist showing `—` indefinitely. (v3 would report `NOT_FOUND` — see §3.2.) +- **Prices freeze outside market hours.** Overnight, at weekends, and on holidays the snapshot returns the last trade of the previous session. The UI looks broken but is correct. This is the main reason the simulator is the default. +- **Free-tier data is 15 minutes delayed**, so prices will not match any other quote source the user has open. +- **Snapshot data is cleared at midnight ET** and repopulates from about 4am ET. Between those times `last_trade` may be absent entirely — which is exactly the `None` case §8 guards. + +--- + +## 10. Verification script + +Run this once a real `MASSIVE_API_KEY` is available. It confirms auth, the multi-ticker snapshot, unit conversion, and the EOD path in one pass. + +```python +# backend/scripts/verify_massive.py +"""Smoke-test the Massive REST API against a live key.""" + +import os +from datetime import UTC, datetime + from massive import RESTClient from massive.rest.models import SnapshotMarketType -async def poll_massive(api_key: str, get_tickers, price_cache, interval: float = 15.0): - """Poll Massive API and update the price cache.""" - client = RESTClient(api_key=api_key) - - while True: - tickers = get_tickers() - if tickers: - snapshots = client.get_snapshot_all( - market_type=SnapshotMarketType.STOCKS, - tickers=tickers, - ) - for snap in snapshots: - price_cache.update( - ticker=snap.ticker, - price=snap.last_trade.price, - previous_close=snap.day.previous_close, - timestamp=snap.last_trade.timestamp, - ) - - await asyncio.sleep(interval) -``` - -## Error Handling - -The client raises exceptions for HTTP errors: -- **401**: Invalid API key -- **403**: Insufficient permissions (plan doesn't include the endpoint) -- **429**: Rate limit exceeded (free tier: 5 req/min) -- **5xx**: Server errors (client has built-in retry with 3 retries by default) - -## Notes - -- The snapshot endpoint returns data for **all requested tickers in one call** — this is critical for staying within rate limits on the free tier -- Timestamps from the API are Unix milliseconds -- During market closed hours, `last_trade.price` reflects the last traded price (may include after-hours) -- The `day` object resets at market open; during pre-market, values may be from the previous session +NANOS_PER_SECOND = 1_000_000_000 +TICKERS = ["AAPL", "GOOGL", "MSFT", "NVDA", "TSLA"] + + +def main() -> None: + key = os.environ["MASSIVE_API_KEY"] + client = RESTClient(api_key=key) + + status = client.get_market_status() + print(f"market: {status.market}") + + snapshots = client.get_snapshot_all(SnapshotMarketType.STOCKS, TICKERS) + print(f"requested {len(TICKERS)}, received {len(snapshots)}") + + for snap in snapshots: + trade = snap.last_trade + if trade is None or trade.price is None: + print(f"{snap.ticker}: no trade data") + continue + seconds = trade.sip_timestamp / NANOS_PER_SECOND + when = datetime.fromtimestamp(seconds, UTC) + print(f"{snap.ticker}: ${trade.price:.2f} at {when:%Y-%m-%d %H:%M:%S} UTC") + + missing = set(TICKERS) - {s.ticker for s in snapshots} + if missing: + print(f"absent from response (unknown or untraded): {sorted(missing)}") + + prev = client.get_previous_close_agg("AAPL") + print(f"AAPL previous close: ${prev.close:.2f}") + + +if __name__ == "__main__": + main() +``` + +```bash +uv run python scripts/verify_massive.py +``` + +Expected: a market status, five priced tickers with timestamps in the recent past (not 55 million years hence), and a previous close. Timestamps far in the future mean the unit divisor is wrong; `AttributeError` means §8.1 has regressed. + +--- + +## 11. Summary — what FinAlly uses + +| Need | Endpoint | SDK call | Cost | +|---|---|---|---| +| Live prices, all watched tickers | `/v2/snapshot/.../tickers` | `get_snapshot_all` | 1 request per poll | +| EOD close, one ticker | `/v2/aggs/ticker/{t}/prev` | `get_previous_close_agg` | 1 request per ticker | +| EOD close, many tickers | `/v2/aggs/grouped/...` | `get_grouped_daily_aggs` | 1 request total | +| Chart backfill | `/v2/aggs/ticker/{t}/range/...` | `get_aggs` | 1 request per ticker | +| Explain a frozen feed | `/v1/marketstatus/now` | `get_market_status` | 1 request | + +The polling design that follows from the 5 req/min free tier — one snapshot request covering the union of watchlist and held positions, every 15 seconds — is specified in `MARKET_INTERFACE.md`. + +## Sources + +- [Full Market Snapshot](https://massive.com/docs/rest/stocks/snapshots/full-market-snapshot) +- [Unified Snapshot](https://massive.com/docs/rest/stocks/snapshots/unified-snapshot) +- [Previous Day Bar](https://massive.com/docs/rest/stocks/aggregates/previous-day-bar) +- [Daily Market Summary](https://massive.com/docs/rest/stocks/aggregates/daily-market-summary) +- [Request limits for Massive's RESTful APIs](https://massive.com/knowledge-base/article/what-is-the-request-limit-for-massives-restful-apis) +- [massive-com/client-python](https://github.com/massive-com/client-python) +- Installed SDK source: `backend/.venv/lib/python3.13/site-packages/massive/` (v2.2.0)