Skip to content

RESULT.md

Latest commit

 

History

History
100 lines (67 loc) · 6.05 KB

RESULT.md

File metadata and controls

100 lines (67 loc) · 6.05 KB
Raw

Phase 1 Result

Date: 2026-05-28 Status: Complete — Phase 2 not justified

Verdict

No. Copy-trading Polymarket's most consistently profitable wallets does not produce excess returns after realistic lag and slippage. The strategy fails the pre-set bar on every dimension of the sensitivity sweep.

The bar (set before any backtest ran, per TASK.md)

A "yes, build Phase 2" decision required:

  1. Positive risk-adjusted return after realistic slippage, and
  2. Sharpe > 0 with tolerable max drawdown, and
  3. The result holds across ≥3 of 4 sensitivity buckets under walk-forward.

The numbers

Single backtest (24 months, lag 15m, $1k per position, strict filter, watchlist top 100):

Metric Value
Eval window 2024-06-01 → 2026-05-01
Signals 1,402 (1,384 resolved)
Capital deployed $1,402,000
Total P&L −$8,706
ROI −0.62%
Sharpe −0.13
Max drawdown 3.9%
Win rate 54.8%

Sensitivity sweep (108-cell grid, 4 dimensions):

Dimension Levels (mean ROI) Holds?
Lag 5m: −0.8% · 15m: −0.8% · 30m: −0.9% · 60m: −1.8% fails
Size $250: −0.3% · $1k: −1.4% · $5k: −1.4% fails
Watchlist N 20: −1.0% · 100: −1.1% · 500: −1.1% fails (degenerate)
MM filter strict: −0.9% · loose: +0.6% · off: −2.9% fails

Buckets held: 0 / 4 → not robust.

What the numbers mean

  1. An edge exists, but it doesn't survive costs. The best cells (short lag + smallest size + loose filter) reach ~+1.5% over 23 months — roughly 0.8%/year, only at the smallest position size where slippage is least punishing. Every other corner of the grid is negative. The cost wall is binding.

  2. The classifier earns its keep. mmFilter off (copying everyone on the leaderboard) is consistently the worst — about −2.9% mean ROI. Filtering to smart-money matters; it's just that the filtered subset still doesn't beat costs at any realistic size.

  3. Win rate > 50% is not enough. The 54.8% win rate means the classifier did pick wallets that are right more often than not. P&L is still negative because losses are larger than wins (asymmetric payoffs). Being directionally right > 50% of the time ≠ being profitable to copy with lag and flat sizing.

  4. Category split shows where the edge is and isn't.

    • Sports: +$40,409 (919 bets) — a real signal
    • Other: −$50,273 (456 bets) — wipes the sports edge out and then some
    • Politics: +$789 · Crypto: +$369

  5. Concentration risk. One wallet contributed +$20,708; several lost $6k–$9k each. The aggregate result hinges on a handful of wallets and bets, not a broad edge.

Caveats (all push the result in the same direction — toward "no")

  • Universe selection bias. The 10 smart_money wallets were sifted out of today's top-50 profit leaderboard — wallets selected on present-day prominence. The walk-forward ranking at each historical T is leak-free, but the candidate pool is not. If anything, this should make the result look better than reality; it didn't.

  • Reconstructed leaderboards use realized-only profit, diverging from Polymarket's live realized+unrealized metric. Snapshots are tagged source = 'reconstructed' vs 'live'.

  • Watchlist is thin. Only 0–4 wallets qualified as smart_money at any given T (10 total in the universe, further narrowed by the 60-day-history requirement and as-of-T classification). The watchlist-size sweep dimension is therefore degenerate — there were never 20+ qualifying wallets, so the N=20/100/500 cells are identical. Inherent low-N noise.

  • 2024 was nearly empty — the universe's history doesn't sustain enough smart-money wallets in the back half of the eval window for the early-T watchlists to be populated.

  • Category breakdown is a keyword heuristic (no stored category column); approximate. By-wallet is the firmer concentration check.

  • MM/arb classifier rule ordering is a non-spec resolution (structural MM → arb → generic MM → smart) so all labels stay reachable; documented in lib/backtest/filters.ts.

Decision

Phase 2 (live alerts, hosted infra, real-time integration) was conditional on a positive Phase 1 result. It is not justified by this data. Do not build it.

A "no" was always one of the two valid Phase 1 outcomes, and is just as informative as a "yes" would have been: a cleanly defensible answer to the question, achieved cheaply, before any capital was risked. The build worked exactly as designed — the result is the result.

What the experiment did not test (out of scope, future-work candidates)

  • A wallet universe beyond the top-50 profit leaderboard (e.g. via sync:trades with a low threshold to catch big directional bettors who don't rank). The slippage wall would likely still bind, but this would convert "no" into "definitively no."
  • Per-wallet position sizing weighted by historical win-rate / margin (rather than flat $1k). Could marginally tighten the edge but unlikely to break through the cost wall.
  • Alternative exit logic beyond hold-to-resolution (e.g. partial exits, source-mirror with a delay). The --source-exit flag exists; it was not the focus of the headline result.
  • Markets the classifier flagged as unknown — the 24 unknowns include wallets with too little resolved history to judge. A longer collection window might surface a few more genuine smart-money wallets, but the sweep result suggests the universe-level edge is too small for that to flip the verdict.

How to reproduce

npm install
npm run db:migrate
npm run sync:leaderboard
npm run sync:wallets -- --limit 50    # ~tens of minutes; pre-screen filters obvious bots
npm run sync:markets                  # Gamma at ~1 req/s; several minutes for ~30k markets
npm run classify:all
npm run backfill:leaderboard -- --from 2024-06-01 --to 2026-05-01
npm run backtest -- --from 2024-06-01 --to 2026-05-01 --lag 15 --size 1000
npm run backtest:sensitivity -- --from 2024-06-01 --to 2026-05-01

See RUNBOOK.md for the full pipeline and flags.