update_markets.py silently retries forever once markets.csv > ~50K rows (gamma API offset cap)

[merplerps]

Repro

With a markets.csv containing > ~50K rows (e.g., from prior runs), invoke:

cd poly_data
python -c "from update_utils.update_markets import update_markets; update_markets()"

Symptom

Indefinite retry loop with no observable progress. No new markets are appended; the script never exits.

Actual cause

Polymarket's gamma API caps offset-based pagination. At offset ~55,000+ it returns:

HTTP 422  {"type":"validation error","error":"offset exceeds maximum allowed for markets list queries"}

update_markets.py treats this as a transient error and retries every 3 seconds forever.

Impact

• Users with substantial historical markets.csv cannot use update_markets() to refresh.
• The silent loop wastes API quota and CPU; foreground users may not notice for hours.
• Discovered while debugging missing market metadata for ~80% of currently-active Polymarket markets.

Suggested fixes

1. Fast-fail on 422 with the specific error message — surface to the operator that offset pagination is exhausted.
2. Cursor-based fetch as the longer-term remedy — query by createdAt > <max_known_created_at> instead of offset=<row_count>. The gamma API supports createdAt filtering.
3. Or recommend the alternative discovery path (e.g., update_utils.utils.update_missing_tokens()) for finding new markets via trades.