Fix reviewed Discogs pagination, comparison tasks, and grading (#109)

.assets-revision

@@ -1,14 +1,8 @@
-# Hugging Face dataset holding WebHarbor's heavy static assets.
-# Pin an immutable merged commit SHA, never a moving branch or refs/pr/<n>.
-#
-# All 46 registered sites use this immutable dataset-main revision.
-# The previous global revision eedc1a7f85118ab835eac38e04678fa85d141973 contains 46 archives: 45 registered-site archives plus the unregistered Bandcamp archive. All 46 are byte-for-byte unchanged here.
-# Merged HF PR #86 adds phet_simulations.tar.gz: 10,319,041 bytes, SHA-256 7ed4831ed1218bf95456a6156f82dbd769b909fb630434e042f6e6cd1987a65f, with 150 managed archive members.
-# This revision therefore contains 47 archives: all 46 registered sites plus the unregistered Bandcamp archive.
-# The revision descends from eedc1a7f85118ab835eac38e04678fa85d141973, which includes the Y Combinator archive from HF PR #83 and the reviewed Versus archive from HF PR #90.
-# Drugs.com retains the corrected archive from HF #102, including 13 attributed DailyMed labels, 13 packaging images and its catalog.
-# assets-manifest.json binds every registered-site archive and the extracted managed tree.
-# The unregistered Bandcamp archive is ignored by fetch_assets.sh.
-# Tracked seed migrations/build-generation remain part of the build contract.
+# All 47 registered sites use this immutable merged HF main revision.
+# Original Discogs HF #24 then reviewed replacement #76 are merged.
+# All 50 previously existing dataset files remain byte-identical.
+# Discogs archive SHA-256: e42edef309a0fb44c2fda86d4e6f1472a3a84535013c700dabe629761e0be594
+# assets-manifest.json binds registered archives and the extracted managed tree.
+# Unregistered dataset bundles are not fetched. Tracked seed migrations still apply.
 repo: ChilleD/WebHarbor
-revision: df9a1b44a8c5639decc450706a20ac838e21716b
+revision: ebe2e47a2ce53196ab8a4b18f22b56c7082b5004

.claude/skills/clone-website/SKILL.md

@@ -204,7 +204,7 @@ Then run the site once locally to produce `instance/<site>.db`, copy it to
 ```bash
 ./scripts/build.sh webharbor:dev
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev
+  -p 8201:8101 -p 41000-41046:40000-40046 webharbor:dev
 
 # your new site is on port 41000 + its index
 curl -so /dev/null -w "%{http_code}\n" http://localhost:41000NN/
@@ -228,7 +228,7 @@ After Phase 1, you should have:
 - `sites/<your_site>/static/` with real CSS/JS/icons (and images under HF assets)
 - `sites/<your_site>/instance_seed/<site>.db` with seeded data
 - Site registered in `websyn_start.sh`, `control_server.py`, `Dockerfile`
-- All 46 sites still return 200 on the alt-port container
+- All 47 sites still return 200 on the alt-port container
 - Byte-identical reset passes
 
 ## Next step

.claude/skills/evolve-env/SKILL.md

@@ -65,15 +65,15 @@ through `BASE_DIR = os.path.dirname(os.path.abspath(__file__))`.
 After any DB-affecting change:
 
 ```bash
-# 1. Stop test container (don't touch user's working container on :40000-40045)
+# 1. Stop test container (don't touch user's working container on :40000-40046)
 docker stop wh-test 2>/dev/null || true
 
 # 2. Rebuild
 ./scripts/build.sh webharbor:dev
 
 # 3. Run on alt ports
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev
+  -p 8201:8101 -p 41000-41046:40000-40046 webharbor:dev
 
 # 4. Reset your site and confirm byte-identity
 curl -X POST http://localhost:8201/reset/<your_site>
@@ -149,7 +149,7 @@ After Phase 3:
 - Every task in `tasks.jsonl` is hand-verified to work end-to-end
 - The mirror has no obvious leaks, broken forms, or empty pages
 - Byte-identical reset passes
-- All 46 sites still return 200
+- All 47 sites still return 200
 
 ## Next step
 

.claude/skills/harden-env/SKILL.md

@@ -130,7 +130,7 @@ Hardening changes seed data. After any DB change:
 ```bash
 ./scripts/build.sh webharbor:dev
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev
+  -p 8201:8101 -p 41000-41046:40000-40046 webharbor:dev
 curl -X POST http://localhost:8201/reset/<your_site>
 docker exec wh-test md5sum \
   /opt/WebSyn/<your_site>/instance/<your_site>.db \

.claude/skills/review-env/SKILL.md

@@ -33,18 +33,18 @@ gh pr checkout <pr-number>
 ./scripts/fetch_assets.sh             # pull the pinned HF revision
 ./scripts/build.sh webharbor:dev
 docker run -d --rm --name wh-review \
-  -p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev
+  -p 8201:8101 -p 41000-41046:40000-40046 webharbor:dev
 ```
 
-Confirm the new/changed site is on the expected port (40000 + index). Note: the image now runs 46 sites (40000-40045).
+Confirm the new/changed site is on the expected port (40000 + index). Note: the image now runs 47 sites (40000-40046).
 
 ### Step 2: The mechanical checks (5 minutes)
 
 Run the same Pre-PR checks the contributor was supposed to run.
 
 ```bash
-# 1. all 46 sites return 200
-for p in $(seq 41000 41045); do
+# 1. all 47 sites return 200
+for p in $(seq 41000 41046); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 
@@ -231,7 +231,7 @@ Leave a structured comment on the PR:
 ## Review: <site_name>
 
 ### Mechanical checks: PASS / FAIL
-- [x] All 46 sites return 200
+- [x] All 47 sites return 200
 - [x] Control plane healthy
 - [x] Byte-identical reset (md5 match)
 - [x] Parallel reset <10s

.claude/skills/seed-database/SKILL.md

@@ -169,10 +169,10 @@ gh pr create
 ./scripts/check_assets.sh                       # every site has instance_seed/
 ./scripts/build.sh webharbor:dev                # docker build succeeds
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev
+  -p 8201:8101 -p 41000-41046:40000-40046 webharbor:dev
 
-# all 46 sites return 200
-for p in $(seq 41000 41045); do
+# all 47 sites return 200
+for p in $(seq 41000 41046); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 

AGENTS.md

@@ -15,7 +15,7 @@ A coding agent (Claude Code, Cursor, Aider, Codex, ...) is reading this. Read on
 
 ## What it is
 
-46 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
+47 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
 
 Two repos:
 - **code** (this one) — Flask apps, control plane, scripts.
@@ -59,17 +59,17 @@ Inside the image, sites live at `/opt/WebSyn/<site>/`. The path predates the ren
 # fresh clone
 ./scripts/fetch_assets.sh                     # pulls assets from HF
 ./scripts/build.sh                            # docker build -t webharbor:dev .
-docker run -e WEBSYN_CONTROL_TOKEN -d -p 8101:8101 -p 40000-40045:40000-40045 webharbor:dev
+docker run -e WEBSYN_CONTROL_TOKEN -d -p 8101:8101 -p 40000-40046:40000-40046 webharbor:dev
 ```
 
 Or use the published image directly:
 
 ```bash
-docker run -e WEBSYN_CONTROL_TOKEN -d -p 8101:8101 -p 40000-40045:40000-40045 \
+docker run -e WEBSYN_CONTROL_TOKEN -d -p 8101:8101 -p 40000-40046:40000-40046 \
   battalion7244/webharbor:latest
 ```
 
-Sites are on `40000`-`40045` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
+Sites are on `40000`-`40046` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
 
 | Method | Path                | Purpose                                   |
 |--------|---------------------|-------------------------------------------|
@@ -147,13 +147,13 @@ python3 -m py_compile sites/<site>/app.py
 
 # 3. run on alt ports (don't collide with anything you already have running)
 docker run -e WEBSYN_CONTROL_TOKEN -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev
+  -p 8201:8101 -p 41000-41046:40000-40046 webharbor:dev
 
 # 4. control plane healthy, all sites alive
 curl -s -H "Authorization: Bearer $WEBSYN_CONTROL_TOKEN" http://localhost:8201/health | python3 -m json.tool | head
 
 # 5. every site renders 200
-for p in $(seq 41000 41045); do
+for p in $(seq 41000 41046); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 

CLAUDE.md

@@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl
 
 ## Existing containers
 
-If a container is already running on `:8101` / `:40000-40045`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41045`).
+If a container is already running on `:8101` / `:40000-40046`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41046`).

CONTRIBUTING.md

@@ -35,7 +35,7 @@ git clone https://github.com/<you>/webharbor && cd webharbor
 ./scripts/fetch_assets.sh                       # pull current assets
 ./scripts/new_site.py mywebsite                 # OR edit an existing site
 ./scripts/build.sh && docker run -e WEBSYN_CONTROL_TOKEN -d --rm \
-  -p 8101:8101 -p 40000-40045:40000-40045 webharbor:dev
+  -p 8101:8101 -p 40000-40046:40000-40046 webharbor:dev
 # iterate locally...
 
 ./scripts/extract_assets.sh ../webharbor-static-pr/   # split assets out

Dockerfile

@@ -1,5 +1,5 @@
 # WebHarbor — slim, self-contained image.
-# 46 Flask mirror sites + control plane on :8101.
+# 47 Flask mirror sites + control plane on :8101.
 
 FROM python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254
 
@@ -156,6 +156,6 @@ RUN cd /opt/WebSyn/y_combinator && rm -rf instance instance_seed && \
 # Fail closed after all registered-site seed migrations/generators.
 RUN python3 /opt/check_seed_databases.py /opt/WebSyn
 
-EXPOSE 8101 40000-40045
+EXPOSE 8101 40000-40046
 
 CMD ["/opt/websyn_start.sh"]

README.md

@@ -34,18 +34,18 @@ WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Cod
 - **Deep features unlocked** — carts, checkouts, accounts, all fully testable
 - **Evolving** — harder tasks drive richer mirrors; the environment grows with agents
 - **RL-ready** — sub-second database resets between rollouts
-- **Community-driven** — 46 sites today, scaling to 100+ together
+- **Community-driven** — 47 sites today, scaling to 100+ together
 
 ## 🚀 Quickstart
 
 Build this checkout to run its registered web environments (published image tags may have an older registry):
 
 ```bash
 ./scripts/build.sh webharbor:dev
-docker run -e WEBSYN_CONTROL_TOKEN -p 8101:8101 -p 40000-40045:40000-40045 webharbor:dev
+docker run -e WEBSYN_CONTROL_TOKEN -p 8101:8101 -p 40000-40046:40000-40046 webharbor:dev
 ```
 
-Then point your agent at `http://localhost:40000` through `http://localhost:40045` to explore 46 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, Healthline, Kaggle, NVIDIA, UC Berkeley, B&H Photo, AccuWeather, GOV.UK, IMDb, NBA, Recreation.gov, BoardGameGeek, CarMax, BabyCenter, Amtrak, Cookpad, Craigslist, Drugs.com, Versus, Y Combinator, and PhET Interactive Simulations`.
+Then point your agent at `http://localhost:40000` through `http://localhost:40046` to explore 47 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, Healthline, Kaggle, NVIDIA, UC Berkeley, B&H Photo, AccuWeather, GOV.UK, IMDb, NBA, Recreation.gov, BoardGameGeek, CarMax, BabyCenter, Amtrak, Cookpad, Craigslist, Drugs.com, Versus, Y Combinator, PhET Interactive Simulations, and Discogs`.
 
 For sub-second reset between rollouts, expose the control plane and call `/reset/<site>`:
 
@@ -64,7 +64,7 @@ git clone https://github.com/aiming-lab/WebHarbor && cd WebHarbor
 
 ### Site registry
 
-This checkout registers **46 sites**. NVIDIA remains at index 28, UC Berkeley remains at index 29, B&H Photo remains at index 30, AccuWeather remains at index 31, GOV.UK remains at index 32, IMDb remains at index 33 and NBA remains at index 34. Recreation.gov remains at index 35, BoardGameGeek remains at index 36, CarMax remains at index 37, BabyCenter remains at index 38, Amtrak remains at index 39, Cookpad remains at index 40, Craigslist remains at index 41, Drugs.com remains at index 42, Versus remains at index 43, Y Combinator remains at index 44, and PhET Interactive Simulations is appended at index 45. Build the image from this checkout to use this registry; publishing source does not update the published Docker image automatically.
+This checkout registers **47 sites**. NVIDIA remains at index 28, UC Berkeley remains at index 29, B&H Photo remains at index 30, AccuWeather remains at index 31, GOV.UK remains at index 32, IMDb remains at index 33 and NBA remains at index 34. Recreation.gov remains at index 35, BoardGameGeek remains at index 36, CarMax remains at index 37, BabyCenter remains at index 38, Amtrak remains at index 39, Cookpad remains at index 40, Craigslist remains at index 41, Drugs.com remains at index 42, Versus remains at index 43, Y Combinator remains at index 44, PhET Interactive Simulations remains at index 45, and Discogs is appended at index 46. Build the image from this checkout to use this registry; publishing source does not update the published Docker image automatically.
 
 | Site | Registry position | Container port | Example local review host port |
 | --- | --- | --- | --- |
@@ -86,17 +86,18 @@ This checkout registers **46 sites**. NVIDIA remains at index 28, UC Berkeley re
 | Versus | 43 | 40043 | 48043 |
 | Y Combinator | 44 | 40044 | 48044 |
 | PhET Interactive Simulations | 45 | 40045 | 48045 |
+| Discogs | 46 | 40046 | 48046 |
 
 `websyn_start.sh`, `control_server.py`, the `Dockerfile` `EXPOSE` line and every
-site's `tasks.jsonl` `web` URL agree on 46 sites and `40000-40045`;
+site's `tasks.jsonl` `web` URL agree on 47 sites and `40000-40046`;
 `scripts/check_site_registry.py` (run by `scripts/check_assets.sh`) fails when they
 drift.
 
 After preparing the candidate assets and building `webharbor:dev`, the local
 review deployment uses:
 
 ```bash
-docker run -e WEBSYN_CONTROL_TOKEN -p 127.0.0.1:48080:8101 -p 127.0.0.1:48000-48045:40000-40045 webharbor:dev
+docker run -e WEBSYN_CONTROL_TOKEN -p 127.0.0.1:48080:8101 -p 127.0.0.1:48000-48046:40000-40046 webharbor:dev
 ```
 
 NVIDIA inherits the site contribution from @KaKituken
@@ -117,7 +118,7 @@ The bundle contains 78 articles and 62 structured guidance sections. That archiv
 is part of the consolidated pinned dataset revision below.
 
 
-The current `.assets-revision` consolidates all **46** registered sites at merged HF commit `df9a1b44a8c5639decc450706a20ac838e21716b`. All 46 archives at the previous global revision `eedc1a7f85118ab835eac38e04678fa85d141973`—45 registered-site archives plus the unregistered Bandcamp archive—are byte-identical; merged HF PR #86 adds `phet_simulations.tar.gz` (10,319,041 bytes, SHA-256 `7ed4831ed1218bf95456a6156f82dbd769b909fb630434e042f6e6cd1987a65f`, 150 managed members). The previous revision includes `y_combinator.tar.gz` from merged HF PR #83 and the reviewed Versus archive from HF PR #90. Drugs.com retains its 13 attributed DailyMed labels and packaging images from HF #101 with the archive-root correction from [HF #102](https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/102). `assets-manifest.json` binds all selected archive hashes and the extracted managed tree. The unregistered Bandcamp archive is not fetched. Seed migrations and build-generated seeds remain part of the build contract.
+The current `.assets-revision` pins all **47** registered sites to merged HF commit `ebe2e47a2ce53196ab8a4b18f22b56c7082b5004`. Original Discogs [HF #24](https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/24) and reviewed replacement [HF #76](https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/76) are merged in order. All 50 previous dataset files remain byte-identical. The reviewed Discogs archive has SHA-256 `e42edef309a0fb44c2fda86d4e6f1472a3a84535013c700dabe629761e0be594`. `assets-manifest.json` binds all selected archives and the extracted managed tree. Unregistered bundles are not fetched; tracked seed migrations and generation remain part of the build contract.
 
 Historical asset integration notes below describe superseded pins, not the current pin.
 

agent_demo/README.md

@@ -30,7 +30,7 @@ export OPENAI_BASE_URL=https://api.openai.com/v1   # or your Azure / vLLM endpoi
 
 ## Run a task
 
-WebHarbor must already be running locally (`docker run -e WEBSYN_CONTROL_TOKEN -p 8101:8101 -p 40000-40045:40000-40045 battalion7244/webharbor:latest`).
+WebHarbor must already be running locally (`docker run -e WEBSYN_CONTROL_TOKEN -p 8101:8101 -p 40000-40046:40000-40046 battalion7244/webharbor:latest`).
 
 Run a single task from a site's `tasks.jsonl`:
 

assets-manifest.json

@@ -72,6 +72,10 @@
       "bytes": 16967864,
       "sha256": "741ef266077cc56c082e5b7dd2df0e43ffa81f7a1c297bb55ac232aabe9e5f5d"
     },
+    "discogs.tar.gz": {
+      "bytes": 111903367,
+      "sha256": "e42edef309a0fb44c2fda86d4e6f1472a3a84535013c700dabe629761e0be594"
+    },
     "drugs_com.tar.gz": {
       "bytes": 2092340,
       "sha256": "8a9924e005d0a8a9f793f9e67bceeba0c4f2dbbe10ccf7f1afeb0a0198427b1b"
@@ -185,8 +189,8 @@
       "sha256": "43c666bda84680f327ce64720b2baa2f9e427acb751fe81900967f267e9176f4"
     }
   },
-  "managed_tree_sha256": "ae41eb3d3e039f8ce9990d33c61a27ca18c16b45d6bbd7af5765295130c5bdbf",
+  "managed_tree_sha256": "ab84cee0be1e68d42962a43eceac08e044e9ecd5054df8b90fd35e1a025fcffe",
   "repo": "ChilleD/WebHarbor",
-  "revision": "df9a1b44a8c5639decc450706a20ac838e21716b",
+  "revision": "ebe2e47a2ce53196ab8a4b18f22b56c7082b5004",
   "version": 1
 }

control_server.py

@@ -38,7 +38,7 @@
     'imdb',
     'nba', 'recreation_gov', 'boardgamegeek', 'carmax', 'babycenter', 'amtrak',
     'cookpad', 'craigslist', 'drugs_com', 'versus', 'y_combinator',
-    'phet_simulations',
+    'phet_simulations', 'discogs',
 ]
 BASE_PORT = 40000
 WEBSYN_DIR = '/opt/WebSyn'