Review: validate IMDb mirror, sourced seed and task grading

.assets-revision

@@ -1,21 +1,24 @@
 # Hugging Face dataset holding WebHarbor's heavy static assets.
 # Pin an immutable commit SHA; never a moving branch or refs/pr/<n>.
 #
-# Merged dataset-main commit from HF asset PR #92.
-# https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/92
-# Its complete asset tree matches the tested candidate commit
-# f09e586eec8bf1bca0bc0881e08b77f3c2a5508e.
+# Merged dataset-main commit from HF asset PR #57 (IMDb review, #33).
+# https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/57
 #
-# Adds bh_photo.tar.gz: 79,658,793 bytes, 511 managed archive members.
-# SHA-256: 867363d5484eb114d647e236991017992d5ac91ae3415996ad43bf654d99bd9a
-# All 32 pre-existing archives are unchanged from the previous main pin
-# c32018ca3b3d67e7b858b1b85fb101aea5090cd7 (Berkeley, HF PR #91).
-# This revision has archives for the 31 sites registered before AccuWeather.
-# The two additional archives (bandcamp and drugs_com) are ignored by fetch_assets.sh.
+# Adds imdb.tar.gz: 39,111,545 bytes, LFS oid
+# 0663d30fe90ed2a0659dfe78ecf7ab1718970f565a64c6855a2c6aaadc606513,
+# and gov_uk.tar.gz: 53,891 bytes, LFS oid
+# 1c483241887bde14e3b629ff4acc10d5f54b46283c22bf8abfde2794360743fd.
+# The 33 archives present at the previous main pin
+# fa1e8a5b9e8e5d0e42764cd658825f4dea088d8f (B&H Photo, HF PR #92) are byte-for-byte
+# unchanged, as are the archives from c32018ca3b3d67e7b858b1b85fb101aea5090cd7
+# (Berkeley, HF PR #91).
+# This revision carries an archive for every registered site except accuweather,
+# which is pinned per-site below.
+# The two further archives (bandcamp and drugs_com) are ignored by fetch_assets.sh.
 # B&H's bundle contains images and external cache; Docker generates its seed
 # from tracked catalog data (sites/bh_photo/.build-generated-seed).
 repo: ChilleD/WebHarbor
-revision: fa1e8a5b9e8e5d0e42764cd658825f4dea088d8f
+revision: e8f59470b76b95d607f288958186cb9b73681d9e
 
 # AccuWeather HF PR #66; immutable archive validated in the task review.
 # Other sites retain their merged and validated asset revision above.

.claude/skills/clone-website/SKILL.md

@@ -204,7 +204,7 @@ Then run the site once locally to produce `instance/<site>.db`, copy it to
 ```bash
 ./scripts/build.sh webharbor:dev
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
+  -p 8201:8101 -p 41000-41033:40000-40033 webharbor:dev
 
 # your new site is on port 41000 + its index
 curl -so /dev/null -w "%{http_code}\n" http://localhost:41000NN/
@@ -228,7 +228,7 @@ After Phase 1, you should have:
 - `sites/<your_site>/static/` with real CSS/JS/icons (and images under HF assets)
 - `sites/<your_site>/instance_seed/<site>.db` with seeded data
 - Site registered in `websyn_start.sh`, `control_server.py`, `Dockerfile`
-- All 30 sites still return 200 on the alt-port container
+- All 34 sites still return 200 on the alt-port container
 - Byte-identical reset passes
 
 ## Next step

.claude/skills/evolve-env/SKILL.md

@@ -65,15 +65,15 @@ through `BASE_DIR = os.path.dirname(os.path.abspath(__file__))`.
 After any DB-affecting change:
 
 ```bash
-# 1. Stop test container (don't touch user's working container on :40000-40029)
+# 1. Stop test container (don't touch user's working container on :40000-40033)
 docker stop wh-test 2>/dev/null || true
 
 # 2. Rebuild
 ./scripts/build.sh webharbor:dev
 
 # 3. Run on alt ports
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
+  -p 8201:8101 -p 41000-41033:40000-40033 webharbor:dev
 
 # 4. Reset your site and confirm byte-identity
 curl -X POST http://localhost:8201/reset/<your_site>
@@ -149,7 +149,7 @@ After Phase 3:
 - Every task in `tasks.jsonl` is hand-verified to work end-to-end
 - The mirror has no obvious leaks, broken forms, or empty pages
 - Byte-identical reset passes
-- All 30 sites still return 200
+- All 34 sites still return 200
 
 ## Next step
 

.claude/skills/harden-env/SKILL.md

@@ -130,7 +130,7 @@ Hardening changes seed data. After any DB change:
 ```bash
 ./scripts/build.sh webharbor:dev
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
+  -p 8201:8101 -p 41000-41033:40000-40033 webharbor:dev
 curl -X POST http://localhost:8201/reset/<your_site>
 docker exec wh-test md5sum \
   /opt/WebSyn/<your_site>/instance/<your_site>.db \

.claude/skills/review-env/SKILL.md

@@ -33,18 +33,18 @@ gh pr checkout <pr-number>
 ./scripts/fetch_assets.sh             # pull the pinned HF revision
 ./scripts/build.sh webharbor:dev
 docker run -d --rm --name wh-review \
-  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
+  -p 8201:8101 -p 41000-41033:40000-40033 webharbor:dev
 ```
 
-Confirm the new/changed site is on the expected port (40000 + index). Note: the image now runs 30 sites (40000-40029).
+Confirm the new/changed site is on the expected port (40000 + index). Note: the image now runs 34 sites (40000-40033).
 
 ### Step 2: The mechanical checks (5 minutes)
 
 Run the same Pre-PR checks the contributor was supposed to run.
 
 ```bash
-# 1. all 30 sites return 200
-for p in $(seq 41000 41029); do
+# 1. all 34 sites return 200
+for p in $(seq 41000 41033); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 
@@ -231,7 +231,7 @@ Leave a structured comment on the PR:
 ## Review: <site_name>
 
 ### Mechanical checks: PASS / FAIL
-- [x] All 30 sites return 200
+- [x] All 34 sites return 200
 - [x] Control plane healthy
 - [x] Byte-identical reset (md5 match)
 - [x] Parallel reset <10s

.claude/skills/seed-database/SKILL.md

@@ -169,10 +169,10 @@ gh pr create
 ./scripts/check_assets.sh                       # every site has instance_seed/
 ./scripts/build.sh webharbor:dev                # docker build succeeds
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
+  -p 8201:8101 -p 41000-41033:40000-40033 webharbor:dev
 
-# all 30 sites return 200
-for p in $(seq 41000 41029); do
+# all 34 sites return 200
+for p in $(seq 41000 41033); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 

AGENTS.md

@@ -4,7 +4,7 @@ A coding agent (Claude Code, Cursor, Aider, Codex, ...) is reading this. Read on
 
 ## What it is
 
-33 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
+34 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
 
 Two repos:
 - **code** (this one) — Flask apps, control plane, scripts.
@@ -48,17 +48,17 @@ Inside the image, sites live at `/opt/WebSyn/<site>/`. The path predates the ren
 # fresh clone
 ./scripts/fetch_assets.sh                     # pulls assets from HF
 ./scripts/build.sh                            # docker build -t webharbor:dev .
-docker run -d -p 8101:8101 -p 40000-40032:40000-40032 webharbor:dev
+docker run -d -p 8101:8101 -p 40000-40033:40000-40033 webharbor:dev
 ```
 
 Or use the published image directly:
 
 ```bash
-docker run -d -p 8101:8101 -p 40000-40032:40000-40032 \
+docker run -d -p 8101:8101 -p 40000-40033:40000-40033 \
   battalion7244/webharbor:latest
 ```
 
-Sites are on `40000`-`40032` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
+Sites are on `40000`-`40033` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
 
 | Method | Path                | Purpose                                   |
 |--------|---------------------|-------------------------------------------|
@@ -136,13 +136,13 @@ python3 -m py_compile sites/<site>/app.py
 
 # 3. run on alt ports (don't collide with anything you already have running)
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41032:40000-40032 webharbor:dev
+  -p 8201:8101 -p 41000-41033:40000-40033 webharbor:dev
 
 # 4. control plane healthy, all sites alive
 curl -s http://localhost:8201/health | python3 -m json.tool | head
 
 # 5. every site renders 200
-for p in $(seq 41000 41032); do
+for p in $(seq 41000 41033); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 

CLAUDE.md

@@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl
 
 ## Existing containers
 
-If a container is already running on `:8101` / `:40000-40032`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41032`).
+If a container is already running on `:8101` / `:40000-40033`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41033`).

CONTRIBUTING.md

@@ -24,7 +24,7 @@ git clone https://github.com/<you>/webharbor && cd webharbor
 ./scripts/fetch_assets.sh                       # pull current assets
 ./scripts/new_site.py mywebsite                 # OR edit an existing site
 ./scripts/build.sh && docker run -d --rm \
-  -p 8101:8101 -p 40000-40031:40000-40031 webharbor:dev
+  -p 8101:8101 -p 40000-40033:40000-40033 webharbor:dev
 # iterate locally...
 
 ./scripts/extract_assets.sh ../webharbor-static-pr/   # split assets out

Dockerfile

@@ -1,5 +1,5 @@
 # WebHarbor — slim, self-contained image.
-# 33 Flask mirror sites + control plane on :8101.
+# 34 Flask mirror sites + control plane on :8101.
 
 FROM python:3.12-slim-bookworm
 
@@ -122,6 +122,6 @@ os.makedirs('instance_seed', exist_ok=True); \
 shutil.copy2('instance/accuweather.db', 'instance_seed/accuweather.db'); \
 print('AccuWeather seed DB generated at build time.')" && rm -rf /opt/WebSyn/accuweather/instance
 
-EXPOSE 8101 40000-40032
+EXPOSE 8101 40000-40033
 
 CMD ["/opt/websyn_start.sh"]

README.md

@@ -36,17 +36,17 @@ WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Cod
 - **Deep features unlocked** — carts, checkouts, accounts, all fully testable
 - **Evolving** — harder tasks drive richer mirrors; the environment grows with agents
 - **RL-ready** — sub-second database resets between rollouts
-- **Community-driven** — 33 sites today, scaling to 100+ together
+- **Community-driven** — 34 sites today, scaling to 100+ together
 
 ## 🚀 Quickstart
 
 One command to run all web environments:
 
 ```bash
-docker run -p 8101:8101 -p 40000-40032:40000-40032 battalion7244/webharbor:latest
+docker run -p 8101:8101 -p 40000-40033:40000-40033 battalion7244/webharbor:latest
 ```
 
-Then point your agent at `http://localhost:40000` through `http://localhost:40032` to explore 33 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, Healthline, Kaggle, NVIDIA, UC Berkeley, B&H Photo, AccuWeather, and GOV.UK`.
+Then point your agent at `http://localhost:40000` through `http://localhost:40033` to explore 34 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, Healthline, Kaggle, NVIDIA, UC Berkeley, B&H Photo, AccuWeather, GOV.UK, and IMDb`.
 
 For sub-second reset between rollouts, expose the control plane and call `/reset/<site>`:
 
@@ -65,9 +65,10 @@ git clone https://github.com/aiming-lab/WebHarbor && cd WebHarbor
 
 ### Site registry
 
-This checkout registers **33 sites**. NVIDIA remains at index 28, UC Berkeley
-remains at index 29, B&H Photo remains at index 30, and AccuWeather is appended
-at index 31. Build the image from
+This checkout registers **34 sites**. NVIDIA remains at index 28, UC Berkeley
+remains at index 29, B&H Photo remains at index 30, AccuWeather remains at index
+31 and GOV.UK remains at index 32, so IMDb (the site under review here) is the
+last entry, registry index 33. Build the image from
 this checkout to use this registry; publishing source does not update the
 published Docker image automatically.
 
@@ -78,17 +79,18 @@ published Docker image automatically.
 | B&H Photo | 30 | 40030 | 48030 |
 | AccuWeather | 31 | 40031 | 48031 |
 | GOV.UK | 32 | 40032 | 48032 |
+| IMDb | 33 | 40033 | 48033 |
 
 `websyn_start.sh`, `control_server.py`, the `Dockerfile` `EXPOSE` line and every
-site's `tasks.jsonl` `web` URL agree on 33 sites and `40000-40032`;
+site's `tasks.jsonl` `web` URL agree on 34 sites and `40000-40033`;
 `scripts/check_site_registry.py` (run by `scripts/check_assets.sh`) fails when they
 drift.
 
 After preparing the candidate assets and building `webharbor:dev`, the local
 review deployment uses:
 
 ```bash
-docker run -p 127.0.0.1:48080:8101 -p 127.0.0.1:48000-48032:40000-40032 webharbor:dev
+docker run -p 127.0.0.1:48080:8101 -p 127.0.0.1:48000-48033:40000-40033 webharbor:dev
 ```
 
 NVIDIA inherits the site contribution from @KaKituken
@@ -100,25 +102,28 @@ passed.
 
 ### Asset delivery status
 
-GOV.UK is registered at index 32 / port 40032 in builds of this source revision.
+GOV.UK is registered at index 32 / port 40032 and IMDb at index 33 / port 40033 in
+builds of this source revision.
 The published Docker Hub image is updated in a separate release. Its reviewed seed uses the
 immutable per-site commit `7c4daf7a6714654c609a9ccf97ab3b2431381791` from
 [HF asset PR #93](https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/93).
-The bundle contains 78 articles and 62 structured guidance sections. The HF PR
-is awaiting maintainer merge; the immutable pin downloads the reviewed bundle
-without changing other sites’ asset revisions.
-
-
-`.assets-revision` pins the merged dataset commit `fa1e8a5b9e8e5d0e42764cd658825f4dea088d8f`
-from [HF asset PR #92](https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/92).
-Its complete asset tree matches the tested candidate commit
-`f09e586eec8bf1bca0bc0881e08b77f3c2a5508e`.
-
-This revision adds `bh_photo.tar.gz` and preserves all 32 existing archives from
-`c32018ca3b3d67e7b858b1b85fb101aea5090cd7` byte-for-byte, including Berkeley and
-NVIDIA. It carries 33 archives for 31 registered sites plus the unregistered
-Bandcamp and Drugs.com archives, which `fetch_assets.sh` ignores. A clean asset
-fetch downloaded and extracted all 31 registered sites successfully.
+The bundle contains 78 articles and 62 structured guidance sections. That archive
+is now part of the pinned dataset revision, and the per-site pin below resolves to
+the same reviewed bundle.
+
+
+`.assets-revision` pins the merged dataset commit `e8f59470b76b95d607f288958186cb9b73681d9e`
+from [HF asset PR #57](https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/57).
+
+This revision adds `imdb.tar.gz` (39,111,545 bytes) and `gov_uk.tar.gz`
+(53,891 bytes); all 33 archives present at
+`fa1e8a5b9e8e5d0e42764cd658825f4dea088d8f` are byte-for-byte unchanged, including
+Berkeley, NVIDIA, B&H Photo and GOV.UK. It carries archives for 33 of the 34
+registered sites; AccuWeather uses its own immutable per-site pin
+`0a73c1c1ac2e47513389a8a1a67601f75c8c4150` in the same file. The remaining entries
+are the unregistered Bandcamp and Drugs.com archives, which `fetch_assets.sh`
+ignores. A clean asset fetch downloaded and extracted all 34 registered sites
+successfully and validated 4,536 managed members for IMDb.
 
 B&H's archive contains images and external cache. The Docker build validates
 its 508 declared assets and generates `instance_seed/bh_photo.db` from the tracked

agent_demo/README.md

@@ -19,7 +19,7 @@ export OPENAI_BASE_URL=https://api.openai.com/v1   # or your Azure / vLLM endpoi
 
 ## Run a task
 
-WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40031:40000-40031 battalion7244/webharbor:latest`).
+WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40033:40000-40033 battalion7244/webharbor:latest`).
 
 Run a single task from a site's `tasks.jsonl`:
 

control_server.py

@@ -31,6 +31,7 @@
     'osu', 'rotten_tomatoes', 'compass', 'walmart_careers',
     'fedex', 'webmd_doctor', 'healthline', 'kaggle',
     'nvidia', 'berkeley', 'bh_photo', 'accuweather', 'gov_uk',
+    'imdb',
 ]
 BASE_PORT = 40000
 WEBSYN_DIR = '/opt/WebSyn'