Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .assets-revision
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
# Hugging Face dataset holding WebHarbor's heavy static assets.
# Pin an immutable merged commit SHA, never a moving branch or refs/pr/<n>.
#
# All 45 registered sites use this immutable dataset-main revision.
# This pin covers the previous 45 registered sites. MEGA is registered in
# this checkout at index 45 / port 40045, but mega.tar.gz is not on this
# revision (HF dataset PR #87 is still open). Do not bump the pin until that
# PR is merged and the combined archive set is re-validated.
# The previous global revision f3ecd60ea35795eef566ee19d67140c16b200f26 contains 45 archives: 44 registered-site archives plus the unregistered Bandcamp archive. All 45 are byte-for-byte unchanged here.
# Merged HF PR #83 adds y_combinator.tar.gz: 121,085,653 bytes, SHA-256 43c666bda84680f327ce64720b2baa2f9e427acb751fe81900967f267e9176f4, with 3,973 managed archive members and 3,971 inventoried runtime assets.
# This revision descends from f3ecd60ea35795eef566ee19d67140c16b200f26, which includes the Versus archive added after 555a9aa0b02946a8bdf873ba1d59902b71564a07.
# Drugs.com retains the corrected archive from HF #102, including 13 attributed DailyMed labels, 13 packaging images and its catalog.
# assets-manifest.json binds every registered-site archive and the extracted managed tree.
# assets-manifest.json binds every archive on this pin and the extracted managed tree.
# The unregistered Bandcamp archive is ignored by fetch_assets.sh.
# Tracked seed migrations/build-generation remain part of the build contract.
repo: ChilleD/WebHarbor
Expand Down
4 changes: 2 additions & 2 deletions .claude/skills/clone-website/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ Then run the site once locally to produce `instance/<site>.db`, copy it to
```bash
./scripts/build.sh webharbor:dev
docker run -d --rm --name wh-test \
-p 8201:8101 -p 41000-41044:40000-40044 webharbor:dev
-p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev

# your new site is on port 41000 + its index
curl -so /dev/null -w "%{http_code}\n" http://localhost:41000NN/
Expand All @@ -228,7 +228,7 @@ After Phase 1, you should have:
- `sites/<your_site>/static/` with real CSS/JS/icons (and images under HF assets)
- `sites/<your_site>/instance_seed/<site>.db` with seeded data
- Site registered in `websyn_start.sh`, `control_server.py`, `Dockerfile`
- All 45 sites still return 200 on the alt-port container
- All 46 sites still return 200 on the alt-port container
- Byte-identical reset passes

## Next step
Expand Down
6 changes: 3 additions & 3 deletions .claude/skills/evolve-env/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,15 +65,15 @@ through `BASE_DIR = os.path.dirname(os.path.abspath(__file__))`.
After any DB-affecting change:

```bash
# 1. Stop test container (don't touch user's working container on :40000-40044)
# 1. Stop test container (don't touch user's working container on :40000-40045)
docker stop wh-test 2>/dev/null || true

# 2. Rebuild
./scripts/build.sh webharbor:dev

# 3. Run on alt ports
docker run -d --rm --name wh-test \
-p 8201:8101 -p 41000-41044:40000-40044 webharbor:dev
-p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev

# 4. Reset your site and confirm byte-identity
curl -X POST http://localhost:8201/reset/<your_site>
Expand Down Expand Up @@ -149,7 +149,7 @@ After Phase 3:
- Every task in `tasks.jsonl` is hand-verified to work end-to-end
- The mirror has no obvious leaks, broken forms, or empty pages
- Byte-identical reset passes
- All 45 sites still return 200
- All 46 sites still return 200

## Next step

Expand Down
2 changes: 1 addition & 1 deletion .claude/skills/harden-env/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ Hardening changes seed data. After any DB change:
```bash
./scripts/build.sh webharbor:dev
docker run -d --rm --name wh-test \
-p 8201:8101 -p 41000-41044:40000-40044 webharbor:dev
-p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev
curl -X POST http://localhost:8201/reset/<your_site>
docker exec wh-test md5sum \
/opt/WebSyn/<your_site>/instance/<your_site>.db \
Expand Down
8 changes: 4 additions & 4 deletions .claude/skills/review-env/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,17 @@ gh pr checkout <pr-number>
./scripts/fetch_assets.sh # pull the pinned HF revision
./scripts/build.sh webharbor:dev
docker run -d --rm --name wh-review \
-p 8201:8101 -p 41000-41044:40000-40044 webharbor:dev
-p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev
```

Confirm the new/changed site is on the expected port (40000 + index). Note: the image now runs 45 sites (40000-40044).
Confirm the new/changed site is on the expected port (40000 + index). Note: the image now runs 46 sites (40000-40045).

### Step 2: The mechanical checks (5 minutes)

Run the same Pre-PR checks the contributor was supposed to run.

```bash
# 1. all 45 sites return 200
# 1. all 46 sites return 200
for p in $(seq 41000 41044); do
curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
done
Expand Down Expand Up @@ -231,7 +231,7 @@ Leave a structured comment on the PR:
## Review: <site_name>

### Mechanical checks: PASS / FAIL
- [x] All 45 sites return 200
- [x] All 46 sites return 200
- [x] Control plane healthy
- [x] Byte-identical reset (md5 match)
- [x] Parallel reset <10s
Expand Down
4 changes: 2 additions & 2 deletions .claude/skills/seed-database/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,9 +169,9 @@ gh pr create
./scripts/check_assets.sh # every site has instance_seed/
./scripts/build.sh webharbor:dev # docker build succeeds
docker run -d --rm --name wh-test \
-p 8201:8101 -p 41000-41044:40000-40044 webharbor:dev
-p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev

# all 45 sites return 200
# all 46 sites return 200
for p in $(seq 41000 41044); do
curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
done
Expand Down
12 changes: 6 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ A coding agent (Claude Code, Cursor, Aider, Codex, ...) is reading this. Read on

## What it is

45 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
46 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.

Two repos:
- **code** (this one) — Flask apps, control plane, scripts.
Expand Down Expand Up @@ -59,17 +59,17 @@ Inside the image, sites live at `/opt/WebSyn/<site>/`. The path predates the ren
# fresh clone
./scripts/fetch_assets.sh # pulls assets from HF
./scripts/build.sh # docker build -t webharbor:dev .
docker run -e WEBSYN_CONTROL_TOKEN -d -p 8101:8101 -p 40000-40044:40000-40044 webharbor:dev
docker run -e WEBSYN_CONTROL_TOKEN -d -p 8101:8101 -p 40000-40045:40000-40045 webharbor:dev
```

Or use the published image directly:

```bash
docker run -e WEBSYN_CONTROL_TOKEN -d -p 8101:8101 -p 40000-40044:40000-40044 \
docker run -e WEBSYN_CONTROL_TOKEN -d -p 8101:8101 -p 40000-40045:40000-40045 \
battalion7244/webharbor:latest
```

Sites are on `40000`-`40044` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
Sites are on `40000`-`40045` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:

| Method | Path | Purpose |
|--------|---------------------|-------------------------------------------|
Expand Down Expand Up @@ -147,13 +147,13 @@ python3 -m py_compile sites/<site>/app.py

# 3. run on alt ports (don't collide with anything you already have running)
docker run -e WEBSYN_CONTROL_TOKEN -d --rm --name wh-test \
-p 8201:8101 -p 41000-41044:40000-40044 webharbor:dev
-p 8201:8101 -p 41000-41045:40000-40045 webharbor:dev

# 4. control plane healthy, all sites alive
curl -s -H "Authorization: Bearer $WEBSYN_CONTROL_TOKEN" http://localhost:8201/health | python3 -m json.tool | head

# 5. every site renders 200
for p in $(seq 41000 41044); do
for p in $(seq 41000 41045); do
curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
done

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl

## Existing containers

If a container is already running on `:8101` / `:40000-40044`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41044`).
If a container is already running on `:8101` / `:40000-40045`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41045`).
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ git clone https://github.com/<you>/webharbor && cd webharbor
./scripts/fetch_assets.sh # pull current assets
./scripts/new_site.py mywebsite # OR edit an existing site
./scripts/build.sh && docker run -e WEBSYN_CONTROL_TOKEN -d --rm \
-p 8101:8101 -p 40000-40044:40000-40044 webharbor:dev
-p 8101:8101 -p 40000-40045:40000-40045 webharbor:dev
# iterate locally...

./scripts/extract_assets.sh ../webharbor-static-pr/ # split assets out
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# WebHarbor — slim, self-contained image.
# 45 Flask mirror sites + control plane on :8101.
# 46 Flask mirror sites + control plane on :8101.

FROM python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254

Expand Down Expand Up @@ -156,6 +156,6 @@ RUN cd /opt/WebSyn/y_combinator && rm -rf instance instance_seed && \
# Fail closed after all registered-site seed migrations/generators.
RUN python3 /opt/check_seed_databases.py /opt/WebSyn

EXPOSE 8101 40000-40044
EXPOSE 8101 40000-40045

CMD ["/opt/websyn_start.sh"]
17 changes: 10 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,18 +47,18 @@ WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Cod
- **Deep features unlocked** — carts, checkouts, accounts, all fully testable
- **Evolving** — harder tasks drive richer mirrors; the environment grows with agents
- **RL-ready** — sub-second database resets between rollouts
- **Community-driven** — 45 sites today, scaling to 100+ together
- **Community-driven** — 46 sites today, scaling to 100+ together

## 🚀 Quickstart

Build this checkout to run its registered web environments (published image tags may have an older registry):

```bash
./scripts/build.sh webharbor:dev
docker run -e WEBSYN_CONTROL_TOKEN -p 8101:8101 -p 40000-40044:40000-40044 webharbor:dev
docker run -e WEBSYN_CONTROL_TOKEN -p 8101:8101 -p 40000-40045:40000-40045 webharbor:dev
```

Then point your agent at `http://localhost:40000` through `http://localhost:40044` to explore 45 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, Healthline, Kaggle, NVIDIA, UC Berkeley, B&H Photo, AccuWeather, GOV.UK, IMDb, NBA, Recreation.gov, BoardGameGeek, CarMax, BabyCenter, Amtrak, Cookpad, Craigslist, Drugs.com, Versus, and Y Combinator`.
Then point your agent at `http://localhost:40000` through `http://localhost:40045` to explore 46 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, Healthline, Kaggle, NVIDIA, UC Berkeley, B&H Photo, AccuWeather, GOV.UK, IMDb, NBA, Recreation.gov, BoardGameGeek, CarMax, BabyCenter, Amtrak, Cookpad, Craigslist, Drugs.com, Versus, Y Combinator, and MEGA`.

For sub-second reset between rollouts, expose the control plane and call `/reset/<site>`:

Expand All @@ -77,7 +77,7 @@ git clone https://github.com/aiming-lab/WebHarbor && cd WebHarbor

### Site registry

This checkout registers **45 sites**. NVIDIA remains at index 28, UC Berkeley remains at index 29, B&H Photo remains at index 30, AccuWeather remains at index 31, GOV.UK remains at index 32, IMDb remains at index 33 and NBA remains at index 34. Recreation.gov remains at index 35, BoardGameGeek remains at index 36, CarMax remains at index 37, BabyCenter remains at index 38, Amtrak remains at index 39, Cookpad remains at index 40, Craigslist remains at index 41, Drugs.com remains at index 42, Versus remains at index 43, and Y Combinator is appended at index 44. Build the image from this checkout to use this registry; publishing source does not update the published Docker image automatically.
This checkout registers **46 sites**. NVIDIA remains at index 28, UC Berkeley remains at index 29, B&H Photo remains at index 30, AccuWeather remains at index 31, GOV.UK remains at index 32, IMDb remains at index 33 and NBA remains at index 34. Recreation.gov remains at index 35, BoardGameGeek remains at index 36, CarMax remains at index 37, BabyCenter remains at index 38, Amtrak remains at index 39, Cookpad remains at index 40, Craigslist remains at index 41, Drugs.com remains at index 42, Versus remains at index 43, Y Combinator remains at index 44, and MEGA is appended at index 45. Build the image from this checkout to use this registry; publishing source does not update the published Docker image automatically.

| Site | Registry position | Container port | Example local review host port |
| --- | --- | --- | --- |
Expand All @@ -98,17 +98,18 @@ This checkout registers **45 sites**. NVIDIA remains at index 28, UC Berkeley re
| Drugs.com | 42 | 40042 | 48042 |
| Versus | 43 | 40043 | 48043 |
| Y Combinator | 44 | 40044 | 48044 |
| MEGA | 45 | 40045 | 48045 |

`websyn_start.sh`, `control_server.py`, the `Dockerfile` `EXPOSE` line and every
site's `tasks.jsonl` `web` URL agree on 45 sites and `40000-40044`;
site's `tasks.jsonl` `web` URL agree on 46 sites and `40000-40045`;
`scripts/check_site_registry.py` (run by `scripts/check_assets.sh`) fails when they
drift.

After preparing the candidate assets and building `webharbor:dev`, the local
review deployment uses:

```bash
docker run -e WEBSYN_CONTROL_TOKEN -p 127.0.0.1:48080:8101 -p 127.0.0.1:48000-48044:40000-40044 webharbor:dev
docker run -e WEBSYN_CONTROL_TOKEN -p 127.0.0.1:48080:8101 -p 127.0.0.1:48000-48045:40000-40045 webharbor:dev
```

NVIDIA inherits the site contribution from @KaKituken
Expand All @@ -129,7 +130,9 @@ The bundle contains 78 articles and 62 structured guidance sections. That archiv
is part of the consolidated pinned dataset revision below.


The current `.assets-revision` consolidates all **45** registered sites at merged HF commit `eedc1a7f85118ab835eac38e04678fa85d141973`. All 45 archives from the previous global revision `f3ecd60ea35795eef566ee19d67140c16b200f26`—44 registered-site archives plus the unregistered Bandcamp archive—are byte-identical; merged HF PR #83 adds `y_combinator.tar.gz` (121,085,653 bytes, SHA-256 `43c666bda84680f327ce64720b2baa2f9e427acb751fe81900967f267e9176f4`). The previous revision already includes the reviewed Versus archive from HF PR #90. Drugs.com retains its 13 attributed DailyMed labels and packaging images from HF #101 with the archive-root correction from [HF #102](https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/102). `assets-manifest.json` binds all selected archive hashes and the extracted tree. The unregistered Bandcamp archive is not fetched. Seed migrations and build-generated seeds remain part of the build contract.
The current `.assets-revision` still pins merged HF commit `eedc1a7f85118ab835eac38e04678fa85d141973`, which contains archives for the previous **45** registered sites. All 45 archives from the previous global revision `f3ecd60ea35795eef566ee19d67140c16b200f26`—44 registered-site archives plus the unregistered Bandcamp archive—are byte-identical; merged HF PR #83 adds `y_combinator.tar.gz` (121,085,653 bytes, SHA-256 `43c666bda84680f327ce64720b2baa2f9e427acb751fe81900967f267e9176f4`). The previous revision already includes the reviewed Versus archive from HF PR #90. Drugs.com retains its 13 attributed DailyMed labels and packaging images from HF #101 with the archive-root correction from [HF #102](https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/102). `assets-manifest.json` binds all selected archive hashes and the extracted tree. The unregistered Bandcamp archive is not fetched. Seed migrations and build-generated seeds remain part of the build contract.

MEGA is registered at index 45 / port 40045, but `.assets-revision` is **not** bumped: `mega.tar.gz` is still waiting on open [HF dataset PR #87](https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/87). `scripts/fetch_assets.sh` / `scripts/check_assets.sh` will fail for `sites/mega` until that PR is merged onto `ChilleD/WebHarbor` `main` and the pin is updated to the immutable merge SHA. Do not pin an unmerged discussion commit.

Historical asset integration notes below describe superseded pins, not the current pin.

Expand Down
2 changes: 1 addition & 1 deletion agent_demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ export OPENAI_BASE_URL=https://api.openai.com/v1 # or your Azure / vLLM endpoi

## Run a task

WebHarbor must already be running locally (`docker run -e WEBSYN_CONTROL_TOKEN -p 8101:8101 -p 40000-40044:40000-40044 battalion7244/webharbor:latest`).
WebHarbor must already be running locally (`docker run -e WEBSYN_CONTROL_TOKEN -p 8101:8101 -p 40000-40045:40000-40045 battalion7244/webharbor:latest`).

Run a single task from a site's `tasks.jsonl`:

Expand Down
2 changes: 1 addition & 1 deletion control_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
'nvidia', 'berkeley', 'bh_photo', 'accuweather', 'gov_uk',
'imdb',
'nba', 'recreation_gov', 'boardgamegeek', 'carmax', 'babycenter', 'amtrak',
'cookpad', 'craigslist', 'drugs_com', 'versus', 'y_combinator',
'cookpad', 'craigslist', 'drugs_com', 'versus', 'y_combinator', 'mega',
]
BASE_PORT = 40000
WEBSYN_DIR = '/opt/WebSyn'
Expand Down
13 changes: 7 additions & 6 deletions sites/drugs_com/tests/test_integration.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"target", "ted", "osu", "rotten_tomatoes", "compass", "walmart_careers",
"fedex", "webmd_doctor", "healthline", "kaggle", "nvidia", "berkeley",
"bh_photo", "accuweather", "gov_uk", "imdb", "nba", "recreation_gov",
"boardgamegeek", "carmax", "babycenter", "amtrak", "cookpad", "craigslist", "drugs_com", "versus", "y_combinator",
"boardgamegeek", "carmax", "babycenter", "amtrak", "cookpad", "craigslist", "drugs_com", "versus", "y_combinator", "mega",
]


Expand All @@ -51,14 +51,15 @@ def load_control_server():
return module


def test_exact_45_site_registry_and_ports():
def test_exact_46_site_registry_and_ports():
assert shell_sites() == control_sites() == EXPECTED
assert EXPECTED.index("rotten_tomatoes") + 40000 == 40021
assert EXPECTED.index("compass") + 40000 == 40022
assert EXPECTED.index("walmart_careers") + 40000 == 40023
assert EXPECTED.index("drugs_com") + 40000 == 40042
assert EXPECTED.index("versus") + 40000 == 40043
assert EXPECTED.index("y_combinator") + 40000 == 40044
assert EXPECTED.index("mega") + 40000 == 40045


def test_task_manifest_uses_port_40042_and_complete_verifiers():
Expand All @@ -71,10 +72,10 @@ def test_task_manifest_uses_port_40042_and_complete_verifiers():
assert all("do not require JSON or verbatim wording" in row["judge_rubric"] for row in rows)


def test_docker_and_docs_use_45_site_range():
def test_docker_and_docs_use_46_site_range():
dockerfile = (ROOT / "Dockerfile").read_text()
assert "45 Flask mirror sites" in dockerfile
assert "EXPOSE 8101 40000-40044" in dockerfile
assert "46 Flask mirror sites" in dockerfile
assert "EXPOSE 8101 40000-40045" in dockerfile
assert "check_asset_inventory.py /opt/WebSyn/drugs_com" in dockerfile
assert "cd /opt/WebSyn/drugs_com" in dockerfile
assert "check_seed_databases.py /opt/WebSyn" in dockerfile
Expand All @@ -83,7 +84,7 @@ def test_docker_and_docs_use_45_site_range():
dockerignore = set((ROOT / ".dockerignore").read_text().splitlines())
assert {"**/.env", "**/.env.*", "**/secrets.json", "**/*.pem", "**/*.key"} <= dockerignore
for relative in ["README.md", "AGENTS.md", "CONTRIBUTING.md", "CLAUDE.md", "agent_demo/README.md"]:
assert "40000-40044" in (ROOT / relative).read_text(), relative
assert "40000-40045" in (ROOT / relative).read_text(), relative


def test_asset_path_contracts_are_synchronized():
Expand Down
5 changes: 5 additions & 0 deletions sites/mega/_health.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
"""Per-site health probe."""


def health():
return {"ok": True, "site": "mega"}
Loading