Skip to content

Add reviewed AccuWeather mirror, tasks, and deterministic verifiers - #98

Merged
QianhuiWu merged 14 commits into
aiming-lab:mainfrom
JeremyJC67:review/pr-93-accuweather
Sep 16, 2026
Merged

QianhuiWu merged 14 commits into
aiming-lab:mainfrom
JeremyJC67:review/pr-93-accuweather

Conversation

@JeremyJC67

@JeremyJC67 JeremyJC67 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Adds the AccuWeather mirror at port 40031 with 20 reviewed tasks, judge rubrics, and deterministic verifiers. Preserves @sarendis56's site contribution from #93 and @JeremyJC67's review work, and includes the subsequent GUI audit fixes.

Corrects mobile search and form layouts, menu destinations, radar/AQI legends, and forecast consistency. Task 8 explicitly requests the alert confirmation and task 17 requests the numeric maximum; grading checks units, city/value attribution, contradictions, navigation, and exact database changes.

Integrates current main while preserving the existing 31 site ports. The Dockerfile generates AccuWeather's reset seed from tracked code. .assets-revision retains the merged global asset pin fa1e8a5b9e8e5d0e42764cd658825f4dea088d8f and independently pins the validated AccuWeather archive at immutable HF commit 0a73c1c1ac2e47513389a8a1a67601f75c8c4150 (HF #66). Asset fetching supports scoped revisions and sites with a generated seed and no required downloaded media.

Validation:

  • 20/20 tasks completed through GUI actions during the audit, with 20 GIFs; all saved runs also pass the official evaluator from the integrated checkout.
  • 111 copied-evidence controls match their expected outcomes; 301 verifier tests and 8 asset-download tests pass.
  • Fresh asset download/extraction succeeds for all 32 sites; the registry and all task URLs agree on 40000–40031.
  • Full Docker build succeeds; all 32 sites are healthy and respond HTTP 200. All 51 tracked AccuWeather runtime files match the checkout.
  • Task 6 passes a fresh scripted GUI replay on the integrated image; reset after its saved-location mutation restores seed byte identity, and process restart preserves it.

Weather is synthetic and radar is illustrative. The secondary LLM judge was not run because its API/model configuration is absent. Detailed corrections and integration evidence are documented in review-reports/accuweather-pr98-fixes.md.

sarendis56 and others added 8 commits September 10, 2026 03:06
- README / AGENTS / CONTRIBUTING / CLAUDE / agent_demo README: 25 sites,
  ports 40000-40024 (the PR bumped Dockerfile and the registries but not
  the shared docs).
- walmart_careers and rotten_tomatoes registry tests: accept the 25-site
  registry; the shared-docs port-range test now derives the range from
  control_server.SITES (same approach as PR aiming-lab#95) instead of a hardcoded
  count.
- sites/accuweather: add .build-generated-seed and .requires-images so
  fetch_assets.sh / check_assets.sh treat the site like rotten_tomatoes
  and walmart_careers (the Dockerfile regenerates instance_seed at build
  time; weather icons come from the HF bundle).
- sites/accuweather/requirements.txt: pin Flask-SQLAlchemy / SQLAlchemy /
  Werkzeug / Jinja2 that app.py imports (the image already installs them).

Port 40024 is provisional: PR aiming-lab#95 (webmd_doctor) claims the same slot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…-anchor

Reviewer grading contract for the 20 AccuWeather tasks (CONTRIBUTING
"Reviewer role" B/C):

- sites/accuweather/verify/verify_lib.py: merriam_webster-style API with
  walmart_careers hardening: package identity (task id, agent_done,
  same loopback host+port for every recorded URL, decodable PNGs),
  navigation gates incl. search-surfaces-target, label-aware numeric
  matchers that catch value/label swaps, negation-aware phrase matchers,
  comparison-winner attribution, SQLite snapshot contract (table set,
  columns, row counts, pinned catalog fingerprint, immutable catalog
  tables, read-only tables, exact stateful row deltas), werkzeug hash
  verification with hashlib only. LLM helpers kept for parity; no
  verdict depends on them.
- verify_0.py .. verify_19.py: one deterministic verifier per task,
  ground truth hardcoded (never in tasks.jsonl).
- tasks.jsonl: verifier_path + judge_rubric on every row; task 5
  re-anchored from "UV index" (not rendered on any page) to the pressure
  reading + air-quality value, keeping the 3-page workflow.
- verify/tests: stdlib-only harness (fixture DB reproduces the frozen
  seed fingerprint, agent.py-format trajectories) covering genuine PASS,
  no-op, wrong task id, shortcut, wrong answers, alternative phrasings,
  unterminated run, mixed origin, corrupt PNG, schema/catalog/seed
  drift, collateral writes and stateful mismatch (293 tests), plus
  run_matrix.py which drives every task through Chromium and grades
  pass / noop / shortcut / wrong / mismatch run dirs with live DB
  snapshots (20/20 tasks OK on carcinoma).
- verify/README.md: the contract per task.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…ndling

Second-pass audit findings against the maintainer merge checklist. Each item
was reproduced with a request and re-verified after the fix.

- Open redirect: POST /login?next=https://evil.example returned 302 to that
  URL; /save/<slug> and /units/<unit> did the same via request.referrer.
  Added local_path()/local_referrer() so only same-origin paths are honoured.
- Fixed SECRET_KEY "accuweather-local-benchmark" let a forged session cookie
  read /account as any user. Now os.environ ACCUWEATHER_SECRET_KEY or
  secrets.token_hex(32), matching walmart_careers / rotten_tomatoes.
- A tampered cookie carrying a non-scalar user_id raised on db.session.get and
  returned 500; current_user() now ignores anything that is not an int.
- /register persisted an empty email, "notanemail" and a 200 000-character
  name, all returning 302 to /account. SQLite does not enforce VARCHAR length,
  so the app now bounds email/name/password and returns 400.
- /alerts/<slug> wrote any alert_type the client sent, including a 3 000-char
  value; now whitelisted to severe/rain/temperature, else 400.
- /settings silently ignored unit=K; now 400.
- No MAX_CONTENT_LENGTH: a 60 MB multipart POST was accepted. Now 64 KB.
- No CSRF protection, unlike 23 of the 25 sites. Added CSRFProtect and a token
  to every POST form (login, register, alerts, settings, save, logout).
- No errorhandler and no error template: 404 rendered the bare Werkzeug page.
  Added templates/error.html and handlers for 400/403/404/405/413/500.
- Removed the orphan POST /units/<unit> route (no template ever rendered a
  control for it).
- weather.html/account.html also pick up the unit letter and UV row here
  because these templates are single-line and their CSRF token cannot be
  staged separately.

Flask-WTF is already installed in the image; added to requirements.txt.
Verified: 297-test unit harness OK, 20/20 real-browser task matrix OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
- The homepage overflowed 192 px horizontally at 768 px because .layout keeps
  fixed 632px/304px columns until the 700 px breakpoint. Added a 960 px
  breakpoint that collapses the grid; 60/60 route x width cells are now clean.
- .tabs carried margin-bottom:315px (an ad placeholder), leaving an empty band
  between the tabs and the card on every location page. Now 24 px / 18 px.
- The active tab was hardcoded to .tabs a:first-child, so /hourly highlighted
  "Current Weather". The macro now marks the tab by request.endpoint and sets
  aria-current.
- Temperatures rendered no unit letter, so Celsius mode showed a bare "26deg".
  Added the letter on the current, hourly, daily and account surfaces.
- Contrast, measured by sampling rendered pixels rather than reading CSS:
  white on the brand orange #f05514 is 3.49:1 and #f15a18 is 3.38:1, both
  under the 4.5:1 body-text bar, and .news-grid small was 4.14:1. Introduced
  an --action token (#c2410c, 5.18:1) for buttons, .premium and the breaking
  strip, keeping the brand orange for decoration only; the orange "Accu" mark
  on the dark header measures 4.77:1 and is unchanged. Added a :focus-visible
  ring (--focus #1a73e8) that clears 3:1 on both the white page and the #222
  header.
- Made the seven dead "Weather News" / "Top Stories" items real links.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…ch gates

Two gaps in our own grading contract, both found by probing the verifiers
rather than reading them.

- screenshots_decode only required width and height >= 1, so a run whose every
  screenshot was a 1x1 PNG passed verify_0 with "all checks passed". The
  checklist asks for "decodable and above a minimum size"; now 200x150.
  The test fixture PNG moves from 2x2 to 320x200 accordingly.
- check_search_surfaces accepted any query token intersecting the target's
  city/region/country/postal. Every US location carries country "United
  States", so a single q=United+States search satisfied
  searched_for_springfield-mo without the agent disambiguating the three
  Springfields. search_surfaces now ignores tokens shared by more than half
  the catalog, derived from CATALOG rather than a hardcoded stop-list, so
  "united"/"states" no longer count while "springfield" and "65806" do.

Added negative tests for both (test_stub_screenshot_fails,
test_catalog_wide_query_does_not_satisfy_the_search_gate,
test_screenshots_reject_stub_sizes, test_search_gate_ignores_catalog_wide_tokens).
Harness 293 -> 297 tests, OK; 20/20 real-browser matrix still OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
- provenance.json gains a sha256 for all eight captured assets, so a future
  re-download can be compared objectively.
- New site README recording what is deliberately simplified (no JavaScript,
  decorative radar, formula-generated forecasts), how the seed is made
  byte-reproducible, and the font notice: Solis-Regular.woff2 ships with SFNT
  name ID 0 ("(c) Copyright AccuWeather, 2019. All rights reserved.") and name
  ID 14 preserved verbatim, carries no licence grant, and can be removed by
  deleting the file and the @font-face rule since the stylesheet already
  declares the Arial fallback. Adds non-affiliation and trademark notices.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…rt 40026)

main gained fedex (40024) and webmd_doctor (40025). Merged rather than rebased so every
contributor and reviewer commit on this branch is preserved verbatim. Shared registry,
Dockerfile EXPOSE, docs and every site-local port reference move to index 26 / port 40026;
the slot stays provisional and the maintainer reassigns it at merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
@sarendis56

Copy link
Copy Markdown
Contributor

Contributor asset follow-up for the remaining archive-layout finding:

HF PR #66 is updated at 0a73c1c1ac2e47513389a8a1a67601f75c8c4150. It now contains only accuweather/static/images/ (eight SVG icons), with no unsupported fonts root, stale seed database, or AppleDouble/Finder metadata. This matches this branch's build-generated seed contract.

I downloaded the uploaded bundle and confirmed it passes validate_asset_archive.py (10 managed members). Your code, tasks, and verifiers are untouched. HF merge and the subsequent .assets-revision pin remain the release steps.

…erge

accuweather ships no instance_seed: its seed is generated at image build time. Merging the
26-site main took upstream's Dockerfile wholesale, which dropped that RUN step, so
websyn_start.sh failed at 'cp -a .../instance_seed' and the container exited 1 before the
control plane came up. Step restored verbatim ahead of EXPOSE.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
@JeremyJC67

Copy link
Copy Markdown
Contributor Author

Independently validated, and it matches your report. Downloaded refs/pr/66 at 0a73c1c1 on the review host and ran the repo's own validator:

bytes: 5524      md5: 522284a9610b5699d3982b48878b3623
members: 10      AppleDouble/DS_Store: 0
archive roots: accuweather/        second level: accuweather/static
validate_asset_archive.py -> validated 10 managed members for accuweather

That clears the blocker. The archive is rooted at accuweather/, carries only the managed static/images root, and no longer ships the unsupported fonts root or the stale seed database — which is right, since this site generates its seed at image build time.

One thing worth knowing that is on my side, not yours: when I merged the 26-site main into this review branch, my conflict resolution took upstream's Dockerfile wholesale and silently dropped the accuweather build-time seed step. The container then died at boot on cp -a .../instance_seed before the control plane came up. I have restored that step verbatim (e9e0809) and the site is being re-verified now. Nothing in your code or assets caused it.

Remaining is release mechanics only: a maintainer merges HF #66, then I pin .assets-revision to the merge sha.

@JeremyJC67

Copy link
Copy Markdown
Contributor Author

Rebased onto the 26-site main — this branch is now site 27 / port 40026

main merged FedEx (40024) and WebMD Doctor (40025) overnight, which put every open site
branch into conflict. This branch has been brought up to date by merging main rather than
rebasing, so every contributor and reviewer commit is preserved verbatim, and re-slotted to
index 26 / port 40026: registry, Dockerfile EXPOSE, the shared docs and every
site-local port reference follow. The slot is provisional as always — say the word at merge
time and I will re-slot in whatever order you land things.

The whole environment was then rebuilt and re-verified at the new slot:

check result
HTTP sweep 27/27 return 200
GET /health ok, 27/27 alive and ready
POST /reset/accuweather → md5 cbddbe8704539f93e5be0f5f87959099, identical to the seed
POST /reset-all ok, 27/27 ready, 1.63 s
verifier no-op matrix 20/20 FAIL
registry + docs tests 4 passed, 3,343 subtests

Worth recording because it was my fault, not yours: my first attempt at this merge resolved the
Dockerfile conflict by taking upstream's copy wholesale, which dropped this site's build-time
seed step. Since accuweather ships no instance_seed, websyn_start.sh then died on
cp -a .../instance_seed and the container exited six seconds after start. The step is restored
verbatim in e9e0809 and the run above is with it in place.

Still blocked only on HF #66 being merged, after which I repin .assets-revision.

JeremyJC67 and others added 5 commits September 13, 2026 08:42
…rt 40026)

main gained fedex (40024) and webmd_doctor (40025). Merged rather than rebased so every
contributor and reviewer commit on this branch is preserved verbatim. Shared registry,
Dockerfile EXPOSE, docs and every site-local port reference move to index 26 / port 40026;
the slot stays provisional and the maintainer reassigns it at merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
…n merge again

Same loss as before: this site ships no instance_seed and generates it at image build
time, but resolving the Dockerfile conflict by taking upstream's copy removes the RUN
block, so websyn_start.sh dies on 'cp -a .../instance_seed' and the container exits 1.
Restored verbatim. Any future merge of main must re-check the Dockerfile for this block.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkNTAZMziDodR6uqa6QBtP
@QianhuiWu QianhuiWu changed the title Review: AccuWeather mirror + task verifiers (site by @sarendis56, verifiers by reviewer) Add reviewed AccuWeather mirror, tasks, and deterministic verifiers Sep 16, 2026
@QianhuiWu
QianhuiWu merged commit a2479a0 into aiming-lab:main Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants