Review: Add Versus mirror + task verifiers (site by @Sun-sunshine06, verifiers by reviewer) (#41) - #111
Merged
Conversation
sites/versus builds its seed at image build time rather than shipping an instance_seed asset, which is fine and matches what osu and rotten_tomatoes already do on main. But seed_benchmark_users() called generate_password_hash() for each account, and werkzeug draws a fresh scrypt salt on every call, so two builds of the same commit produced different seed bytes: 96d9d1dc... and 568bfb9d... The logical content was identical both times (5 categories, 20 products, 4 users, 3 saved comparisons hash the same with password_hash excluded); only the salts moved. The in-image reset contract still held, because the seed is copied from that same generation, so this never surfaced as a failing md5 check -- what it broke is pinning the seed hash in review evidence or in a verifier. PYTHONHASHSEED=0, which osu uses for its build-time seed, does not help here. Freeze the hash the way osu freezes BENCHMARK_PASSWORD_HASH instead. Two builds now agree: 231149c4... twice.
23 of the 25 sites on main install CSRFProtect; versus did not, and a POST to /compare/<l>-vs-<r>/save carrying Origin and Referer of another site was accepted and wrote the row. Install CSRFProtect, add the token to the two posting forms, and declare Flask-WTF in the site's requirements (the image already pins 1.2.2). Also stop pre-filling the benchmark email and password into the login inputs. The credentials stay in the task text, which is this repo's convention, but a page that ships them in value attributes lets an agent sign in without having read anything.
The missing /favicon.ico was the only console error on every page load.
Reviewer-side work per CONTRIBUTING: the contributor supplied 10 task definitions and no verifiers. This adds a deterministic verifier per task and a judge_rubric, and reworks the questions that could not be graded honestly. Answer leakage. Versus prints Score, Price and Year on every product card, and /rankings lists all 20 products ordered with score, price and category and needs no filter. Three of the original tasks were therefore answerable without opening anything: "which of these two scores higher" (--0), "highest ranked graphics card" (--2), and "highest scoring smartphone under $1000" (--6). The fix is not to hide the ranking -- it is a real page of the site -- but to ask for a fact the list does not carry. Every spec value (camera score, ANC score, megapixels, burst, VRAM, power, benchmark, battery hours, weight, display) is rendered only on detail and comparison pages, so each rewritten question now ends in one of those. A no-op stateful task. The only task that changed state asked Alice to save the RTX 4080 Super vs RX 7900 XTX comparison, which seed_benchmark_users() already saves for her, and save_comparison() de-duplicates. The after state was identical whether or not the agent acted. It now targets a pair she does not have, and a second save task uses Bob, who starts with none. An ambiguous answer. "A camera with 40 megapixels or more" had two valid answers (Nikon Z8 at 45.7 and Fujifilm X-T5 at 40.0). It now asks for the highest count, which is unique. Ground truth is derived from the passed initial_db rather than frozen into the verifier, so the expected answer moves with the seed and a stale verifier fails loudly instead of grading against a dead value. Where a question implies a superlative, unique_extreme() returns None on a tie and the verifier fails closed rather than picking one. Malformed or missing input produces a structured FAIL, never a traceback. Stateful tasks read the after_db and also assert the pair was absent before, so a task that regresses into a no-op is caught by the verifier itself. 17 tasks, 17 verifiers, 17 rubrics.
Each test fails against the code as submitted, so a later change that reintroduces one of these fails here rather than inside a benchmark run: seed byte-reproducibility across two builds, the two site registries agreeing and the task URLs deriving their port from them rather than freezing it, every task carrying a verifier that exists plus a rubric and no answer key, the task count staying in the review guide's range, no save task being already satisfied at seed state, and CSRF protection being installed with tokens on the posting forms.
simpleArgParser builds its parser from the dataclass field types, so `from __future__ import annotations` turned every field into the string 'str' and every verifier died with "'str' is not callable" before it could grade anything. The merriam_webster reference lib does not use it either.
The adversarial matrix caught this: every navigation check was a bare substring match on the path, so taking a canonical trajectory and rewriting its URLs from localhost:40024 to localhost:40007 -- same paths, different mirror -- still passed all 17 verifiers. A run on a different site is not evidence that the agent visited this one. step_urls() now keeps only steps on this site's own origin, and the port is derived from control_server.py's registry rather than frozen, so a registry reorder moves the verifier with it instead of silently accepting whatever site now owns the old port. WH_SITE_ORIGINS overrides it for a harness that maps the site to another address. The two save tasks also now require a final answer that exists and does not deny having saved. They are graded on the state change, so a badly worded report of a correctly performed action still passes; a report that contradicts the state does not.
Upstream advanced to f20b5ee while this review was open, merging the FedEx mirror (aiming-lab#82) and WebMD Doctor (aiming-lab#95). Both took registry slots, so the three registry files conflicted again. Resolved by keeping the slots those two already merged into and appending versus after them: fedex stays on 40024, webmd_doctor on 40025, versus takes 40026. No earlier site moves. The Dockerfile site count and EXPOSE range follow the 27-site registry, and the two new build-time seed steps are kept alongside this branch's. Task rows are re-pointed from :40024 to :40026. The regression test derives the port from both launchers rather than freezing it, so it follows this move on its own; so does the verifiers' origin check, which is why that was made derived rather than constant in 0cbdc87. The Versus seed step now also clears instance/ and __pycache__ after copying the seed, matching what osu, rotten_tomatoes, fedex and webmd_doctor do. The image no longer ships a stale instance dir that websyn_start.sh would overwrite at boot anyway.
…ted synthetic art Two changes that could not be cleanly separated -- the art had to be designed against the new palette, and both touch the same templates and app.py. ## Interface The mirror shared no design language with versus.com: light blue-and-white template, a round "VS" badge, two flat sections and a one-line footer, against a near-black to deep-purple editorial site. Rebuilt from the evidence captured before the source started refusing this client: dark ground with the purple wash and the wave cutout, lowercase wordmark, the oversized two-line "compare everything" hero with its accent underline, pill controls, score rings on products, four-column footer. The comparison page also carried the wrong shape. The source resolves a comparison per area with a margin per row, not as one overall number, so the table now has a Margin column with the leading side marked and the ruling band reads "leads in N of M areas". What the site *declares* the winner is deliberately unchanged -- still the higher Versus Score, via winner() -- because tasks 9 and 16 are graded on that, and changing it would have moved the answer under the verifier rather than fixing the presentation. Colours are read off screenshots, not sampled from the source stylesheet: the 403 landed before exact values could be extracted. The CSS header records them as estimates rather than claiming they are exact. Adds /about, stating which values are sourced and which are synthetic; the footer carries the short version on every page. The Versus Score is synthetic and now says so in the product, not only in review notes. Not mirrored, and not pretended otherwise: the editorial/blog mosaic, the ~100 other categories, the glossary, the locale switcher. ## Art The art was drawn per request by a Flask route: no files, no hashes, no gate, and a placeholder that did not say it was one. It is still synthetic, deliberately. versus.com stayed blocked across repeated probes and no attempt was made to work around that. Freely licensed photography for these exact models could not be matched reliably either -- a Wikimedia Commons sweep found a freely licensed candidate for 18 of 20 products, but strict model matching showed most hits were the wrong item (a OnePlus 8 for the OnePlus 12, an A7R IV for the A7 IV, a 4070 Ti Super for the 4070 Super, a card-slot close-up for the Nikon Z8, earbuds for over-ear headphones). Shipping those would put false product facts into a benchmark built on factual navigation, which is worse than art that admits what it is. So the art follows the precedent already on main -- webmd_doctor ships Pillow-drawn avatars and gradient panels "instead of photography" -- and is held to the same contract: generate_art.py is deterministic (no RNG, clock or locale, Pillow's bundled font, fixed PNG compression, no ancillary chunks); generated_asset_inventory.json pins every path, byte length and SHA-256; check_generated_assets.py enforces coverage, size, hash and PNG decode and runs in the Docker build. Tiles are build products -- static/images/ stays gitignored and the inventory is what travels in Git. Each tile carries a visible SYNTHETIC ART label. No task answer depends on reading an image: all 17 verifiers are deterministic and never open one. This is a deviation from the reviewer checklist's "Real images" line and NOTICE.md says so rather than glossing it. It is the maintainers' call.
An independent blind reviewer passed all 17 executions and, on a run it passed, recorded that the final answer had been emitted at chrome-error://chromewebdata/ after a failed navigation. Every fact-bearing read had already happened, so the answer was right. All 17 deterministic verifiers passed that run too, and that is the actual finding. The navigation checks only look for evidence that the right page *was* opened, and step_urls() filters off-origin steps out on purpose, so a run that crashed and then answered was indistinguishable from a clean one. Evidence that hides its own failure grades as clean. Reproduced before fixing: an error_page_ending fixture, built by rewriting a canonical bundle's done step to a chrome-error URL, passed verify_0, verify_7 and verify_10 unmodified. answered_on_site() requires the step carrying the final answer to sit on this site's origin, and terminal_state_is_sound() wires it into all 17 verifiers. The fixture now fails everywhere, the clean runs still pass, and the real bundle that prompted this correctly failed and was re-recorded rather than accepted. Pinned in tests: answered_on_site must reject a chrome-error terminal step, and every verify_*.py must carry the check.
Upstream advanced to a437bd4, merging Healthline (aiming-lab#105), which took index 26 and port 40026 -- the slot this branch was using. Resolved by taking upstream's registry verbatim and appending versus after healthline: healthline keeps 40026, versus takes 40027. No earlier site moves. Dockerfile site count and EXPOSE range follow the 28-site registry, and the task rows and site README follow the slot. The regression test derives the port from both launchers rather than freezing it, so it follows this move on its own; so does the verifiers' origin check. Recorded trajectories carry the old port, so the runs are re-recorded at 40027 rather than graded with an origin override.
The terminal-state regression hardcoded localhost:40026. Healthline took that slot upstream, versus moved to 40027, and the test started failing against correct code. Third instance of one mistake: a frozen port in the verifier's navigation check (silently accepted trajectories from another mirror), a frozen port in the adversarial foreign-origin fixture (silently stopped testing anything), and now one here. This one at least failed loudly rather than going quiet, which is the behaviour the other two lacked. All three now derive the port from control_server.py's registry.
The verifier derived its expected origin from control_server.py's registry, so every recorded trajectory expired the next time an upstream merge re-slotted the site. That happened four times in one day -- 40015, 40024, 40026, 40027, 40028 -- and each move meant re-recording 22 executions to grade evidence that had nothing wrong with it. At that point the checker is the thing breaking, not the evidence. The port was also buying less than it looked. What binds a run to this environment is the before/after seed SHA-256 in its manifest and the code SHA, neither of which rots. A genuine cross-site replay is caught by the paths, not the port: the other mirrors serve /article/, /section/ and /track/, not /item/ and /compare/. The foreign_origin fixture only ever rewrote the port while keeping versus's own paths, which is not a replay anyone could actually perform. run_origin() now takes the origin from the trajectory's own start_url, so the check is internal consistency: every fact-bearing step, and the step carrying the final answer, must sit on the origin the run started from. That keeps the property that earned its place -- a step recorded on chrome-error://chromewebdata/ after a failed navigation is still not evidence that a page was seen -- and drops the property that only made evidence perishable. Verified rather than assumed: with the site re-slotted to 40028 and the runs still recorded at 40027, all 17 canonical executions verify with no re-recording, and both error_page_ending and foreign_origin still fail as they should (foreign_origin now fails for a better reason -- the run is internally inconsistent). Also carries the upstream merge that re-slots versus behind kaggle to 40028.
…es a re-slot The terminal-state test still called site_origins(), which the previous commit replaced. It errored rather than passing quietly, which is the behaviour this suite is for -- but I pushed before running it, so it went out broken. Rewritten against run_origin(): an arbitrary port is used deliberately, since the check is now internal consistency rather than today's registry value. Adds the case the change exists for -- a run recorded at 40027 still verifies after the site is re-slotted -- and the case that a trajectory with no start_url has no origin to be consistent with.
…inate
CONTRIBUTING asks for "≥ ~20 records per major filter" so that filters,
pagination and search look plausible. The contribution shipped 4 products per
category across 5 categories, and no surface had enough rows to paginate. That
is the project's own line, not a new one -- it predates this review.
The five electronics categories cannot be enlarged from any free citable source
available here, and this was established rather than assumed. A SPARQL count of
digital cameras holding both mass and release date returns zero; the same query
shape returns 2489 cities and 3582 universities. Wikipedia infoboxes yielded all
four needed fields for 2 of 22 camera candidates. Consumer audio and mid-range
watches mostly have no article and no Wikidata mass or price claims. Manufacturer
pages carry the figures but Sony and Garmin return 403 here, and Bose's own
ld+json on a headphone page describes the earbuds. Filling them from memory was
not an option -- that is the rule that made the Versus Score a disclosed
synthetic value.
So the site gains the two categories its own source site advertises ("Smartphones,
cities, graphics cards, universities") and that are sourceable at depth: 52
cities and 35 universities, each row carrying its Q-id, the property id behind
every figure, and the fetch time, in data/catalogue_wikidata.json.
Both new categories clear the ≥20 line; the five original ones still do not, and
the PR says so.
Pagination is added to the category listings and the rankings, 8 per page, with
true ranks carried across pages rather than restarting at 1.
Data quality is cross-checked rather than trusted. Rows whose own claims
contradict each other are dropped at seed time: one paired a state population
with a city area, giving 102,298 people per km². The first attempt at this named
a hand-written Q-id, which turned out not to be the offending item's id, so the
bad row shipped anyway -- it is now a rule over the claims. Entities sharing an
English label are dropped on both sides rather than disambiguated, since a task
naming one would be ambiguous on the page.
A caveat that is documented rather than solved: population and area are
independent claims and need not describe the same administrative boundary, so a
metro population against a city-proper area can still inflate a density within
the plausibility bound. No task is written against the density figure.
price, release_year and the third spec are now optional, since a city has no
list price and the source gives universities no third figure. Templates and the
comparison table omit what is absent instead of rendering an empty column.
Raibows
added a commit
to TabsPhasers/WebHarbor
that referenced
this pull request
Sep 14, 2026
… brand hue
Measured against rendered pixels: --nv-green-d #5e9400 was 3.68:1 as text on white
(card series label, newsroom category, article eyebrow, active pill, green links)
and the star gold #f5a623 was 2.03:1, both below the 4.5:1 body-text requirement;
the --nv-green #76b900 focus ring was 2.41:1 on white and 2.25:1 on #f7f7f7, below
the 3:1 non-text requirement, and .field input:focus{outline:none} suppressed the
ring on form fields entirely.
Tokens are now role-based: --brand stays decorative (dark surfaces, button fill,
logo), --action #4a7500 is 5.48:1 on white and 5.11:1 on #f7f7f7 for text,
--focus #4a7500 clears 3:1 on white, #f7f7f7, #000, aiming-lab#111 and #1a1a1a, and --star
#9c6200 is 5.05:1. test_ui_contract.test_09_contrast_tokens derives the thresholds
from the stylesheet; the pixel audits confirm 0 text failures and 8/8 focus
indicators at 3.84-5.48:1.
Evidence: _wh_review_tools/pr107-fixes/fixes/M1M2/{after,mutation}.txt
The seventeen existing tasks are all consumer electronics, so the two categories that carry real depth were not exercised by anything. These five are, and they are the ones that need the depth: finding a named city among 52 means paging or searching, and the two superlatives compare 52 and 35 candidates rather than 4. Ground truth is derived from initial_db as everywhere else, and each superlative was checked for a unique answer with a clear runner-up before the task was written: most populous Shanghai 24,870,895 against Beijing 21,893,095; longest teaching Bologna 1088 against Sapienza 1303. No task is written against the derived density figure. Population and area are independent Wikidata claims that need not describe the same administrative unit, and that caveat is recorded in data/README.md rather than papered over. The listings and rankings now render the pager they gained, showing the range and page count, so paging is reachable by clicking rather than only by editing a URL.
The sourced categories brought the list to 22. No merged site ships more than 20 -- healthline, webmd_doctor, merriam_webster and osu are all at 20, compass 19, fedex 18 -- so 22 would have been the outlier. The two removed are the ones the S2 matrix already marked shallow: a two-click category read of a single spec, and a picker build with no fact extraction. Neither loses coverage. Finding a city among 52 exercises the same category listing with paging, and the two save tasks build their pair in the picker. Ids are contiguous after the trim; verifiers renumbered with them.
The adversarial wrong-task replay caught it: a run of task 17 -- compare Shanghai and Beijing by area -- passed verifier 18 outright. Task 18 asked for the area of the most populous city, which is also Shanghai, so task 17's answer carried both the name and the figure, and its trajectory had opened the same detail page. That is not a verifier hole so much as two tasks pointed at the same fact about the same entity. Task 18 now asks for the population of the largest city by area, which is Ganzhou, and carries a real distractor: Nanyang has a larger population on a smaller footprint, so reading "biggest" as population gives the wrong answer.
Upstream advanced to 7ace1c7, merging NVIDIA (aiming-lab#107) at index 28. versus follows at index 29, port 40029. No earlier site moves; the Dockerfile count and EXPOSE range, the task rows and the site README follow the slot. Fifth re-slot of the day, and the first that cost nothing: the origin check now binds to each run's own start_url rather than the registry, so all 20 recorded executions -- still carrying 40028 -- verify unchanged against the site on 40029. Earlier moves meant re-recording 22 bundles each time.
An independent blind review failed Versus--16 on a run that every deterministic check had passed. The answer asserted "88500.0 students" while the panel text pasted after it carried the real 96945, and the numeric check searched the whole answer, so the quoted evidence satisfied a claim that contradicted it. 88500 matches no university on the site; it was a figure the guided run asserted without ever reading it. Two layers, both fixed. The run script is corrected to the sourced figure. More importantly the check is: claims_number, claims_money and claims_product now match only within the claim region -- the text before the first sign of quoted page text. An answer that states its figure up front passes; one that only quotes does not. Without that, an agent could paste a page and satisfy every numeric and naming check without committing to an answer. Reproduced before fixing: a wrong_claim_correct_dump fixture passed verify_0, verify_16 and verify_18 unmodified. Getting the boundary right took two attempts -- a greedy label pattern matched from the start of the answer and put the cut at zero, and a "head or text" fallback then returned the whole string, which would have reopened the hole it was closing. State-graded tasks (7, 11, 19) are deliberately exempt: they are judged on the database row, not the prose, so only a denial fails them. The matrix records that as an expectation rather than leaving it as an unexplained pass. Also pins that sites/versus/verify holds exactly the expected files. A rename left sixteen stray "verify_N 2.py" copies there; they were never committed but they crashed the harness that globs the directory to decide what to grade.
jackjin1997
marked this pull request as ready for review
September 14, 2026 15:30
Contributor
Author
|
Marking this ready for review. Final verification was run from a cold checkout of this branch, not from my working tree — the tree carries no
Grading: 20 canonical runs 20/20, five legitimate alternative routes 5/5, adversarial matrix 261 cases, 0 expectation mismatches, 0 tracebacks. An independent blind review (separate session, no access to the verifiers, this branch or these PRs; packet and all 165 files hash-verified) returned 19 PASS / 1 FAIL — and the failure was one every deterministic verifier had passed. A run asserted "88500 students" while the page text pasted after it carried the real 96945, and the numeric check searched the whole answer, so the quoted evidence satisfied a claim that contradicted it. Answer checks now match only within the claim region, so pasting a page no longer counts as answering it. Reproduced with a fixture before fixing, and pinned. Two things are for you to rule on, not settled here:
Also carried honestly: the Versus Score is synthetic and says so on Site and mirror by @Sun-sunshine06 (#41); source-level review by @Susan571. |
jackjin1997
marked this pull request as draft
September 14, 2026 16:00
Contributor
Author
|
Owner image feedback is addressed at Fresh affected checks: source/output reproduction 107/107, repository asset gate 107/107, regression suite 21/21, functional/negative audit 19/19, remote archive hash matches, and live desktop/390px/320px visual checks show undistorted 4:3 cards. The PR description now carries the full current report and reproduction command. Keeping this Review PR Draft until the Owner confirms the live visual result. Reviewer will not merge either PR. |
jackjin1997
marked this pull request as ready for review
September 16, 2026 06:41
Contributor
|
Hi @jackjin1997, following up as the original contributor of #41. Thank you for preserving the authorship and for the image, UI, and grading work. I checked the current public status: head Could you please confirm the next steps and who owns each remaining item?
Is there anything you need me to address on the original contribution, or should I leave the site changes with this reviewer branch to avoid duplicating work? This is a coordination follow-up, not a new visual approval or a claim that I reran the tests locally. |
Contributor
|
Thanks for your contribution! @jackjin1997 @Sun-sunshine06 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reviewer pass over original contribution #41 by @Sun-sunshine06. The site commit remains attributed to the original author; later commits are reviewer fixes and verification.
Current candidate
6e807fcc0a35a03bd901270d396b48254bd8706f5d7a4e8c594028b845cbdcc180619e438c3a22abMERGEABLE/CLEANThe final merge preserved UC Berkeley on 40029 and B&H Photo on 40030, then appended Versus on 40031. All 20 task URLs were updated to the registered port. Task prompts, rubrics, verifier paths and upstream URLs are byte-for-byte unchanged apart from that port field.
Source-backed imagery and task-path UI
The rejected programmatically drawn tiles are gone. All 107 entity cards use exact, locally stored 960×720 WebP images:
The task-path presentation was also tightened without changing task semantics:
The Owner accepted this task-path-focused visual result on 2026-09-16.
Asset candidate
The image bundle is in paired Hugging Face dataset PR #90, still open at immutable commit
464c121a3cc6dacd318df3f76f461e70901e6e24.versus.tar.gzis 12 MB with SHA-256093656316188dd7960177c9a187262de860e50c9f27a3b6658407f86d16f5cb3. The uploaded archive was downloaded from that immutable commit and passedvalidate_asset_archive.pywith 111 managed members.Reviewers can fetch the candidate before the HF PR is merged:
The repository pin is the current merged HF main revision for the other 31 sites. Maintainers still need to merge HF PR #90 and update the repository pin in the coordinated final integration.
Verification
websyn_start.sh,control_server.py, DockerEXPOSEand everytasks.jsonlURLThe port re-slot does not invalidate the frozen runs: verifiers derive the expected origin from each run's own
start_url, and a dedicated regression test confirms older runs survive a registry move.The full 32-site image was not rebuilt on this host because only 7 GiB was free, below the repository review workflow's 10 GiB reuse floor and 50 GiB unknown-build floor. The final merge took the two newly integrated sites unchanged from current upstream main; combined integration was checked through the registry validator and Versus-specific affected tests. No claim of a fresh full-image build is made.
Known limits and maintainer actions
Site and mirror by @Sun-sunshine06; source-level review by @Susan571; reviewer fixes and verification by @jackjin1997.